feat(evals): add QA suites for 18 Habitat scenes

[ruthwikdasyam]

Contribution path

Builds on the merged Habitat environment support in #4126.

Problem

Add scene-specific QA evaluations for Habitat datasets.

Solution

• 210 user-reviewed questions across 18 scenes: HSSD, HM3D, ReplicaCAD, and Habitat test scenes.
• Explicit EvalCase definitions with shared scoring helpers and fresh environments.
• Numbered suite names, JPEG overview images, and consolidated context in SCENES.md.
• Co-located regression tests.

How to Test

With Habitat and the HM3D example dataset installed:

dimos evals run dimos.evals.suites.habitat.hm3d.hm3d_scene_3 --agent dimos.evals.agents.pi --limit 1

117 focused tests and pre-commit checks passed. Live-agent runs and HSSD runtime navmesh coverage remain to be validated.

AI assistance

OpenCode with GPT-6 Astra implemented the suites and checks; the user visually reviewed the scenes and reference answers.

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 98.24047% with 6 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/scorers.py	73.91%	3 Missing and 3 partials ⚠️

@@            Coverage Diff             @@
##             main    #4195      +/-   ##
==========================================
+ Coverage   79.45%   79.46%   +0.01%     
==========================================
  Files        1485     1519      +34     
  Lines      140563   141152     +589     
  Branches    12062    12087      +25     
==========================================
+ Hits       111685   112168     +483     
- Misses      25462    25566     +104     
- Partials     3416     3418       +2     

Components	Coverage Δ
Tests	95.19% <100.00%> (-0.16%)	⬇️

Flag	Coverage Δ
OS-ubuntu-24.04-arm	75.07% <98.24%> (+0.05%)	⬆️
OS-ubuntu-latest	75.91% <98.24%> (+0.05%)	⬆️
Py-3.10	75.69% <98.24%> (+0.05%)	⬆️
Py-3.11	75.89% <98.24%> (+0.05%)	⬆️
Py-3.12	75.90% <98.24%> (+0.05%)	⬆️
SelfHosted-Linux	36.72% <61.05%> (+0.05%)	⬆️
SelfHosted-macOS	37.01% <61.05%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/evals/suites/habitat/hm3d/hm3d_scene_1.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hm3d/hm3d_scene_2.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hm3d/hm3d_scene_3.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_1.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_10.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_2.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_3.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_4.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_5.py	100.00% <100.00%> (ø)
dimos/evals/suites/habitat/hssd/hssd_scene_6.py	100.00% <100.00%> (ø)
... and 13 more

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.