feat(evals): Habitat navigation benchmark: text-only arms, planner arm, video, parallel cases

[spomichter]

Contribution path

Draft. Stacked on feat/typesafe-agent (the TypeSafe module this benchmark evaluates), so the diff is the benchmark only. The TypeSafe v2 rewrite is #4212; the eval needs only set_trace_dir, world_state and agent_idle from the module and rebases onto whichever lands. Spec: engineering/typesafe-nav-benchmark/spec.md in the agents repo.

Problem

Compare the TypeSafe reactive agent, dimOS's planner, and coding agents with and without dimOS on long-horizon go-to-object navigation in real Habitat scenes, on an even text-only footing, with navigation-quality metrics, per-case video, and parallel execution.

Solution

One launch serves every arm: habitat-nav mcp-server demo-objects nav-skills, plus the agent's own modules.

• Habitat native publishes odom (PoseStamped) and a body-frame lidar scan, so type-safe-agent and raw-robot-bridge autoconnect by name. No relays, no remap blueprints.
• nav-skills: go_to(x, y, wait_s), stop_navigation, finish over the planner's goal / goal_reached / stop_movement topics. Goals carry the robot's floor z: HM3D houses are multi-level and a z=0 goal snaps onto furniture.
• TopicAgent (dimos/evals/agents/topic.py): evaluate whatever is already running. Publishes the instruction (text, or its (x, y) as a PointStamped) on one topic and returns when another goes truthy; trajectory from a module's set_trace_dir trace. The planner arm and the TypeSafe arm are two configurations of it.
• Raw bridge: topics allowlist, world_state/json (the same builder TypeSafe reads) and finished/json; ROBOT.md lists only the served topics.
• HabitatEnvironment: a navmesh tour from the scene file is driven before the task so the planner has a map (it plans only over observed surface, like dimsim_house explores first); task_start_ts; record_topics.
• nav_metrics: reached, time to object (run duration when never reached), facing, bumps (held command without displacement), path length, straightness, turn reversals; written as nav_metrics.json beside the recording.
• Sim: raw_topics, extra_env, --rerun-open none, opt-in rrd, video (Xvfb + dimos-viewer + ffmpeg, viewer window only).
• dimos evals run --case --parallel --container --repeat: one dimos evals run per case, in a container when given; docker/eval/Dockerfile.
• Suite dimos.evals.suites.habitat_nav with an example HM3D scene file (furniture-sized boxes, 4 cases, tour). The scene file layout is the detection3d_array_to_dict one from Ruthwik/feat/dimsim object annotations #4208 plus cases and tour.

How to Test

# planner alone
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.topic \
  --set send=goal --set send_type=point --set done=goal_reached --set done_type=Bool
# TypeSafe agent (TYPESAFE_API_KEY)
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.topic \
  --set 'modules=["type-safe-agent"]' --set trace=TypeSafeAgent
# coding agent with dimOS / without dimOS
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.dimcode --set model=gpt-6-astra
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.pi --set no_dimos=true --set model=gpt-6-astra

Verified on the example scene, chair case (score 0.8 each): planner reached in 10.9 s; Pi + Astra without dimOS wrote a Zenoh observer and navigator and reached in 34 s; dimcode + Astra with dimOS reached in 16 s with one go_to. viewer.mp4 captured headless. 285 unit tests, ruff, mypy and pre-commit pass.

Not yet run: the TypeSafe arm (no key on the box) and the container path of --parallel (Docker registry unreachable on the box; job command unit-tested).

Checklist

• I have read and approved the CLA.