Conversation
GEAR-SONIC planner+encoder+decoder ported from the Matrix parity reimplementation into a ControlTask claiming all 29 joints. All 27 locomotion modes RPC-reachable via task_invoke; twist via the standard duck-typed hook; upper-body targets are encoder hints only (never post-decoder overrides). Armature-derived SONIC PD gains carried over exactly, including the doubled ankle/waist entries. Blueprint unitree-g1-sonic-wbc with the standard sim/real dual backend. Verified standalone against raw MuJoCo: 40 s continuous choreography (walk 4 m, squat, boxing, dance) with zero falls and zero NaN at 0.23 ms/step policy latency on CPU onnxruntime.
Groot-style URDF static robot + joint-state overrides, entity key matching zenoh's stripped topic naming (world/g1_joints). Guarded off under DIMOS_TRANSPORT=zenoh: callable factories in rerun_config do not survive zenoh module-deploy serialization (msgpack -> dicts -> validation reject); applies to any blueprint, works under LCM.
SONIC's native protocol as a first-class task input: wire codec for the 1280-byte packed-header format (round-trip verified against NVIDIA's own builders), command/planner/pose topic routing polled non-blocking from compute(), StreamedMotionMerger ported faithfully (sliding window, catch-up, protocol v1/v2/v3, encode-mode selection), SMPL + wrist encoder fields for mode 2, wire-order 17-DoF upper-body targets with velocities, direct planner commands overriding twist mapping, g1_debug state PUB. E2E verified in sim against the live blueprint driven ONLY by NVIDIA's sender code: planner-mode walk, then protocol-v1 streamed-motion playback of the macarena reference clip - 110 planner + 110 pose messages, zero decode errors, zero tick errors, no falls (base z 0.79 at end).
play_motion_clip/stop_motion_clip/list_motion_clips on the task: reference
clips (50 Hz CSVs, IsaacLab joint order) load into a StreamedMotion and run
through the exact streamed-motion path the ZMQ pose topic uses, heading
re-anchored on start. Verified in the sim blueprint over RPC: macarena
(1375 frames) and dance_in_da_party (497) play end to end, no falls,
clean return to planner idle after stop.
data/sonic ships via LFS (encoder+decoder+774MB planner, observation
config, 13 reference clips); blueprint model paths now default to
LfsPath("sonic") with SONIC_MODEL_DIR/SONIC_PLANNER_PATH still
overriding, so the GR00T-WholeBodyControl checkout is no longer required.
Also restores the LFS pointer for reloc_and_nav_to.webp that the first
SONIC commit accidentally hydrated (132 B -> 1 MB blob).
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
End-effector teleop through SONIC's native interface: set_vr_3point RPC and the ZMQ planner-topic vr fields feed root-relative wrist/head poses (9 pos + 12 quat, C++ GatherVR3Point buffered convention - sender applies offsets and root normalization) into encoder obs [901:922]. Teleop mode builds the C++-faithful observation: zeroed buffer, mode scalar, lowerbody joint history from the planner trajectory using the mujoco-order-in-isaaclab index set, single-frame anchor orientation. Stale VR data (>0.5 s) reverts to planner obs; locomotion keeps running underneath. Verified: unit test pins every field offset against policy_parameters.hpp and observation_config.yaml; sim E2E - hands-raised targets streamed at 15 Hz swing the elbows 1.28 rad while base z stays 0.77-0.80, and the arms return to idle after the stream stops.
NVIDIA's SONIC v1.1 (HF nvidia/GEAR-SONIC sonic_v1_1) is now the shipped checkpoint: heading-normalized target orientations + wrist-pose augmentation, the intended model for whole-body teleoperation. The pipeline selects the obs layout by the encoder's input width (1762 original / 1751 v1.1 - root_z dropped, anchor fields reordered, every downstream offset shifted) and switches the anchor-orientation left quat to robot heading only (C++ orientation_mode 1). The original release still runs via SONIC_MODEL_DIR; data/sonic and the LFS archive carry v1.1. Fixes a heading bug v1.1 exposed (latent under the original checkpoint): motion-source switches kept the previous source's heading delta, so post-clip planner trajectories stayed anchored to the clip's mocap heading and the policy turned instead of walking. Heading now re-anchors on every source switch (C++ reinitialize_heading_ semantics). PICO 5-sensor teleop path (full-body SMPL from headset + controllers + foot trackers, pose-topic protocol v3): accept the pico server's body_quat_w field name, consume vr_position/vr_orientation from the pose topic, and apply heading_increment (joystick yaw) into the heading delta. Verified on v1.1 in sim: walk 4.28 m; macarena tracks end to end; VR 3-point elbows 1.34 rad with base z 0.786-0.807; clip->VR->walk regression 4.19 m; pico-faithful SMPL stream (NVIDIA's pack_pose_message, exact field set) runs encoder mode 2 with stable stance and +0.9 rad heading_increment yaws the robot +0.72 rad. Obs layouts for both variants unit-pinned against the C++ registry and both observation_config.yaml files.
Drop the checkpoint-variant layout table: we ship exactly one policy (SONIC v1.1 in the LFS archive), so the obs layout is flat module constants again and the anchor math is unconditionally heading-normalized. A checkpoint with a different encoder width now fails at load with a clear error instead of being silently supported. Smoke on the stripped build: clip -> walk 4.21 m, base z 0.788.
Greptile SummaryThis change adds the G1 SONIC v1.1 whole-body-control task, coordinator commands, ZMQ command ingestion, motion playback, the G1 blueprint, and MuJoCo support. It is not ready to merge because malformed SONIC ZMQ frames with zero-length fields can raise unhandled exceptions during controller dispatch. Earlier reports that SONIC was included in Ubuntu blueprint validation, that velocity RPC bypassed declared-command validation, and that command-wire heading changes were ignored were disproved by exercising the current behavior. Confidence Score: 3/5Not safe to merge until malformed SONIC wire frames are rejected before controller dispatch. One verified input-driven controller crash remains in the SONIC wire decoder and dispatch path. Files Needing Attention: dimos/control/tasks/g1_sonic_wbc_task/zmq_wire.py
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| _SONIC_RELEASE_DIR = Path(_env_model_dir) if _env_model_dir else LfsPath("sonic") | ||
| _env_planner = os.environ.get("SONIC_PLANNER_PATH") | ||
| _SONIC_PLANNER_PATH = Path(_env_planner) if _env_planner else LfsPath("sonic/planner_sonic.onnx") |
There was a problem hiding this comment.
unitree-g1-sonic-wbc is not in SELF_HOSTED_BLUEPRINTS, so it is included in UBUNTU_BLUEPRINTS. The normal Ubuntu validation imports this blueprint, and module-level task configuration converts these SONIC LfsPath values to strings. That invokes get_data() and fails with RuntimeError: Missing required tools: git-lfs on the Ubuntu runner. Add this blueprint to SELF_HOSTED_BLUEPRINTS so the LFS-backed configuration is validated in the intended environment.
Artifacts
Narrow SONIC blueprint validation reproduction script
- Authored script computes Ubuntu selection from the production allowlist and invokes the production `_check_blueprint` path, with an in-memory self-hosted comparison mode; takeaway: it exercises the finding without changing production code.
Captured SONIC reproduction script source
- Executed SHA-256 and source capture of the authored reproduction script; takeaway: the test harness used for both observed runs is traceable.
Ubuntu-selected SONIC blueprint import fails without Git LFS
- The executed production `_check_blueprint("unitree-g1-sonic-wbc")` run reports `selected_ubuntu=True`, no Git LFS, and fails through `LfsPath.__str__` and `get_data`; takeaway: the reported import-time LFS failure is reproduced.
Exact pytest SONIC Ubuntu blueprint validation failure
- The exact `test_blueprint_is_valid[unitree-g1-sonic-wbc]` pytest node fails at module line 245 after `LfsPath` calls `get_data` and detects missing `git-lfs`; takeaway: normal Ubuntu blueprint validation has the same failure.
In-memory self-hosted routing excludes SONIC from Ubuntu validation
- The same reproduction script with only an in-memory self-hosted routing addition reports `selected_ubuntu=False` and does not call the import check; takeaway: adding the blueprint to the self-hosted allowlist prevents this Ubuntu failure.
| "start", | ||
| "set_locomotion_mode", |
There was a problem hiding this comment.
G1SonicWBCTask implements set_velocity_command, but TASK_EXPOSES does not declare it. As a result, ControlCoordinator.task_invoke takes the undeclared reflective-dispatch path, emits an undeclared-command warning, and does not apply the coordinator's command signature validation. Add set_velocity_command to the exposure list so callers receive the normal validated RPC behavior.
Artifacts
Narrow coordinator and registry harness source
- This authored harness registers a task with the production velocity-command signature and invokes it through the real registry and ControlCoordinator path, showing the manifest-dependent behavior.
Coordinator output with the shipped g1_sonic_wbc manifest
- The executed shipped-manifest run shows the command absent from coordinator commands, undeclared-dispatch warnings, and the raw method bad-keyword error, confirming the finding.
Coordinator output after declaring set_velocity_command
- The executed comparison run adds only the missing registry exposure and shows coordinator signature validation rejecting the bad keyword while the valid command still dispatches, proving the intended fix.
…on to self-hosted Both from PR review: set_velocity_command was reachable only via the coordinator's undeclared reflective-dispatch fallback (warning, no signature validation); and the blueprint stringifies LfsPath model paths at import, which on the plain Ubuntu runner means a git-lfs failure or an 836MB pull during blueprint validation - same reason unitree-g1-nav-sim is self-hosted.
Conflict in test_all_blueprints.py: main reorganized SELF_HOSTED_BLUEPRINTS (openarm added, nav blueprints removed with the asset-manager migration); kept main's list plus unitree-g1-sonic-wbc, which still stringifies LfsPath model paths at import. Post-merge hardening found by the runtime smoke: pyzmq was never a declared dependency (the task's ZMQ wire rode a manually installed package) - added to project deps. And a missing pyzmq used to make _zmq_start retry the import and log a warning on every 50 Hz compute tick, which starved the control loop until the robot collapsed; the wire now disables itself permanently after one failed start, one warning. Smoke on merged main: arm + walk 3.56 m, base z 0.787.
| if cmd.delta_heading is not None: | ||
| logger.info( | ||
| "ZMQ delta_heading received (not yet applied)", | ||
| task=self._name, | ||
| delta_heading=cmd.delta_heading, | ||
| ) |
There was a problem hiding this comment.
_on_wire_command receives a decoded cmd.delta_heading but only logs it as received; it never calls self._pipeline.apply_heading_increment(...). A SONIC command message can therefore successfully carry an operator yaw adjustment while the controller silently ignores it. Forward the decoded value to the pipeline when it is present.
Artifacts
SONIC command delta-heading reproduction script
- This authored script constructs a packed SONIC command message, decodes it with the repository codec, and invokes the repository command handler to assert that delta heading reaches the pipeline; takeaway: the script is the executable reproduction.
SONIC command delta-heading decode capture
- This executed decode-only capture shows the packed command message preserves `planner=True` and `delta_heading=0.75` before command handling; takeaway: the wire decoder correctly receives the heading increment.
SONIC command delta-heading handler failure
- This executed handler capture shows `heading_increments=[]` after handling the decoded command and ends in the assertion that the heading increment was not applied; takeaway: the handler discards a valid operator heading command.
SONIC prior concerns validation log
- This executed source assertion log confirms `set_velocity_command` is in `TASK_EXPOSES`, the SONIC blueprint is self-hosted, and Ubuntu selection subtracts self-hosted blueprints; takeaway: both prior concerns are fixed.
Two mechanisms from the C++ gamepad manager that the port lacked, found by chasing a kneel-entry crash (instant walking-context -> kneel switch made the planner emit a violent drop): - set_mode now stages floor postures exactly like gamepad_manager.hpp: crawling is entered through kneel, elbow crawling through crawling, exits reverse the ladder, one rung per 2 s dwell, advanced from step(). - Forced modes apply the C++ applySpeedAndHeight table: kneel/squat/crawl send the planner height command (0.4 / 0.4 / 0.3) - without it the planner emits a floor-collapse descent instead of a supported kneel - and the crawl/boxing families their canonical 0.7 speed. Also applies command-topic delta_heading (incremental yaw, was decoded and only logged). Verified in the standalone harness at real-time planner pacing: kneel holds at z 0.41 and stands back up; single kneel likewise; crawl -> kneel -> stand recovery works; full 27-mode + 13-clip take with a live chained floor section, zero falls. Genuine policy limitation left: lying face down has no get-up (the C++ transition logic never chains out of lying either).
…feat/g1-sonic-wbc
| shape = tuple(int(v) for v in f.get("shape", [1])) | ||
| nbytes = int(np.prod(shape)) * dtype.itemsize | ||
| if offset + nbytes > len(payload): | ||
| raise ValueError( | ||
| f"{topic}: payload underrun at field {name!r} ({offset + nbytes} > {len(payload)})" | ||
| ) | ||
| arr = np.frombuffer(payload, dtype=dtype, count=int(np.prod(shape)), offset=offset) | ||
| msg.fields[name] = arr.reshape(shape).copy() |
There was a problem hiding this comment.
decode() accepts a declared shape such as [0] and returns an empty array. The task poller assumes decoded control arrays have elements: command fields raise IndexError at flat[0], planner movement raises during reshape(3), and pose data reaches pose handling before failing. A malformed SONIC ZMQ frame can therefore abort a controller tick instead of being rejected. Reject empty, zero, negative, and otherwise invalid dimensions in the decoder, and validate the expected control-field schemas before dispatch.
TomCC7
left a comment
TomCC7
left a comment
There was a problem hiding this comment.
I don't see why we need zmq protocol involved here. @Nabla7 wondering why you choosed this from the first place. do you intend to let our integration work with some external nvidia tool? if not I think only porting the model will be the better option.
There was a problem hiding this comment.
I have no idea why we keep adding model (which was on huggingface) to lfs. should just pull from huggingface
| if len(body) < HEADER_SIZE: | ||
| raise ValueError(f"{topic}: body shorter than header ({len(body)} < {HEADER_SIZE})") | ||
| header_json = body[:HEADER_SIZE].rstrip(b"\x00") | ||
| header: dict[str, Any] = json.loads(header_json) |
There was a problem hiding this comment.
can use pydantic model directly instead of manual decode and field validation
| error: str | None = None | ||
|
|
||
|
|
||
| def infer_protocol_version(fields: dict[str, NDArray]) -> int: |
There was a problem hiding this comment.
this should better be determined once and cached instead of inferring every frame?
| if self._active_protocol is None: | ||
| self._active_protocol = protocol | ||
| elif self._active_protocol != protocol: | ||
| result.error = f"protocol version changed {self._active_protocol} -> {protocol}" |
There was a problem hiding this comment.
I wonder in what case would version change
Review feedback: nothing points NVIDIA's ZMQ tooling at dimos, and the teleop work built on this task talks to the pipeline directly (zenoh), so the socket layer was dead code. Removes zmq_wire.py, the task's socket/poll/publish plumbing, and the pyzmq dependency. The streamed motion merger and the pose-fields contract stay; that is what teleop consumes.
Review feedback: the version is a property of the sender, not of each chunk. Establish it from the first chunk, hold it until reset(), and keep the per-chunk field check purely as a mid-stream sender-change guard (the C++ endpoint errors on that too).
2 participants
Adds NVIDIA GEAR-SONIC v1.1 as a unified 29-joint G1 controller, with a standalone Zenoh blueprint for MuJoCo and hardware.
ControlCoordinatoraccepts velocity commands, selectable locomotion modes, motion clips and optional numeric reference targets. The controller runs independently of headset integrations.Tasks load and prepare their initial hold before adapters activate. Hardware starts unarmed in dry-run, ramps from the measured pose when armed, and continuously refreshes hold commands. SONIC faults latch damping; the independent DDS publisher also checks command and feedback freshness. SONIC opts into bounded Zenoh delivery so busy subscribers discard old targets.
dimos-sonic-modelsexplicitly installs the pinned v1.1 policy, planner and observation configuration from Hugging Face, plus 13 clips from pinned NVIDIA sources, and verifies SHA-256 hashes in the shared DimOS cache.bin/hardware/g1/setup-sonicdetects JetPack 5/6, checks CUDA/cuDNN prerequisites, and installs the matching ONNX Runtime and model assets. Setup and RPC examples.Floor commands keep squat/kneeling stationary and stop crawl translation at neutral input. Face-down mode 7 is excluded from the selectable menu, matching NVIDIA's deployment menu.
Validation:
Limits: the combined installer has not been executed on a Jetson, and this update has no new physical-robot activation. Crawl stability, timed floor transitions and hardware watchdog timing still need validation. Interrupted simulation startup can hit the existing stale-SHM attachment race; the first validation attempt hit it, and a clean restart passed. Full-repository CI remains separate from these local checks.