feat(g1): add PICO SONIC WebXR teleoperation

AGENTS.md

@@ -44,7 +44,7 @@ dimos restart          # stop + re-run with same original args
 | `xarm-perception-agent` | xArm | real | gpt-5.6-luna | ✓ | Manipulation + perception + agent |
 | `xarm-perception-sim-agent` | xArm | sim | gpt-5.6-luna | ✓ | Manipulation + perception + agent, sim |
 | `xarm7-planner-coordinator` | xArm7 | real | — | — | Trajectory planner coordinator |
-| `teleop-quest-xarm7` | xArm7 | real | — | — | Quest VR teleop |
+| `teleop-webxr-xarm7` | xArm7 | real | — | — | WebXR teleop |
 | `dual-xarm6-planner-coordinator` | xArm6×2 | mock | — | — | Dual-arm motion planner |
 
 Run `dimos list` for the full list.

CONTEXT.md

@@ -0,0 +1,45 @@
+# Teleoperation
+
+This context defines the operator-tracking language used when human motion guides a robot through dimOS.
+
+## Language
+
+**Full-body SONIC teleoperation**:
+An operating mode in which tracked motion of the operator's legs, pelvis, torso, arms, wrists, and head guides the G1 through SONIC. Articulated fingers are excluded.
+_Avoid_: Full-body tracking, three-point teleoperation
+
+**WebXR body snapshot**:
+A set of named operator-joint poses observed together in one WebXR reference space.
+_Avoid_: SMPL frame, SONIC pose
+
+**Native-equivalent SONIC pose**:
+An operator pose whose meaning matches the native SONIC PICO input, independently of the transport used to deliver it.
+_Avoid_: Partial SONIC pose, policy-minimal pose
+
+**SONIC retargeting**:
+The dimOS-owned interpretation of a WebXR body snapshot as a native-equivalent SONIC pose.
+_Avoid_: Packet packing, vendor conversion
+
+**Motion Tracker calibration**:
+The PICO-managed process that establishes the operator skeleton and tracker relationship before a WebXR session.
+_Avoid_: Teleoperation alignment
+
+**Teleoperation alignment**:
+The per-engagement association between a complete operator skeleton and the G1 heading. It does not reshape or rescale the operator.
+_Avoid_: Calibration, body normalization
+
+**Full-body engagement**:
+The interval while the operator holds the WebXR deadman control and fresh, complete body poses guide SONIC. Ending engagement returns SONIC to planner control without disarming the policy.
+_Avoid_: Policy arming, robot activation
+
+**Complete operator pose**:
+A WebXR body snapshot containing every joint required to produce one native-equivalent SONIC pose. Partial snapshots do not advance the robot target.
+_Avoid_: Best-effort pose, partial target
+
+**Operator motion source**:
+The engaged operator pose as SONIC's sole whole-body motion reference. Planner locomotion resumes after full-body engagement ends.
+_Avoid_: Hybrid locomotion, simultaneous planner control
+
+**Live PICO simulation test**:
+An acceptance session in which a physical PICO and Motion Trackers guide the MuJoCo G1 through the production WebXR path.
+_Avoid_: Synthetic replay, converter smoke test

MUJOCO_LOG.TXT

@@ -0,0 +1,17 @@
+WARNING Thu Aug 27 10:29:16 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 10:39:37 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 10:43:33 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 10:56:21 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 11:09:50 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 11:11:17 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 11:12:41 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 11:27:40 2026: OpenGL error 0x502 in or before mjr_makeContext
+
+WARNING Thu Aug 27 11:32:50 2026: OpenGL error 0x502 in or before mjr_makeContext

bin/hardware/g1/setup-sonic-jp5

@@ -0,0 +1,196 @@
+#!/usr/bin/env bash
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Install the pinned CUDA ONNX Runtime wheel for a JetPack 5 G1 PC2.
+set -euo pipefail
+
+REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)
+CUDA_HOME_JP5=/usr/local/cuda-11.8
+SONIC_VENV="$REPO_ROOT/.venv-sonic-jp5"
+SONIC_PYTHON_VERSION=3.10
+ORT_AUTO_VERSION=1.23.3
+ORT_DISTRIBUTION_VERSION=1.18.1.11.8
+ORT_VERSION=1.18.1
+CHECK_ONLY=false
+FAILED=0
+
+usage() {
+    echo "usage: bin/hardware/g1/setup-sonic-jp5 [--check]"
+}
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --check) CHECK_ONLY=true ;;
+        -h|--help) usage; exit 0 ;;
+        *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
+    esac
+    shift
+done
+
+pass() {
+    echo "PASS  $1"
+}
+
+fail() {
+    echo "FAIL  $1" >&2
+    FAILED=$((FAILED + 1))
+}
+
+[[ -f "$REPO_ROOT/pyproject.toml" && -f "$REPO_ROOT/uv.lock" ]] \
+    && pass "DimOS source checkout: $REPO_ROOT" \
+    || fail "run this script from a DimOS source checkout"
+
+[[ $(uname -m) == aarch64 ]] \
+    && pass "architecture: aarch64" \
+    || fail "expected the G1 PC2 aarch64 host, found $(uname -m)"
+
+if [[ -r /etc/nv_tegra_release ]] && head -n 1 /etc/nv_tegra_release | grep -q '^# R35'; then
+    pass "Jetson Linux: $(head -n 1 /etc/nv_tegra_release)"
+else
+    fail "expected Jetson Linux R35 (JetPack 5)"
+fi
+
+[[ -e "$CUDA_HOME_JP5/lib64/libcudart.so.11.0" ]] \
+    && pass "CUDA 11.8 runtime: $CUDA_HOME_JP5/lib64/libcudart.so.11.0" \
+    || fail "CUDA 11.8 runtime is missing from $CUDA_HOME_JP5"
+
+[[ -e "$CUDA_HOME_JP5/compat/libcuda.so" ]] \
+    && pass "CUDA 11.8 compatibility driver: $CUDA_HOME_JP5/compat/libcuda.so" \
+    || fail "cuda-compat-11-8 is missing"
+
+[[ -e /usr/lib/aarch64-linux-gnu/libcudnn.so.8 ]] \
+    && pass "cuDNN 8: /usr/lib/aarch64-linux-gnu/libcudnn.so.8" \
+    || fail "cuDNN 8 is missing"
+
+command -v uv >/dev/null 2>&1 && pass "uv: $(command -v uv)" || fail "uv is missing"
+
+if [[ "$FAILED" -ne 0 ]]; then
+    cat >&2 <<'EOF'
+
+Prerequisites failed. This setup does not flash JetPack and does not install
+system packages. For CUDA, use NVIDIA's CUDA 11.8 archive and select:
+  Linux / aarch64-jetson / Ubuntu / 20.04 / deb
+Then install the versioned packages:
+  sudo apt-get install cuda-runtime-11-8 cuda-compat-11-8
+https://developer.nvidia.com/cuda-11-8-0-download-archive
+EOF
+    exit 1
+fi
+
+if [[ "$CHECK_ONLY" == true ]]; then
+    echo "JetPack 5 SONIC prerequisites passed."
+    exit 0
+fi
+
+export PATH="$CUDA_HOME_JP5/bin:$PATH"
+export LD_LIBRARY_PATH="$CUDA_HOME_JP5/compat:$CUDA_HOME_JP5/lib64:/usr/lib/aarch64-linux-gnu${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+
+if [[ ! -x "$SONIC_VENV/bin/python" ]] \
+    || [[ $("$SONIC_VENV/bin/python" -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') != "$SONIC_PYTHON_VERSION" ]]; then
+    uv venv --clear --python "$SONIC_PYTHON_VERSION" "$SONIC_VENV"
+fi
+DIMOS_ALLOW_MISSING_COCKPIT=1 VIRTUAL_ENV="$SONIC_VENV" uv sync \
+    --active \
+    --python "$SONIC_VENV/bin/python" \
+    --locked \
+    --no-default-groups \
+    --extra unitree-dds \
+    --inexact \
+    --no-install-package gtsam-extended \
+    --no-install-package onnxruntime
+uv pip install \
+    --python "$SONIC_VENV/bin/python" \
+    "numpy==1.26.4" \
+    pip
+
+"$SONIC_VENV/bin/python" -m pip uninstall --yes \
+    onnxruntime \
+    onnxruntime-gpu \
+    onnxruntime-gpu-extended \
+    onnxruntime-gpu-extended-auto
+"$SONIC_VENV/bin/python" -m pip install \
+    --no-cache-dir \
+    "onnxruntime-gpu-extended-auto==$ORT_AUTO_VERSION"
+
+"$SONIC_VENV/bin/python" - \
+    "$ORT_AUTO_VERSION" \
+    "$ORT_DISTRIBUTION_VERSION" \
+    "$ORT_VERSION" <<'PY'
+import sys
+from importlib.metadata import version
+
+try:
+    import onnxruntime as ort
+except ImportError as exc:
+    msg = "onnxruntime-gpu-extended installation did not provide onnxruntime"
+    raise ImportError(msg) from exc
+import onnxruntime as ort
+
+expected_auto, expected_distribution, expected_runtime = sys.argv[1:]
+actual_auto = version("onnxruntime-gpu-extended-auto")
+actual_distribution = version("onnxruntime-gpu-extended")
+actual_runtime = ort.__version__
+providers = ort.get_available_providers()
+
+expected = (expected_auto, expected_distribution, expected_runtime)
+actual = (actual_auto, actual_distribution, actual_runtime)
+if actual != expected:
+    raise SystemExit(f"unexpected ONNX Runtime packages: expected {expected}, found {actual}")
+if "CUDAExecutionProvider" not in providers:
+    raise SystemExit(f"CUDAExecutionProvider unavailable; ONNX Runtime exposes {providers}")
+
+print(
+    "PASS  ONNX Runtime: "
+    f"auto={actual_auto} distribution={actual_distribution} runtime={actual_runtime}"
+)
+print(f"PASS  execution providers: {providers}")
+PY
+
+"$SONIC_VENV/bin/python" "$REPO_ROOT/bin/hardware/g1/setup-sonic-models"
+
+"$SONIC_VENV/bin/python" - <<'PY'
+try:
+    import open3d
+except ImportError as exc:
+    msg = "JetPack-compatible Open3D is missing from the SONIC environment"
+    raise ImportError(msg) from exc
+import open3d
+
+from dimos.cli.dimos import cli_main
+from dimos.core.o3dpickle import register_picklers
+from dimos.robot.unitree.g1.blueprints.basic.unitree_g1_sonic_webxr_teleop import (
+    unitree_g1_sonic_webxr_teleop,
+)
+
+register_picklers()
+print(f"PASS  Open3D: {open3d.__version__}")
+print("PASS  DimOS CLI and SONIC WebXR blueprint imports")
+PY
+
+cat <<EOF
+
+JetPack 5 SONIC environment is ready.
+
+Before each run:
+  export DIMOS_TRANSPORT=zenoh
+  export PATH=$CUDA_HOME_JP5/bin:\$PATH
+  export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu\${LD_LIBRARY_PATH:+:\$LD_LIBRARY_PATH}
+  export LD_LIBRARY_PATH=$CUDA_HOME_JP5/lib64:\$LD_LIBRARY_PATH
+  export LD_LIBRARY_PATH=$CUDA_HOME_JP5/compat:\$LD_LIBRARY_PATH
+  source $SONIC_VENV/bin/activate
+  dimos hardware g1 sonic-doctor
+
+Use the activated environment's dimos command directly. Do not use uv run.
+EOF

bin/hardware/g1/setup-sonic-models

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+
+"""Download and verify the official NVIDIA SONIC policy bundles."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+from pathlib import Path
+
+try:
+    from huggingface_hub import hf_hub_download
+except ImportError as exc:
+    msg = "huggingface_hub is required; run this command through the DimOS environment"
+    raise ImportError(msg) from exc
+from huggingface_hub import hf_hub_download
+
+from dimos.utils.data import get_data
+
+REPO_ID = "nvidia/GEAR-SONIC"
+REVISION = "6733128a3d8a523b1418b06bca3cdf61c8b0987f"
+FILES: dict[str, str | None] = {
+    "sonic_v1_1/model_encoder.onnx": "fb97de22819b2057b41459802128d91723d91a25f0ad73e7bfc41a9cf8365bae",
+    "sonic_v1_1/model_decoder.onnx": "34bae8570d4a4421a5391a5c2befd745d4a02d182ec539e5f9da44c091c67509",
+    "sonic_v1_1/observation_config.yaml": None,
+    "low_latency/model_encoder.onnx": "60be43157f57d812f38bdbb740a5de5d5d070e8840d9edc16f02a91a6d06255b",
+    "low_latency/model_decoder.onnx": "c4ac2e74045e7cbfb568f15e6bf47ea7ce023df7a94322af50be223e0a628bab",
+    "low_latency/observation_config.yaml": None,
+}
+
+
+def _sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for chunk in iter(lambda: stream.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def _verify(path: Path, expected_sha: str | None) -> None:
+    if not path.is_file():
+        raise RuntimeError(f"missing {path}")
+    if expected_sha is not None:
+        actual_sha = _sha256(path)
+        if actual_sha != expected_sha:
+            raise RuntimeError(
+                f"bad SHA-256 for {path}: expected {expected_sha}, found {actual_sha}"
+            )
+    print(f"PASS  SONIC model: {path}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--check", action="store_true")
+    parser.add_argument(
+        "--profile",
+        choices=("all", "sonic-v1.1", "sonic-low-latency"),
+        default="all",
+    )
+    parser.add_argument(
+        "--destination",
+        type=Path,
+        default=None,
+    )
+    args = parser.parse_args()
+
+    default_destination = Path(__file__).resolve().parents[3] / "data" / "sonic"
+    destination = args.destination or default_destination
+    if args.destination is None and not args.check and not destination.is_dir():
+        print("Materializing the shared SONIC planner and motion data...")
+        destination = get_data("sonic")
+
+    prefix = {
+        "all": None,
+        "sonic-v1.1": "sonic_v1_1/",
+        "sonic-low-latency": "low_latency/",
+    }[args.profile]
+    selected_files = {
+        filename: expected_sha
+        for filename, expected_sha in FILES.items()
+        if prefix is None or filename.startswith(prefix)
+    }
+    for filename, expected_sha in selected_files.items():
+        path = destination / filename
+        try:
+            _verify(path, expected_sha)
+            continue
+        except RuntimeError:
+            if args.check:
+                raise
+
+        if not args.check:
+            path = Path(
+                hf_hub_download(
+                    repo_id=REPO_ID,
+                    filename=filename,
+                    revision=REVISION,
+                    local_dir=destination,
+                )
+            )
+        _verify(path, expected_sha)
+
+
+if __name__ == "__main__":
+    main()