Skip to content

refactor(imitation): isolate LeRobot policy runtime - #3315

Open
TomCC7 wants to merge 15 commits into
fix/openyam-learning-baselinefrom
feat/experimental-lerobot-policy
Open

TomCC7 wants to merge 15 commits into
fix/openyam-learning-baselinefrom
feat/experimental-lerobot-policy

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jul 31, 2026

Copy link
Copy Markdown
Member

Contribution path

Problem

  • LeRobot's dependency environment conflicts with the host stack.
  • Policy chunks need explicit execution, validation, preemption, and cancellation through the coordinator.

Solution

  • Run LeRobot in an isolated Python project behind the host LeRobotPolicyModule contract.
  • Own generic rollout behavior here: checkpoint/live-input preflight, ordered action chunks, freshness checks, and explicit start/stop.
  • Add named trajectory execution/cancellation, joint-limit validation, accepted-command telemetry, and idle-hold support to the control layer.
  • Keep cancellation ordered after in-flight submissions and report uncertain cancellation.
  • Publish debounced Quest button edges while retaining continuous grip/trigger inputs.
  • Keep robot-specific rollout composition in feat(imitation): add controlled OpenYAM policy rollout #3855; collection consumes accepted-command telemetry in feat(imitation): add native OpenYAM collection #3854.

API shape

LeRobotPolicyModule
  inputs: color_image: Image, coordinator_joint_state: JointState
          button_pressed: Buttons (optional operator input)
  config: policy_path, task, device, joint_names, fps,
          image_width, image_height, max_observation_age_s,
          trajectory_task_name, rollout_button
  control: PolicyControlSpec → execute_trajectory / cancel_trajectory
  • RolloutControlSpec: preflight_rollout(), start_rollout(), stop_rollout(), rollout_status().
  • Preflight loads/validates the checkpoint and current inputs without sending motion.
  • This is the existing single-camera LeRobot contract, not a generalized backend/remapping API.

How to Test

Exercise the isolated runtime contract without robot hardware:

cd dimos/imitation/policy/lerobot/python
uv run --isolated --locked --group tests --with-editable ../../../../../ python -m pytest
  • At this layer, 398 affected host tests and 27 isolated-runtime tests passed In the previous cleanup. The final stack also passed 413 control/Quest regression tests and all 36 isolated policy/preparation tests.

  • Scoped type checks and commit hooks passed.

  • Physical rollout and cancellation remain hardware release checks; mock tests are not safety certification.

  • The repository-wide Clippy hook reaches an existing dim_slam flake failure resolving git+file:../../../..; that unrelated flake is unchanged.

AI assistance

  • Review cleanup updated by OpenAI Codex (agent).
  • OpenAI Codex (GPT-5) assisted extensively with implementation, tests, documentation, and stack cleanup.

Checklist

  • I have read and approved the CLA.

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.41697% with 7 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/imitation/policy/lerobot/module.py 92.00% 5 Missing and 1 partial ⚠️
dimos/teleop/webxr/extensions.py 93.75% 1 Missing ⚠️
@@                        Coverage Diff                        @@
##           fix/openyam-learning-baseline    #3315      +/-   ##
=================================================================
+ Coverage                          79.57%   79.60%   +0.02%     
=================================================================
  Files                               1498     1501       +3     
  Lines                             141657   141887     +230     
  Branches                           12130    12141      +11     
=================================================================
+ Hits                              112730   112947     +217     
- Misses                             25502    25517      +15     
+ Partials                            3425     3423       -2     
Components Coverage Δ
Tests 95.38% <100.00%> (+0.01%) ⬆️
Flag Coverage Δ
OS-ubuntu-24.04-arm 75.18% <96.67%> (+0.03%) ⬆️
OS-ubuntu-latest 76.01% <96.67%> (+0.02%) ⬆️
Py-3.10 75.79% <96.67%> (+0.03%) ⬆️
Py-3.11 75.99% <96.67%> (+0.02%) ⬆️
Py-3.12 76.00% <96.67%> (+0.02%) ⬆️
SelfHosted-Large 30.63% <38.74%> (+0.02%) ⬆️
SelfHosted-Linux 36.69% <40.22%> (+<0.01%) ⬆️
SelfHosted-macOS 37.03% <40.22%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...mos/control/tasks/teleop_ik_task/teleop_ik_task.py 90.28% <100.00%> (ø)
...ontrol/tasks/teleop_ik_task/test_teleop_ik_task.py 100.00% <100.00%> (ø)
dimos/imitation/policy/lerobot/test_module.py 100.00% <100.00%> (ø)
dimos/robot/all_blueprints.py 100.00% <ø> (ø)
.../robot/manipulators/common/test_teleop_priority.py 100.00% <100.00%> (ø)
...bot/manipulators/dual_openyam/blueprints/teleop.py 100.00% <ø> (ø)
...robot/manipulators/dual_openyam/test_blueprints.py 100.00% <100.00%> (ø)
...os/robot/manipulators/openarm/blueprints/teleop.py 93.75% <ø> (ø)
.../robot/manipulators/openarm/test_openarm_teleop.py 100.00% <100.00%> (ø)
...os/robot/manipulators/openyam/blueprints/teleop.py 100.00% <ø> (ø)
... and 7 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
Comment thread dimos/experimental/robot_policy/lerobot.py Outdated
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from a8ab2d8 to 486d959 Compare July 31, 2026 23:35
@TomCC7
TomCC7 changed the base branch from main to feat/galaxea-a1z-hardware July 31, 2026 23:35
@TomCC7
TomCC7 marked this pull request as ready for review July 31, 2026 23:58
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 1, 2026
@TomCC7 TomCC7 mentioned this pull request Aug 1, 2026
3 tasks
Base automatically changed from feat/galaxea-a1z-hardware to main August 4, 2026 08:05
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 01aa5d0 to 18ffcc7 Compare August 14, 2026 20:03
@mintlify

mintlify Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
dimensional 🟢 Ready View Preview Aug 14, 2026, 8:04 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 14, 2026
@TomCC7
TomCC7 changed the base branch from main to cc/feat/openyam-macos-support August 14, 2026 20:05
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 18ffcc7 to cda4f1f Compare August 14, 2026 21:37
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from f6a2f76 to 2a46dfb Compare August 14, 2026 21:40
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 2a46dfb to 884a430 Compare August 14, 2026 21:56
@TomCC7
TomCC7 marked this pull request as draft August 14, 2026 21:57
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 884a430 to 2ffca94 Compare August 18, 2026 00:07
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 89cc9f6 to b8a7241 Compare September 1, 2026 17:00
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from b8a7241 to 0c8996a Compare September 1, 2026 17:16
@TomCC7
TomCC7 force-pushed the feat/experimental-lerobot-policy branch from 0c8996a to 564d788 Compare September 1, 2026 17:17
Comment thread dimos/imitation/policy/lerobot/python/dimos_lerobot/runtime.py Outdated
Comment thread dimos/imitation/policy/lerobot/python/dimos_lerobot/runtime.py Outdated
@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This change adds a host-side LeRobot policy integration backed by an isolated Python runtime, including rollout lifecycle handling and joint-command publication.

Two failures were reproduced. Standard wheel installations omit the isolated runtime manifest required to start LeRobot, so rollout startup raises FileNotFoundError. Separately, stopping a rollout can still emit one arm command and one gripper command after the stop request. These issues must be fixed before merge.

Confidence Score: 2/5

Not safe to merge because wheel-installed LeRobot rollouts cannot start and stopped rollouts can still publish robot commands.

The reproduced packaging failure blocks the feature in standard deployments, while the reproduced stop race can issue commands after a stop request.

Files Needing Attention: MANIFEST.in and dimos/imitation/policy/lerobot/python/dimos_lerobot/runtime.py

T-Rex T-Rex Logs

What T-Rex did

  • Executed the installed-wheel runtime-project execution script and captured its output to document the runtime path for the P1 finding.
  • Executed the deterministic stop-to-publication window test and captured the output showing commands after the stop.
  • Produced and documented a third finding-proof for a posted P1 finding.
  • Validated the general-contract scenario by running a controlled sequence where stop_requested_before_arm_publish=True and recording the subsequent one arm and one gripper command after the stop.

View all artifacts

T-Rex Ran code and verified through T-Rex

Comments Outside Diff (1)

  1. General comment

    P1 Stop request can be followed by arm and gripper command publication

    • Bug
      • A deterministic controlled-hook execution reached the post-predict check with stop unset, issued _stop_policy immediately afterward, and observed an arm command plus gripper command publish while the stop event was already set.
    • Cause
      • _run_rollout checks _stop_event at lines 333-336 but performs publication later at lines 347-354 without a synchronization mechanism or a second stop check that closes the intervening window.
    • Fix
      • Serialize stop and publication with a shared lock/protocol, or perform a final stop-state check immediately before each publication while ensuring a concurrent stop cannot occur between that check and publish.

    T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "fix(imitation): tighten LeRobot runtime ..." | Re-trigger Greptile

Comment thread MANIFEST.in Outdated
Comment thread dimos/imitation/policy/lerobot/python/dimos_lerobot/runtime.py Outdated
Comment thread dimos/teleop/webxr/module.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants