feat(imitation): add native OpenYAM collection

[TomCC7]

Contribution path

• PRD: Manipulation SDK preview, discussion #3923.
• Stacked on refactor(imitation): isolate LeRobot policy runtime #3315.

Problem

• Fixed recorder ports and a separate dataset profile can disagree about stream names, camera counts, and joint order.
• Offline preparation needs the recording's original schema even after the robot configuration changes.

Solution

• Introduce the final CollectionProfile and recorder factory here, before robot presets depend on them.

• Generate typed input ports before autoconnect. Capture each distinct source once, even when several dataset features project from it.

• Save schema.json beside recording.mcap or recording.db in a new recording directory.

• Prepare MCAP/SQLite through the saved schema, with LeRobot conversion isolated from host dependencies and HDF5 available through the Python API.

• Declare raw-source meaning with source_kind, not a selectable sampling algorithm. Snapshots retain nearest alignment; partial joint-position updates reconstruct a causal target state before feature projection.

• Read each raw source once per episode, including command history before the episode. Share evaluation between inspect and prepare.

• Validate shared-source consistency and reject malformed updates or missing initial joints. Do not borrow future commands or fabricate missing targets.

• Define EpisodeControlSpec and canonical episode messages. Stopping leaves an active episode incomplete; incomplete/discarded episodes are excluded from export.

• Add the ordinary openyam-quest-collection Blueprint, with measured state and accepted joint commands.

• Keep native build/schema pins, camera startup/shutdown fixes, and recorder integration tests in this owning layer.

• Remove obsolete dynamic-profile DataPrep CLI paths. Keep the Python recorder used by Piper, xArm, and G1.

• Pin Livox, RealSense, and dimSLAM to the shared native message revision and update the standalone lockfiles.

• Retry pytest only when the initial test step failed, not after native setup/build failures.

API shape

Declarations below omit validation/default details:

class CollectionFeature(FeatureSpec):
    message_type: type[Any]
    # FeatureSpec supplies stream, field, dtype, shape, names, and:
    source_kind: Literal["snapshot", "joint_position_updates"] = "snapshot"

class CollectionProfile(BaseConfig):
    name: str
    robot_type: str
    observations: dict[str, CollectionFeature]
    actions: dict[str, CollectionFeature]
    sync: SyncConfig
    quality: QualityConfig

def collection_recorder(
    *,
    profile: CollectionProfile,
    recording: Path | None = None,
    format: Literal["mcap", "sqlite"] = "mcap",
    instance_name: str = "recorder",
) -> Blueprint: ...

collect = autoconnect(
    my_robot,
    *my_cameras,
    collection_recorder(profile=MY_PROFILE),
    EpisodeMonitorModule.blueprint(instance_name="episodes"),
)

• source_kind="snapshot" applies to measured state and images, with existing nearest-frame tolerance. source_kind="joint_position_updates" applies to partial target updates: omitted joints retain their last target across ticks and episode boundaries.
• Source meaning is independent of whether a feature is an observation or action. Shared-source projections must agree.
• There is no legacy sampling alias or automatic schema migration. Existing recordings are not rewritten.
• Profiles have no registry. Ordinary Blueprint discovery selects the setup; the profile name is recording metadata.
• The graph constructs cameras, hardware, and transports; image features do not create devices.
• Add camera features and matching producers without a recorder subclass. Normal remappings resolve different names.
• Source ports must be nonreserved Python identifiers with importable, native-encodable message types. The recorder reserves status: In[EpisodeStatus].
• EpisodeControlSpec exposes get_status() and command("start" | "save" | "discard" | "toggle").

recordings/session-001/
├── schema.json
└── recording.mcap  # or recording.db

• Existing directories are rejected. Move/copy the complete directory.
• Python preparation: RecordingSchema.read(directory).dataprep_config(directory, output), then run_lerobot_dataprep(config) or run_dataprep(config).
• Only prepare trusted recordings, with their custom message packages installed.
• CLI panels come in feat(imitation): add CLI learning workflow #3931; dual-arm presets come in feat(imitation): add profile-driven dual-arm collection #3942.

How to Test

With OpenYAM, Quest, and a wrist RGB camera configured:

dimos --can-port follower_l run openyam-quest-collection --recorder.recording recordings/session-001 --episodes.task "pick up the cube"

• Quest B starts/saves; Y discards. Repeat with a new directory and --recorder.format sqlite.

• In the previous cleanup: 149 focused host tests, 36 isolated LeRobot tests, 4 native MCAP/SQLite round-trip tests, and 6 registry checks passed.

• The native OpenYAM recording test also passes using synthetic messages; real collection remains a hardware release check.

• Current review cleanup: 140 dataprep/collection tests passed at this layer, including source-kind serialization, shared-source single reads, causal reconstruction, and native MCAP/SQLite round trips with 1/2/4 cameras. Mypy passed for the three changed production files.

• Keep raw-store round-trip interpretation tests in dataprep/test_recording.py; raw capture itself remains unaligned.

• Generic camera-count and 14-D conversion tests use synthetic profiles rather than importing the later dual-arm preset.

• CI repair: locked workspace Clippy and Rust tests passed; the Nix recorder build passed with the existing vendoring hash. All seven native MCAP/SQLite interoperability tests passed, and all three Cargo graphs resolve one shared message source.

AI assistance

• Review cleanup updated by OpenAI Codex (agent).
• OpenAI Codex (GPT-5) assisted extensively with implementation, tests, documentation, and stack cleanup.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 7 Tests Failed:

Tests completed	Failed	Passed	Skipped
6077	7	6070	102

View the top 2 failed test(s) by shortest run time

dimos.experimental.memory.test_rust_recorder_e2e::test_native_collection_directory_preserves_schema_and_messages[mcap]

Stack Traces | 0.001s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

dimos.experimental.memory.test_rust_recorder_e2e::test_native_collection_directory_preserves_schema_and_messages[sqlite]

Stack Traces | 0.001s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

View the full list of 5 ❄️ flaky test(s)

dimos.experimental.memory.test_rust_recorder_e2e::test_cli_recording_uses_existing_binary_for_both_formats[mcap]

Flake rate in main: 34.78% (Passed 30 times, Failed 16 times)

Stack Traces | 0.002s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

dimos.experimental.memory.test_rust_recorder_e2e::test_cli_recording_uses_existing_binary_for_both_formats[sqlite]

Flake rate in main: 34.78% (Passed 30 times, Failed 16 times)

Stack Traces | 0.001s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

dimos.experimental.memory.test_rust_recorder_e2e::test_rust_artifact_is_readable_by_python_memory2[mcap]

Flake rate in main: 34.78% (Passed 30 times, Failed 16 times)

Stack Traces | 0.001s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

dimos.experimental.memory.test_rust_recorder_e2e::test_rust_artifact_is_readable_by_python_memory2[sqlite]

Flake rate in main: 34.78% (Passed 30 times, Failed 16 times)

Stack Traces | 30.8s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

dimos.experimental.memory.test_rust_recorder_e2e::test_tf_records_over_zenoh_and_replays_through_python

Flake rate in main: 34.78% (Passed 30 times, Failed 16 times)

Stack Traces | 0.002s run time

@pytest.fixture(scope="module")
    def rust_recorder_executable() -> Path:
        # Build locally. Should already be built in CI.
        if not _EXECUTABLE.is_file():
>           subprocess.run(
                [
                    "nix",
                    "--extra-experimental-features",
                    "nix-command flakes",
                    "build",
                    "-L",
                    ".#dimos-memory-recorder",
                    "--no-write-lock-file",
                ],
                cwd=_RUST_PACKAGE,
                check=True,
            )


.../experimental/memory/test_rust_recorder_e2e.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
kwargs = {'cwd': PosixPath('.../experimental/memory/rust')}
process = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
stdout = None, stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', '--no-write-lock-file']' returned non-zero exit status 1.

capture_output = False
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../experimental/memory/rust')}
popenargs  = (['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '-L', '.#dimos-memory-recorder', ...],)
process    = <Popen: returncode: 1 args: ['nix', '--extra-experimental-features', 'nix-co...>
retcode    = 1
stderr     = None
stdout     = None
timeout    = None

.../github/home/.local.../uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[TomCC7]

why is this necessary can you confirm? or maybe need to be a separate PR?

[TomCC7]

is this CI change necessary? is that because we added turbo jpeg encoding support for rust recorder???

[TomCC7]

if we remove support for turbojpeg encoding can we avoid changing this file?