Skip to content

Install ovrtx-extensions from the wheelhouse and enable OVRTX crash upload - #7651

Draft
mataylor-nvidia wants to merge 2 commits into
isaac-sim:developfrom
mataylor-nvidia:mataylor/ovrtx-extensions-crash-upload
Draft

Install ovrtx-extensions from the wheelhouse and enable OVRTX crash upload#7651
mataylor-nvidia wants to merge 2 commits into
isaac-sim:developfrom
mataylor-nvidia:mataylor/ovrtx-extensions-crash-upload

Conversation

@mataylor-nvidia

@mataylor-nvidia mataylor-nvidia commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Description

Adds ovrtx-extensions as a second wheelhouse package next to ovphysx, and uses it to turn on Omniverse crash-report upload for the OVRTX runtime.

ovrtx-extensions is published to NVIDIA's internal Artifactory index (package) rather than PyPI or pypi.nvidia.com, so the wheelhouse is the only path that can reach it. That path is already gated to develop (USE_OVPHYSX_WHEELHOUSE), which keeps this off pull requests from forks and off every release branch.

What changed

  • .github/workflows/build.yamlwheelhouse-packages becomes ovphysx ovrtx-extensions in the three jobs that already mount the wheelhouse: isaaclab_ov, rendering-correctness-kitless (legacy), and rendering-correctness-kitless (ovstage).
  • .github/actions/run-tests/run_tests.sh — sets ISAACLAB_OVRTX_CRASH_UPLOAD=1 on the test container when, and only when, the wheelhouse package list contains ovrtx-extensions.
  • source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_crash_reporter.py — applies the --/crashreporter/... settings through ovrtx_extensions.apply_settings, gated on that variable.
  • source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py — calls it from OVRTXRenderer.__init__, after redirect_shader_cache and before Renderer(OVRTX_CONFIG).
  • .github/workflows/config.yaml — comment recording what the wheelhouse image has to contain, since it is empty today and a name missing from it fails the job under pip --no-index.
  • Tests in source/isaaclab_ov/test/test_ovrtx_crash_reporter.py, plus a changelog fragment for isaaclab_ov.

Settings applied

--/crashreporter/enabled=true
--/crashreporter/url="https://services.nvidia.com/submit"
--/crashreporter/product="Omniverse.ovrtx"
--/crashreporter/version="<importlib.metadata.version('ovrtx')>"
--/crashreporter/preserveDump=true
--/crashreporter/gatherUserStory=false
--/crashreporter/devOnlyOverridePrivacyAndForceUpload=true
--/crashreporter/alwaysUpload=true

Why the settings are applied at renderer construction

The first revision of this PR ran the shim from a setup step in run_tests.sh. That was wrong: carb settings are process-local, so they died with the short-lived helper, and the pytest process that actually builds the renderer — and actually crashes — was never configured. Fixed in baee515dc5 by moving the call into OVRTXRenderer.__init__.

It takes the RendererConfig for the same reason redirect_shader_cache does. Applying a setting can be what first loads the ovrtx library, ovrtx_initialize runs exactly once per process with the config of whichever caller loads it first (ovrtx._src.bindings._LibraryLoader.create_bindings), and ovrtx_extensions has no config of its own to hand it. Loading the bindings with the renderer's real config first is what keeps the log_file_path, log_level and keep_system_alive that tools/ovrtx_log.py replays per test.

Still inert until the wheelhouse image is set. ovphysx_wheelhouse_image is "" on develop today, so the wheelhouse path is disabled and the env var is never set. It also needs an image that ships an ovrtx-extensions wheel plus its dependencies — pip --no-index cannot fall back to an index.

Type of change

  • New feature (non-breaking change which adds functionality)

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

source/isaaclab_ov/test/test_ovrtx_crash_reporter.py covers the settings contract, that every setting is applied, that the renderer's config reaches the applier, that an unset variable touches nothing, and that an explicit request without ovrtx_extensions raises. It runs with no GPU, no ovrtx runtime and no renderer, mirroring test_ovrtx_shader_cache_redirect.py.

Add ovrtx-extensions alongside ovphysx in the develop-gated wheelhouse
package list, so the three jobs that already mount the wheelhouse
(isaaclab_ov and both kitless rendering-correctness variants) pick it up.

With the package present, run tools/enable_ovrtx_crash_upload.py after the
offline install to turn on Omniverse crash-report upload for the OVRTX
runtime. The script no-ops everywhere else: ovrtx-extensions ships on
NVIDIA's internal index rather than PyPI, so it is never in the package
list outside those jobs.

The script runs as its own process on purpose. Applying an ovrtx setting is
what first loads the ovrtx library, and ovrtx_initialize runs once per
process with the config of whichever caller loads it first, so applying
these ahead of the renderer inside pytest would take the renderer's log
path, log level and keep-alive away from it.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The develop-gated wheelhouse wiring installs ovrtx-extensions in the three intended CI jobs, but the crash-upload configuration is applied only in a short-lived helper process and therefore does not configure the pytest processes whose renderer crashes the change is intended to capture.

  • Design and architecture: The standalone helper conflicts with the documented process-local OVRTX initialization model. It exits before pytest starts, so renderer construction and test execution remain unaffected while the jobs gain an additional required step. Apply the settings in the renderer process alongside construction with the real RendererConfig, or defer this integration until that path is available.
  • API: No public package API under source/ changes. The existing workflow input shapes and develop-only wheelhouse gating are preserved, and no source-package changelog fragment is required.
  • Implementation: The wheelhouse package list and conditional shell dispatch are internally consistent, and the wheelhouse image requirements are documented. However, the invocation at run_tests.sh:379 cannot enable crash reporting for pytest because tools/enable_ovrtx_crash_upload.py runs in a separate interpreter that terminates before line 418 starts pytest.

Significant concerns. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

Comment thread .github/actions/run-tests/run_tests.sh Outdated

case \" \${TEST_WHEELHOUSE_PACKAGES} \" in
*\" ovrtx-extensions \"*)
./isaaclab.sh -p tools/enable_ovrtx_crash_upload.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Design Architecture — Crash settings do not reach pytest process

The helper runs as its own interpreter and exits before pytest starts (line 418), while the script's docstring states these settings and ovrtx_initialize are per process. As written the step enables no crash upload for any test while adding a new hard-failure path (return 1 under set -e) to three jobs. Consider applying the settings at renderer construction with the real RendererConfig, or deferring the step until that is possible.

@greptile-apps

greptile-apps Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with no actionable new defects or outstanding findings identified.

Summary

  • Propagates the crash-upload opt-in environment variable to applicable test containers.
  • Applies crash-reporter settings before OVRTX renderer construction while preserving the renderer configuration used for runtime initialization.
  • Adds unit coverage for opt-in behavior, settings, version attribution, configuration forwarding, and missing-extension failures.
  • Documents the wheelhouse image requirements and adds a changelog fragment.

Diagram

sequenceDiagram
    participant Workflow as Build workflow
    participant Runner as run_tests.sh
    participant Pytest as Pytest container
    participant OVRTX as OVRTXRenderer
    participant Reporter as ovrtx_extensions
    participant Renderer as ovrtx.Renderer

    Workflow->>Runner: wheelhouse-packages includes ovrtx-extensions
    Runner->>Pytest: Install wheelhouse packages with --no-index
    Runner->>Pytest: "ISAACLAB_OVRTX_CRASH_UPLOAD=1"
    Pytest->>OVRTX: Construct renderer
    OVRTX->>OVRTX: Redirect shader cache
    OVRTX->>Reporter: Apply crash-report settings
    OVRTX->>Renderer: Construct with OVRTX_CONFIG
Loading

Comment thread .github/actions/run-tests/run_tests.sh Outdated
The setup-step shim applied the settings in a short-lived helper process. Carb
settings are process-local, so they died with it: pytest started afterwards in
its own process, built the renderer there, and a crash in that process was
never configured to upload anything.

Move the shim into isaaclab_ov as ovrtx_crash_reporter and call it from
OVRTXRenderer.__init__ before Renderer() is constructed, so the settings apply
in the process that renders and crashes. It takes the RendererConfig for the
same reason redirect_shader_cache does: applying a setting can be what loads
the ovrtx library, ovrtx_initialize runs once per process, and ovrtx_extensions
has no config of its own to hand it, so the renderer's config is loaded first
rather than losing its log sink, log level and keep-alive.

Behavior stays CI-only through ISAACLAB_OVRTX_CRASH_UPLOAD, which run_tests.sh
sets on the container only when the wheelhouse installs ovrtx-extensions. The
variable set without the package raises, since that means a broken install
rather than an unsupported runtime.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant