Install ovrtx-extensions from the wheelhouse and enable OVRTX crash upload - #7651
Install ovrtx-extensions from the wheelhouse and enable OVRTX crash upload#7651mataylor-nvidia wants to merge 2 commits into
Conversation
Add ovrtx-extensions alongside ovphysx in the develop-gated wheelhouse package list, so the three jobs that already mount the wheelhouse (isaaclab_ov and both kitless rendering-correctness variants) pick it up. With the package present, run tools/enable_ovrtx_crash_upload.py after the offline install to turn on Omniverse crash-report upload for the OVRTX runtime. The script no-ops everywhere else: ovrtx-extensions ships on NVIDIA's internal index rather than PyPI, so it is never in the package list outside those jobs. The script runs as its own process on purpose. Applying an ovrtx setting is what first loads the ovrtx library, and ovrtx_initialize runs once per process with the config of whichever caller loads it first, so applying these ahead of the renderer inside pytest would take the renderer's log path, log level and keep-alive away from it.
There was a problem hiding this comment.
Isaac Lab Review Bot
The develop-gated wheelhouse wiring installs ovrtx-extensions in the three intended CI jobs, but the crash-upload configuration is applied only in a short-lived helper process and therefore does not configure the pytest processes whose renderer crashes the change is intended to capture.
- Design and architecture: The standalone helper conflicts with the documented process-local OVRTX initialization model. It exits before pytest starts, so renderer construction and test execution remain unaffected while the jobs gain an additional required step. Apply the settings in the renderer process alongside construction with the real
RendererConfig, or defer this integration until that path is available. - API: No public package API under
source/changes. The existing workflow input shapes and develop-only wheelhouse gating are preserved, and no source-package changelog fragment is required. - Implementation: The wheelhouse package list and conditional shell dispatch are internally consistent, and the wheelhouse image requirements are documented. However, the invocation at
run_tests.sh:379cannot enable crash reporting for pytest becausetools/enable_ovrtx_crash_upload.pyruns in a separate interpreter that terminates before line 418 starts pytest.
Significant concerns. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
|
|
||
| case \" \${TEST_WHEELHOUSE_PACKAGES} \" in | ||
| *\" ovrtx-extensions \"*) | ||
| ./isaaclab.sh -p tools/enable_ovrtx_crash_upload.py |
There was a problem hiding this comment.
🟡 Warning · Design Architecture — Crash settings do not reach pytest process
The helper runs as its own interpreter and exits before pytest starts (line 418), while the script's docstring states these settings and ovrtx_initialize are per process. As written the step enables no crash upload for any test while adding a new hard-failure path (return 1 under set -e) to three jobs. Consider applying the settings at renderer construction with the real RendererConfig, or deferring the step until that is possible.
The setup-step shim applied the settings in a short-lived helper process. Carb settings are process-local, so they died with it: pytest started afterwards in its own process, built the renderer there, and a crash in that process was never configured to upload anything. Move the shim into isaaclab_ov as ovrtx_crash_reporter and call it from OVRTXRenderer.__init__ before Renderer() is constructed, so the settings apply in the process that renders and crashes. It takes the RendererConfig for the same reason redirect_shader_cache does: applying a setting can be what loads the ovrtx library, ovrtx_initialize runs once per process, and ovrtx_extensions has no config of its own to hand it, so the renderer's config is loaded first rather than losing its log sink, log level and keep-alive. Behavior stays CI-only through ISAACLAB_OVRTX_CRASH_UPLOAD, which run_tests.sh sets on the container only when the wheelhouse installs ovrtx-extensions. The variable set without the package raises, since that means a broken install rather than an unsupported runtime.
Description
Adds
ovrtx-extensionsas a second wheelhouse package next toovphysx, and uses it to turn on Omniverse crash-report upload for the OVRTX runtime.ovrtx-extensionsis published to NVIDIA's internal Artifactory index (package) rather than PyPI orpypi.nvidia.com, so the wheelhouse is the only path that can reach it. That path is already gated todevelop(USE_OVPHYSX_WHEELHOUSE), which keeps this off pull requests from forks and off every release branch.What changed
.github/workflows/build.yaml—wheelhouse-packagesbecomesovphysx ovrtx-extensionsin the three jobs that already mount the wheelhouse:isaaclab_ov,rendering-correctness-kitless (legacy), andrendering-correctness-kitless (ovstage)..github/actions/run-tests/run_tests.sh— setsISAACLAB_OVRTX_CRASH_UPLOAD=1on the test container when, and only when, the wheelhouse package list containsovrtx-extensions.source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_crash_reporter.py— applies the--/crashreporter/...settings throughovrtx_extensions.apply_settings, gated on that variable.source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py— calls it fromOVRTXRenderer.__init__, afterredirect_shader_cacheand beforeRenderer(OVRTX_CONFIG)..github/workflows/config.yaml— comment recording what the wheelhouse image has to contain, since it is empty today and a name missing from it fails the job underpip --no-index.source/isaaclab_ov/test/test_ovrtx_crash_reporter.py, plus a changelog fragment forisaaclab_ov.Settings applied
Why the settings are applied at renderer construction
The first revision of this PR ran the shim from a setup step in
run_tests.sh. That was wrong: carb settings are process-local, so they died with the short-lived helper, and the pytest process that actually builds the renderer — and actually crashes — was never configured. Fixed inbaee515dc5by moving the call intoOVRTXRenderer.__init__.It takes the
RendererConfigfor the same reasonredirect_shader_cachedoes. Applying a setting can be what first loads the ovrtx library,ovrtx_initializeruns exactly once per process with the config of whichever caller loads it first (ovrtx._src.bindings._LibraryLoader.create_bindings), andovrtx_extensionshas no config of its own to hand it. Loading the bindings with the renderer's real config first is what keeps thelog_file_path,log_levelandkeep_system_alivethattools/ovrtx_log.pyreplays per test.Still inert until the wheelhouse image is set.
ovphysx_wheelhouse_imageis""ondeveloptoday, so the wheelhouse path is disabled and the env var is never set. It also needs an image that ships anovrtx-extensionswheel plus its dependencies —pip --no-indexcannot fall back to an index.Type of change
Release backport
developChecklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitchecks with./isaaclab.sh --formatsource/<pkg>/changelog.d/for every touched package (do not editCHANGELOG.rstor bumpextension.toml— CI handles that)CONTRIBUTORS.mdor my name already exists theresource/isaaclab_ov/test/test_ovrtx_crash_reporter.pycovers the settings contract, that every setting is applied, that the renderer's config reaches the applier, that an unset variable touches nothing, and that an explicit request withoutovrtx_extensionsraises. It runs with no GPU, no ovrtx runtime and no renderer, mirroringtest_ovrtx_shader_cache_redirect.py.