[Docker] Rebuild the kit-less image as a single stage - #7671
Draft
hujc7 wants to merge 9 commits into
Draft
Conversation
The kit-less image is the only multi-stage Dockerfile, and ENV does not cross stages: the builder's uv configuration was never repeated in the runtime stage. uv resolves the project environment from UV_PROJECT_ENVIRONMENT alone, so any in-container `uv run` ignored VIRTUAL_ENV, targeted /workspace/isaaclab/.venv and reinstalled the whole locked dependency set. Where Compose or CI bind-mounts the host checkout over source/, the editable rebuild then wrote egg-info into files the image's uid cannot touch and the command died with "Permission denied". Repeat the four uv settings in the runtime stage. UV_NO_SYNC is what keeps `uv run` on the environment the image shipped; the venv is already synced from the lock, so re-deriving it can only diverge from it. Every existing validation step calls `python` directly, which is why nothing caught this. Add a `uv run` step to the kit-less validation action.
The stage split existed to keep a C toolchain out of the runtime, but nothing
in the lock builds a compiled source distribution: 0 of 222 installed
distributions are locally built on x86_64. Dropping build-essential, cmake and
python3-dev removes the toolchain outright, and with one stage the uv settings
are declared once instead of being repeated across a stage boundary the
runtime never inherited.
Install through a uv cache mount as Dockerfile.base and Dockerfile.curobo do.
The mount targets ${HOME}/.cache/uv, not root's, because this image sets HOME
for the whole build; UV_CACHE_DIR pins the two together so an unmatched mount
cannot bake the cache into the layer.
Assert the fix on the built image rather than on the Dockerfile text: the new
image invariant runs uv run inside the image and checks it resolves the shipped
environment. kitless-docker.yml never passed verify-test-path, so image
invariants had never run against this image at all.
Measured on desk, RTX PRO 6000: 15.46 GB image, 109 s cold build, OVRTX
renderer constructs on GPU, warp and newton import on CUDA, uv run resolves
/opt/isaaclab-venv with source mounted read-only.
The verify-test-path input added in the previous commit runs the image invariants with ``uv run`` on the runner itself, and the GPU runners have no uv on PATH, so the step failed with exit 127 before the tests ran. build.yaml already carries the same setup-uv step ahead of its own verify-test-path for this reason; kitless-docker.yml only had one in its ubuntu-latest changes job.
Restore the build toolchain for arm64 only. psutil 5.9.8 and pyopengl-accelerate
3.1.10 publish no aarch64 wheels, so uv builds them from sdist there; the earlier
claim that nothing in the lock compiles held only on x86_64, which is the one
architecture it was measured on. Guarded by dpkg --print-architecture as
Dockerfile.base does, so the x86_64 image keeps shipping no compiler.
Stop setting UV_NO_SYNC in the image. It suppressed the editable rebuild that
fails on a bind-mounted source tree, but it also stopped uv run --extra <name>
installing an extra the image does not carry, which breaks the documented
--extra leapp flow. The setting now travels with the caller that mounts the
source read-only, which is the validate action.
Retarget UV_PYTHON at the venv after the sync, as Dockerfile.base and
Dockerfile.curobo already do, so a runtime uv pip install does not resolve to
the externally-managed system interpreter.
Pre-create ${HOME}/.cache so BuildKit cannot leave the cache-mount parent
root-owned inside a 750 home, and select the image invariants in the workflow's
change patterns so an edit to them rebuilds the image they assert against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The kit-less image drops from 159 to 114 Dockerfile lines and stops installing a C toolchain
it never uses on x86_64, while a
uv runinvariant runs against the built image for the firsttime. Cold build 109 s, image unchanged at 15.46 GB.
Description
Stacked on #7628 — that PR carries the four runtime
uvsettings for NVBug 6732972 and shouldmerge first. Until it does, this PR's diff shows its commits too.
1. Single stage
Dockerfile.kitlesswas the repo's only image whose final stage was not the stage that ran theinstall, which is why
ENVhad to be declared twice and why nvbugs 6732972 was possible at all.Merging the stages removes that class of bug rather than patching it.
The builder stage existed to keep a compiler out of the runtime. Measured: 0 of 222 installed
distributions are locally built on x86_64, so
build-essential,cmakeandpython3-devwere339 MB of unused toolchain.
psutil==5.9.8andpyopengl-accelerate==3.1.10publish no aarch64wheels, so those build deps are restored for arm64 only, as
Dockerfile.basealready does.2. uv cache mount
Kit-less was the only image without one; base and cuRobo have used
--mount=type=cachesince#7405. The mount targets
${HOME}/.cache/uvrather than root's, because this image setsHOMEfor the whole build — an unmatched target silently bakes the cache into the layer.
3. Built-image invariant
kitless-docker.ymlnever passedverify-test-path, sodocker/test/test_image_invariants.pyhad never run against this image. It now does, with a new assertion that
uv runresolves theshipped environment. Verified to fail on an image with
UV_PROJECT_ENVIRONMENTremoved and passon the fixed one.
4. Validation
Measured on an RTX PRO 6000 (Blackwell), amd64:
--no-cachebuild--gpus alluv runwithsource/bind-mounted read-only/opt/isaaclab-venv, no stray.venvdocker/testuv run isaaclab -farm64 is not verified. PR CI builds amd64 only; the aarch64 path is exercised by
publish-images.yamlafter merge. The conditional is confirmed inert on amd64.A two-phase
uv sync(deps, then workspace) was measured and rejected: it made a source-editrebuild 6.9× slower (468 s vs 68 s) and the image 14.9 GB larger, because the second sync
rewrites the venv in a later layer.
Type of change
Release backport
developChecklist
pre-commitchecks with./isaaclab.sh --formatsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there