From 30857e4672e00946f1de58a74dc8aee1470d9b07 Mon Sep 17 00:00:00 2001 From: Mike Sarahan Date: Thu, 10 Sep 2026 12:58:25 -0500 Subject: [PATCH] Prevent release image bootstrap mismatches (#767) **Posted by Codex (GPT-5) on Michael Sarahan's behalf. Treat this message as LLM-generated.** ## Why The 26.12 version update triggered the release workflow before `v26.12.00a` existed. Image producers derive their namespace from the most recent reachable tag, so they published successful `26.10-*` images while the newly updated unified devcontainers requested `26.12-*` images. The alpha tag appeared nine minutes after the workflow began. A later retry of failed jobs could not recover because GitHub did not rerun the already-successful producer jobs. This creates an avoidable bootstrap dependency: the version-update workflow needs to publish the new namespace before the tag that currently defines that namespace can be created. The committed `VERSION` file is already updated atomically with the consumer definitions and is therefore the appropriate source of truth for producer tags. The incident also exposed a second class of mismatch: `matrix.yml` can change an image component version without the checked-in unified devcontainers changing their `BASE` references. Without validation, that drift is only discovered after all producer builds complete and consumers attempt to pull an image that will never be published. PR #766 applies the UCX alignment to `release/26.10` for forward-merging. This branch currently contains the equivalent main-branch alignment; that portion will disappear from this PR's diff after the forward merge reaches `main`. ## Changes - derive Linux and Windows producer namespaces from the committed `VERSION` file instead of `git describe` - validate that `VERSION` uses `YY.MM.PP` format - reconstruct all publishable Linux image tags from `matrix.yml`, including OS-qualified tags and OS-free aliases - reject any checked-in RAPIDS unified devcontainer `BASE` that the current matrix cannot produce - run that validation in pre-commit and as a fatal release prerequisite before Linux producers and downstream unified devcontainers run - add ShellCheck coverage for the release-bootstrap scripts touched here ## Why the previous sequence cannot recur After these changes, merging a version update changes both producer and consumer namespaces through the same committed `VERSION` value. The producer no longer depends on whether the alpha tag existed when the workflow checkout occurred. The release also stops before building images if a consumer base does not correspond to the current matrix, so component-version drift is reported directly rather than surfacing as a late missing-manifest error. ## Validation - `pre-commit run --all-files` - zizmor - devcontainer base-image validation - ShellCheck for the release-bootstrap scripts - positive validation against the 26.12 / UCX 1.21 configuration - negative validation with a deliberate UCX 1.19 base, which failed and identified the affected file and unproducible tag - `git diff --check` Incident: https://github.com/rapidsai/devcontainers/actions/runs/34280170418/job/102279620120 --------- Co-authored-by: Paul Taylor <178183+trxcllnt@users.noreply.github.com> --- .github/actions/devcontainer-json/action.sh | 8 ++- .../build-test-and-push-windows-image.yml | 4 +- .github/workflows/release.yml | 17 +++++- .github/workflows/test.yml | 14 +++++ .pre-commit-config.yaml | 7 +++ ci/validate-devcontainer-bases.sh | 57 +++++++++++++++++++ 6 files changed, 103 insertions(+), 4 deletions(-) create mode 100755 ci/validate-devcontainer-bases.sh diff --git a/.github/actions/devcontainer-json/action.sh b/.github/actions/devcontainer-json/action.sh index c769c34c9..b58846d42 100755 --- a/.github/actions/devcontainer-json/action.sh +++ b/.github/actions/devcontainer-json/action.sh @@ -1,13 +1,17 @@ #! /usr/bin/env bash # cd to the repo root -cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../../../"; +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "${script_dir}/../../../" || exit 1 os="${1:-"ubuntu:22.04"}"; features="${2:-"[]"}"; container_env="${3:-"null"}"; -VERSION="$(git describe --abbrev=0 --tags --first-parent | sed 's/[a-zA-Z]//g' | cut -d '.' -f -2)"; +# VERSION is updated before the alpha tag is created during release rollover. +# Use it as the source of truth so a release workflow triggered by that update +# publishes the new image namespace instead of the previous tagged version. +VERSION="$(cut -d '.' -f 1-2 VERSION)"; tag="$(node -p "$(cat < !x.hide).map(({ name = '', version = '', suffix = '' }) => { if (name.includes(':')) { diff --git a/.github/workflows/build-test-and-push-windows-image.yml b/.github/workflows/build-test-and-push-windows-image.yml index 368016d38..895d3eb3d 100644 --- a/.github/workflows/build-test-and-push-windows-image.yml +++ b/.github/workflows/build-test-and-push-windows-image.yml @@ -54,7 +54,9 @@ jobs: repo="$INPUT_REPO"; cl="$(echo "$INPUT_FEATURES" | jq -r '.[1].version')"; cuda="$(echo "$INPUT_FEATURES" | jq -r '.[0].version')"; - version="$(git describe --abbrev=0 --tags --first-parent | sed 's/[a-zA-Z]//g' | cut -d '.' -f -2)"; + # VERSION changes before the alpha tag exists during release rollover. + # Reading it directly keeps producer tags aligned with their consumers. + version="$(cut -d '.' -f 1-2 VERSION)"; base_tag="cuda${cuda}-cl${cl}"; tag_without_os="${version}-${base_tag}"; cat <&2 + exit 1 +fi +short_version="${full_version%.*}" + +expected_tags="$(mktemp)" +trap 'rm -f "${expected_tags}"' EXIT + +# Recreate the visible part of every Linux image name from matrix.yml. Hidden +# features affect image contents but, by design, do not appear in image tags. +yq --yaml-fix-merge-anchor-to-spec -eMo json matrix.yml \ + | jq -r --arg version "${short_version}" ' + .include[] + | select(.os != "windows") + | (.os | gsub(":"; "")) as $os + | .images[] + | (.features + | map( + select(.hide != true) + | (.name | split("/")[-1] | split(":")[0]) + + (.version // "" | tostring) + + (.suffix // "" | tostring) + ) + | (. + [$os]) + | join("-") + ) as $name + # The release publishes both the OS-qualified tag and an OS-free alias. + | [ + $version + "-cpp-" + $name, + $version + "-cpp-" + ($name | sub("-" + $os + "$"; "")) + ] + | .[] + ' \ + | sort -u > "${expected_tags}" + +status=0 +while IFS=$'\t' read -r file base; do + [[ "${base}" == rapidsai/devcontainers:* ]] || continue + tag="${base#rapidsai/devcontainers:}" + if ! grep -Fqx -- "${tag}" "${expected_tags}"; then + echo "${file}: BASE '${base}' is not produced by matrix.yml" >&2 + status=1 + fi +done < <( + find .devcontainer -name devcontainer.json -exec \ + jq -r '[input_filename, (.build.args.BASE // "")] | @tsv' {} + +) + +exit "${status}"