Skip to content

CI: Harden prior-branch artifact lookup #2643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rwgk merged 3 commits into from
Aug 19, 2026
Merged

CI: Harden prior-branch artifact lookup #2643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8a95bec
Harden prior-branch artifact lookup
rwgk Aug 15, 2026
51f5006
Merge branch 'main' into issue-2642-harden-prior-branch-artifact-lookup
rwgk Aug 18, 2026
619e507
Filter prior-branch runs by success before limiting
rwgk Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/build-wheel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,10 +586,14 @@ jobs:

OLD_BRANCH=$(yq '.backport_branch' ci/versions.yml)
OLD_BASENAME="cuda-bindings-python${PYTHON_VERSION_FORMATTED}-cuda*-${{ inputs.host-platform }}*"
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id --branch "${OLD_BRANCH}" NVIDIA/cuda-python "CI")
OLD_WHEEL_ARTIFACT_PATTERN="${OLD_BASENAME}[0-9a-f]"
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id \
--branch "${OLD_BRANCH}" \
--artifact "${OLD_WHEEL_ARTIFACT_PATTERN}" \
NVIDIA/cuda-python "CI")
PREV_BINDINGS_DIR="cuda_bindings/dist-prev"

gh run download $LATEST_PRIOR_RUN_ID -p ${OLD_BASENAME} -R NVIDIA/cuda-python
gh run download "${LATEST_PRIOR_RUN_ID}" -p "${OLD_BASENAME}" -R NVIDIA/cuda-python
rm -rf ${OLD_BASENAME}-tests # exclude cython test artifacts
ls -al $OLD_BASENAME
mkdir -p "${PREV_BINDINGS_DIR}"
Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/test-wheel-linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,17 +219,26 @@ jobs:

OLD_BRANCH=${{ needs.compute-matrix.outputs.OLD_BRANCH }}
OLD_BASENAME="cuda-bindings-python${PYTHON_VERSION_FORMATTED}-cuda*-${{ inputs.host-platform }}*"
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id --branch "${OLD_BRANCH}" NVIDIA/cuda-python "CI")
OLD_WHEEL_ARTIFACT_PATTERN="${OLD_BASENAME}[0-9a-f]"
LOOKUP_ARGS=(
--branch "${OLD_BRANCH}"
--artifact "${OLD_WHEEL_ARTIFACT_PATTERN}"
)
if ${{ inputs.test-python }}; then
LOOKUP_ARGS+=(--artifact cuda-python-wheel)
fi
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id \
"${LOOKUP_ARGS[@]}" NVIDIA/cuda-python "CI")

gh run download $LATEST_PRIOR_RUN_ID -p ${OLD_BASENAME} -R NVIDIA/cuda-python
gh run download "${LATEST_PRIOR_RUN_ID}" -p "${OLD_BASENAME}" -R NVIDIA/cuda-python
rm -rf ${OLD_BASENAME}-tests # exclude cython test artifacts
ls -al $OLD_BASENAME
mkdir -p "${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}"
mv $OLD_BASENAME/*.whl "${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}"/
rmdir $OLD_BASENAME

if ${{ inputs.test-python }}; then
gh run download $LATEST_PRIOR_RUN_ID -p cuda-python-wheel -R NVIDIA/cuda-python
gh run download "${LATEST_PRIOR_RUN_ID}" -p cuda-python-wheel -R NVIDIA/cuda-python
ls -al cuda-python-wheel
mv cuda-python-wheel/*.whl .
rmdir cuda-python-wheel
Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/test-wheel-windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,17 +199,26 @@ jobs:
run: |
OLD_BRANCH=$(yq '.backport_branch' ci/versions.yml)
OLD_BASENAME="cuda-bindings-python${PYTHON_VERSION_FORMATTED}-cuda*-${{ inputs.host-platform }}*"
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id --branch "${OLD_BRANCH}" NVIDIA/cuda-python "CI")
OLD_WHEEL_ARTIFACT_PATTERN="${OLD_BASENAME}[0-9a-f]"
LOOKUP_ARGS=(
--branch "${OLD_BRANCH}"
--artifact "${OLD_WHEEL_ARTIFACT_PATTERN}"
)
if ${{ inputs.test-python }}; then
LOOKUP_ARGS+=(--artifact cuda-python-wheel)
fi
LATEST_PRIOR_RUN_ID=$(./ci/tools/lookup-run-id \
"${LOOKUP_ARGS[@]}" NVIDIA/cuda-python "CI")

gh run download $LATEST_PRIOR_RUN_ID -p ${OLD_BASENAME} -R NVIDIA/cuda-python
gh run download "${LATEST_PRIOR_RUN_ID}" -p "${OLD_BASENAME}" -R NVIDIA/cuda-python
rm -rf ${OLD_BASENAME}-tests # exclude cython test artifacts
ls -al $OLD_BASENAME
mkdir -p "${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}"
mv $OLD_BASENAME/*.whl "${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}"/
rmdir $OLD_BASENAME

if ${{ inputs.test-python }}; then
gh run download $LATEST_PRIOR_RUN_ID -p cuda-python-wheel -R NVIDIA/cuda-python
gh run download "${LATEST_PRIOR_RUN_ID}" -p cuda-python-wheel -R NVIDIA/cuda-python
ls -al cuda-python-wheel
mv cuda-python-wheel/*.whl .
rmdir cuda-python-wheel
Expand Down
104 changes: 90 additions & 14 deletions ci/tools/lookup-run-id
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#
# Two modes:
# --tag <tag> Find the successful CI run triggered by a tag push.
# --branch <branch> Find the latest successful CI run on a branch.
# --branch <branch> Find the latest qualifying successful CI run on a branch.
#
# Outputs the run ID on stdout. All diagnostic messages go to stderr.
# When --head-sha is passed, a second line with the run's head SHA is printed.
Expand All @@ -24,11 +24,14 @@ Usage:
Options:
--tag <tag> Find run by git tag (requires local git repo with the tag)
--branch <branch> Find latest successful run on the given branch
--artifact <glob> Require an unexpired artifact matching this shell glob
(repeatable; branch mode only)
--head-sha Also print the run's head commit SHA (second line)

Examples:
$0 --tag v13.0.1 NVIDIA/cuda-python
$0 --branch main NVIDIA/cuda-python
$0 --branch 12.9.x --artifact 'cuda-bindings-python313-*' NVIDIA/cuda-python
$0 --branch main --head-sha NVIDIA/cuda-python "CI"
EOF
exit 1
Expand All @@ -38,15 +41,21 @@ EOF
MODE=""
REF=""
HEAD_SHA_FLAG=0
ARTIFACT_PATTERNS=()

while [[ $# -gt 0 ]]; do
case "${1}" in
--tag)
[[ $# -ge 2 ]] || usage
[[ -n "${MODE}" && "${MODE}" != "tag" ]] && { echo "Error: --tag and --branch are mutually exclusive" >&2; exit 1; }
MODE="tag"; REF="${2}"; shift 2 ;;
--branch)
[[ $# -ge 2 ]] || usage
[[ -n "${MODE}" && "${MODE}" != "branch" ]] && { echo "Error: --tag and --branch are mutually exclusive" >&2; exit 1; }
MODE="branch"; REF="${2}"; shift 2 ;;
--artifact)
[[ $# -ge 2 ]] || usage
ARTIFACT_PATTERNS+=("${2}"); shift 2 ;;
--head-sha)
HEAD_SHA_FLAG=1; shift ;;
-h|--help)
Expand All @@ -69,6 +78,11 @@ WORKFLOW_NAME="${1:-CI}"

if [[ -z "${REPOSITORY}" ]]; then usage; fi

if [[ "${MODE}" != "branch" && ${#ARTIFACT_PATTERNS[@]} -gt 0 ]]; then
echo "Error: --artifact is only supported with --branch" >&2
exit 1
fi

# ── Prerequisite checks ──
if [[ -z "${GH_TOKEN:-}" ]]; then
echo "Error: GH_TOKEN environment variable is required" >&2
Expand All @@ -86,28 +100,90 @@ done
if [[ "${MODE}" == "branch" ]]; then
echo "Looking up latest successful '${WORKFLOW_NAME}' run on branch: ${REF}" >&2

RUN_ID=$(gh run list \
-b "${REF}" \
-L 1 \
-w "${WORKFLOW_NAME}" \
-s success \
-R "${REPOSITORY}" \
--json databaseId \
| jq -r '.[0].databaseId // empty')
RUN_DATA=$(gh run list \
--repo "${REPOSITORY}" \
--branch "${REF}" \
--workflow "${WORKFLOW_NAME}" \
--status success \
--json databaseId,workflowName,status,conclusion,headSha,headBranch,createdAt,url \
--limit 100)
Comment on lines +103 to +109

@mdboom mdboom Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude:

Branch-mode lookup now fetches --status completed --limit 100 runs (any conclusion: success, failure, cancelled, …) and filters for conclusion == "success" client-side in the jq step, replacing the old query which filtered -s success (i.e. conclusion=success) directly at the GitHub API level with -L 1. I confirmed via gh run list --help that --status/-s accepts both status values (completed, in_progress, …) and conclusion values (success, failure, …) through the same flag — so the old code was already using the more precise "only success" filter, unbounded. The new code is bounded to the 100 most recent completed runs regardless of outcome. If a backport/maintenance branch (exactly the kind of branch this feature targets — see 12.9.x in the new tests) accumulates 100+ failing/cancelled scheduled runs since its last success (plausible for a stagnant branch that isn't actively fixed), the real successful run falls outside the fetched window and the script now reports "No successful run found" even though a usable run exists. This is compounded by the new artifact-existence filtering, which discards additional candidates from that already-truncated window before giving up. This directly undermines the PR's stated goal of "choosing the newest usable run." Fix: use --status success (still bounded by --limit 100, but only counting real successes, matching prior semantics) instead of --status completed.

@rwgk rwgk Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. You're right that --status completed --limit 100 bounded the search before our client-side success filter, so enough newer failed or cancelled runs could hide an older usable success.

I changed branch-mode lookup to use --status success, while retaining the client-side .conclusion == "success" check as defense in depth. The 100-result candidate budget now counts successful runs; artifact validation can fall back across those candidates without failed/cancelled runs crowding them out.

I also updated the fake gh implementation to model server-side status filtering before the limit and added a regression case with 100 newer failed runs followed by one older successful run. That case fails with --status completed and passes with --status success.

Fixed in 619e507. The focused tests pass (5 tests), as do Bash syntax checking, ShellCheck, and the full pre-commit suite.


CANDIDATE_RUNS=$(echo "${RUN_DATA}" | jq -r \
--arg branch "${REF}" '
map(select(
.headBranch == $branch
and .conclusion == "success"
))
| sort_by(.createdAt, .databaseId)
| reverse
| .[]
| [.databaseId, .headSha, .createdAt]
| @tsv
')

if [[ -z "${CANDIDATE_RUNS}" ]]; then
echo "Error: No successful '${WORKFLOW_NAME}' run found on branch '${REF}'" >&2
exit 1
fi

if [[ ${#ARTIFACT_PATTERNS[@]} -gt 0 ]]; then
echo "Requiring unexpired artifacts matching:" >&2
printf ' - %s\n' "${ARTIFACT_PATTERNS[@]}" >&2
fi

RUN_ID=""
HEAD_SHA=""
while IFS= read -r candidate; do
IFS=$'\t' read -r candidate_id candidate_sha candidate_created_at <<< "${candidate}"

missing_patterns=()
if [[ ${#ARTIFACT_PATTERNS[@]} -gt 0 ]]; then
if ! ARTIFACT_NAMES=$(gh api --paginate \
"repos/${REPOSITORY}/actions/runs/${candidate_id}/artifacts?per_page=100" \
--jq '.artifacts[] | select(.expired == false) | .name'); then
echo "Error: Failed to list artifacts for run ${candidate_id}" >&2
exit 1
fi

for pattern in "${ARTIFACT_PATTERNS[@]}"; do
pattern_matched=0
while IFS= read -r artifact_name; do
# The caller supplies a shell glob, so the RHS must remain unquoted.
# shellcheck disable=SC2053
if [[ -n "${artifact_name}" && "${artifact_name}" == ${pattern} ]]; then
pattern_matched=1
break
fi
done <<< "${ARTIFACT_NAMES}"
if [[ ${pattern_matched} == 0 ]]; then
missing_patterns+=("${pattern}")
fi
done
fi

if [[ ${#missing_patterns[@]} -gt 0 ]]; then
echo "Skipping run ${candidate_id} (${candidate_created_at}); missing unexpired artifact(s): ${missing_patterns[*]}" >&2
continue
fi

RUN_ID="${candidate_id}"
HEAD_SHA="${candidate_sha}"
break
done <<< "${CANDIDATE_RUNS}"

if [[ -z "${RUN_ID}" ]]; then
echo "Error: No successful '${WORKFLOW_NAME}' run found on branch '${REF}'" >&2
echo "Error: No successful '${WORKFLOW_NAME}' run on branch '${REF}' has all required artifacts" >&2
exit 1
fi

echo "Found run ID: ${RUN_ID}" >&2
echo "${RUN_ID}"

if [[ "${HEAD_SHA_FLAG}" == 1 ]]; then
HEAD_SHA=$(gh run view "${RUN_ID}" \
-R "${REPOSITORY}" \
--json headSha \
| jq -r '.headSha')
if [[ -z "${HEAD_SHA}" ]]; then
echo "Error: Run ${RUN_ID} has no head SHA" >&2
exit 1
fi
echo "Head SHA: ${HEAD_SHA}" >&2
echo "${HEAD_SHA}"
fi
Expand Down
Loading
Loading