Skip to content

CI: Harden prior-branch artifact lookup - #2643

Open
rwgk wants to merge 3 commits into
NVIDIA:mainfrom
rwgk:issue-2642-harden-prior-branch-artifact-lookup
Open

CI: Harden prior-branch artifact lookup#2643
rwgk wants to merge 3 commits into
NVIDIA:mainfrom
rwgk:issue-2642-harden-prior-branch-artifact-lookup

Conversation

@rwgk

@rwgk rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes #2642.

CUDA Python CI reuses cuda.bindings wheel artifacts from the configured prior-major branch when it builds and tests the combined cuda.core wheel. The existing lookup trusted the first successful run returned by gh run list without verifying that the artifacts required by the requesting matrix row were present and unexpired.

This change makes branch-mode lookup choose the newest usable run from a bounded candidate set:

  • retrieve up to 100 completed runs, filter for the exact branch and successful conclusion, and sort explicitly by creation time with a run-ID tie-breaker;
  • accept repeatable --artifact patterns and require every pattern to match an unexpired artifact in the same run;
  • skip candidates with missing or expired requirements and continue to the next candidate;
  • update the wheel build and Linux/Windows wheel-test consumers to declare the artifacts they subsequently download; and
  • preserve tag mode, branch mode without artifact requirements, and --head-sha output.

This intentionally does not eliminate the possibility that GitHub's server-side branch filter returns a stale or incomplete bounded candidate set; that broader lookup redesign remains outside #2642.

Testing

  • TestVenv/bin/python -m pytest -v --noconftest ci/tools/tests/test_lookup_run_id.py (4 passed)
  • bash -n ci/tools/lookup-run-id
  • shellcheck ci/tools/lookup-run-id
  • pre-commit run --all-files
  • PR Continue merging main into 13.4.x #2641 CI run 31870063964 exercised the extracted change across prior-branch wheel-build and Linux wheel-test consumers; every exercised lookup selected a usable run and passed.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Sort completed branch runs explicitly and choose the newest successful run whose required artifacts are still available. Validate bindings and metapackage artifacts at all prior-branch download sites, and cover fallback and compatibility behavior with standalone tests.
@copy-pr-bot

copy-pr-bot Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Aug 15, 2026
@rwgk rwgk added this to the cuda.bindings next milestone Aug 15, 2026 — with ChatGPT Codex Connector
@rwgk rwgk added bug Something isn't working P1 Medium priority - Should do labels Aug 15, 2026 — with ChatGPT Codex Connector
@rwgk rwgk self-assigned this Aug 15, 2026
@rwgk

rwgk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8a95bec

@github-actions

Copy link
Copy Markdown

@rwgk
rwgk marked this pull request as ready for review August 16, 2026 04:15
@rwgk
rwgk requested a review from mdboom August 16, 2026 04:15
@rwgk
rwgk enabled auto-merge (squash) August 17, 2026 03:08
Comment thread ci/tools/lookup-run-id
Comment on lines +103 to +109
RUN_DATA=$(gh run list \
--repo "${REPOSITORY}" \
--branch "${REF}" \
--workflow "${WORKFLOW_NAME}" \
--status completed \
--json databaseId,workflowName,status,conclusion,headSha,headBranch,createdAt,url \
--limit 100)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude:

Branch-mode lookup now fetches --status completed --limit 100 runs (any conclusion: success, failure, cancelled, …) and filters for conclusion == "success" client-side in the jq step, replacing the old query which filtered -s success (i.e. conclusion=success) directly at the GitHub API level with -L 1. I confirmed via gh run list --help that --status/-s accepts both status values (completed, in_progress, …) and conclusion values (success, failure, …) through the same flag — so the old code was already using the more precise "only success" filter, unbounded. The new code is bounded to the 100 most recent completed runs regardless of outcome. If a backport/maintenance branch (exactly the kind of branch this feature targets — see 12.9.x in the new tests) accumulates 100+ failing/cancelled scheduled runs since its last success (plausible for a stagnant branch that isn't actively fixed), the real successful run falls outside the fetched window and the script now reports "No successful run found" even though a usable run exists. This is compounded by the new artifact-existence filtering, which discards additional candidates from that already-truncated window before giving up. This directly undermines the PR's stated goal of "choosing the newest usable run." Fix: use --status success (still bounded by --limit 100, but only counting real successes, matching prior semantics) instead of --status completed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. You're right that --status completed --limit 100 bounded the search before our client-side success filter, so enough newer failed or cancelled runs could hide an older usable success.

I changed branch-mode lookup to use --status success, while retaining the client-side .conclusion == "success" check as defense in depth. The 100-result candidate budget now counts successful runs; artifact validation can fall back across those candidates without failed/cancelled runs crowding them out.

I also updated the fake gh implementation to model server-side status filtering before the limit and added a regression case with 100 newer failed runs followed by one older successful run. That case fails with --status completed and passes with --status success.

Fixed in 619e507. The focused tests pass (5 tests), as do Bash syntax checking, ShellCheck, and the full pre-commit suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure P1 Medium priority - Should do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: CI: Harden prior-branch artifact lookup against unstable run ordering and unavailable artifacts

2 participants