CI: Harden prior-branch artifact lookup - #2643
Conversation
Sort completed branch runs explicitly and choose the newest successful run whose required artifacts are still available. Validate bindings and metapackage artifacts at all prior-branch download sites, and cover fallback and compatibility behavior with standalone tests.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test 8a95bec |
|
| RUN_DATA=$(gh run list \ | ||
| --repo "${REPOSITORY}" \ | ||
| --branch "${REF}" \ | ||
| --workflow "${WORKFLOW_NAME}" \ | ||
| --status completed \ | ||
| --json databaseId,workflowName,status,conclusion,headSha,headBranch,createdAt,url \ | ||
| --limit 100) |
There was a problem hiding this comment.
From Claude:
Branch-mode lookup now fetches --status completed --limit 100 runs (any conclusion: success, failure, cancelled, …) and filters for conclusion == "success" client-side in the jq step, replacing the old query which filtered -s success (i.e. conclusion=success) directly at the GitHub API level with -L 1. I confirmed via gh run list --help that --status/-s accepts both status values (completed, in_progress, …) and conclusion values (success, failure, …) through the same flag — so the old code was already using the more precise "only success" filter, unbounded. The new code is bounded to the 100 most recent completed runs regardless of outcome. If a backport/maintenance branch (exactly the kind of branch this feature targets — see 12.9.x in the new tests) accumulates 100+ failing/cancelled scheduled runs since its last success (plausible for a stagnant branch that isn't actively fixed), the real successful run falls outside the fetched window and the script now reports "No successful run found" even though a usable run exists. This is compounded by the new artifact-existence filtering, which discards additional candidates from that already-truncated window before giving up. This directly undermines the PR's stated goal of "choosing the newest usable run." Fix: use --status success (still bounded by --limit 100, but only counting real successes, matching prior semantics) instead of --status completed.
There was a problem hiding this comment.
Thanks, good catch. You're right that --status completed --limit 100 bounded the search before our client-side success filter, so enough newer failed or cancelled runs could hide an older usable success.
I changed branch-mode lookup to use --status success, while retaining the client-side .conclusion == "success" check as defense in depth. The 100-result candidate budget now counts successful runs; artifact validation can fall back across those candidates without failed/cancelled runs crowding them out.
I also updated the fake gh implementation to model server-side status filtering before the limit and added a regression case with 100 newer failed runs followed by one older successful run. That case fails with --status completed and passes with --status success.
Fixed in 619e507. The focused tests pass (5 tests), as do Bash syntax checking, ShellCheck, and the full pre-commit suite.
Description
Fixes #2642.
CUDA Python CI reuses
cuda.bindingswheel artifacts from the configured prior-major branch when it builds and tests the combinedcuda.corewheel. The existing lookup trusted the first successful run returned bygh run listwithout verifying that the artifacts required by the requesting matrix row were present and unexpired.This change makes branch-mode lookup choose the newest usable run from a bounded candidate set:
--artifactpatterns and require every pattern to match an unexpired artifact in the same run;--head-shaoutput.This intentionally does not eliminate the possibility that GitHub's server-side branch filter returns a stale or incomplete bounded candidate set; that broader lookup redesign remains outside #2642.
Testing
TestVenv/bin/python -m pytest -v --noconftest ci/tools/tests/test_lookup_run_id.py(4 passed)bash -n ci/tools/lookup-run-idshellcheck ci/tools/lookup-run-idpre-commit run --all-filesChecklist