Harden CI against transient Bazel/emsdk fetch timeouts.#239
Conversation
Mirror emsdk downloads and prefetch/retry network-bound Bazel steps so jobs fail less often on SocketTimeoutException during repo resolution. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce CI flakiness caused by transient network timeouts during Bazel external repository resolution, especially around emsdk acquisition for WASM builds, by overriding the emsdk source download URLs and adding explicit prefetch/retry steps in CI workflows.
Changes:
- Override the
emsdkBazel module source to prefer GitHub tarballs (with an alternate CDN URL) while keeping integrity pinned. - Add
bazel fetchprefetch steps with simple retry loops to CI jobs to warm external repositories before builds/tests. - Add build retry loops for WASM and Windows workflows to mitigate transient failures during analysis/fetch.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| MODULE.bazel.lock | Updates lockfile hashes/digests to reflect the new module resolution behavior. |
| MODULE.bazel | Adds an archive_override for emsdk to use alternate download URLs with pinned integrity. |
| .github/workflows/ci_windows.yml | Adds prefetch and build retry logic to reduce transient external fetch failures on Windows CI. |
| .github/workflows/ci_wasm.yml | Adds prefetch and build retry logic to reduce transient external fetch failures in WASM CI. |
| .github/workflows/ci_macos.yml | Adds prefetch-with-retry step to warm external repos before building on macOS CI. |
| .github/workflows/ci_linux.yml | Adds prefetch-with-retry step to warm external repos before building on Linux CI. |
Restart Bazel instead of expunging between build retries, and warm external repos for Linux/macOS sanitizer jobs the same way. Co-authored-by: Cursor <cursoragent@cursor.com>
zzcgumn
left a comment
zzcgumn
left a comment
There was a problem hiding this comment.
I like this idea but am wondering why we are mixing bazelisk and bazel commands. I don't think it makes any difference on CI runners as I expect there to be only one bazel version that bazelisk can resolve to use.
|
We should always use bazelisk. Otherwise a local bazel might differ from the repo bazel, causing MODULE.bazel.lock mismatches. I've added 'alias bazel=bazelisk' to my |
Mirror emsdk downloads and prefetch/retry network-bound Bazel steps so jobs fail less often on SocketTimeoutException during repo resolution.