Skip to content

fix(cli): retry transient gateway errors from the Checkly API [RED-861] [ship] - #1439

Merged
sorccu merged 2 commits into
mainfrom
simo/red-861-retry-transient-502s
Aug 14, 2026
Merged

fix(cli): retry transient gateway errors from the Checkly API [RED-861] [ship]#1439
sorccu merged 2 commits into
mainfrom
simo/red-861-retry-transient-502s

Conversation

@sorccu

@sorccu sorccu commented Aug 14, 2026

Copy link
Copy Markdown
Member

Linear: RED-861

What

Adds bounded, backoff-based retries for transient upstream errors to the CLI's axios REST layer, so one-off gateway blips (e.g. a single 502) no longer fail deploy/test/trigger runs — or our e2e CI — outright.

  • New packages/cli/src/rest/retry.ts: a retry rejection-handler registered as the first response interceptor on the shared instance (before the error-mapping interceptor), re-dispatching api.request(error.config) so the request interceptor refreshes auth headers on each attempt.
  • Retried (GET/HEAD/OPTIONS automatically): 502/503/504 (Retry-After used when ≤ 10 s, else jittered backoff), 429 (Retry-After ≤ 10 s honored; > 10 s → no retry; absent → backoff), and no-response network errors (ECONNRESET, ETIMEDOUT, ECONNABORTED, EAI_AGAIN, ERR_NETWORK, EPIPE).
  • Never retried: 408 (long-poll callers own that cadence), plain 500 (often deterministic), aborted/canceled requests, stream responses (deploy-progress SSE), and stream request bodies (single-use gzip payloads).
  • Mutations can opt in per request via a new checklyRetry: true config flag where the endpoint is known idempotent; no endpoints are opted in yet (mechanism only).
  • Policy: 3 total attempts, full-jitter exponential backoff (base 250 ms, backoff delay cap 2 s); an explicit server Retry-After is trusted for longer, up to 10 s per delay. Each retry emits a checkly:cli:rest:retry debug line (DEBUG=checkly:cli:rest*).
  • The backoff timer stays ref'd while a caller awaits the retry (so the process can't exit mid-backoff) and is cancelled immediately if the request's abort signal fires (so an abandoned retry can't hold the process open).

Testing

  • 22 new unit tests covering the retry policy matrix, Retry-After parsing, abort behavior, stream guards, and the production interceptor wiring/order.
  • Verified end-to-end against a local always-502 server: the CLI makes 3 attempts, logs both retry lines, surfaces ServerError, exits 1; aborting mid-backoff settles immediately instead of waiting out the delay.

Known limitations

  • checkly api sets validateStatus: () => true, so it never rejects and never retries (raw passthrough semantics).
  • The login flow builds its own interceptor-less axios instances and is not covered.
  • deploy/test submit requests can never retry (single-use gzip stream bodies); opting in plain-JSON mutations (e.g. env add, trigger) is a candidate follow-up.

🤖 Generated with Claude Code

sorccu and others added 2 commits August 14, 2026 12:03
Add bounded, backoff-based retries to the shared axios REST instance so
one-off 502/503/504/429 gateway blips (and no-response network errors)
no longer fail commands and CI pipelines outright.

GET/HEAD/OPTIONS retry automatically (3 attempts, full-jitter
exponential backoff, base 250ms, per-delay cap 2s, Retry-After
honored); mutations can opt in via a new checklyRetry request-config
flag where known idempotent. 408, aborted requests, and stream
requests/bodies are never retried. Each retry emits a
checkly:cli:rest:retry debug line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Split the single delay cap into two knobs: jittered backoff stays
capped at 2s, while an explicit Retry-After header is now honored
verbatim up to 10s (for 429 and 502/503/504 alike). A 429 asking for
more than 10s still gives up; a 5xx asking for more falls back to
backoff. Renames maxDelayMs to maxBackoffDelayMs to keep the pair
self-describing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sorccu
sorccu force-pushed the simo/red-861-retry-transient-502s branch from 2f1c5a2 to 60e5eee Compare August 14, 2026 03:07
@sorccu sorccu changed the title fix(cli): retry transient gateway errors from the Checkly API [RED-861] fix(cli): retry transient gateway errors from the Checkly API [RED-861] [ship] Aug 14, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: ship/show PR from a same-repo branch.

@sorccu
sorccu merged commit 7a4c1bb into main Aug 14, 2026
18 checks passed
@sorccu
sorccu deleted the simo/red-861-retry-transient-502s branch August 14, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant