Retry only transient Helix failures in Job Monitor

[mmitche]

Summary

• retry Job Monitor Helix operations only when the failure is classified as transient
• classify generated RestApiException failures using HTTP status codes
• return deterministic client failures such as HTTP 400 after the first attempt while retaining retries for 408, 429, 5xx, network, I/O, and timeout failures

Addresses the retry-amplification defect observed in dotnet/arcade#17502 and dnceng work item 12784. The prior implementation retried every exception five times, including invalid cancellation requests.

Validation

• added focused coverage for generated REST API status classification and single-attempt non-transient failures
• git diff --check passes
• focused local build was attempted, but Arcade's SDK bootstrap could not extract into .dotnet because the active sandbox denied file creation; PR validation should run the full tests

[Copilot]

🟢 Approval recommended

The change is small, targeted, and includes new tests that directly validate the updated transient classification and retry behavior.

Pull request overview

This PR refines Helix Job Monitor retry behavior to avoid retry amplification by retrying only failures classified as transient, including generated Helix REST client exceptions classified by HTTP status code.

Changes:

• Stop retries early for non-transient failures in HelixService.RetryAsync by consulting TransientFailureDetector.
• Extend transient classification to include RestApiException using the HTTP response status code.
• Add focused unit tests covering RestApiException status classification and verifying non-transient failures are not retried.

File summaries

File	Description
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/TransientFailureDetectorTests.cs	Adds unit coverage for transient classification of generated REST API exceptions by status code.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/HelixServiceTests.cs	Adds a regression test ensuring non-transient failures are not retried.
src/Microsoft.DotNet.Helix/JobMonitor/TransientFailureDetector.cs	Adds RestApiException handling to transient detection based on HTTP status.
src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs	Updates retry loop to retry only when failures are classified as transient.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.