Core: Add pluggable BackoffStrategy SPI for Tasks retries

[wombatu-kun]

Part of #7528.

Motivation

Retry backoff in Tasks.runTaskWithRetry was a single hardcoded exponential-plus-jitter formula with no way to substitute another algorithm.

What this PR does

SPI introduction. Introduces a pluggable SPI org.apache.iceberg.util.BackoffStrategy selected via a single property retry.strategy-impl, loaded reflectively via the standard DynConstructors no-arg constructor + initialize(Map) pattern that catalog-impl, io-impl, and lock-impl already use. The new ExponentialBackoffStrategy reproduces the previous inline formula and jitter exactly, and is the default whenever no strategy is configured — so existing callers see byte-for-byte identical per-attempt wait times.

Every production .exponentialBackoff(...) site that had access to a properties map is wired through the SPI: commits, REST catalog handlers, metadata refresh, status checks, in-memory and DynamoDB lock managers, the Hive Metastore lock (via Hadoop Configuration), REST scan polling, and Spark cleanup via FileIO.properties(). OAuth2 token refresh has no usable properties map at its call site, so it uses an explicitly-constructed default.

API cleanup. Tasks.Builder.exponentialBackoff(long, long, long, double) was carrying two unrelated concerns: per-attempt formula parameters (min/max/scale) and the total retry budget (totalTimeout). The new SPI made the conflation visible — at every wired site the formula args became dead weight whenever the configured strategy actually won. The method is replaced by two: Tasks.Builder.totalTimeoutMs(long) (the retry budget — aligned with the operator-facing property name commit.retry.total-timeout-ms) and Tasks.Builder.backoffStrategy(BackoffStrategy) (the per-attempt wait formula). A new BackoffStrategies.from(Map, long, long, double) overload preserves operator-tunable formula defaults: it returns the custom strategy when retry.strategy-impl is set, otherwise a fresh ExponentialBackoffStrategy with the provided params. The internal field maxDurationMs is renamed totalTimeoutMs to match the same operator-facing name.

A follow-up PR will add an AIMD strategy on top of this SPI without further core changes. The implementation is already prepared on branch issue/7528-aimd-backoff-strategy (two commits: an onSuccess() callback added to BackoffStrategy plus Tasks wiring, then AIMDBackoffStrategy with tests and docs) and will be filed against main once this PR is merged.

Compatibility

For the SPI surface, behavior is byte-for-byte unchanged unless retry.strategy-impl is set — the default fallback uses the same exponential-plus-jitter formula with the same parameters as before.

For the builder API, Tasks.Builder.exponentialBackoff(long, long, long, double) is removed. The behavior is fully recoverable via the new two-call form .totalTimeoutMs(total).backoffStrategy(BackoffStrategies.from(props, min, max, scale)). The revapi exception against the 1.11.0 baseline documents the removal and points at the replacement.

Tests

TestBackoffStrategies covers the loader: happy path with initialize invocation, null/empty/missing-key returns, unknown class name, missing no-arg constructor, non-strategy class, initialize exception propagation, and the new 4-arg from(Map, long, long, double) overload across three branches (custom strategy from properties, default exponential, null properties). TestTasks covers Tasks integration: custom strategy is used, null is a no-op, the returned milliseconds actually drive Thread.sleep (timing-verified), and a single strategy instance is shared across parallel worker threads. TestBackoffStrategyCommit exercises end-to-end commit retry through the SPI across all four format versions.

---

AI Disclosure

• Model: Claude Opus 4.7
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Introduce a pluggable BackoffStrategy SPI for Tasks retries.

[wombatu-kun]

PR isn't the cause of Kafka Connect CI failure. I've fixed Kafka Connect integration test flakiness in separate PR #16438

[wombatu-kun]

Hi @nastra @jackye1995 @singhpk234
This is ready for review when someone has a moment. Happy to split it up or adjust the approach based on feedback.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[wombatu-kun]

not stale

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.