Bound an exchange by one request timeout - #2314
Conversation
TimeoutsHolder anchors the request deadline on its own construction, and a redirect, a retry and an auth replay each build a new one for the same future. Every hop therefore starts the budget again: with maxRedirects=5 a chain can legitimately run for six times the configured requestTimeout. The javadoc claims requestTimeout is the maximum time until the response is completed, which is not what happens. Add isUseAbsoluteRequestDeadline(), off by default, which anchors the deadline on when the exchange was submitted instead, so a later hop gets whatever is left of the budget rather than a fresh one. Off by default because turning it on shortens exchanges that rely on the per-attempt behaviour; the getRequestTimeout() javadoc now describes what actually happens and points at the flag either way. Settable per request as well as per client, following the followRedirect pattern: a nullable Boolean on Request that overrides the config value. Resolved once, in newNettyResponseFuture, and kept on the NettyResponseFuture rather than read from the request per hop. The first attempt put it on Request alone, and the two override tests failed in opposite directions because Redirect30xInterceptor rebuilds the request for the next hop from a hand-picked set of fields: the override was dropped mid-exchange and the config value took over. Anything carried only on the request has that problem, so the flag lives on the exchange, which is also what it describes. A redirect target cannot change it, which is right - the budget belongs to the caller. DefaultRequest keeps its existing public constructor, delegating to a new one that takes the flag as a trailing argument. Inserting the parameter beside followRedirect instead was a binary-incompatible change to a public constructor, which revapi correctly rejected. Claude Code on behalf of Pavel Ptashyts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| // A deadline already behind us is scheduled at zero rather than negative, so the task still runs and | ||
| // still cancels its read-timeout sibling, which is bookkeeping only it does. | ||
| requestTimeout = newTimeout(new RequestTimeoutTimerTask(nettyResponseFuture, requestSender, this, requestTimeoutInMs), | ||
| Math.max(requestTimeoutMillisTime - unpreciseMillisTime(), 0L)); |
There was a problem hiding this comment.
Clamping to zero does not stop the hop, it only delays the abort. The wheel timer fires on tick boundaries, so with a deadline already behind us we still take a permit, still take a connection and still write the request, then fail it a tick later. On a 307 that means the POST body reached the redirect target while the caller is handed a TimeoutException that reads like nothing was sent. I think this wants a remaining budget check in sendNextRequest that fails the exchange before the write.
| // deadline here hands each of those hops a fresh budget, so a chain of n hops runs for n times the | ||
| // configured timeout; anchoring it on the future bounds the exchange as a whole instead. Which one | ||
| // applies is the caller's choice, per request or per client. | ||
| requestTimeoutMillisTime = (nettyResponseFuture.isUseAbsoluteRequestDeadline() |
There was a problem hiding this comment.
getStart is currentTimeMillis, but the timer that actually fires runs on nanoTime. Until now a clock step could only distort the hop in flight. With an absolute anchor, a backward NTP correction of a few seconds effectively removes a 600ms deadline for the rest of the chain, and a forward one aborts every later hop instantly on a healthy connection. An absolute anchor really wants a monotonic source.
| requestTimeoutMillisTime = (nettyResponseFuture.isUseAbsoluteRequestDeadline() | ||
| ? nettyResponseFuture.getStart() : unpreciseMillisTime()) + requestTimeoutInMs; | ||
| // A deadline already behind us is scheduled at zero rather than negative, so the task still runs and | ||
| // still cancels its read-timeout sibling, which is bookkeeping only it does. |
There was a problem hiding this comment.
This justification does not hold. readTimeout is still null at this point, it is only set once scheduleReadTimeout runs after the write, and requestTimeout is not assigned yet either because the constructor registers with the timer first. In the case you are describing the task cancels nothing.
| * | ||
| * @return {@code true} to treat the request timeout as a deadline for the whole exchange | ||
| */ | ||
| default boolean isUseAbsoluteRequestDeadline() { |
There was a problem hiding this comment.
Hardcoding false here means the property only ever works through the builder. Someone on a custom config or a wrapper sets org.asynchttpclient.useAbsoluteRequestDeadline, gets nothing, and nothing is logged, while a sibling service on the builder honours it. Either delegate to defaultUseAbsoluteRequestDeadline or say in the javadoc that the property is builder only.
| * rather than beside {@code followRedirect} so the original signature | ||
| * stays intact for callers that build a request without the builder. | ||
| */ | ||
| public DefaultRequest(String method, |
There was a problem hiding this comment.
This is the part we cannot take back later. A 27 arg public constructor gets pinned by revapi, the next per request option makes it 28, and the two parameter lists have to stay in lockstep with no compiler help if they drift, since the tail is all reference types. Only RequestBuilderBase.build calls it. Package private, or a small options holder, costs nothing today.
| rb.file = file; | ||
| rb.followRedirect = followRedirect; | ||
| rb.useAbsoluteRequestDeadline = useAbsoluteRequestDeadline; | ||
| rb.requestTimeout = requestTimeout; |
There was a problem hiding this comment.
Since you are already in this copy block, readTimeout is missing from it. Set a signature calculator and a per request read timeout silently falls back to the config 60s. Exactly the hand copied field list problem this PR is about.
| * would silently revert to the config value partway through the exchange, which is exactly the case this | ||
| * setting exists for. | ||
| */ | ||
| public boolean isUseAbsoluteRequestDeadline() { |
There was a problem hiding this comment.
Holding it on the future is the right call, but the Request now disagrees with the behaviour. After a redirect, getTargetRequest().getUseAbsoluteRequestDeadline() is null, so a filter, a signature calculator or a handler reading the request thinks we are on per attempt timeouts while the future is enforcing a deadline. Redirect30xInterceptor can carry it across for free. While in there, it also drops readTimeout, which resets a 500ms read timeout to the config default on every hop after the first.
|
|
||
| Throwable cause = runAndAwait(baseConfig(), null); | ||
|
|
||
| assertNull(cause, "per-attempt timeouts should let both hops run, got " + cause); |
There was a problem hiding this comment.
This one and the two other assertNull tests only check that nothing threw. Drop the Location header or break followRedirect and they all still pass, having exercised a single hop. Assert the final 200 or the second URI so they can actually fail.
| return AsyncHttpClientConfigHelper.getAsyncHttpClientConfig().getBoolean(ASYNC_CLIENT_CONFIG_ROOT + FAILED_IP_COOLDOWN_ENABLED_CONFIG); | ||
| } | ||
|
|
||
| public static boolean defaultUseAbsoluteRequestDeadline() { |
There was a problem hiding this comment.
Nothing asserts this default. AsyncHttpClientDefaultsTest covers the other flags, and if the key is ever dropped or misspelled getBoolean returns false quietly, which is also the intended default, so the regression stays invisible until someone flips it. A direct TimeoutsHolder test for the anchor and the clamp would help too, the five new tests are all wall clock.
Problem
TimeoutsHolderanchors the request deadline on its own construction:A redirect, a retry and an auth replay all continue the same exchange on the same
NettyResponseFuture, but each builds a new holder for it. Every hop therefore starts thebudget again, so with
maxRedirects=5a chain can legitimately run for six times theconfigured
requestTimeout. Nothing carries an absolute deadline across hops:NettyResponseFuture#getStart()exists but is only read for a diagnosticagein a log line.The
getRequestTimeout()javadoc says it is "the maximum time an AsyncHttpClient waits untilthe response is completed", which is not what happens.
Change
AsyncHttpClientConfig#isUseAbsoluteRequestDeadline(), off by default, anchors thedeadline on when the exchange was submitted instead, so a later hop gets whatever is left of
the budget rather than a fresh one.
Off by default because turning it on shortens exchanges that rely on the per-attempt
behaviour, which is a behaviour change even if the current one contradicts the docs. The
getRequestTimeout()javadoc now describes what actually happens and points at the flag, soit stops being wrong either way.
Settable per request as well as per client, following the existing
followRedirectpattern: a nullable
BooleanonRequestthat overrides the config value.Where the flag lives, and why not on the request
It is resolved once, in
newNettyResponseFuture, and kept on theNettyResponseFuture.The first attempt kept it only on
Requestand the two override tests failed in oppositedirections.
Redirect30xInterceptorrebuilds the request for the next hop from a hand-pickedset of fields, so the override was silently dropped mid-exchange and the config value took
over — precisely in the case the setting exists for. Anything carried only on the request has
that problem, and every future site that rebuilds a request would have to remember it.
Keeping it on the exchange also says the right thing: the deadline describes the exchange, and
a redirect target cannot change it, because the budget belongs to the caller.
API compatibility
revapipasses. Everything added is additive, and one place needed care:DefaultRequestkeeps its existing public constructor, which now delegates to a new onetaking the flag as a trailing argument. Inserting the parameter beside
followRedirectinstead was a binary-incompatible change to a public constructor, and revapi correctly
rejected it — a 27-parameter overload is not pretty, but it is what keeps direct callers
working.
Request#getUseAbsoluteRequestDeadline()is adefaultmethod returning null, so existingimplementations are unaffected.
Tests
AbsoluteRequestDeadlineTestruns two hops of 400 ms against a 600 ms budget, so each hop fitson its own and the pair does not. Five cases:
Timing-based, so
@RepeatedIfExceptionsTest, matching the neighbouring timeout tests.Verification
mvnw clean verify— BUILD SUCCESS, 1474 tests, 0 failures, 0 errors, 26 skipped. ErrorProne, NullAway and Revapi all clean.
Caveat on the testing gate:
AGENTS.mdrequires the build to run on JDK 11 and no JDK 11 isinstalled on this machine, so it was run on JDK 17 (also in the CI matrix). The JDK 11 leg
of CI on this PR is the real gate.
Relationship to #2313
Both touch the same three or four lines of the
TimeoutsHolderconstructor — #2313 replacesnewTimeout(...)with anarm(...)that can target an event loop, this one changes theanchor and the delay. They are otherwise independent and can be reviewed independently.
Whichever merges second will conflict there; happy to rebase this one on top of #2313, or the
other way round, whichever you prefer to take first.
Noticed while running the suite
Two timing tests are flaky under load and unrelated to this change, mentioned only so a red
run is not mistaken for this PR:
SemaphoreTest.checkAcquireTime(three methods) allow 400 ms for a 100 ms timeout and use@RepeatedTest(10), which does not retry, unlike thecheckReleasetests beside them. Thisfailed one cell of thirteen on Arm request timeouts on an event loop #2313's CI (macOS, JDK 21) at 420 ms.
NettyRequestThrottleTimeoutTest.testRequestTimeouttakes ~31 s against a 30 s latch even onan idle machine, and releases its throttle permit only from
onThrowable, so a single requestcompleting instead of timing out deadlocks the remaining threads.
Happy to send a separate PR for both.
Claude Code on behalf of @pavel-ptashyts
🤖 Generated with Claude Code