fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient - #20151
fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient#20151shoemoney wants to merge 2 commits into
Conversation
Fix verified RED->GREEN. Broker masks 429/503 HTML as JsonParseException 0x3c at DirectDruidClient.java:242
| break; | ||
| } | ||
| } | ||
| if (statusCode == 429 || statusCode == 503) { |
There was a problem hiding this comment.
this if block should be placed in front of above if block which detects whether the response is html body
| } | ||
| throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(msg); | ||
| } | ||
| if (isHtmlContentType || isHtmlBody) { |
There was a problem hiding this comment.
I think we should re-organise the new code into two methods which follow current checkTotalBytesLimit and checkQueryTimeout, so in this handleResponse it looks like sth like:
...
checkQueryTimeout();
checkStatusCode(); // which checks 429 and 503
checkHtmlResponse();
checkTotalBytesLimit(response.getContent().readableBytes());
...
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 1 |
| P3 | 0 |
| Total | 2 |
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 1 |
| P3 | 0 |
| Total | 2 |
Reviewed 1 of 1 changed files.
This is an automated review by Codex GPT-5.6-Luna(max)
| preview = preview.substring(0, Math.min(preview.length(), 256)); | ||
| msg = StringUtils.format("%s: %s", msg, preview); | ||
| } | ||
| throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(msg); |
There was a problem hiding this comment.
[P1] Thrown response errors are lost by Netty
In production, NettyHttpClient assigns the result of handleResponse only after it returns. This throw therefore leaves its response null; Netty completes the future with null, and JsonParserIterator converts that into ResourceLimitExceededException instead of QueryCapacityExceededException. The HTML branch has the same issue. Propagate the original exception through the transport failure path and add a Netty-backed test.
| final ChannelBuffer contentBuffer = response.getContent(); | ||
| boolean isHtmlContentType = contentType != null && contentType.toLowerCase().contains("text/html"); | ||
| boolean isHtmlBody = false; | ||
| if (contentBuffer.readableBytes() > 0) { |
There was a problem hiding this comment.
[P2] Chunked HTML bodies bypass detection
For chunked responses without a text/html Content-Type, the initial HttpResponse has no body and the HTML bytes arrive through later HttpChunks, which handleChunk enqueues without inspection. Such responses still reach JSON parsing and fail on '<'. Preserve prefix state across callbacks and inspect the first non-whitespace byte before enqueueing.
…d bodies NettyHttpClient discarded exceptions thrown from HttpResponseHandler#handleResponse by resolving the future to null before rethrowing, so callers never saw the failure. Now the future is failed with the original exception. DirectDruidClient's 429/503 and HTML detection only inspected the buffer attached to the initial HttpResponse, which is empty for chunked replies. handleChunk now retries the same first-non-whitespace-byte check against each chunk until the prefix resolves, so HTML delivered via chunked transfer is still caught before JSON parsing. Also fixes an ImportOrder violation from the prior commit.
|
Fixed both in 60f2b82: NettyHttpClient now fails the future with the original exception instead of resolving to null, and handleChunk retries the HTML-prefix check against each chunk until it resolves. Also fixed the checkstyle import order failing CI. |
Description
Broker can return HTTP 429 or 503 with an HTML error page (for example from a load balancer or reverse proxy) instead of the expected JSON. DirectDruidClient at 242 enqueues the raw body and later JsonParserIterator tries to parse it as JSON, which fails with a JsonParseException for 0x3c ('<') and masks the real capacity error.
This fix addresses three related issues:
Changes:
This preserves the correct exception type for capacity errors and avoids HTML being misreported as a JSON parse failure.
Testing
Release note
Fix broker HTML 429/503 responses being masked as JsonParseException in DirectDruidClient. NettyHttpClient now propagates exceptions from response handlers instead of dropping them.
Key changed/added classes in this PR
This PR has: