Skip to content

fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient - #20151

Open
shoemoney wants to merge 2 commits into
apache:masterfrom
shoemoney:fix/broker-429-html
Open

fix(druid): handle 429/503 HTML before JSON parse in DirectDruidClient#20151
shoemoney wants to merge 2 commits into
apache:masterfrom
shoemoney:fix/broker-429-html

Conversation

@shoemoney

@shoemoney shoemoney commented Aug 25, 2026

Copy link
Copy Markdown

Description

Broker can return HTTP 429 or 503 with an HTML error page (for example from a load balancer or reverse proxy) instead of the expected JSON. DirectDruidClient at 242 enqueues the raw body and later JsonParserIterator tries to parse it as JSON, which fails with a JsonParseException for 0x3c ('<') and masks the real capacity error.

This fix addresses three related issues:

  1. NettyHttpClient silently drops exceptions thrown from handleResponse(), resolving the future to null instead of propagating the error. This masked capacity exceptions as successful null results.
  2. DirectDruidClient checks response status and Content-Type before JSON parse in handleResponse().
  3. For chunked responses without Content-Type headers, HTML can arrive via HttpChunks after the initial response. DirectDruidClient now inspects each chunk until the prefix is resolved.

Changes:

  • NettyHttpClient: propagate exceptions from the handler via retVal.setException() instead of resolving to null
  • DirectDruidClient: check status 429/503 and HTML content-type/prefix in handleResponse()
  • DirectDruidClient: retry prefix check in handleChunk() for chunked bodies

This preserves the correct exception type for capacity errors and avoids HTML being misreported as a JSON parse failure.

Testing

  • RED to GREEN verified locally
  • DirectDruidClientTest, 12 tests pass before and after the change
  • NettyHttpClientTest added: regression test for exception propagation from handleResponse
  • Formatter blast radius limited to modified files

Release note

Fix broker HTML 429/503 responses being masked as JsonParseException in DirectDruidClient. NettyHttpClient now propagates exceptions from response handlers instead of dropping them.


Key changed/added classes in this PR
  • NettyHttpClient (exception propagation fix)
  • NettyHttpClientTest (new regression test)
  • DirectDruidClient (HTML detection and status code checks)

This PR has:

  • been self-reviewed

Fix verified RED->GREEN. Broker masks 429/503 HTML as JsonParseException 0x3c at DirectDruidClient.java:242
break;
}
}
if (statusCode == 429 || statusCode == 503) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if block should be placed in front of above if block which detects whether the response is html body

}
throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(msg);
}
if (isHtmlContentType || isHtmlBody) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should re-organise the new code into two methods which follow current checkTotalBytesLimit and checkQueryTimeout, so in this handleResponse it looks like sth like:

...
checkQueryTimeout();
checkStatusCode(); // which checks 429 and 503
checkHtmlResponse();
checkTotalBytesLimit(response.getContent().readableBytes());
...

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 1
P3 0
Total 2
Severity Findings
P0 0
P1 1
P2 1
P3 0
Total 2

Reviewed 1 of 1 changed files.


This is an automated review by Codex GPT-5.6-Luna(max)

preview = preview.substring(0, Math.min(preview.length(), 256));
msg = StringUtils.format("%s: %s", msg, preview);
}
throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(msg);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Thrown response errors are lost by Netty

In production, NettyHttpClient assigns the result of handleResponse only after it returns. This throw therefore leaves its response null; Netty completes the future with null, and JsonParserIterator converts that into ResourceLimitExceededException instead of QueryCapacityExceededException. The HTML branch has the same issue. Propagate the original exception through the transport failure path and add a Netty-backed test.

final ChannelBuffer contentBuffer = response.getContent();
boolean isHtmlContentType = contentType != null && contentType.toLowerCase().contains("text/html");
boolean isHtmlBody = false;
if (contentBuffer.readableBytes() > 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Chunked HTML bodies bypass detection

For chunked responses without a text/html Content-Type, the initial HttpResponse has no body and the HTML bytes arrive through later HttpChunks, which handleChunk enqueues without inspection. Such responses still reach JSON parsing and fail on '<'. Preserve prefix state across callbacks and inspect the first non-whitespace byte before enqueueing.

…d bodies

NettyHttpClient discarded exceptions thrown from HttpResponseHandler#handleResponse
by resolving the future to null before rethrowing, so callers never saw the failure.
Now the future is failed with the original exception.

DirectDruidClient's 429/503 and HTML detection only inspected the buffer attached to
the initial HttpResponse, which is empty for chunked replies. handleChunk now retries
the same first-non-whitespace-byte check against each chunk until the prefix resolves,
so HTML delivered via chunked transfer is still caught before JSON parsing.

Also fixes an ImportOrder violation from the prior commit.
@shoemoney

Copy link
Copy Markdown
Author

Fixed both in 60f2b82: NettyHttpClient now fails the future with the original exception instead of resolving to null, and handleChunk retries the HTML-prefix check against each chunk until it resolves. Also fixed the checkstyle import order failing CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants