fix: send SSE headers before prefill and beat during it - #735
Open
Javinator9889 wants to merge 1 commit into
Open
Javinator9889 wants to merge 1 commit into
Javinator9889 wants to merge 1 commit into
Conversation
write_streaming_response() sends the HTTP headers on its first call, and in the streaming chat path that call cannot happen until the first token is generated -- insert() runs to completion first. On a long prompt that is minutes during which the client receives zero bytes, not even response headers, and cannot distinguish a working server from a dead one. It times out and retries, and the retry queues behind the request still running. Emit an SSE comment before prefill and every five seconds during it. Lines starting with ':' are comments that clients ignore, but they put the headers on the wire and keep the connection alive. is_cancelled() is already invoked once per prefill chunk, so it carries the heartbeat with no new plumbing. Measured on qwen3.5:9b with a 9038-token prompt: time to first byte drops from the full prefill (23s) to 2ms. Answer, usage chunk and [DONE] unchanged. Refs ROCm#733
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the fourth defect in #733.
HttpSession::write_streaming_responsesends the HTTP headers on its firstcall, and in the streaming chat path that call cannot happen until the first
token is generated —
insert()runs to completion first.For a large prompt that is minutes during which the client receives zero bytes,
not even response headers. It cannot distinguish a working server from a dead
one, so it times out and retries, and the retry queues behind the request still
running.
This emits an SSE comment before prefill and every five seconds during it. Lines
beginning with
:are comments that clients ignore, but they put the headers onthe wire and keep the connection demonstrably alive.
is_cancelled()is alreadyinvoked once per prefill chunk, so it carries the heartbeat with no new plumbing.
Measured on
qwen3.5:9bwith a 9038-token prompt:The answer, the usage chunk and the
[DONE]terminator are unchanged; threeheartbeats arrived before the first content chunk.
This does not make prefill faster — it stops the client concluding the server is
dead while it works.