fix(models): classify streaming timeouts as retriable + allow retry with partial chunks#1745
Open
gdeyoung wants to merge 9 commits into
Open
fix(models): classify streaming timeouts as retriable + allow retry with partial chunks#1745gdeyoung wants to merge 9 commits into
gdeyoung wants to merge 9 commits into
Conversation
…tection with risk scoring
… guard, CC protocol, GitHub config
…nce and plugin-first rule
…ocs against framework
Fixed 10 broken tools (voice_stt, voice_tts, image_gen, video_gen, media_pipeline, thinking_council, content_extraction, enrichment_tool, knowledge_manager, kg_dreamer) Root causes found and fixed: - helpers/ namespace collision (7 tools) - Tool filename mismatch (7 tools) - Missing Tool inheritance (3 tools) - Extension filename deduplication (23 extensions renamed) - kg_dreamer: syntax error + missing export + missing factories Enforcement architecture: - 23 uniquely-named tool_execute_before extensions - 13 tools verified blocked from main agent - Subordinates pass through freely Other fixes: - backup_manager SSH credential resolution (3-tier fallback) - Deleted stale plugins/plugins/ mirror - Deleted 7 empty helpers/ dirs - Created 7 prompt files, kg_dreamer factories.py - ulimit raised to 65536 - Platform: 20 registered tools, 64 skills, 50 plugins
…ith partial chunks Two bugs in models.py prevented recovery from mid-stream httpx.ReadTimeout (originating as aiohttp.SocketTimeoutError) during streaming completions: 1. _is_transient_litellm_error() did not classify httpx.ReadTimeout, ConnectTimeout, or PoolTimeout as transient — so streaming timeouts were treated as permanent errors and never retried. 2. The got_any_chunk guard in unified_call() blocked ALL retries once any streaming data was received, even for transient errors. When an LLM backend stalls mid-stream (e.g., model pausing during long reasoning chains), the aiohttp socket read timeout fires and propagates as httpx.ReadTimeout via LiteLLM custom aiohttp transport. The retry logic should treat this as a transient error and attempt recovery. Changes: - Add httpx.ReadTimeout, ConnectTimeout, PoolTimeout to transient_types - Modify retry condition: transient errors retry even with partial chunks; non-transient errors with partial chunks still bail immediately
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When an LLM backend stalls mid-stream (e.g., model pausing during long reasoning chains), the aiohttp socket read timeout fires and propagates as
httpx.ReadTimeoutvia LiteLLM's custom aiohttp transport. The retry logic inunified_call()fails to recover from this error due to two bugs:Bug 1: Timeout exceptions not classified as transient
_is_transient_litellm_error()only checks OpenAI SDK exception classes and HTTP status codes.httpx.ReadTimeout(mapped fromaiohttp.SocketTimeoutErrorby LiteLLM's custom transport) is not in the transient types list, so streaming timeouts are treated as permanent errors and never retried.Bug 2:
got_any_chunkguard blocks ALL retriesThe retry condition blocked all retries once any streaming data was received — even for transient errors:
This means a mid-stream stall after receiving the first token(s) is never retried.
Fix
Change 1: Add httpx timeout exceptions to transient types
Change 2: Allow retries for transient errors with partial chunks
Transient errors (timeouts, connection errors) are retried even with partial data — the stall is transient and the full request will likely succeed on retry. Non-transient errors with partial chunks still bail immediately.
Testing
py_compilehttpx.ReadTimeoutinstances are now classified as transientgot_any_chunk=Truenow enter retry loopImpact
This affects every A0 user using streaming LLM completions with LiteLLM's aiohttp transport. Without this fix, any mid-stream stall results in an unrecoverable error with no retry attempt.