Skip to content

fix(models): classify streaming timeouts as retriable + allow retry with partial chunks#1745

Open
gdeyoung wants to merge 9 commits into
agent0ai:mainfrom
gdeyoung:fix/streaming-timeout-retry
Open

fix(models): classify streaming timeouts as retriable + allow retry with partial chunks#1745
gdeyoung wants to merge 9 commits into
agent0ai:mainfrom
gdeyoung:fix/streaming-timeout-retry

Conversation

@gdeyoung

@gdeyoung gdeyoung commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Problem

When an LLM backend stalls mid-stream (e.g., model pausing during long reasoning chains), the aiohttp socket read timeout fires and propagates as httpx.ReadTimeout via LiteLLM's custom aiohttp transport. The retry logic in unified_call() fails to recover from this error due to two bugs:

Bug 1: Timeout exceptions not classified as transient

_is_transient_litellm_error() only checks OpenAI SDK exception classes and HTTP status codes. httpx.ReadTimeout (mapped from aiohttp.SocketTimeoutError by LiteLLM's custom transport) is not in the transient types list, so streaming timeouts are treated as permanent errors and never retried.

# Before — missing httpx timeout types
transient_types = (
    getattr(openai, "APITimeoutError", Exception),
    getattr(openai, "APIConnectionError", Exception),
    ...
)

Bug 2: got_any_chunk guard blocks ALL retries

The retry condition blocked all retries once any streaming data was received — even for transient errors:

# Before — no retry if ANY chunk received
if got_any_chunk or not _is_transient_litellm_error(e) or attempt >= max_retries:
    raise

This means a mid-stream stall after receiving the first token(s) is never retried.

Fix

Change 1: Add httpx timeout exceptions to transient types

transient_types = (
    getattr(openai, "APITimeoutError", Exception),
    getattr(openai, "APIConnectionError", Exception),
    ...
    # Streaming socket timeouts
    httpx.ReadTimeout,
    httpx.ConnectTimeout,
    httpx.PoolTimeout,
)

Change 2: Allow retries for transient errors with partial chunks

is_transient = _is_transient_litellm_error(e)
if attempt >= max_retries:
    raise
if not is_transient and got_any_chunk:
    raise

Transient errors (timeouts, connection errors) are retried even with partial data — the stall is transient and the full request will likely succeed on retry. Non-transient errors with partial chunks still bail immediately.

Testing

  • Syntax verified via py_compile
  • Error classification verified: httpx.ReadTimeout instances are now classified as transient
  • Retry condition verified: transient errors with got_any_chunk=True now enter retry loop

Impact

This affects every A0 user using streaming LLM completions with LiteLLM's aiohttp transport. Without this fix, any mid-stream stall results in an unrecoverable error with no retry attempt.

Agent Zero and others added 9 commits June 3, 2026 11:53
Fixed 10 broken tools (voice_stt, voice_tts, image_gen, video_gen, media_pipeline,
thinking_council, content_extraction, enrichment_tool, knowledge_manager, kg_dreamer)

Root causes found and fixed:
- helpers/ namespace collision (7 tools)
- Tool filename mismatch (7 tools)
- Missing Tool inheritance (3 tools)
- Extension filename deduplication (23 extensions renamed)
- kg_dreamer: syntax error + missing export + missing factories

Enforcement architecture:
- 23 uniquely-named tool_execute_before extensions
- 13 tools verified blocked from main agent
- Subordinates pass through freely

Other fixes:
- backup_manager SSH credential resolution (3-tier fallback)
- Deleted stale plugins/plugins/ mirror
- Deleted 7 empty helpers/ dirs
- Created 7 prompt files, kg_dreamer factories.py
- ulimit raised to 65536
- Platform: 20 registered tools, 64 skills, 50 plugins
…ith partial chunks

Two bugs in models.py prevented recovery from mid-stream httpx.ReadTimeout
(originating as aiohttp.SocketTimeoutError) during streaming completions:

1. _is_transient_litellm_error() did not classify httpx.ReadTimeout,
   ConnectTimeout, or PoolTimeout as transient — so streaming timeouts
   were treated as permanent errors and never retried.

2. The got_any_chunk guard in unified_call() blocked ALL retries once
   any streaming data was received, even for transient errors.

When an LLM backend stalls mid-stream (e.g., model pausing during long
reasoning chains), the aiohttp socket read timeout fires and propagates
as httpx.ReadTimeout via LiteLLM custom aiohttp transport. The retry
logic should treat this as a transient error and attempt recovery.

Changes:
- Add httpx.ReadTimeout, ConnectTimeout, PoolTimeout to transient_types
- Modify retry condition: transient errors retry even with partial chunks;
  non-transient errors with partial chunks still bail immediately
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants