Skip to content

fix: preserve Anthropic streaming input and cache usage - #4360

Merged
dgageot merged 1 commit into
mainfrom
fix/anthropic-stream-usage
Sep 18, 2026
Merged

dgageot merged 1 commit into
mainfrom
fix/anthropic-stream-usage

Conversation

@dgageot

@dgageot dgageot commented Sep 18, 2026

Copy link
Copy Markdown
Member

When Anthropic streams a response, input token counts and cache read/write
metrics arrive in the message_start event, while message_delta carries
only output tokens. The previous code built its usage snapshot from the
message_delta event alone, so input tokens and cache usage were silently
dropped from every streaming call.

Both the standard and beta adapter paths now map the SDK's accumulated
usage object (which the SDK merges across all events) instead of the
message_delta snapshot. Explicit delta counts, including zero, still
override the accumulated value, so the behavior for fields that arrive in
message_delta is unchanged. A raw-capture failure no longer suppresses
usage entirely — the code falls through to the accumulated snapshot.
track_usage opt-out and fresh per-call snapshots are maintained.

Tests cover: field omission from message_delta (input/cache preserved),
cumulative updates, zero-value replacement, streams with no message_start,
disabled tracking, and raw-capture failures.

Full task test passes with the two TestLoadExamples subtests for
dmr.yaml and unload_on_switch.yaml excluded — both require a local
Docker Model Runner that is not available in CI and fail identically on
the unmodified baseline.

Signed-off-by: David Gageot <david.gageot@docker.com>
@dgageot
dgageot requested a review from a team as a code owner September 18, 2026 12:42
@dgageot
dgageot added this pull request to the merge queue Sep 18, 2026
@aheritier aheritier added area/models LLM model integrations and model providers kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Sep 18, 2026
Merged via the queue into main with commit 90988b7 Sep 18, 2026
17 checks passed
@dgageot
dgageot deleted the fix/anthropic-stream-usage branch September 18, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/models LLM model integrations and model providers kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants