fix(pydantic_ai): capture Anthropic/Google reasoning tokens - #745
Conversation
`_extract_response_metrics` only read `details["reasoning_tokens"]`, which is the key pydantic-ai uses for OpenAI. Anthropic reasoning is stashed under `thinking_tokens` and Google under `thoughts_tokens`, so `completion_reasoning_tokens` was silently empty for those providers (reported for Anthropic in a customer trace set: 16,587/16,587 llm spans with no reasoning metric, all provider=anthropic). Read whichever of the three normalized keys is present. Verified end-to-end by driving pydantic-ai's real `_map_usage` from an Anthropic response with thinking tokens through the extractor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7846dfc38c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| usage = RequestUsage(input_tokens=10, output_tokens=20, details={details_key: 128}) | ||
| response = SimpleNamespace(parts=[], usage=usage) |
There was a problem hiding this comment.
In the checked test_pydantic_ai_integration matrix, this regression test manually inserts each expected key into RequestUsage.details, so it cannot catch the provider-specific failure that matters here: Anthropic or Google changing—or failing to populate—the key while the extractor test continues to pass. Exercise at least the Anthropic/Google mapping through a real provider response and checked-in cassette, keeping this synthetic test only as supplemental coverage.
AGENTS.md reference: AGENTS.md:L182-L184
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
replaced the synthetic test with a real cassette to repro pre / post fix
Address review feedback: the synthetic parametrized test verifies the extractor reads each provider key, but not that pydantic-ai actually populates thinking_tokens from a real Anthropic response. Add a cassette-backed test that runs a real Agent against Anthropic (extended thinking enabled) through pydantic-ai's own _map_usage, asserting the leaf chat span surfaces completion_reasoning_tokens. Keeps the synthetic test as supplemental coverage (per AGENTS.md VCR-first rule). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 participants
Problem
In the pydantic-ai integration,
_extract_response_metricsreads reasoning tokens only fromusage.details["reasoning_tokens"], the key pydantic-ai uses for OpenAI. But pydantic-ai stashes the same normalized quantity under a different key per provider:detailskeyreasoning_tokensthinking_tokensthoughts_tokensSo
completion_reasoning_tokenswas silently empty for Anthropic and Google. Nothing failed, nothing warned the charts just look like reasoning was never used.Reported by a customer: in their non-prod agent platform, 16,587 of 16,587 llm spans over 30 days had no reasoning metric, all
provider = anthropic.pydantic-ai's own source confirms the counts are equivalent, the Anthropic mapping comment (
models/anthropic.py) says thinking tokens are "a readable subset of the output total ... matchingreasoning_tokenson OpenAI andthoughts_tokenson Google."Fix
Read whichever of the three normalized keys is present in
details.Verification
Reproduced end-to-end by driving pydantic-ai's real
_map_usagefrom an AnthropicBetaMessagecarryingoutput_tokens_details.thinking_tokens=180:completion_reasoning_tokens = None(OpenAI captured 180)completion_reasoning_tokens = 180.0Added a parametrized regression test covering all three provider keys.
Blast radius
_extract_response_metrics; no other metric or provider path changes.reasoning_tokensstill read first).