fix(openai): report realtime transcription usage per session - #2521
rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 7330703 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| /** Audio input tokens, a subset of inputTokens when reported by the provider. */ | ||
| inputAudioTokens?: number; |
There was a problem hiding this comment.
🟡 Remote usage drops audio tokens
When inputAudioTokens reaches remote usage, sessionUsageToProto omits it. Remote consumers lose the audio-token billing breakdown.
Learn more
The session usage type now preserves audio input tokens, and local reports serialize them. Remote sessions use a separate protobuf conversion. That conversion copies STT input tokens, output tokens, and duration, but not inputAudioTokens in sessionUsageToProto. Every remote usage event and getSessionUsage response therefore drops this new value.
Example: A realtime transcription records 10 input tokens, including 8 audio tokens. Local session.usage contains inputAudioTokens: 8, while a remote client receives only 10 input tokens and cannot recover the audio subset.
Recommended fix: Add the audio-token field to the protocol's STTModelUsage, update @livekit/protocol, and map su.inputAudioTokens in sessionUsageToProto. Add coverage for both remote usage events and getSessionUsage responses.
Was this helpful? React with 👍 or 👎 to provide feedback.
| sttUsage.inputAudioTokens = | ||
| (sttUsage.inputAudioTokens ?? 0) + (metrics.inputAudioTokens ?? 0); |
There was a problem hiding this comment.
🟡 Unknown audio tokens become zero
For metrics without inputAudioTokens, ModelUsageCollector stores zero in the optional field. flatten() reports a measured zero instead of unavailable data.
Learn more
inputAudioTokens is optional because most STT providers do not report an audio-token breakdown. The collector initializes the field to zero and also assigns zero whenever an incoming metric omits it. Consequently, flatten() includes inputAudioTokens: 0 even though no provider supplied that measurement. The report serializer hides this through zero filtering, but direct session usage consumers still receive the false value.
Example: Deepgram emits an STT metric with no inputAudioTokens. The collector returns inputAudioTokens: 0, making consumers interpret the breakdown as reported and free rather than unavailable.
Recommended fix: Do not initialize inputAudioTokens in getSTTUsage. Increment it only when metrics.inputAudioTokens !== undefined, preserving undefined until a provider reports the field.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#7303.
Reports OpenAI realtime transcription duration and token usage as streamed STT metrics. Preserves audio-token breakdowns in session usage and reports, and attributes each metric to the session current transcription model.
Source diff coverage
livekit-agents/livekit/agents/metrics/base.py: adapted toagents/src/metrics/base.ts; added total/audio token fields and an internal resolver because JS metrics are structural objects rather than Pydantic models.livekit-agents/livekit/agents/metrics/usage.py: ported toagents/src/metrics/model_usage.ts; preserves and aggregates input audio tokens separately.livekit-agents/livekit/agents/metrics/usage_collector.py: ported toagents/src/metrics/usage_collector.ts; extends the deprecated summary with STT token totals.livekit-agents/livekit/agents/metrics/utils.py: ported toagents/src/metrics/utils.ts; logs input, output, total, and input-audio tokens.livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py: adapted toplugins/openai/src/realtime/realtime_model.tsandplugins/openai/src/realtime/api_proto.ts; explicit TypeScript wire validation replaces Pydantic validation, and provider seconds are converted to framework milliseconds.tests/test_metrics_usage.py: adapted toagents/src/metrics/model_usage.test.ts; ports omitted/explicit total-token behavior and STT token aggregation coverage.tests/test_realtime/test_openai_realtime_model.py: adapted toplugins/openai/src/realtime/realtime_model.test.tsandagents/src/voice/report.test.ts; ports duration usage, token usage, malformed usage, session model updates, session aggregation, and report serialization.Target-only release/generated artifacts:
agents/etc/agents.api.md,plugins/openai/etc/agents-plugin-openai.api.md, and.changeset/realtime-transcription-session-usage.md.Validation
pnpm build: passed (all 40 workspace packages).agentssuite run: 2712 passed, 5 skipped, 4 unrelatedtelemetry/loop_monitor.test.tsfailures because this host exposes process rather than thread CPU accounting.plugins/openai/src/ws/llm.ts:127no-misused-promiseserror.cue-clivoice validation attempted; the configured Cue project returned401 Unauthorized - invalid API key, so no live framework-event assertion was available.Ported from livekit/agents#7303
Original PR description
Supersedes #5029, adopting @bml1g12's original transcription usage implementation.
Report OpenAI realtime ASR duration and token usage as STT metrics. Preserve audio-token counts in session usage and reports, attribute usage to each session's current transcription model, and mark realtime transcription as streamed.
input_tokensincludes both audio and text;input_audio_tokensis the audio subset.Addresses AGT-3525.
Initial prompt and agent context
Model: GPT-6