feat(convozen): add Akshara STT and Ragini TTS plugin - #7319
aminaashrafch wants to merge 10 commits into
Conversation
Adds livekit-plugins-convozen for ConvoZen's Akshara speech-to-text and Ragini text-to-speech, covering nine Indian languages including code-mixed speech. Akshara returns one transcript per request and emits no interim results, so the STT declares streaming=False and Agent.stt_node wraps it in stt.StreamAdapter using the session VAD to segment utterances. Ragini synthesizes a whole request at a time, so the TTS declares streaming=False and is wrapped by tts.StreamAdapter. Neither has a websocket path. Ragini returns a chunked WAV container whose RIFF length fields are placeholders, so the package depends on livekit-agents[codecs] and lets AudioEmitter decode it rather than parsing headers itself. Keyterms configured on the AgentSession are forwarded to Akshara's keywords field. Akshara also returns a score, which is an unbounded log-probability rather than a [0, 1] confidence; it is exposed through SpeechData.metadata and confidence is left at its default.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| if res.status != 200: | ||
| body = (await res.text())[:_MAX_ERROR_BODY] | ||
| raise APIStatusError( | ||
| message=f"ConvoZen Akshara returned {res.status}: {body}", | ||
| status_code=res.status, | ||
| request_id=None, | ||
| body=body, | ||
| ) |
There was a problem hiding this comment.
🟨 STT errors expose provider content
A non-200 response embeds body in APIStatusError. Retry logs and exception telemetry can expose provider content without PII tagging.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
In 5e45276 I set aligned_transcript=False explicitly, documented that word_timestamps values are relative to each recognized utterance (docstring and README), and added a test pinning the capability with word_timestamps=True. If you'd like batch STTs to support alignment, I'm happy to follow up with a StreamAdapter change that propagates the capability and offsets forwarded timings by each segment's position in the stream.
Akshara returns word timings relative to each recognized utterance, and stt.StreamAdapter forwards them without offsetting them onto the audio stream, so they do not meet the aligned_transcript contract. Declare aligned_transcript=False explicitly, document the timing reference in the word_timestamps docstring and README, and pin the capability in tests.
# Conflicts: # livekit-agents/pyproject.toml
There was a problem hiding this comment.
Devin Review found 2 new potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| if is_given(language) or is_given(lang_tags): | ||
| # Re-derive so that changing the language alone also moves the hint, | ||
| # and so an explicit tag list is validated the same way as at __init__. | ||
| self._opts.lang_tags = _resolve_lang_tags( | ||
| self._opts.language, | ||
| lang_tags if is_given(lang_tags) else NOT_GIVEN, | ||
| ) |
There was a problem hiding this comment.
🟡 Explicit language hints are discarded
Calling update_options(language=...) replaces explicit lang_tags with one derived tag. Later code-mixed recognition loses its configured language hints.
Learn more
lang_tags can either be explicit configuration or a default derived from language. The current options object stores only the resolved list, so update_options cannot distinguish those cases. A language-only update always resolves a fresh list and overwrites explicit code-mixing hints.
Example: Construct STT(language="hi", lang_tags=["hi", "en"]), then call update_options(language="ta"). The next request sends ["ta"] instead of retaining the explicitly configured ["hi", "en"].
Recommended fix: Track whether lang_tags is explicit or derived. Recompute it on a language-only update only when it was derived; update that state whenever lang_tags itself is supplied.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Adds
livekit-plugins-convozen, integrating ConvoZen's Akshara speech-to-text and Ragini text-to-speech models. Both cover nine Indian languages (Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Tamil, Telugu), including code-mixed speech.Install with
pip install "livekit-agents[convozen]".Design
STTdeclaresstreaming=False.Agent.stt_nodewraps it instt.StreamAdapter, so a VAD is required on theAgentSession.TTSdeclaresstreaming=Falseand is wrapped bytts.StreamAdapter. Audio streams back as a chunked WAV, decoded byAudioEmitter(hence thelivekit-agents[codecs]dependency).keywordsfield (keyterms=True).scoreis an unbounded log-probability rather than a[0, 1]confidence, so it's exposed viaSpeechData.metadataandconfidenceis left unset.from Noneand report only the exception type, since aiohttp errors carry request headers including the API key.Testing
tests/test_plugin_convozen.py: 29 hermetic unit tests (pytest --unit)make format-checkandmake lintpass;livekit.plugins.convozentype-checks undermake type-checkAgentSessionwith Silero VAD against a local LiveKit server: caller speech → STT → agent turn → TTS audio back to the callerNotes
uv.lockisn't included; regenerating it locally changed unrelated Python version markers, and CI'suv syncresolves it.