feat(agents): support declarative prompt-voice agents (managed model) - #9364
Draft
v1212 wants to merge 3 commits into
Draft
feat(agents): support declarative prompt-voice agents (managed model)#9364v1212 wants to merge 3 commits into
v1212 wants to merge 3 commits into
Conversation
Add azd support for a new 'prompt-voice' agent kind that creates a managed speech-to-speech (voice) agent on Azure AI Foundry. - yaml: new prompt-voice kind + VoiceAgent authoring struct/parsing - map: translate authoring kind prompt-voice -> data-plane kind voice, defaulting the audio pipeline (PCM16@24k, server_vad, whisper-1, DragonHD default voice) and v1 implicit managed model_type - agent_api: VoiceAgentDefinition wire structs + CreateVoiceAgent with Foundry-Features: VoiceAgents=V1Preview preview header - project: voice-aware agent_definition read/write + isolated deployVoiceAgent deploy path (container path unchanged) - init: --kind/--voice flags, voice manifest synthesis, prompt option Scope: prompt-voice + managed model only. BYOM, hosted-voice, tools, avatar, and cascaded models are follow-ups. Draft: needs further end-to-end session testing and optimization.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
- map_voice_test.go: cover CreateVoiceAgentAPIRequest defaults/overrides, managed enforcement, BYOM rejection, missing-model error, and the isOpenAIVoice/buildVoiceConfig voice-type selection - parse_voice_test.go: cover prompt-voice manifest parsing and ValidateAgentDefinition (ok / missing model.id / self_deployed rejected) - azure.ai.agent.json: add prompt-voice to the kind enum and document the voice service properties (modelType/model/instructions/voice/store) - cspell.yaml: allow BYOM
The azure.ai.projects synthesis copy must stay byte-identical to the azure.ai.agents copy (TestAgentsSynthesisCopyMatches). Mirror the prompt-voice comment update made in the agents synthesizer.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds managed declarative prompt-voice agents across initialization, configuration, API mapping, and deployment.
Changes:
- Adds voice-agent YAML/schema models and validation.
- Maps voice manifests to Foundry’s preview API contract.
- Adds voice-specific initialization and deployment paths with tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
schemas/azure.ai.agent.json |
Adds voice-agent schema fields. |
internal/synthesis/synthesizer.go |
Marks voice agents as non-container. |
internal/project/service_target_agent.go |
Adds voice deployment handling. |
internal/project/agent_definition.go |
Adds inline voice configuration conversion. |
internal/pkg/agents/agent_yaml/yaml.go |
Defines voice authoring models. |
internal/pkg/agents/agent_yaml/parse.go |
Parses and validates voice manifests. |
internal/pkg/agents/agent_yaml/parse_voice_test.go |
Tests voice parsing and validation. |
internal/pkg/agents/agent_yaml/map.go |
Maps voice manifests to API requests. |
internal/pkg/agents/agent_yaml/map_voice_test.go |
Tests voice request mapping. |
internal/pkg/agents/agent_api/operations.go |
Adds the preview create operation. |
internal/pkg/agents/agent_api/models.go |
Defines voice API wire models. |
internal/cmd/init.go |
Adds voice initialization flags and flow. |
internal/cmd/init_from_templates_helpers.go |
Adds the interactive voice option. |
cspell.yaml |
Adds voice-related terminology. |
Comment on lines
+1079
to
+1082
| if va, isVoice, vErr := VoiceAgentFromResolvedService(serviceConfig, p.projectPath); vErr != nil { | ||
| return nil, vErr | ||
| } else if isVoice { | ||
| return p.deployVoiceAgent(ctx, serviceConfig, va, azdEnv, progress) |
Comment on lines
+1286
to
+1290
| // Prompt-voice fast path: when --kind prompt-voice is set without a | ||
| // manifest, there is no source to scaffold and no template/language to | ||
| // choose. Synthesize a declarative (managed) voice manifest and route it | ||
| // through the manifest flow (which skips the init-mode / template / | ||
| // language prompts and code scaffolding). Mirrors the --image fast path. |
Comment on lines
+1299
to
+1304
| if flags.image != "" { | ||
| return exterrors.Validation( | ||
| exterrors.CodeInvalidParameter, | ||
| "--kind prompt-voice cannot be combined with --image", | ||
| "a voice agent is managed and has no container image; drop --image", | ||
| ) |
Comment on lines
+1785
to
+1787
| for key, value := range map[string]string{ | ||
| fmt.Sprintf("AGENT_%s_NAME", serviceKey): agentObject.Name, | ||
| fmt.Sprintf("AGENT_%s_ENDPOINT", serviceKey): baseEndpoint, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds azd support for a new declarative
prompt-voiceagent kind that creates a managed speech-to-speech (voice) agent on Azure AI Foundry, end-to-end throughazd init→azure.yaml→azd deploy.Scope (intentionally narrow):
prompt-voice+ managed model only. BYOM, hosted-voice, tools, avatar, and cascaded models are deliberately left as follow-up PRs.What's included
agent_yaml/yaml.go,parse.go): newprompt-voiceauthoring kind +VoiceAgentstruct/parsing/validation.agent_yaml/map.go): translate authoring kindprompt-voice→ data-plane service kindvoice; default the audio pipeline (PCM16 @ 24 kHz,server_vad,whisper-1, DragonHD default voice); v1 = implicitmodel_type: managed.agent_api/models.go,operations.go):VoiceAgentDefinitionwire structs +CreateVoiceAgentwith the required preview headerFoundry-Features: VoiceAgents=V1Preview.project/agent_definition.go,service_target_agent.go): voice-aware definition read/write + an isolateddeployVoiceAgentdeploy path — the existing hosted/container path is byte-for-byte unchanged.cmd/init.go,init_from_templates_helpers.go):--kind/--voiceflags, voice manifest synthesis, and a new prompt option.azure.ai.agent.jsongains theprompt-voicekind + voice service properties; unit tests cover the map translation, voice-type selection, and manifest parse/validation.Compatibility
All changes are additive — new
casebranches with untoucheddefaults andomitemptyfields. Existingazd init/ invoke / deploy / list flows and the hosted/workflow code paths are unchanged.Testing
go build ./...,go vet ./..., andcspellclean.map_voice_test.go,parse_voice_test.go) plus the existing suite pass.proj-voice): an agent created via the realCreateVoiceAgentAPIRequest+CreateVoiceAgentcode appears in the project list and its storeddefinitionis byte-identical to the service's own managed reference agent (kind:voice/model_type:managed/ audio pipeline / voice config). Connect reachessession.created.Known gaps (why this is a draft)
session.audio.output.voicestring coercion at the Voice Live layer, and a demo-clientsession.updateframing issue) — being investigated; not specific to this change.Follow-ups
self_deployed) model type