[Blazor] Add multimodal message input components by kotlarmilos · Pull Request #68672 · dotnet/aspnetcore

kotlarmilos · 2026-08-20T21:13:40Z

Overview

This is position 10 in the native Components.AI stack and depends directly on #68335. This three-commit layer adds reusable multimodal message input components for attachments, recorded audio, live speech, submission, and cancellation, then exercises them in a production-oriented vehicle claim sample. The shipping UI runtime remains provider-neutral while the sample uses the real UIAgent -> AGUIChatClient -> HTTP/SSE -> AGUI.Server path and Microsoft Foundry for chat, vision, transcription, and grounded repair research.

Design

MessageInput becomes the composition boundary for text and DataContent. It owns the draft, attachments, validation state, submission state, and cancellation state through MessageInputContext. MessageAttachButton, AudioCaptureButton, LiveSpeechButton, MessageSendButton, and MessageStopButton consume that shared context instead of duplicating composer state.

Recorded voice and live voice have separate behavior. Recorded voice streams interim browser recognition into the textbox and sends the completed audio to the configured transcription callback. Live voice submits finalized browser transcripts directly as conversation turns, keeps listening across model responses and recoverable failures, and stops only when the user explicitly disables it or browser speech recognition reports a fatal error. Assistant responses remain text-only.

Media enters the existing content pipeline through DataContentBlock, DataContentHandler, and MediaContent. This keeps images and audio represented as Microsoft.Extensions.AI.DataContent throughout the agent pipeline instead of introducing provider-specific message types. Microsoft.AspNetCore.Components.Media.Audio provides the corresponding audio rendering surface. Audio capture uses the browser-native MediaRecorder and SpeechRecognition APIs without third-party JavaScript dependencies.

Retries now treat each protocol round as a transaction. A failed user message or continuation is not committed to conversation history, partial response blocks are removed, and retry replays the exact failed round without duplicating the rendered request, text, attachments, tool results, or approval responses.

Implementation

The vehicle claim sample demonstrates the complete composition model:

Text, drag-and-drop photos, file selection, recorded voice, and continuous live voice share one MessageInput.
Claim state crosses the AG-UI boundary through official state snapshot and RFC 6902 delta events.
Damage visualization uses a client-side UI action.
Claim submission uses the AG-UI approval and resume flow.
Production requests use Microsoft Foundry for conversation, image analysis, transcription, and public web research.
E2E tests replace only IClaimAssistantBackend. AGUIChatClient, HTTP, SSE serialization, AGUI.Server, client tools, state mapping, and approval continuations remain production code.
Model-provided research links are restricted to normalized absolute HTTP and HTTPS URLs.
Evidence limits are enforced consistently in the composer, workflow, and the bounded claim-agent request endpoint.
The agent base address supports an explicit ClaimAgent:BaseAddress and otherwise resolves from NavigationManager.BaseUri, preserving reverse-proxy path prefixes.

The sample also covers light, dark, and high-contrast themes, keyboard submission and cancellation, accessible attachment and voice controls, cumulative evidence reassessment, approval rejection, and retry recovery.

Outcome

Validation class	Result
Components.AI runtime and component tests	139 passed
Components.Media tests	15 passed
ClaimApp browser scenarios	22 passed, 3 configuration-gated Foundry scenarios skipped
ClaimApp build	0 warnings, 0 errors
Diff validation	`git diff --check` passed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ab398e0-c70e-427e-93a8-134ee977fb17

javiercn

More feedback.

I'm still not done, I think there are some issues with how the InputMessage component works witht the underlying input sources (like audio)

dariatiurina

Both of my notes here are applicable in more places than what I have marked.

Azure Identity can wrap a canceled token request in an AuthenticationFailedException. Normalize that specific case to an OperationCanceledException so canceling voice transcription does not terminate the Blazor circuit. Add regression coverage for the wrapped cancellation.

ilonatommy · 2026-09-09T07:53:26Z

Recorded voice and live voice have separate behavior. Recorded voice streams interim browser recognition into the textbox and sends the completed audio to the configured transcription callback. Live voice submits finalized browser transcripts directly as conversation turns, keeps listening across model responses and recoverable failures, and stops only when the user explicitly disables it or browser speech recognition reports a fatal error. Assistant responses remain text-only.

I have doubts about "live voice".
Auto-submit feature is added to the component that has weaker recognition skills. I understand that it's for hands-free mode but e.g. CopilotKit has only one component (CopilotChatAudioRecorder) for transcription/voice and it auto-sends after backend double checked the input. We're proposing the auto-submit in the component that doesn't have the backend's dobule-check correction.

What about simplifying to one component only? We would support 3 modes: backend speech recognition only (equivalent of recorded voice with ShowInterimTranscript="false"), backend + browser recognition (equivalent of recorded voice with ShowInterimTranscript="true") and browser only (equivalent of live voice). On the top of that, the app can choose if to allow edits or auto-submit.

ilonatommy · 2026-09-09T08:10:53Z

Recorded voice and live voice have separate behavior. Recorded voice streams interim browser recognition into the textbox and sends the completed audio to the configured transcription callback. Live voice submits finalized browser transcripts directly as conversation turns, keeps listening across model responses and recoverable failures, and stops only when the user explicitly disables it or browser speech recognition reports a fatal error. Assistant responses remain text-only.

I have doubts about "live voice". Auto-submit feature is added to the component that has weaker recognition skills. I understand that it's for hands-free mode but e.g. CopilotKit has only one component (CopilotChatAudioRecorder) for transcription/voice and it auto-sends after backend double checked the input. We're proposing the auto-submit in the component that doesn't have the backend's dobule-check correction.

What about simplifying to one component only? We would support 3 modes: backend speech recognition only (equivalent of recorded voice with ShowInterimTranscript="false"), backend + browser recognition (equivalent of recorded voice with ShowInterimTranscript="true") and browser only (equivalent of live voice). On the top of that, the app can choose if to allow edits or auto-submit.

Double checking in #68227, section "Journey 8 — Ask by voice" has no requirements about hands-free mode nor about components separation. I underlines multiple times that all transcriptions have to be editable.

dictation is another way to author the message, never a separate pipeline that sends behind their back.

Recognition is imperfect, so they stop dictating and edit what was heard with the keyboard

and correctable before it is sent

Add configurable speech recognition modes, auto-submit, and continuous listening to AudioCaptureButton, replacing LiveSpeechButton.

ilonatommy · 2026-09-09T08:59:18Z

Demo of redesigned AudioCaptureButton from 820ac12. All three buttons use same component with different modes and auto-submit settings. The advantage of this design: user can combine auto-submission with backend transcription - the setup that makes most sense for hands-free usage (not possible in the original design).

one-component.mp4

ilonatommy · 2026-09-09T09:29:08Z

Edge cases. Permission revoked mid-session. No speech API in the browser at all, which must degrade to the control simply not being offered. Dictating while an answer streams should queue exactly as typing does (journey 1). Interim versus final transcription results should not cause the box to flicker.

From #68227 (comment). We're missing tests for this, I am working on covering that gap.

Fixed in 84f8832.

…oads and causes compiler errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 94653859-e420-46e8-828f-80dd291e33e5

Keep audio button content stable while microphone permission is pending and render accessible requesting, denied, and unavailable feedback above the message composer. Support custom messages, rendering, state inspection, and dismissal. Harden late permission cleanup, update ClaimApp for the current media API, and add unit and browser coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f835455-8409-4733-b968-69f7d3b9dd90

kotlarmilos · 2026-09-14T14:19:27Z

@javiercn @dariatiurina I resolved the remaining comments to resolve stale threads. Please un-resolve threads where you want to keep the discussion open. @dariatiurina will isolate the sample app in a follow-up PR to make it easier for review.

Keep this change focused on the reusable multimodal message input components. The vehicle claim sample, the DojoClient.E2E.Tests -> Components.AI.E2E.Tests rename, and the ClaimApp scenario tests move to a follow-up change stacked on top of this branch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

dariatiurina · 2026-09-15T09:48:17Z

I divided PR into two: multimodal support and ClaimApp sample app. I couldn't do stack PR, so ClaimApp PR a bit ugly here: #69308

kotlarmilos marked this pull request as ready for review August 21, 2026 08:00

kotlarmilos requested a review from a team as a code owner August 21, 2026 08:00

javiercn force-pushed the javiercn-components-ai-09-predictive-state branch from dd8b97e to 489a976 Compare August 28, 2026 21:22

dariatiurina deleted the branch dotnet:main September 2, 2026 12:34

dariatiurina closed this Sep 2, 2026

kotlarmilos added 4 commits September 2, 2026 17:08

feat(ai): add multimodal message input components

7342755

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a

feat(ai): add multimodal claim assistant sample

2c88ea1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a

test(ai): cover multimodal claim workflows

3b5f4ec

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a

refactor(claimapp): simplify Azure OpenAI configuration

11410ef

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a

kotlarmilos reopened this Sep 2, 2026

kotlarmilos changed the base branch from javiercn-components-ai-09-predictive-state to main September 2, 2026 15:14

kotlarmilos force-pushed the milosk/components-ai-10-production-app branch from b67462b to 11410ef Compare September 2, 2026 15:14

kotlarmilos requested review from a team, BrennanConroy, SamMonoRT, Youssef1313, cincuranet, davpetr, halter73, rokonec and wtgodbe as code owners September 2, 2026 15:14

kotlarmilos removed request for a team, BrennanConroy, Youssef1313, cincuranet, halter73, rokonec and wtgodbe September 2, 2026 15:21