Skip to content

feat: add configurable voice transcription - #6860

Open
LauraGPT wants to merge 1 commit into
ChatGPTNextWeb:mainfrom
LauraGPT:codex/funasr-transcription-endpoint-6803
Open

LauraGPT wants to merge 1 commit into
ChatGPTNextWeb:mainfrom
LauraGPT:codex/funasr-transcription-endpoint-6803

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Jul 23, 2026

Copy link
Copy Markdown

Summary

  • Adds opt-in voice transcription using a configurable OpenAI-compatible multipart /v1/audio/transcriptions endpoint, model name, and optional bearer token.
  • Web requests use NextChat's existing same-origin proxy; desktop requests target the configured endpoint directly.
  • Transcripts are appended to the current draft without automatically sending a chat message.
  • Includes English and Simplified Chinese settings, configuration coverage, and mocked recorder/request tests.
  • Defaults to http://localhost:8000/v1 and model sensevoice, without bundling a speech runtime. Endpoint availability, accepted recording formats, and model-name support are deployment requirements, not verified interoperability with every compatible server.

Closes #6803.

Outstanding Issues

This PR is not ready to claim complete recorder cleanup or the server-held credential boundary requested in #6803.

Recorder Lifecycle

On published head 4fd2ab76508367bba254b5dda307fe36f0538264, startAudioRecording() registers onerror and onstop only when the caller invokes stop(). If the recorder errors or stops before that call, its acquired stream tracks are not released by this utility. A subsequent stop() rejects with Audio recording is not active before releasing them.

A deterministic, device-free probe invoked the published utility with injected recorder/stream doubles, then delivered terminal events before calling stop(). Both the error and spontaneous-stop cases observed zero track releases instead of one. This is a reproduced utility-level cleanup failure, not a claim of real-browser/device testing. A bounded lifecycle correction and regression tests are pending; no source fix has been pushed.

Credential and Deployment Boundary

The same-origin proxy removes a browser-to-service CORS requirement for the web path; it does not make configured credentials server-only. The browser sends the endpoint in x-base-url and the optional token in Authorization. The token resides in the client access store, whose persistence uses IndexedDB. It is not hidden from the user's browser. The existing proxy also has provider-specific authorization behavior, so universal token-forwarding compatibility is not established by this PR's mocked tests.

For web deployments, localhost refers to the NextChat server/container's network context, not the browser user's computer. The service must be reachable from that server. The desktop path uses its own device's network context. Microphone access still depends on browser/device support and permissions.

The server-held credential requirement in #6803 remains unmet by the current client-configured implementation. Do not treat this description as relaxing that requirement or as a completed security review.

Verification

Fresh Checks: 2026-09-22

Published head is unchanged. Checks used the existing checkout dependencies, Node 24.18.0, and repository-pinned Yarn 1.22.19:

  • Native yarn test:ci --runInBand --runTestsByPath for the four transcription suites: 4 suites, 19 tests passed. Existing IndexedDB hydration warnings were printed.
  • Additional recorder terminal-event probe: failed both cases described above. Existing passing tests do not cover those cases.
  • Tracked files and index remain unchanged. No source edits, commit, push, deployment, or CI bypass in this pass.
  • Current GitHub head readback returned no check runs and no Actions runs. That is not a passing hosted-CI result.

No fresh full-suite, TypeScript, lint, build, browser microphone, or real transcription-service validation was performed this pass.

Historical Checks Only

Earlier comments reported 38 suites / 176 tests, TypeScript validation, targeted ESLint, and production build success on this head. They also described an unused-imports/no-unused-imports lint-rule crash in unchanged app/constant.ts. Those are historical results, not rerun or reconfirmed by the current focused checks.

This PR was developed with AI coding agents; the present pass is a self-review, not independent maintainer approval.

@LauraGPT

LauraGPT commented Jul 26, 2026

Copy link
Copy Markdown
Author

Rebased onto current main (defdcdb5) and updated the implementation at signed+DCO head 4fd2ab76508367bba254b5dda307fe36f0538264.

The web path now uses NextChat’s same-origin proxy instead of requiring browser CORS from the transcription server; desktop builds continue to call local endpoints directly. Blank model values are rejected before upload, and the settings copy now follows the existing English/Simplified Chinese locale structure.

Exact-head validation:

  • 38/38 Jest suites, 176/176 tests
  • yarn tsc --noEmit
  • targeted ESLint: 0 errors (4 existing warnings in unchanged portions of chat.tsx)
  • production yarn build

The only full-lint limitation remains the repository tooling crash in unchanged app/constant.ts:1 (unused-imports/no-unused-imports reading loc); the build completes with that warning.

@codeCraft-Ritik codeCraft-Ritik left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really solid work. I appreciate that the optimization focuses on eliminating unnecessary allocations while maintaining the same parser behavior. The additional regression tests around slicing, concatenation, and lazy evaluation make this a well-rounded change. 👏 @LauraGPT

@LauraGPT
LauraGPT force-pushed the codex/funasr-transcription-endpoint-6803 branch from 46f6736 to 640c0b8 Compare August 17, 2026 04:05
Signed-off-by: zhifu gao <18321252+LauraGPT@users.noreply.github.com>
@LauraGPT
LauraGPT force-pushed the codex/funasr-transcription-endpoint-6803 branch from 640c0b8 to 4fd2ab7 Compare August 17, 2026 04:07
@LauraGPT

Copy link
Copy Markdown
Author

@Leizhenpeng, could you review this configurable voice-transcription integration when convenient? It addresses #6803 without bundling a speech runtime: web deployments use the existing same-origin proxy, desktop builds can call local OpenAI-compatible /v1/audio/transcriptions endpoints directly, and the default behavior remains unchanged.

I revalidated the unchanged signed+DCO head 4fd2ab7 against the current main@defdcdb55d850cd12c4c657eb83729fd66e215c0 on 2026-08-30:

  • 38/38 Jest suites, 176/176 tests
  • yarn tsc --noEmit
  • git diff --check
  • PR is mergeable with no unresolved review threads

I tried GitHub’s formal reviewer-request API first, but fork authors do not have RequestReviewsByLogin permission. I will respond promptly to any maintainer feedback and will not repeat the ping.

@LauraGPT

LauraGPT commented Sep 2, 2026

Copy link
Copy Markdown
Author

Revalidated the current PR head locally against the current main ancestry:

  • yarn test:ci --runInBand for all four transcription suites: 4 suites, 19 tests passed.
  • yarn build: completed successfully, including Next.js type checking and production output generation.

The only lint diagnostic is an existing unused-imports/no-unused-imports rule crash while linting untouched app/constant.ts; the same diagnostic is surfaced non-fatally by next build. Correction (2026-09-22): the web request uses a same-origin proxy, but the browser sends the configured endpoint in the x-base-url header and the optional token in Authorization. The token is held in the client's persisted access store, not exclusively on the server. The prior statement that the endpoint is absent from client requests was inaccurate. Desktop calls are direct. The test/build results above are historical; the current PR body records fresh focused results and an unresolved recorder-cleanup failure.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add FunASR/SenseVoice as a local STT engine

2 participants