Skip to content

docs: warn when --ubatch-size is too small in --embeddings mode#25841

Open
chinmaykamjith001 wants to merge 1 commit into
ggml-org:masterfrom
chinmaykamjith001:fix-embeddings-ubatch-size
Open

docs: warn when --ubatch-size is too small in --embeddings mode#25841
chinmaykamjith001 wants to merge 1 commit into
ggml-org:masterfrom
chinmaykamjith001:fix-embeddings-ubatch-size

Conversation

@chinmaykamjith001

Copy link
Copy Markdown

Overview

In --embeddings mode with pooling, there was a silent HTTP 500 error due to undocumented --ubatch-size behavior. This happens when input token size exceeds --ubatch-size. This PR aims to document that behavior and include appropriate warnings to users.

In embeddings/pooling configuration, the entire input must fit within a single ubatch of --ubatch-size which is unlike normal inference where the input is split into chunks. This is not documented anywhere, leading users to set small values (e.g. 8 or 512) and get unexplained 500 errors

Changes:

  1. common/arg.cpp: server-specific note to --ubatch-size --help to explain this limitation of embeddings mode
  2. tools/server/server.cpp: Display startup warning when --embeddings mode is active with pooling enabled. Warns users to size --ubatch-size appropriately for their use case.
  3. tools/server/README.md: Added [!IMPORTANT] callouts in both /embeddings and /reranking endpoint docs.

Additional information

Fixes #25293
Related: #6263 (fixed the crash but left the behavior undocumented)

Requirements

  • I have read and agree with the contributing guidelines
    -YES

  • AI usage disclosure: Yes, AI was used in final verification process of whether my code contained any apparent errosr/unintended effects.

@chinmaykamjith001
chinmaykamjith001 requested review from a team as code owners July 17, 2026 18:29
@github-actions github-actions Bot added documentation Improvements or additions to documentation server labels Jul 17, 2026
@chinmaykamjith001 chinmaykamjith001 changed the title server: warn when --ubatch-size is too small in --embeddings mode docs: warn when --ubatch-size is too small in --embeddings mode Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation server

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docs] --ubatch-size behavior in --embeddings mode (pooling) is undocumented — causes silent 500 errors

1 participant