Skip to content

thinking_budget: none is silently ignored on OpenAI-compatible endpoints, and provider_opts cannot pass the vendor off switch #4309

Description

@IsmaelMartinez

Open-weight reasoning models (Qwen3, DeepSeek, GLM) think by default, and every OpenAI-compatible server or vendor exposes a request field to turn it off: llama.cpp, vLLM, SGLang and mlx_lm accept chat_template_kwargs: {"enable_thinking": false}; llama.cpp, vLLM, SGLang, Ollama, Groq and Cerebras accept reasoning_effort: "none"; DeepSeek uses thinking: {"type": "disabled"}. docker agent can send none of them: thinking_budget is only acted on for OpenAI reasoning model names (modelinfo.UsesReasoningEffort), a disabled budget is normalised to nil in applyModelDefaults before any client sees it (which also leaves the documented DMR reasoning-budget: 0 path dead), and provider_opts forwarding is limited to the sampling allowlist. The model therefore reasons on every call and max_tokens can be spent entirely on reasoning.

Measured with v1.139.0 on macOS against mlx_lm.server 0.31.3 serving mlx-community/Qwen3.6-35B-A3B-8bit:

models:
  local:
    provider: openai
    model: mlx-community/Qwen3.6-35B-A3B-8bit
    base_url: http://localhost:8080/v1
    temperature: 0

docker agent run --exec --json agent.yaml - with a 7 kB prompt took 80–200 s per call and emitted 4.5k–13k agent_choice_reasoning events; the same prompt with chat_template_kwargs sent directly returns in about 3 s. thinking_budget: none, provider_opts: {chat_template_kwargs: {enable_thinking: false}} and a /no_think prefix all changed nothing (209–231 reasoning events on a three-word probe); a proxy confirms the body carries only model, messages, stream, temperature. With max_tokens: 2048 the whole budget went to reasoning: zero agent_choice events, exit 0, a warning event, no error. provider: dmr with runtime_flags: ["--reasoning-budget", "0"] on qwen3.6:35b-a3b-q8_0 still produced 218 reasoning events (whether the flag reached llama.cpp is unconfirmed; a failed _configure is only logged at debug level).

Proposed:

  1. provider_opts.extra_body (object) merged verbatim into the chat-completions body for the openai and dmr clients, on every provider, as the general escape hatch (extra_body: {reasoning_effort: none} on groq, extra_body: {thinking: {type: disabled}} on deepseek, ...).
  2. thinking_budget: none / 0 on a user-supplied base_url (not Azure/ChatGPT, model name not an OpenAI one) sends chat_template_kwargs: {"enable_thinking": false} on Chat Completions with the existing 256-token max_tokens floor; the dmr client sends the same per request alongside llamacpp.reasoning-budget, which also covers its MLX and SGLang engines.
  3. dmr: internal no-thinking clones (MCP sampling) no longer call the model-wide _configure. Model Runner replaces a model's stored configuration on every call, so a sampling call already wiped the agent's reasoning-budget (verified: 4096 → {} in docker model configure show); with the disabled budget preserved it would set it to 0 instead. Title and compaction clones already skipped it.

Follow-up, not in scope: a per-alias off switch in the alias registry so thinking_budget: none works on built-in aliases such as ollama (which honours reasoning_effort: none) without extra_body.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/docker-model-runnerDocker Model Runner (DMR) local inferencearea/providers/openaiFor features/issues/fixes related to the usage of OpenAI models

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions