Recover malformed Engine tool calls

[Baiju Meswani (baijumeswani)]

This change prevents malformed native Qwen tool-call output from being returned as a successful-looking assistant
response when Foundry Local uses the Engine backend.

Automatic tool choice remains natural and unconstrained on the first generation attempt. If a qualified Qwen model
produces a narrowly recognized malformed native tool call before any semantic output, Foundry Local discards that
attempt, closes its Engine Request, and retries once on a fresh Engine Request using the existing tool-only guidance.

The retry is buffered and accepted only when it produces a complete, schema-valid structured tool-call batch. If
recovery is unavailable or fails, the logical request fails through the existing internal/inference error path and no
assistant turn is committed.

This PR is stacked on baijumeswani/qwen-auto-tools / #1104.

Motivation

A qualified Qwen model can occasionally produce output such as:

<tool_call>
<function=rename_session>
<title>Trapping Rain Water Solution </parameter>
</function>
</tool_call>

The output clearly attempts to invoke a declared tool, but its parameter framing is malformed. Before this change,
the strict native decoder correctly refused to execute it, but the rejected XML became visible assistant text. If the
model then emitted EOS, the request ended with a normal stop result, leaving the coding harness with neither a
usable response nor an executable tool call.

The recovery policy separates model-protocol failure from ordinary assistant text without constraining normal
automatic generation.

Flow

Each generation remains speculative until Foundry Local accepts and commits it:

Committed transcript + current user input
                |
                v
   Attempt 1: natural automatic generation
                |
        +-------+------------------+
        |                          |
  normal response or         qualified malformed
  valid native tool call     native tool-call attempt
        |                          |
        v                          v
 publish and commit       recovery eligibility check
                                   |
                     +-------------+-------------+
                     |                           |
              not eligible                  eligible
          (semantic output, stop,        (Engine, auto,
           limit, cancellation,           natural end,
           unsupported schema, etc.)      no semantic output)
                     |                           |
                     v                           v
           existing inference error       checked close of
           and no transcript commit        first Engine Request
                                                |
                                                v
                                     fresh Engine Request
                                     with tool-only guidance
                                                |
                                      +---------+---------+
                                      |                   |
                                valid strict call     invalid retry
                                      |                   |
                                      v                   v
                               publish and commit    existing inference
                               accepted attempt      error; commit nothing

Implementation

Engine-only malformed classification

The native Qwen decoder can now distinguish a recovery-eligible structural failure from an ordinary rejected
candidate.

Recovery eligibility is deliberately narrow:

• exact qualified Qwen tool-call framing was entered;
• the function name resolves to a declared tool;
• the tool has a schema supported by the native decoder;
• the failure is structural or framing-related.

Undeclared tools, unsupported schemas, unknown or duplicate parameters, missing required parameters,
schema-incompatible values, fenced examples, reserved nested framing, oversized candidates, and mixed valid/invalid
batches retain the existing rejected-visible fallback behavior.

The recovery-aware parser mode is selected only for automatic tool calls on ChatBackendKind::kEngine. The classic
Generator path retains its previous byte-for-byte behavior and never retries.

No semantic-output rollback

Foundry Local retries only when no semantic generated output preceded the malformed candidate. Semantic output
includes visible text, reasoning, raw tool output, or a parsed structured call.

This keeps streaming behavior honest: output that has already been exposed is never retracted. If a safe prefix was
streamed before a malformed candidate, the malformed bytes remain suppressed, the stream terminates through the
existing error path, and no transcript turn is committed.

Fresh checked Engine Request

The malformed attempt is never committed to the canonical transcript. Its generated tokens do exist in the first
Engine Request, so that Request cannot be continued safely.

Recovery therefore:

1. explicitly and synchronously closes the completed first Engine Request;
2. reuses the already captured transcript, current input, tool snapshot, options, system prefix, and prepared prompt;
3. creates a fresh Engine Request;
4. changes only the internal output policy to tool-only guidance;
5. performs at most one retry.

The retry does not use Engine rewind, classic Generator rewind, a hidden corrective message, or automatic prefix
caching. Prefix reuse can remain a future performance optimization; it is not required for correctness.

Strict, schema-aware retry

The guided retry is fully buffered before publication. It must produce:

• a complete canonical structured-tool envelope;
• one or more advertised tool calls;
• exact canonical fields;
• parameter objects that satisfy the effective declared schema;
• an atomically valid complete batch.

The retry path does not accept legacy parser repairs such as missing end markers, missing braces, alternate argument
aliases, singleton shorthand, unknown properties, missing required values, or incompatible value types.

If the retry is canceled, truncated, malformed, empty, text-only, reasoning-only, or otherwise invalid, Foundry Local
publishes no retry output and commits no turn.

Existing API and usage contracts

• No public C API, C++ API, provider option, error enum, or model-dialect API is added.
• No ONNX Runtime GenAI change is required.
• Existing SDK, Chat Completions, and Responses error translation is reused.
• Public usage and finish reason describe the accepted retry attempt.
• Classic Generator behavior is unchanged.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 16, 2026 4:40am UTC

[Copilot]

🟡 Changes recommended

Strict retry validation can accept duplicate JSON fields and inconsistently handles nested anyOf schemas.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds Engine-only recovery for malformed Qwen native tool calls while preserving Generator behavior.

Changes:

• Classifies narrowly recoverable malformed tool output.
• Retries once with strict tool-only guidance on a fresh Engine request.
• Adds recovery, cancellation, streaming, usage, and fallback tests.

File summaries

File	Description
web_service_test.cc	Extends failure-envelope assertions.
tool_call_stream_accumulator_test.cc	Tests malformed-output classification.
chat_session_test.cc	Covers Engine recovery behavior.
tool_call_stream_accumulator.h	Propagates malformed status.
tool_call_payload_parser.h	Adds malformed disposition.
qwen_xml_tool_call_decoder.h	Exposes recovery-aware parsers.
qwen_xml_tool_call_decoder.cc	Implements classification and strict retry parsing.
onnx_engine_chat_stream.h	Adds explicit close support.
onnx_engine_chat_stream.cc	Implements checked, idempotent closing.
chat_session.cc	Implements one-shot guided recovery.
chat_generator.h	Adds the generator close contract.
chat_generator.cc	Provides the default no-op close.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.