Skip to content

Fix shell tool truncating real output that fit within max_output_length - #5027

Open
arpankernel wants to merge 1 commit into
openai:mainfrom
arpankernel:fix-shell-output-truncation-counts-command-decoration
Open

arpankernel wants to merge 1 commit into
openai:mainfrom
arpankernel:fix-shell-output-truncation-counts-command-decoration

Conversation

@arpankernel

Copy link
Copy Markdown

Summary

ShellAction.execute truncates shell output twice against the same budget, so a command whose output already fits within max_output_length can still have real output chopped — or lost entirely.

In src/agents/run_internal/tool_actions.py (the ShellResult branch):

if max_output_length is not None:
    normalized = truncate_shell_outputs(normalized, max_output_length)  # bounds the stdout/stderr payload
output_text = render_shell_outputs(normalized)                          # adds "$ <command>", "stderr:", "exit code:" decoration
if max_output_length is not None:
    output_text = output_text[:max_output_length]                       # re-clamps the DECORATED text to the same budget

truncate_shell_outputs already bounds the combined stdout+stderr to max_output_length (its docstring: "Truncate shell output streams to a maximum combined length"). render_shell_outputs then prepends the $ <command> line and other framing, so the second output_text[:max_output_length] charges that framing against the same budget and cuts real output that already fit. With a command set and a small budget, the entire payload is dropped and the model sees only the $ <command> prefix.

Fix

max_output_length bounds the output streams, not the human-readable framing, so drop the redundant second clamp for positive budgets. The only load-bearing case for it was max_output_length == 0, where render_shell_outputs substitutes a (no output) placeholder for an empty command; that is preserved by collapsing to an empty string (matching the prior behavior).

Test plan

  • Added test_shell_tool_max_output_length_does_not_count_command_decoration: a command (echo hi) with stdout="0123456789" and max_output_length=6 now yields "$ echo hi\n012345" (stdout bounded to the 6-char budget, decoration not counted). Before the fix this test produces "$ echo" — the whole real stdout is lost.
  • tests/test_shell_tool.py, tests/test_shell_call_serialization.py pass (46), including the existing zero/negative max_output_length cases.
  • ruff check, ruff format --check, and mypy are clean on the changed files.

Issue number if applicable

None — found by inspection; no existing issue.

Checks

  • I added new tests to account for change in behavior
  • I ran make lint / make mypy on the changed files
  • I made sure tests pass

`ShellAction.execute` truncated shell output twice against the same budget.
`truncate_shell_outputs(normalized, max_output_length)` already bounds the
combined stdout+stderr payload to the budget. `render_shell_outputs` then adds
decoration (the `$ <command>` prefix, `stderr:`, `exit code:` lines, and blank
separators), and the code re-clamped that decorated string with
`output_text[:max_output_length]` — charging the decoration against the same
budget and cutting real output that already fit. With a command set and a small
budget the entire payload could be lost, leaving only the `$ <command>` prefix.

`max_output_length` bounds the output streams (per `truncate_shell_outputs`),
not the human-readable framing, so drop the second clamp for positive budgets.
The only load-bearing case for it was `max_output_length == 0`, where
`render_shell_outputs` emits a "(no output)" placeholder; that is preserved by
collapsing to an empty string. Add a regression test.
@arpankernel

Copy link
Copy Markdown
Author

One design note for reviewers, since this touches the meaning of max_output_length:

This PR treats max_output_length as a bound on the output streams (stdout/stderr), which is what truncate_shell_outputs already enforces and what feeds both the rendered output and the structured raw_item["output"]. The $ <command> / stderr: / exit code: decoration is treated as framing that is not charged against that budget. A consequence is that the rendered output string can now exceed max_output_length by the length of that framing.

If you instead intend max_output_length as a hard cap on the rendered text the model sees (framing included), I am happy to switch to reserving room for the decoration (truncate the payload to max_output_length - len(decoration)) so the rendered output stays within the cap while still preserving as much real output as possible. Just let me know which contract you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant