Keep the signal. Stop rereading the same failures.
Token-Saver compresses the terminal output your AI coding assistant reads —
git diff, pytest, npm install, terraform plan, kubectl, docker — to
leave more room for the code, instructions, and next decision. Compression runs
locally, without another model call or an output upload.
New in v3: Delta. Run your tests, edit, run again: see which failures are new, changed, or unchanged. Fresh diagnostics keep their full details; repeated ones can become a short inventory, with details available on demand. Delta is an experimental, opt-in feature in 3.0.0 for supported pytest and Ruff commands in Claude Code.
36 specialized processors cover git, pytest, jest, cargo, go, docker, kubernetes, terraform, pulumi, helm, ansible, aws, gcloud, and more. Their rules reduce progress logs and boilerplate, with failure fixtures and optional quality contracts checking important diagnostics.
Ordinary compression supports Claude Code and Antigravity CLI. It runs automatically after installation. Results depend on the command and output; already concise or unsupported output may stay unchanged.
Why developers use Token-Saver:
- 💸 Less output to process — many benchmark scenarios save 60-99% of estimated output tokens; billing impact depends on your usage and provider.
- 🔁 See what changed — opt-in Delta compares repeated pytest/Ruff runs and keeps retained details one command away.
- 🪟 Bigger effective context — fit more real work into the same context window.
- 🎯 Tested preservation — failure fixtures check important diagnostics; add quality contracts for your own workflows.
- 🔌 Install once, forget it — works automatically in the background, no prompts to change.
- 🛡️ Local compression — pure regex/parsing with no network calls in the compression path.
- Before & After
- Quick Start
- Why Token-Saver Exists
- Who It's For
- What Gets Compressed — Real Examples
- How It Compares
- How It Works
- Precision Guarantees
- Installation
- CLI Reference
- Delta: Changes Between Runs
- Compression Quality Gates
- Processors
- Configuration
- Tuning Recipes
- Custom Processors
- Savings Tracking
- Security & Privacy
- Performance
- FAQ
- Troubleshooting
- Known Limitations
- Project Structure
- Tests
- Contributing
| Command | Raw Output | Compressed | Savings |
|---|---|---|---|
git diff (5 files, 20 context lines each) |
2,270 tokens | 546 tokens | 76% |
git diff --stat (25 files) |
526 tokens | 21 tokens | 96% |
git log --oneline (50 entries) |
568 tokens | 117 tokens | 79% |
pytest (500 tests, 2 failures) |
6,744 tokens | 307 tokens | 95% |
pytest (200 passed, 30 warnings) |
3,543 tokens | 80 tokens | 98% |
jest (50 suites, all passing) |
570 tokens | 32 tokens | 94% |
npm install (220 packages) |
3,843 tokens | 4 tokens | 99.9% |
pip install -r requirements.txt (30 packages) |
1,797 tokens | 88 tokens | 95% |
cargo build (120 crates) |
934 tokens | 21 tokens | 98% |
docker build (20 steps) |
1,682 tokens | 207 tokens | 88% |
curl download (100 progress lines) |
2,122 tokens | 0 tokens | 100% |
ruff check (110 violations, 4 rules) |
1,475 tokens | 186 tokens | 87% |
eslint (55 violations, 2 rules) |
1,292 tokens | 144 tokens | 89% |
tree (350+ lines) |
1,840 tokens | 136 tokens | 93% |
find . -name '*.py' (205 results) |
1,294 tokens | 217 tokens | 75% |
These are locked to tests/compression_baselines.json and gated by CI (tests/test_compression_ratchet.py) — if a change makes any of them compress worse, the build fails. Run python3 scripts/audit_compression.py to see the full scenario set, or token-saver benchmark '<command>' to measure savings on your own workloads.
Not every command compresses, and that's deliberate. cat large_file.py, git status -s, and tsc with 7 type errors all sit in the same baseline file at 0% — they're already dense, or the model needs them verbatim. Token-Saver returns the original bytes rather than shaving a few tokens at the cost of correctness.
/plugin marketplace add ppgranger/token-saver
/plugin install token-saver
(Also available on Anthropic's official community marketplace — see Installation for both options.)
That's the whole setup. Ask Claude to run anything — "run the tests", "show me the diff", "what changed in the last 10 commits" — and the output is compressed before it reaches the model. Then:
token-saver statsWant to see it work before installing it?
git clone https://github.com/ppgranger/token-saver.git && cd token-saver
python3 bin/token-saver benchmark 'git log --oneline -50'
python3 bin/token-saver benchmark 'git diff' --show-removed
python3 bin/token-saver explain 'docker compose logs | grep error'Verbose tool output competes with your code and instructions for context. A
pytest run with 200 passing tests or an install with hundreds of progress
lines often repeats information that can be represented much more compactly.
After an edit, another run can repeat the same failures too.
Token-Saver reduces that repetition using local parsing. Delta extends this to consecutive runs, while keeping a current diagnostic inventory and retrievable details. Whether this improves task completion, response time, or billing needs measurement in the agent workflow that uses it.
There are three common ways to attack this:
- Summarize with another LLM. Accurate-ish, but it costs a second inference per command, adds seconds of latency, and is non-deterministic — the same
pytestrun can be summarized two different ways. - Truncate blindly. Free and instant, but it's exactly how you lose the one stack-trace line that mattered.
- Parse the format you already know.
git diffhas a grammar.pytesthas a summary line.npm installhas a progress phase and a result phase. Format-aware parsing can remove repetitive output while retaining tested diagnostics, deterministically and without another model call.
Token-Saver is the third approach, applied to 36 command families. It sits between the CLI and your AI assistant, applying format-specific rules whose preservation behavior you can test against your own captured output.
- Developers paying per token on Claude Code, whether via API credits or a plan with usage limits. Tool output is often the largest single category of input tokens in an agentic session.
- Teams running agents in CI, where a long pipeline command (
terraform plan,pytest,gradle build) is piped straight into a model. - Anyone hitting context limits mid-task — the "conversation is too long" wall usually arrives right after a big
npm installor a widegit diff. - Monorepo and infrastructure work, where a single
terraform planorkubectl describeis thousands of tokens of mostly-unchanged resource attributes. - Privacy-sensitive environments that can't route output through a second model or a hosted service. Token-Saver is pure local Python with no network calls in the compression path.
It's probably not for you if your agent mostly reads and writes source files and rarely runs shell commands — file reads go through Claude Code's own Read tool, which Token-Saver doesn't intercept.
Everything below is actual Token-Saver output, produced by the same scenario corpus the CI ratchet uses.
Passing tests collapse to a count. Every failure keeps its full traceback, assertion diff, file, and line number.
[500 tests passed]
=================================== FAILURES ===================================
__________________________ test_compression_ratio ______________________________
def test_compression_ratio():
engine = CompressionEngine()
result = engine.compress('git status', large_output)
> assert len(result[0]) < len(large_output) * 0.5
E AssertionError: assert 1500 < 1000
...
FAILED tests/test_engine.py::test_compression_ratio - AssertionError: assert 1500 < 1000
FAILED tests/test_processors.py::test_diff_context_trim - AssertionError: assert 42 < 10
========================= 2 failed, 510 passed ==
Violations are grouped by rule with a count and a couple of concrete examples, so the model learns the pattern to fix instead of reading 45 near-identical lines.
110 issues across 4 rules:
E501: 45 occurrences in 45 files
src/module/e501_00.py:10:5: E501 Line too long
src/module/e501_01.py:11:6: E501 Line too long
... (43 more)
F401: 30 occurrences in 30 files
src/module/f401_00.py:10:5: F401 imported but unused
... (28 more)
Found 110 errors.
Build succeeded.
Nothing actionable was lost: the raw output was 220 added <pkg>@<ver> lines and a progress bar. Had the install failed, the error and the failing package would survive verbatim — see Precision Guarantees.
[121 crates compiled]
Finished dev [unoptimized + debuginfo] target(s) in 45.23s
Unified diffs keep every +/- line and every @@ hunk header, with surrounding unchanged context trimmed to max_diff_context_lines (default 3):
diff --git a/src/module_0.py b/src/module_0.py
@@ -10,30 +10,32 @@ def some_function_0():
# This is context line 9 that hasn't changed and takes up tokens
- old_value = compute_something(0)
- return old_value
+ new_value = compute_something_better(0)
+ cached = cache.get(new_value)
+ if cached:
+ return cached
+ return new_value
# This is trailing context line 0 that is unchanged and wastes tokens
The directory shape is preserved down to the truncation point, with an explicit marker and the original totals, so the model knows what it isn't seeing:
.
├── src/
│ ├── engine.py
│ ├── config.py
│ ├── processors/
│ │ ├── git.py
...
... (287 lines truncated)
20 directories, 310 files
Variables whose names look sensitive (SECRET, PASSWORD, CREDENTIAL, API_KEY, GITHUB_TOKEN, and bare KEY/TOKEN/AUTH/PWD at letter boundaries — so PATH, AUTHOR, and MONKEY are left alone) have their values replaced. Token-Saver returns this redacted output even when it isn't smaller than the input: a redacted result is never traded back for the raw one to satisfy a compression threshold.
| Token-Saver | LLM summarizer | Blind truncation | Response caching | |
|---|---|---|---|---|
| Extra inference cost | None | One call per command | None | None |
| Additional work | Python startup + parsing | Another model call | Truncation | Cache lookup |
| Deterministic | Yes | No | Yes | Yes |
| Diagnostic preservation | Tested fixtures + configurable quality contracts | Depends on prompt/model | No semantic checks | Depends on cached content |
| Works offline | Yes | Needs a model | Yes | Usually not |
Understands git diff |
Yes | Sort of | No | N/A |
For a sourced comparison with RTK and Context Mode, including integration caveats, see the full comparison.
Built around explicit SOLID boundaries. Processors parse output, a shared policy decides command eligibility, adapters execute commands, and dedicated modules own storage and presentation. Delta comparison and statistics formatting are pure functions; engine dependencies and telemetry writers can be supplied independently. The architecture guide documents the contracts, compatibility guarantees, tests, and remaining couplings.
CLI command --> Specialized processor --> Compressed output
|
36 processors
(git, test, cargo, go, build,
lint, package_list, python_install,
maven_gradle, bun, network, docker,
kubectl, terraform, pulumi, cdktf,
nix, mise, env, search, system_info,
gh, db_query, cloud_cli, ansible,
helm, syslog, ssh, jq_yq, just, act,
structured_log, file_listing,
file_content, generic)
The engine (CompressionEngine) maintains a priority-ordered chain of processors.
The first processor that can handle the command (can_handle()) produces the
compressed output. GenericProcessor serves as a fallback and always matches last.
When a specialized processor doesn't achieve the minimum compression ratio, the engine tries the generic processor as a fallback before returning uncompressed output.
After the specialized processor runs, a lightweight cleanup pass (clean())
strips residual ANSI codes and collapses consecutive blank lines.
- Eligibility. The hook first decides whether the command is safe to wrap at all — see Security & Privacy. Redirections,
sudo, editors, unquoted newlines, background&, and shell-construct fragments are all left alone. - Size gates. Output shorter than
min_input_lengthis returned untouched. Output longer thanmax_output_bytes(default 10 MB) is hard-capped with an explicit note before compression, so a pathological command can't be held in memory in full. - Routing. Processors are tried in priority order; the first
can_handle()match wins. - Failure routing. If the command exited non-zero and the matched processor hasn't opted into
handles_failure, the engine deliberately routes to the generic processor instead. A specialized processor's heuristics are tuned for the success shape of a tool's output; failure text often isn't recognized, and an unrecognized line is exactly what a loop drops silently. - Chaining. A processor may declare
chain_toto hand its result to a second one, bounded bymax_chain_depth. - Cleanup. ANSI escape stripping and blank-line collapsing.
- Critical-line recovery. The engine diffs the compressed output against the original for error-shaped lines that vanished, and re-appends up to
recover_critical_linesof them under an explicit[token-saver] N error line(s) recoveredmarker. This is a backstop covering all 36 processors — and every future one — rather than an audit that has to be repeated per processor. - Ratio gate. If the result isn't meaningfully smaller, the original is returned. Exception: if the processor redacted secrets, the redacted output always wins, regardless of size.
- Tracking. Sizes — never content — go to a local SQLite database.
The two platforms use different mechanisms:
Claude Code (PreToolUse hook):
1. Claude wants to run `git status`
2. PreToolUse hook intercepts the command
3. Rewrites to: python3 wrap.py 'git status'
4. wrap.py executes the original command
5. Compresses the output
6. Claude receives the compressed version
Claude Code's PreToolUse hook cannot modify output after execution. The only way to reduce tokens is to rewrite the command to go through a wrapper that executes, compresses, and returns the result.
Because that rewrite is a source-to-source shell transformation, the rewritten
command is syntax-checked (sh -n -c) before it runs. If the rewrite wouldn't
parse, the original command runs uncompressed — a missed saving, never a
broken command.
Antigravity CLI (AfterTool hook):
1. Antigravity executes the command
2. AfterTool hook receives the raw output
3. Compresses the output
4. Replaces it via {"decision": "deny", "reason": "<compressed output>"}
Antigravity CLI allows direct output replacement through the deny/reason mechanism.
git add -A && git commit -m wip && git push is a single Bash tool call, but three
commands with three different output shapes. Token-Saver splits the chain, validates
each segment independently, injects unique boundary markers, and compresses each
segment with its own processor — so the git push output is handled by the git
processor even though it shares a shell with two other commands. If any segment is
ineligible, the whole chain runs untouched.
Compression is aggressive on noise, conservative on signal:
- Tiny outputs are never touched, and a result that doesn't actually get smaller is discarded in favor of the original (both thresholds are configurable — see Configuration)
- Failure fixtures verify specific errors, tracebacks and changed lines; grouping and bounded recovery can omit other details
- Source code files (
cat *.py,cat *.ts, ...) pass through unchanged — the model needs exact content - Secrets in
.env.production,.env.local, and other.env.*variants are automatically redacted before reaching the model (.env,.env.example, and.env.templatepass through unchanged, by design — see the note below) - Processors remove or group progress bars, passing tests, installation logs, ANSI codes and other repetitive content according to their documented rules
- Generic truncation includes an omission marker; format-specific processors can also summarize or omit lines
- 1300+ tests including precision-specific tests that verify every critical piece of data survives compression, and a compression-ratchet suite that fails CI if a real-world scenario's compression ratio regresses
Two of these are enforced structurally rather than by convention:
TestFailureHandlingruns a realistic failing-command fixture through every processor, at every exit code (0,1, and unknown), and asserts the failure reason is still present. A processor that claimshandles_failure = Truehas to earn it; one that doesn't gets routed around.- Critical-line recovery (step 7 above) restores recognized error lines within a configurable cap, except after redaction where restoring raw lines could expose secrets.
These are protections on tested output formats, not a universal lossless guarantee. Use quality gates to assert the exact diagnostics your workflow requires.
Note on
.env:.env,.env.example, and.env.templateare intentionally left untouched (not redacted, not compressed) — these are the files you're most likely actively editing, where exact values matter. Other.env.*variants (.env.production,.env.local, ...), which you're more likely to be reading than editing, get their values redacted. If youcat .envdirectly, treat that output as sensitive the same way you would without Token-Saver installed.
- Python 3.10+
- Claude Code and/or Antigravity CLI
No third-party Python packages are required — Token-Saver uses only the standard library.
Two marketplaces carry Token-Saver — pick one.
Anthropic's official community marketplace (claude-community), no setup beyond adding it once:
/plugin marketplace add anthropics/claude-plugins-community
/plugin install token-saver@claude-community --scope project
This mirror is a snapshot pinned to a specific commit, refreshed periodically rather than on every push — if you want the exact latest commit on main, use the self-hosted marketplace below instead.
Self-hosted marketplace (this repo, always current):
/plugin marketplace add ppgranger/token-saver
/plugin install token-saver
Or test directly from a local clone:
git clone https://github.com/ppgranger/token-saver.git
claude --plugin-dir ./token-savergit clone https://github.com/ppgranger/token-saver.git
cd token-saver
python3 install.py --target claude # Claude Code only
python3 install.py --target antigravity # Antigravity CLI only
python3 install.py --target both # Both platformsThe manual installer registers token-saver as a native Claude Code plugin
(equivalent to /plugin install). It appears in /plugin list and hooks,
skills, and commands are managed natively by Claude Code.
The repo/zip can be deleted after installation. Token-Saver copies everything
it needs to ~/.token-saver/ and the platform plugin directories.
python3 install.py --target claude --link # Symlinks instead of copiesChanges in the source directory are immediately applied. Do not delete the repo in this mode.
python3 install.py --uninstall # Remove from all platforms
python3 install.py --uninstall --keep-data # Keep stats DBPlugin install: Claude Code handles updates automatically when you refresh the marketplace.
Manual install: Run token-saver update from anywhere, or:
cd token-saver && git pull && python3 install.py --target claudeGitHub releases: Both methods check for new releases via the GitHub API. The token-saver update CLI command and the SessionStart hook notification work regardless of install method. The remote lookup is cached for 24h and fails open — a network outage or an API rate limit never blocks a session.
If you previously installed token-saver v1.x:
cd token-saver
git pull
python3 install.py --target claudeThe installer automatically:
- Removes legacy hooks from
~/.claude/settings.json(no longer needed) - Removes the old
~/.claude/plugins/token-saver/directory - Installs to the plugin cache as a native Claude Code plugin
- Registers in
enabledPluginsandinstalled_plugins.json
You can also run token-saver update from anywhere to auto-upgrade.
Do NOT install token-saver via BOTH /plugin install AND python3 install.py
simultaneously — this could register the plugin twice. Use one method or the
other. The same applies across marketplaces: install from either
claude-community or the self-hosted ppgranger/token-saver marketplace,
not both.
To switch from manual to marketplace:
python3 install.py --uninstall --target claude
/plugin marketplace add ppgranger/token-saver
/plugin install token-saver- Copies (or symlinks) files to:
- Core:
~/.token-saver/(CLI, updater, shared source) - Claude Code:
~/.claude/plugins/cache/token-saver-marketplace/token-saver/ - Antigravity CLI:
~/.gemini/antigravity-cli/plugins/token-saver/
- Core:
- Registers as a native Claude Code plugin in
installed_plugins.jsonandenabledPlugins - Installs
token-saverCLI to~/.local/bin/ - Stamps the current version into plugin manifests
- Cleans up any legacy
token-savingor v1.x installation
After installation the token-saver command is available. If ~/.local/bin is not on your PATH, the installer prints instructions.
| Command | What it does |
|---|---|
token-saver version |
Print the installed version |
token-saver stats |
Show session + lifetime savings |
token-saver stats --json |
Same, machine-readable |
token-saver update |
Check GitHub for a new release and apply it |
token-saver benchmark '<cmd>' |
Run a command and report its compression |
token-saver benchmark '<cmd>' --dry-run |
Show which processor would match, without executing |
token-saver benchmark '<cmd>' --show-removed |
Line/byte breakdown of what compression removed |
token-saver benchmark '<cmd>' --stdin |
Compress output piped on stdin instead of executing |
token-saver benchmark '<cmd>' --format json |
Machine-readable benchmark |
token-saver explain '<cmd>' |
Explain how a command is routed — or why it's excluded |
token-saver explain '<cmd>' --format json |
Same, machine-readable |
token-saver compress '<cmd>' |
Read UTF-8 stdin and emit compressed text; never execute the command label |
token-saver compress '<cmd>' --exit-code 1 --max-tokens 1500 |
Apply failure-aware routing and check an estimated output budget |
token-saver compress '<cmd>' --format json |
Compressed output plus measurements and budget verdict |
token-saver replay quality.json |
Check captured fixtures against preservation and budget contracts |
token-saver replay quality.json --format json |
Content-free quality report for CI |
token-saver delta show RUN |
Read a retained, sanitized Delta snapshot without rerunning its command |
token-saver delta show RUN --diagnostic EXACT_ID |
Read full details for one diagnostic in that snapshot |
token-saver delta clear |
Delete all retained Delta snapshots and reset comparisons |
explain is the fastest way to answer "why didn't that get compressed?":
$ token-saver explain 'git status'
Token-Saver Explain
========================================
Command: git status
Compressible: yes
Reason: matched a compressible processor pattern
Processor: git
$ token-saver explain 'cat x.py > out.txt'
Command: cat x.py > out.txt
Compressible: no
Reason: contains output redirection (>, >>, 2>, &>)
Excluded by: output redirection
Processor: file_contentAnd --stdin lets you test compression against output you already have, with no re-execution:
pytest > /tmp/out.txt 2>&1
token-saver benchmark 'pytest' --stdin --show-removed < /tmp/out.txtNew in 3.0.0: spend context on the next change. Delta is experimental and
disabled by default.
Delta compares consecutive pytest and ruff check results in Claude Code so
the agent can see what changed after an edit. Commands still execute normally.
New or changed diagnostics retain their full details; unchanged diagnostics keep
their identity and summary, with a command to retrieve the details locally.
| In your edit–test loop | What Delta reports |
|---|---|
| First supported run | A baseline with full failure details. |
| Same failures again | UNCHANGED with names and summaries when that saves output. |
| An assertion or traceback changes | CHANGED with its full current details. |
| A new regression appears | NEW with its full current details. |
| A previous failure disappears | PASSED only with explicit passing evidence; otherwise NOT OBSERVED. |
| You need the original diagnostic | token-saver delta show RUN --diagnostic EXACT_ID, without rerunning tests. |
One repeated run from the real-command fixture benchmark
returns this instead of 4,079 characters of ordinary compressed output
(RUN replaces the generated snapshot identifier):
[token-saver delta] pytest | exit 1
3 failed, 21 passed in 0.02s
UNCHANGED test_fixture.py::test_payload — AssertionError: payload diagnostic
UNCHANGED test_fixture.py::test_status — AssertionError: status diagnostic
UNCHANGED test_fixture.py::test_cache — AssertionError: cache diagnostic
Details: token-saver delta show RUN [--diagnostic ID]
The complete five-run fixture sequence saves 32.5% of output characters for pytest and 8.3% for Ruff, compared with ordinary compression, including one targeted detail retrieval. These are measured fixture results, not universal savings or billing figures. A complete snapshot read is more expensive: it erases the gain in the Ruff scenario. The guide includes every result and the reproduction command.
Enable it explicitly in your global configuration, or before launching Claude:
export TOKEN_SAVER_DELTA_ENABLED=trueThe comparison stays within the same session, working directory, and exact
command. Each Delta response includes the current exit status, totals, and diagnostic
inventory. A missing failure is NOT OBSERVED, not confirmed fixed; PASSED
requires an explicit passing test result in the current output. Use pytest -v
to expose those per-test results.
Delta is disabled by default. Enabling it retains sanitized output snapshots locally for 24 hours and at most 100 runs by default. It currently applies only to supported single commands through the Claude wrapper, with a host session identifier. Unsupported or ambiguous output uses ordinary compression.
The first run and new or changed failures can produce more output than v2 when preserving full details requires it. For unchanged repeats, Token-Saver uses ordinary sanitized compression whenever Delta would be as large or larger. Ordinary compression can also win for new failures if it retains every full new or changed diagnostic. Valid snapshots still establish comparison baselines. The reproducible Delta scenario compares five synthetic pytest captures, including detail retrieval. It does not measure agent task success or billing savings; the existing savings table above still describes ordinary compression. See the Delta guide for supported formats, retrieval, storage, and limits.
Make compression testable on your workflow: set per-command and total token
budgets, require a minimum measured reduction, and name literal messages that
must survive. replay checks saved fixtures without re-running commands; a typo
in a required message fails against the original fixture too.
python3 bin/token-saver replay examples/quality-replay.json
python3 bin/token-saver compress 'pytest -v' --exit-code 1 --max-tokens 1500 < examples/fixtures/pytest_output.txtAn over-budget result returns status 1 and retains the compressed output;
it never truncates more text just to satisfy a limit. Invalid input returns 2.
Token counts use ceil(characters / chars_per_token) and remain estimates,
not exact tokenizer or billing counts. JSON replay reports omit command strings,
raw output and expected text; compress --format json includes the compressed output.
Neither command stores captured output or savings history.
See the manifest format and CI example. These additions
are available in this checkout; use python3 bin/token-saver to try them before
a release is published.
Each processor handles a family of commands. The first one that matches
(can_handle()) processes the output. Detailed documentation for each
processor is in docs/processors/.
| # | Processor | Priority | Commands | Docs |
|---|---|---|---|---|
| 1 | Package List | 15 | pip list/freeze, npm ls, conda list, gem list, brew list | package_list.md |
| 2 | just | 18 | just --list, just --summary (recipe listing compaction) | just.md |
| 3 | act | 19 | act (run GitHub Actions locally via nektos/act) | act.md |
| 4 | Git | 20 | status, diff, log, show, push/pull/fetch, branch, stash, reflog, blame, cherry-pick, rebase, merge | git.md |
| 5 | Test | 21 | pytest, jest, vitest, mocha, cargo test, go test, rspec, phpunit, bun test, npm/yarn/pnpm test, dotnet test, swift test, mix test | test_output.md |
| 6 | Cargo | 22 | cargo build, check, doc, update, bench | cargo.md |
| 7 | Go | 23 | go build, vet, mod, generate, install | go.md |
| 8 | Python Install | 24 | pip install, poetry install/update/add, uv pip install, uv sync | python_install.md |
| 9 | Build | 25 | npm/yarn/pnpm build/install, cargo build, make, cmake, tsc, webpack, vite, next build, turbo, nx, bazel, sbt, mix compile, docker build | build_output.md |
| 10 | Cargo Clippy | 26 | cargo clippy (multi-line block grouping with span/help preservation) | cargo_clippy.md |
| 11 | Lint | 27 | eslint, ruff, flake8, pylint, clippy, mypy, prettier, biome, shellcheck, hadolint, rubocop, golangci-lint | lint_output.md |
| 12 | Maven/Gradle | 28 | mvn, ./mvnw, gradle, ./gradlew (download stripping, task noise removal) | maven_gradle.md |
| 13 | Bun | 29 | bun install, add, remove, update | bun.md |
| 14 | Network | 30 | curl, wget, http/https (httpie) | network.md |
| 15 | Docker | 31 | ps, images, logs, pull/push, inspect, stats, compose up/down/build/ps/logs | docker.md |
| 16 | Kubernetes | 32 | kubectl/oc get, describe, logs, top, apply, delete, create | kubectl.md |
| 17 | Terraform | 33 | terraform/tofu plan, apply, destroy, init, output, state list/show | terraform.md |
| 18 | Environment | 34 | env, printenv (with secret redaction) | env.md |
| 19 | Search | 35 | grep -r, rg, ag, fd, fdfind | search.md |
| 20 | System Info | 36 | du, wc, df | system_info.md |
| 21 | GitHub CLI | 37 | gh pr/issue/run list/view/diff/checks/status | gh.md |
| 22 | Database Query | 38 | psql, mysql, sqlite3, pgcli, mycli, litecli | db_query.md |
| 23 | Cloud CLI | 39 | aws, gcloud, az (JSON/table/text output compression) | cloud_cli.md |
| 24 | Ansible | 40 | ansible-playbook, ansible (ok/skipped counting, error preservation) | ansible.md |
| 25 | Helm | 41 | helm install/upgrade/list/template/status/history | helm.md |
| 26 | Syslog | 42 | journalctl, dmesg (head/tail with error extraction) | syslog.md |
| 27 | SSH/SCP | 43 | non-interactive ssh, scp (remote command output compression) | ssh.md |
| 28 | JQ/YQ | 44 | jq, yq (large JSON/YAML output compaction) | jq_yq.md |
| 29 | Structured Log | 45 | stern, kubetail (JSON Lines grouping by level) | structured_log.md |
| 30 | Pulumi | 46 | pulumi up, preview, destroy, refresh | pulumi.md |
| 31 | CDKTF | 47 | cdktf deploy, diff, destroy, synth | cdktf.md |
| 32 | Nix | 48 | nix build/develop/eval/run, nix-build, nix-shell | nix.md |
| 33 | mise | 49 | mise install, use, upgrade (runtime version manager) | mise.md |
| 34 | File Listing | 50 | ls, find, tree, exa, eza, rsync | file_listing.md |
| 35 | File Content | 51 | cat, head, tail, bat, less, more (content-aware: code, config, log, CSV) | file_content.md |
| 36 | Generic | 999 | Any command (fallback: ANSI strip, dedup, truncation) | generic.md |
Lower priority numbers run first. The gaps between numbers are intentional — they leave room for custom processors to slot in ahead of, or behind, any built-in.
Thresholds are configurable via JSON file or environment variables. Precedence, lowest to highest:
built-in defaults < ~/.token-saver/config.json < ./.token-saver.json < TOKEN_SAVER_* env vars
~/.token-saver/config.json:
{
"enabled": true,
"min_input_length": 1,
"min_compression_ratio": 0.0,
"max_diff_hunk_lines": 50,
"max_log_entries": 10,
"max_file_lines": 100,
"generic_truncate_threshold": 200,
"debug": false
}Values are coerced to the type of their default, and a value that can't be coerced is ignored rather than propagated — {"max_chain_depth": "deep"} leaves the default in place instead of reaching arithmetic code downstream.
Every key can be overridden with the TOKEN_SAVER_ prefix:
export TOKEN_SAVER_MAX_LOG_ENTRIES=50
export TOKEN_SAVER_DEBUG=true
# Disable compression entirely (bypass mode)
export TOKEN_SAVER_ENABLED=falseList-valued keys take a comma-separated string: TOKEN_SAVER_DISABLED_PROCESSORS=file_content,network.
Drop a .token-saver.json in your repository root to override global settings:
{
"max_diff_hunk_lines": 300,
"generic_truncate_threshold": 1000,
"max_log_entries": 50
}Project settings are merged with global settings. Token-Saver walks up parent directories (like .gitignore resolution) to find the nearest .token-saver.json, stopping at your home directory or the filesystem root. Useful for monorepos or projects with atypical output patterns (large Terraform plans, verbose test suites, etc.).
Security note: project files cannot set user_processors_dir (loads Python
code), disabled_processors or redaction_allowlist (can weaken redaction), or
delta_enabled, delta_retention_hours, and delta_max_runs (control local
output retention). Set these only in global ~/.token-saver/config.json or via
TOKEN_SAVER_* environment variables. A project file that sets them has those
keys dropped with a debug-log line, not coerced. Cloning a repository cannot
opt you into retaining its command output.
This table is generated from src/config.py's _DEFAULTS dict — that file is the
source of truth if the two ever disagree, and
tests/test_readme_config_sync.py fails CI if
they drift apart.
| Parameter | Default | Description |
|---|---|---|
enabled |
true |
Master switch -- set to false to bypass all compression |
min_input_length |
1 |
Minimum threshold (characters) to attempt compression |
min_compression_ratio |
0.0 |
Minimum gain to apply compression (0 = apply any gain) |
wrap_timeout |
300 |
Wrapper timeout in seconds |
max_diff_hunk_lines |
50 |
Max lines per hunk in git diff |
max_diff_context_lines |
3 |
Context lines kept before/after each change in diffs |
max_log_entries |
10 |
Max entries in git log/reflog |
max_file_lines |
100 |
Threshold before file content compression kicks in |
file_keep_head |
80 |
Lines kept from the start of file (fallback strategy) |
file_keep_tail |
30 |
Lines kept from the end of file (fallback strategy) |
generic_truncate_threshold |
200 |
Generic truncation threshold |
generic_keep_head |
100 |
Lines kept from the start (generic) |
generic_keep_tail |
50 |
Lines kept from the end (generic) |
generic_keep_critical |
20 |
Max error/failure lines rescued from the truncated middle (0 disables) |
recover_critical_lines |
20 |
Max error lines the engine re-appends when a processor dropped them (0 disables) |
ls_compact_threshold |
15 |
Items before ls compaction |
find_compact_threshold |
20 |
Results before find compaction |
tree_compact_threshold |
30 |
Lines before tree truncation |
lint_example_count |
2 |
Examples shown per lint rule |
lint_group_threshold |
3 |
Occurrences before grouping by rule |
file_code_head_lines |
15 |
Import/header lines to preserve in code files |
file_code_body_lines |
2 |
Body lines kept per function/class definition |
file_log_context_lines |
2 |
Context lines around errors in log files |
file_csv_head_rows |
3 |
Data rows kept from start of CSV files |
file_csv_tail_rows |
2 |
Data rows kept from end of CSV files |
search_max_per_file |
3 |
Max match lines shown per file |
search_max_files |
15 |
Max files shown in search results |
kubectl_keep_head |
5 |
Lines kept from start of kubectl logs |
kubectl_keep_tail |
10 |
Lines kept from end of kubectl logs |
docker_log_keep_head |
5 |
Lines kept from start of docker logs |
docker_log_keep_tail |
10 |
Lines kept from end of docker logs |
git_branch_threshold |
15 |
Branches before compaction |
git_stash_threshold |
5 |
Stash entries before truncation |
max_traceback_lines |
30 |
Max traceback lines before truncation |
db_max_rows |
20 |
Max rows shown per database query result |
db_prune_days |
90 |
Stats retention in days |
chars_per_token |
4 |
Chars-per-token ratio used to estimate token counts (display only) |
cargo_warning_example_count |
2 |
Example warnings shown per cargo/clippy lint |
cargo_warning_group_threshold |
3 |
Occurrences before grouping cargo warnings |
jq_passthrough_threshold |
50 |
Below this many lines, jq/yq output passes through unchanged |
max_chain_depth |
3 |
Maximum processor chain depth (chain_to) |
max_output_bytes |
10000000 |
Hard cap on output length (chars) before compression runs (0 disables) |
debug |
false |
Enable debug logging |
user_processors_dir |
"" (falls back to ~/.token-saver/processors/) |
Directory for custom processors — global config / env var only, not project config |
disabled_processors |
[] |
Processor names to disable (env: comma-separated) — global config / env var only, not project config |
redaction_allowlist |
[] |
Env var name patterns exempt from secret redaction — global config / env var only, not project config |
delta_enabled |
false |
Experimental repeated-run diagnostics and local output retention — global config / env var only, not project config |
delta_retention_hours |
24 |
Delta snapshot lifetime in hours, from 1 to 168 — global config / env var only, not project config |
delta_max_runs |
100 |
Maximum retained Delta snapshots across all scopes, from 1 to 1000 — global config / env var only, not project config |
Parameters marked global config / env var only cannot be set from a
project-level .token-saver.json — see the security note in
Per-Project Configuration.
"I want maximum savings and I trust the precision tests." The defaults already apply
any gain at all (min_compression_ratio: 0.0), so turn the truncation thresholds down:
{ "generic_truncate_threshold": 80, "max_log_entries": 5, "max_diff_context_lines": 1 }"I'm doing a careful code review and want full diffs." Loosen the diff limits in a
project-level .token-saver.json:
{ "max_diff_hunk_lines": 1000, "max_diff_context_lines": 10 }"Terraform plans in this repo are enormous and I need all of them."
{ "generic_truncate_threshold": 2000, "max_output_bytes": 50000000 }"One processor is mangling a tool I care about." Disable just that one — globally, not from a project file:
export TOKEN_SAVER_DISABLED_PROCESSORS=jq_yqThe generic processor can't be disabled; it's the fallback that provides clean().
"Turn everything off for this session."
export TOKEN_SAVER_ENABLED=false"Show me what's happening."
export TOKEN_SAVER_DEBUG=trueYou can extend Token-Saver with your own processors for commands not covered by the built-in 36.
- Create a Python file with a class inheriting from
src.processors.base.Processor - Implement
can_handle(),process(),name, and setpriority - Copy the file to
~/.token-saver/processors/
# Example: install the ansible processor
cp examples/custom_processor/ansible_output.py ~/.token-saver/processors/A minimal processor looks like this:
import re
from src.processors.base import Processor
class TfsecProcessor(Processor):
priority = 16 # a free slot: 15 is package_list, 18 is just
hook_patterns = [r"^tfsec\b"]
@property
def name(self) -> str:
return "tfsec"
def can_handle(self, command: str) -> bool:
return bool(re.match(r"tfsec\b", command.strip()))
def process(self, command: str, output: str) -> str:
keep = [ln for ln in output.splitlines() if "Result" in ln or "CRITICAL" in ln]
return "\n".join(keep) if keep else outputTwo details worth knowing:
hook_patternsis what the PreToolUse hook matches before any output exists;can_handle()is what the engine matches after. They must agree, andtests/test_pattern_consistency.pyenforces that for every built-in processor — a command that gets wrapped but never routed is pure overhead.- Returning
outputunchanged is a supported, first-class outcome. It signals "I looked and decided not to compress", which the engine records differently from a failed compression attempt.
User processors are auto-discovered on every invocation. A broken processor (syntax error, missing import) is skipped with a warning — it never crashes the engine.
See examples/custom_processor/ for a complete example with documentation, and CONTRIBUTING.md if you'd like to upstream one.
Token-Saver records every compression in a local SQLite database:
~/.token-saver/savings.db
- savings: each individual compression (timestamp, command, processor, sizes, platform)
- sessions: aggregated totals per session (first/last activity, total original/compressed, command count)
On every session start, the SessionStart hook displays a summary:
[token-saver] Lifetime: 342 cmds, 307.2k tokens saved (67.3%) | Session: 5 cmds, 11.3k tokens saved (72.1%)
If a newer version is available, the notification is appended:
[token-saver] Lifetime: 342 cmds, 307.2k tokens saved (67.3%) | Update available: v1.0.1 -> v1.1.0 -- Run: token-saver update
token-saver stats
token-saver stats --jsonToken-Saver Statistics
========================================
Session
----------------------------------------
Commands compressed: 12
Original tokens: 61.3k tokens
Compressed tokens: 15.5k tokens
Saved: 45.8k tokens (74.7%)
Lifetime
----------------------------------------
Sessions: 47
Commands compressed: 342
Original tokens: 461.0k tokens
Compressed tokens: 147.3k tokens
Saved: 307.2k tokens (67.3%)
Top Processors
----------------------------------------
git 142 cmds, 121.8k tokens saved
test 89 cmds, 78.0k tokens saved
build 45 cmds, 49.7k tokens saved
Token counts are estimated from a chars_per_token ratio (default 4) — they're for
comparison and trend-watching, not for reconciling an invoice.
- Auto-pruning of records older than 90 days (configurable via
db_prune_days) - Automatic recovery on database corruption
- Thread-safe (reentrant lock on all operations)
- WAL mode for concurrent write performance
Nothing, in the compression path. Compression is pure local Python — regex and string parsing, no network, no model call, no telemetry. The only outbound request Token-Saver ever makes is an optional GitHub release check (cached 24h, 1s timeout, fails open), which sends nothing but a standard HTTP GET.
The stats database records command strings, processor names, size measurements, and timestamps; it does not store captured output. Command strings can themselves contain sensitive information.
Opting into Delta additionally stores sanitized diagnostic
snapshots, including captured output context, in a separate private local
database. Defaults are 24-hour retention and 100 snapshots; expiry is enforced
when the store is accessed. Redaction recognizes specific secret formats and
cannot detect every secret. Disable delta_enabled to stop new captures and run
token-saver delta clear to remove retained snapshots. Deletion does not promise
secure erasure from filesystem snapshots or backups.
- No shell injection: commands are passed through
shlex.quote()when rewriting - Syntax-checked rewrites: the rewritten command is validated with
sh -n -cbefore it runs; an invalid rewrite falls back to the original command - Fail-open: if the hook fails (Python error, missing file, timeout), the original command executes normally. A broken Token-Saver costs you savings, not a working shell.
- Secret redaction: the
envprocessor automatically redacts values of variables matching*KEY*,*SECRET*,*TOKEN*,*PASSWORD*,*CREDENTIAL*patterns, preventing accidental leakage into AI context windows — and a redacted result is never discarded in favor of the raw one by the compression-ratio gate - Untrusted project config: processor-loading, redaction, and Delta retention settings listed as global-only above are ignored in a repo-local
.token-saver.json - Signal forwarding: the wrapper propagates SIGINT/SIGTERM to the child process
- Exclusions: commands with complex pipes, redirections,
sudo, editors,ssh, unquoted newlines, background&, or shell-construct fragments (for,while,if,case) are never intercepted - Safe trailing pipes: simple trailing pipes (
| head,| tail,| wc,| grep,| sort,| uniq,| cut) are allowed - Chained commands:
&∧chains are supported — each segment is validated individually, and one ineligible segment disqualifies the whole chain - Self-protection: commands containing
token-saverorwrap.pyare not intercepted (prevents recursion)
Quote-awareness matters here: echo "a > b" contains a > inside quotes and is not a
redirection, while 2>&1 is a file-descriptor duplication rather than a write to a file.
Exclusion scanning uses a quote-aware tokenizer (src/shell_syntax.py) rather than a
naive substring search, so quoted metacharacters don't cause false exclusions — or,
worse, false inclusions.
Measured on macOS, Python 3.12, warm filesystem cache:
| Time | |
|---|---|
Bare sh -c 'echo hi' |
~5ms |
python3 -c pass (interpreter startup floor) |
~99ms |
Full wrap.py round-trip |
~158ms |
| Wrapper versus bare shell | ~153ms |
| Portion beyond Python startup | ~60ms |
These previously recorded timings include interpreter startup in the full wrapper cost. The roughly 60ms remainder is not the total overhead users pay. Measure the full round-trip on your machine: an extra 153ms is about 15% of a one-second command, or 0.5% of a thirty-second command. Short commands can make the overhead more noticeable than the output reduction.
max_output_bytes (default 10,000,000, currently counted as characters) limits
the text passed to compression after capture. It does not limit subprocess
buffering or guarantee a bound on every parser's execution time. Delta adds
parsing, comparison, and local SQLite storage; the timings above are for ordinary
compression and do not establish its overhead.
Does Token-Saver send my code or output to any server? No. Compression is entirely local — regex and string parsing in Python's standard library. The only network call is an optional GitHub release check, cached for 24 hours and silently skipped offline.
Will it hide an error from me or from the model?
That's the failure mode the preservation tests target. Known failures route
around processors that have not opted into failure handling, and critical-line
recovery covers selected omissions. These protections are tested on a fixture
corpus; unfamiliar formats and configured truncation can still lose useful
context. Use quality contracts for your workload and inspect --show-removed
when evaluating a rule. Delta retains full new or changed diagnostics for its
supported formats, with details retrievable while the snapshot is retained.
How much will I actually save?
It depends on your command mix, output, and detail retrievals. The tables above
measure individual fixture scenarios, not complete agent sessions. Run
token-saver stats for recorded command-output estimates, or
token-saver benchmark '<command>' for one command. Delta detail retrievals add
output and are not included in the ordinary savings tracker; account for them
separately when evaluating a complete workflow.
Does it work with the Claude API or the Claude Agent SDK directly?
Not as a drop-in — Token-Saver ships as a Claude Code plugin and an Antigravity CLI
plugin. But the engine is importable (from src.engine import CompressionEngine) and
has no dependency on either platform, so wiring it into your own agent loop is a few
lines.
Does it work on Windows?
Yes. Windows is a supported platform, with a token-saver.cmd launcher, an
%APPDATA%-based data directory, and UTF-8 stdio forcing at every entry point.
Does it slow down my shell? No — Token-Saver only runs inside your AI assistant's tool calls. Your interactive terminal is untouched.
What happens if Token-Saver crashes? The hook fails open: the original command runs, uncompressed, exactly as it would without Token-Saver installed. Same for a Python error, a missing file, or a timeout.
Can I use it alongside other token-reduction MCP servers? Yes. Token-Saver operates at the output level; caching and delegation tools operate at the task or invocation level. See docs/comparison.md.
Why not just tell the model "be brief", or pipe through | tail -50?
Because the model doesn't control tool output, and tail -50 doesn't know the error was
on line 12. Format-aware compression keeps the 12 lines that matter out of 500 — blind
truncation keeps the last 50 and hopes.
Does it compress files Claude reads with the Read tool?
No. Token-Saver intercepts Bash commands; reads via the native file tool don't pass
through it. It does handle cat, head, tail, and bat run as shell commands —
though source code files pass through unchanged by design.
Can a repository I clone attack me through .token-saver.json?
Project configuration cannot load processors through user_processors_dir,
change disabled_processors or redaction_allowlist, or enable or adjust Delta
retention through delta_enabled, delta_retention_hours, or delta_max_runs.
Those settings require global configuration or environment variables. Other
project settings still affect compression behavior; review them when needed.
How do I turn it off temporarily?
export TOKEN_SAVER_ENABLED=false, or set {"enabled": false} in
~/.token-saver/config.json.
Is a specific processor's behavior documented anywhere?
Yes — docs/processors/ has a page per processor covering what it
matches, what it keeps, what it drops, and its config knobs.
A command isn't being compressed. Ask why:
token-saver explain 'the exact command'The most common answers are a redirection (>), a sudo, a complex pipe, or output
that was simply too small to be worth touching.
Compression looks wrong for one tool. Reproduce it in isolation and see what got removed:
token-saver benchmark 'the exact command' --show-removedIf it's genuinely wrong, disable that one processor
(export TOKEN_SAVER_DISABLED_PROCESSORS=<name>) and please
open an issue with the raw output.
Stats show nothing. Verify the plugin is actually loaded (/plugin in Claude Code)
and that token-saver version resolves. If you installed manually, confirm
~/.local/bin is on your PATH.
I see [token-saver] N error line(s) recovered. That's the safety net working: a
processor dropped lines that looked like errors and the engine put them back. Worth
reporting — the processor should have kept them itself.
Everything is broken and I want out.
export TOKEN_SAVER_ENABLED=false # immediate, this session
python3 install.py --uninstall # permanentI need to see what the hook is doing.
export TOKEN_SAVER_DEBUG=true
python3 scripts/wrap.py --dry-run 'git status'- Does not compress commands with complex pipelines, redirections (
> file), or||chains - Simple trailing pipes are supported (
| head,| tail,| wc,| grep,| sort,| uniq,| cut) - Chained commands (
&&,;) are supported — each segment is validated individually sudo,ssh,vimcommands are never intercepted; remotersync(with host:path) is excluded but localrsyncis compressible- Long diff compression truncates per-hunk, not per-file: a diff with many small hunks is not reduced
- The generic processor only deduplicates consecutive identical lines, not similar lines
- Token counts in stats are estimated from a chars-per-token ratio, not a real tokenizer
- Experimental Delta covers supported single pytest/Ruff commands in Claude Code; it needs an explicit host session and an unchanged command to compare runs
- Delta retains local output when enabled; fresh diagnostics and detail retrieval can outweigh repeated-run savings
- Antigravity CLI: the deny/reason mechanism may have side effects if other plugins use the same hook
token-saver/
├── .claude-plugin/ # Plugin metadata
│ ├── plugin.json # Plugin manifest
│ └── marketplace.json # Marketplace catalog for distribution
├── hooks/ # Native hook declarations
│ └── hooks.json # Claude Code reads this automatically
├── skills/ # Agent skills
│ └── token-saver-config/
│ └── SKILL.md
├── commands/ # Slash commands
│ └── token-saver-stats.md
├── scripts/ # Python hook scripts
│ ├── __init__.py # Package init (prevents namespace conflicts)
│ ├── hook_pretool.py # PreToolUse hook (Claude Code)
│ ├── wrap.py # CLI wrapper (Claude Code)
│ ├── hook_session.py # SessionStart hook wrapper
│ └── audit_compression.py # Compression ratio corpus + report
├── antigravity/ # Antigravity CLI specific files
│ ├── antigravity-plugin.json # Antigravity plugin metadata
│ ├── hooks.json # Antigravity hook definitions
│ └── hook_aftertool.py # AfterTool hook (Antigravity CLI)
├── bin/ # CLI executables
│ ├── token-saver # Unix CLI wrapper
│ └── token-saver.cmd # Windows CLI wrapper
├── src/ # Shared source code
│ ├── __init__.py # Version (__version__) + data_dir()
│ ├── chain_utils.py # Chained command splitting (&&, ;)
│ ├── cli.py # CLI entry point (version/stats/benchmark/explain)
│ ├── config.py # Configuration system + project trust boundary
│ ├── console.py # UTF-8 stdio forcing
│ ├── engine.py # Compression engine (orchestrator)
│ ├── diagnostics.py # Immutable processor diagnostic contract
│ ├── delta.py # Compare and render completed runs
│ ├── delta_redaction.py # Recognized secret masking for snapshots
│ ├── delta_store.py # Private, bounded snapshot persistence
│ ├── delta_cli.py # Retrieve details and clear snapshots
│ ├── hook_session.py # SessionStart hook (stats + update notif)
│ ├── platforms.py # Platform detection + I/O abstraction
│ ├── shell_syntax.py # Quote-aware shell scanning (exclusions)
│ ├── stats.py # Stats display
│ ├── tracker.py # SQLite tracking
│ ├── version_check.py # GitHub update check
│ └── processors/ # 36 auto-discovered processors
│ ├── __init__.py
│ ├── base.py # Abstract Processor class
│ ├── critical.py # Shared "is this line an error?" predicate
│ ├── utils.py # Shared utilities (diff compression)
│ ├── package_list.py # pip list/freeze, npm ls, conda list
│ ├── git.py # git status/diff/log/show/blame/push/pull
│ ├── test_output.py # pytest/jest/cargo/go/dotnet/swift/mix test
│ ├── build_output.py # npm/cargo/make/webpack/tsc/turbo/nx/docker build
│ ├── lint_output.py # eslint/ruff/pylint/clippy/mypy/shellcheck/hadolint
│ ├── network.py # curl/wget/httpie
│ ├── docker.py # docker ps/images/logs/inspect/stats/compose
│ ├── kubectl.py # kubectl get/describe/logs/apply/delete/create
│ ├── terraform.py # terraform plan/apply/init/output/state
│ ├── env.py # env/printenv (with secret redaction)
│ ├── search.py # grep/rg/ag/fd/fdfind
│ ├── system_info.py # du/wc/df
│ ├── gh.py # gh pr/issue/run list/view/diff/checks
│ ├── db_query.py # psql/mysql/sqlite3/pgcli/mycli/litecli
│ ├── cloud_cli.py # aws/gcloud/az
│ ├── ansible.py # ansible-playbook/ansible
│ ├── helm.py # helm install/upgrade/list/template/status
│ ├── syslog.py # journalctl/dmesg
│ ├── file_listing.py # ls/find/tree/exa/eza/rsync
│ ├── file_content.py # cat/bat (content-aware compression)
│ └── generic.py # Universal fallback
├── docs/
│ ├── comparison.md # Token-Saver vs. adjacent tools
│ └── processors/ # Per-processor documentation
├── examples/
│ └── custom_processor/ # Worked example of a user processor
├── installers/ # Modular installer package
│ ├── common.py # Shared constants + utilities
│ ├── claude.py # Claude Code installer (native plugin registration)
│ └── antigravity.py # Antigravity CLI installer
├── install.py # Installer entry point
├── CLAUDE.md # Plugin instructions
├── tests/
│ ├── test_engine.py # Engine + registry tests
│ ├── test_processors.py # Per-processor tests (the largest file by far)
│ ├── test_hooks.py # Hook pattern + integration tests
│ ├── test_precision.py # Precision preservation tests
│ ├── test_pattern_consistency.py # hook_patterns vs. can_handle() agreement, per processor
│ ├── test_core.py # Shared compression core tests
│ ├── test_tracker.py # SQLite + concurrency tests
│ ├── test_config.py # Configuration tests
│ ├── test_console.py # UTF-8 stdio forcing tests
│ ├── test_readme_config_sync.py # README parameter table vs. _DEFAULTS drift guard
│ ├── test_version_check.py # Version check + fail-open tests
│ ├── test_version_consistency.py # Manifest/marketplace version drift guard
│ ├── test_cli.py # CLI subcommand tests
│ ├── test_user_processors.py # Custom processor loading tests
│ ├── test_installers.py # Installer utility tests
│ ├── test_install_smoke.py # End-to-end installer smoke tests
│ ├── failure_fixtures.py # One failing-command fixture per processor
│ ├── compression_baselines.json # Recorded ratios guarded by the ratchet test
│ └── test_compression_ratchet.py # Fails if any scenario compresses worse
├── pyproject.toml # Python project config + Ruff rules
├── CONTRIBUTING.md # Developer guide
├── LICENSE # Apache 2.0
└── README.md
python3 -m pytest tests/ -v1,300+ tests (run python3 -m pytest tests/ --collect-only -q | tail -1 for the exact, current count) covering:
- test_engine.py: compression thresholds, processor priority, ANSI cleanup, generic fallback, hook pattern coverage across all supported commands
- test_processors.py: each processor with nominal and edge cases, chained command routing, all subcommands (blame, inspect, stats, compose, apply/delete, init/output/state, fd, exa, httpie, dotnet/swift/mix test, shellcheck/hadolint/biome, traceback truncation, ansible, helm, syslog, parameterized tests, coverage, docker compose logs, tsc typecheck, .env redaction, minified files, search directory grouping, git lockfiles/stat grouping)
- test_hooks.py: matching patterns for all supported commands, exclusions (pipes, sudo, editors, redirections, remote rsync, backgrounded
&, unquoted newlines, shell-construct fragments), subprocess integration, global options (git, docker, kubectl), chained commands (shared shell state,&&short-circuit,;continue, unterminated-segment markers), safe trailing pipes - test_precision.py: verification that every critical piece of data survives compression (filenames, hashes, error messages, stack traces, line numbers, rule IDs, diff changes, warning types, secret redaction under every fallback path, unhealthy pods, terraform changes, unmet dependencies) — includes
TestFailureHandling, which runs a realistic failing-command fixture through every processor and asserts the failure reason is never lost, for every processor, at every exit code - test_compression_ratchet.py: locks each real-world scenario's compression ratio, processor routing, and was-compressed status to a recorded baseline — a silent regression fails CI, not just a code review
- test_pattern_consistency.py: every processor's
hook_patterns(used by the PreToolUse hook, before any output exists) must agree with itscan_handle()(used by the engine, after output exists) — otherwise a command gets wrapped but never actually routed - test_readme_config_sync.py: parses this README's parameter table and cross-checks every row against
src.config._DEFAULTS— documentation drift fails CI instead of shipping - test_core.py: shared compression core (decision, pass-through-on-error, audit logging) and the platform hook end-to-end
- test_tracker.py: CRUD, concurrency, corruption recovery, session tracking, stats CLI
- test_config.py: defaults, env overrides, invalid values, project-config trust boundaries
- test_version_check.py: version parsing, comparison, fail-open on errors
- test_cli.py: version/stats/help/explain subcommands, bin script execution
- test_user_processors.py: custom processor discovery and loading from
~/.token-saver/processors/ - test_installers.py / test_install_smoke.py: version stamping, legacy migration, CLI install/uninstall, end-to-end install smoke tests
Lint and type checks match CI:
ruff check .
ruff format --check .
mypy src/ scripts/ --ignore-missing-importsTo diagnose issues:
# Test compression on a command without replacing the output
python3 scripts/wrap.py --dry-run 'git status'
# Explain routing / exclusion decisions
token-saver explain 'git status'
# See exactly what compression removed
token-saver benchmark 'git diff' --show-removed
# Enable debug logging
export TOKEN_SAVER_DEBUG=true
# Check stats
token-saver stats
# Check version
token-saver versionContributions are welcome — especially new processors for tools that aren't covered yet. Read CONTRIBUTING.md for the development workflow, then:
- Add your processor to
src/processors/ - Add tests to
tests/test_processors.pyand a failing fixture totests/failure_fixtures.py - Add a scenario to
scripts/audit_compression.pyso the ratchet guards your ratio - Document it in
docs/processors/and add a row to the Processors table - Run
ruff check . && mypy src/ scripts/ --ignore-missing-imports && python3 -m pytest tests/
Apache 2.0 — see LICENSE.