fix(bench): skip an unsupported slot count instead of aborting the sweep

[rakhimovv]

Fixes #146. Sending this per @MR-1124's offer in the issue thread — happy to take review notes on it.

Revised after review. Two things in the first version were wrong and are corrected below: the row counts I quoted were undercounted by half, and the justification I gave for catching rather than pre-checking did not hold. Details at the end.

Problem

benchmarks/bench_offload_cache_copy.py derives its default slot counts as
[profile.experts, int(0.4 * profile.layers * profile.experts)]. For minimax-m2.5-marlin
(L=62, E=256) the second value is 6348, which OffloadMoeCache rejects — the marlin
backend is capped at 992 slots. The ValueError was uncaught, so a plain
python benchmarks/bench_offload_cache_copy.py died on the 4th of 8 profiles.

The part that makes it worth fixing rather than documenting: the four profiles after it
never ran, and the output left behind reads as a completed sweep.

Fix

Catch it in print_table, skip that one combination with a printed line, and report the
skip in the exit status.

Clamping to the cap was the alternative and is worse — it silently changes the size the row
measures, so a reader comparing tables would be comparing different configurations without
being told.

Three details:

• The cache is built before the header is printed. Previously print_table printed the
  profile header and column titles first, so a skip at the old call site would leave an
  orphan header with no rows under it — the same shape the abort produced.
  validate_rebuild runs in __post_init__ ahead of the allocations, so the refused path
  costs nothing.
• The exit status distinguishes two cases that are not alike. A derived default that one
  backend cannot satisfy is expected output and stays 0, so the documented no-arg sweep is
  not red forever. Naming a geometry by hand and getting nothing — or measuring nothing at
  all — returns 1. This follows the shape bench_decode_moe.py already uses for a failed
  backend.
• Catching rather than pre-checking is because the guard also covers the num_experts
  floor, so a pre-check would duplicate the rule, not just the number. (MARLIN_MAX_CACHE_SIZE
  is a public constant in the module this file already imports from, so "the number would
  drift" would not have been a real argument.) A device OOM is a RuntimeError, so it still
  aborts — which is right, since it is not a statement about the geometry being illegal.

Before / after

Default invocation, no arguments, on bd372b6:

	tables	rows	profiles reached	exit
before	7 complete + 1 orphan header	112	4 of 8	traceback
after	15	240	8 of 8	0

15 tables rather than 16 is the point: 8 profiles × 2 slot counts, minus the one refused
combination, which is now reported instead of fatal. Every table is 16 rows (4 batch sizes ×
4 miss rates).

After, the skip is one line and names the reason, and the run ends with a tally:

minimax-m2.5-marlin @ cache_slots=6348 (47.1 GiB): SKIPPED -- moe_cache_size=6348 exceeds
the marlin backend's slot limit of 992 (vLLM moe_align_block_size caps padded experts at
1024); reduce moe_cache_size or force --nvfp4-backend triton

skipped 1 of 16 combinations: minimax-m2.5-marlin@6348

Asking for that geometry explicitly now fails instead of passing quietly:

$ python benchmarks/bench_offload_cache_copy.py --models minimax-m2.5-marlin --cache-slots 6348
skipped 1 of 1 combinations: minimax-m2.5-marlin@6348
$ echo $?
1

Explicitly naming the profiles that used to be unreachable is unchanged — still 128 rows,
exit 0:

python benchmarks/bench_offload_cache_copy.py \
  --models minimax-m2.5-triton gpt-oss-20b gpt-oss-120b glm4.7-nvfp4

"8 of 8" is a property of an 80 GB card, not of this change. The default sweep's largest
slot cache is glm4.7-nvfp4 @ 5696 = 70.5 GiB, then minimax-m2.5-triton @ 6348 = 47.2 GiB.
On a smaller GPU the sweep still dies partway through with a device OOM — that is a separate
limitation of the default slot counts and out of scope here.

Tested on

• GPU: NVIDIA H100 80GB HBM3 (sm_90), one GPU of four in the box
• NVIDIA driver: 580.126.16
• CPU: Intel Xeon Platinum 8462Y+, 56 threads; 2015 GiB system RAM
• OS: Linux 5.15.0-157-generic
• CUDA toolkit: nvcc release 13.1, V13.1.115
• torch 2.11.0+cu130, Python 3.12.13
• FreeToken 0.1.2, branched from bd372b6
• Install: uv pip install -e ".[accel,dev]"

The benchmark is synthetic, so no checkpoint is involved.

CUDA_VISIBLE_DEVICES=2 PYTHONPATH=python:. python benchmarks/bench_offload_cache_copy.py

No test is added: nothing under tests/ imports benchmarks/, and the guard is only
reachable with a CUDA device.

What was wrong in the first version

• Row counts. I quoted 56 / 120 / 64. The real figures are 112 / 240 / 128. My count came
  from grep '^[0-9]', and rows are printed with f"{batch_size:2d}", which right-aligns —
  so bs=1 and bs=4 begin with a space and were silently dropped, losing exactly half of
  every table. The table counts and the profile counts were unaffected.
• The reason for catch-over-pre-check. I wrote that pre-checking would mean copying 992
  into a file where it would drift. MARLIN_MAX_CACHE_SIZE is public and already importable
  here, so that was not the real argument; the real one is the num_experts floor, above.
• Exit status. The first version returned 0 even when a run measured nothing at all.

[MR-1124]

Checked the diff — looks right to me. Builds the cache before printing the header, asks the backend for the limit rather than hardcoding it, matches what we discussed. Thanks for the quick fix!

[rakhimovv]

Thanks for the review. Before it goes further I have to walk back two things I told you, both my error — I pushed corrections and updated the description.

The row counts were wrong. I quoted 56 before / 120 after / 64 for the subset. The real figures are 112 / 240 / 128. Rows print with f"{batch_size:2d}", which right-aligns, so bs=1 and bs=4 start with a space — and the grep '^[0-9]' I counted with dropped exactly half of every table. The table counts (8 → 15) and profile counts (4 → 8) were unaffected, so the shape of the result stands; only the row numbers were understated. Sorry for handing you figures you then repeated.

"Asks the backend for the limit rather than hardcoding it" was not a real argument. MARLIN_MAX_CACHE_SIZE is a public constant in freetoken.moe.offload_cache, and the benchmark already imports from that module — so a pre-check would not have been "hardcoding" anything. The actual reason to catch is that the same guard covers the num_experts floor, so pre-checking would duplicate the rule, not just the number. I have corrected the comment, which also said the cap was 1024 when it is 992.

One behaviour change while I was in there: the first version exited 0 even when a run measured nothing. Now print_table reports whether it measured anything and main tallies, following bench_decode_moe.py's shape. The exit status separates two cases that are not alike — a derived default one backend cannot satisfy stays 0, so the documented no-arg sweep is not red forever, while naming a geometry by hand and getting nothing returns 1:

$ python benchmarks/bench_offload_cache_copy.py --models minimax-m2.5-marlin --cache-slots 6348
skipped 1 of 1 combinations: minimax-m2.5-marlin@6348
$ echo $?
1

Also worth flagging since the description previously implied otherwise: "8 of 8 profiles" is a property of an 80 GB card. glm4.7-nvfp4 @ 5696 slots needs a 70.5 GiB slot cache, so on a smaller GPU the sweep still dies partway with a device OOM. That is a separate limitation of the default slot counts, and I have left it out of scope rather than quietly widening this PR.

Re-review whenever suits.

[MR-1124]

Thanks for the transparency and the fix. Checked the second diff — the exit-status logic matches what you described (0 for a partial default skip, 1 if everything's skipped or an explicit geometry fails), and the corrected comment reads right (992 limit, num_experts floor, RuntimeError still aborts). Appreciate you catching your own mistakes here.