`ft bench bw` picks `hybrid` on Ampere, but `offload` is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090)

[GlidLov]

ft bench bw picks hybrid on Ampere, but offload is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090)

Summary

On a 2x RTX 3090 (SM86) host, ft bench bw measures the CPU MoE kernel at 56.2 GB/s vs 12.3 GB/s for PCIe, reports a 4.57x ratio, and selects hybrid. In actual serving, hybrid delivers 0.67 tok/s while forcing --moe-backend offload on the same machine and model delivers 5.58 tok/s.

The auto-selected backend is 8.3x slower than the one it rejected.

The calibrated CPU number does not survive contact with serving: at 0.67 tok/s and ~3.07 GB of expert weights per token, the effective CPU-MoE rate is roughly 2 GB/s, about 28x below what ft bench bw measured for the same kernel, dtype and workload minutes earlier.

Environment

FreeToken	0.1.2 (PyPI, freetoken[accel])
Model	deepseek-ai/DeepSeek-V4-Flash-0731 (fp8 dense, expert_dtype: fp4)
GPU	2x RTX 3090 24GB, SM86, driver 595.84
CPU	Intel Xeon Gold 6252 @ 2.10GHz, avx512f, 46 cores available to the container
RAM	503 GB
CUDA / torch	13.0 / 2.11.0+cu130
OS	Linux x86_64 (container)

Reproduction

ft bench bw --model dsv4 -o bwprofile.json

ft serve --model-path /path/to/DeepSeek-V4-Flash-0731 \
  --tp-size 2 --moe-backend hybrid --moe-cache-auto \
  --host 0.0.0.0 --port 1919
# -> 0.67 tok/s

ft serve --model-path /path/to/DeepSeek-V4-Flash-0731 \
  --tp-size 2 --moe-backend offload --moe-cache-auto \
  --host 0.0.0.0 --port 1919
# -> 5.58 tok/s

Decode measured over a streamed 60-token completion, first token excluded, after a warmup request.

ft bench bw output

  host 82d6485125bb   gpu cuda:0 (NVIDIA GeForce RTX 3090)   cpu 24c/24t
  ceilings: CPU STREAM read 62.3  |  PCIe linear H2D 12.3  D2H 13.2  GB/s   (threshold 2.0x)

  dsv4  H=4096 I=2048 E=256 top_k=6
    format      expert       CPU-MoE   PCIe-gather  CPU/PCIe  backend
    ds_fp4    12.75 MB     56.2 GB/s     12.3 GB/s     4.57x  hybrid
       overlapped: CPU-MoE 44.8 + PCIe 11.3 GB/s -> hybrid fetches 20.1% of misses

Measured results

backend	CPU threads	decode	TTFT (short prompt)
hybrid (auto-selected)	23	0.58 tok/s	5.8 s
hybrid	44	0.67 tok/s	3.5 s
offload	n/a	5.58 tok/s	2.5 s
offload, --memory-ratio 0.95	n/a	5.95 tok/s	2.4 s

The last row is included to show the ceiling is not cache-related: raising memory-ratio from the 0.9 default grew the expert pool from 624 to 779 slots (5.7% -> 7.1% residency) for only +6.6% throughput. Expert-cache residency cannot be pushed much further on this host — the ~24 GB of non-expert weights (167 GB total minus 143 GB of experts) already occupy roughly half of the 48 GB of VRAM, leaving ~10-12 GB for all pools combined.

Doubling the CPU pool from 23 to 44 threads yields only +15%, which is itself evidence that the CPU path is not thread-bound but limited by something else — most likely per-layer synchronisation. With top_k=6 over 43 MoE layers there are 258 expert activations per token, each a separate small GEMV with its own sync point, whereas the benchmark measures a tight loop over one expert.

Note on TP=2 and #62

All runs use --tp-size 2, and #62 documents that OffloadMoeCache is TP-unaware, making TP=2 a regression for offloaded MoE. That applies here too, so the absolute numbers below are pessimistic.

TP=1 is not available as a control on this hardware: the non-expert weights are ~25 GB in total (~12.7 GB per rank at TP=2, confirmed by VRAM accounting), which does not fit a single 24 GB card alongside any cache. Anyone reproducing on >=32 GB cards can and should check TP=1.

The hybrid vs offload comparison is unaffected: both ran at TP=2 with identical flags, so the 8.3x gap is attributable to the backend alone. If anything, #62 means both figures would rise together at TP=1.

Config was otherwise identical across runs (--moe-cache-auto, cache resolved to 624 slots / 5.7% residency in every case), so the difference is attributable to the backend alone.

Why this matters

The wrong call is made silently. --moe-backend auto resolves to hybrid here and produces the same 0.67 tok/s; nothing in the logs or in ft ctl stats points at the backend as the cause. Without a reason to try offload by hand, the reasonable conclusion is that the engine is unusable on this hardware, when in fact it is 8.3x faster one flag away.

Suggestions

1. Validate the calibration against a short end-to-end decode. After picking a backend, run a few tokens through it and compare achieved throughput to the predicted rate; if the gap exceeds some factor, fall back and warn.
2. Make the benchmark representative of serving. Measure the CPU path across a full layer sweep with the real synchronisation pattern (258 activations/token), not a single-expert loop. A per-activation fixed cost would then show up in the model.
3. Expose runtime cache/backend effectiveness. ft ctl stats reports decode_tps but not expert cache hit rate or the realised CPU-vs-PCIe split. Those would make this diagnosable from the outside, and would also cover the Desktop-UI side of [Feature Request]: Add real-time expert cache hit rate and PCIe transfer telemetry to Desktop UI and CLI #76.
4. Consider raising the selection threshold for hybrid on architectures without native FP4, or excluding it there until the CPU-side cost model accounts for per-activation overhead.

Secondary issues found in the same session

-lnccl link failure with --tp-size > 1. The JIT build of freetoken__pynccl.so fails on a clean uv pip install "freetoken[accel]":

/usr/bin/ld: cannot find -lnccl: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
RuntimeError: ninja exited with status 1
ERROR Backend worker is gone and cannot be restarted; stopping the API server

torch ships nvidia/nccl/lib/libnccl.so.2 without the libnccl.so devel symlink that -lnccl requires. Workaround:

NCCLDIR=$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/nccl/lib
ln -sf libnccl.so.2 $NCCLDIR/libnccl.so
export LIBRARY_PATH="$NCCLDIR:$LIBRARY_PATH"

The build could resolve the NCCL path from torch.utils.cpp_extension / the installed nvidia-nccl-* package instead of relying on a system -lnccl.

--moe-cache-size accepts a value that makes startup fail. ft ctl cache reported the moe pool as resizable to 256..828, but --moe-cache-size 828 aborts the backend:

ValueError: DSV4 KV budget -284406580 bytes cannot fit the minimal pool
(31 pages incl. the window working-set floor 31); raise memory_ratio or
lower max_running_req/max_seq_len

The advertised upper bound does not account for the KV/SWA floor, so the value the tool itself reports as the maximum is not actually selectable. Possibly the same budget-solve area as #111 and #150.

Kernel cache directory is not created. Every start recompiles kernels:

freetoken/models/deepseek_v4/weight.py:70: UserWarning: Specified kernel cache
directory could not be created! This disables kernel caching

What worked

For balance: the model itself runs correctly on SM86. fp8 dense weights and fp4 experts are dequantised at runtime with no kernel errors, dsv4_sparse attention is selected automatically, and output quality is fine. The only hard blocker to getting it up was the NCCL symlink above. The problem is backend selection, not Ampere support as such.

[ascorb12]

Ran the same hybrid vs offload comparison on different hardware and cant reproduce it, they come out the same here.

Gemma-4-26B-A4B-it bf16 (47 GB), 32,768 ctx, single 4090:

hybrid  (auto-picked): 13.8 / 15.9 tok/s
offload (forced):      15.4 / 16.0 tok/s

Same prompt, same 32k pool, only the backend flag changed. ft bench bw picked hybrid for bf16 here at a 2.27x ratio and that turned out fine, no big gap either direction.

Setup is 7800X3D, single RTX 4090 (SM89), 63 GB RAM, WSL2 Ubuntu 24.04, built from main, CUDA 13.0, driver 610.88. The CPU MoE executor only got 6 threads pinned here vs your 46 cores.

Fair warning on how much this is worth, its a different model and a different dtype than yours (bf16 Gemma vs fp4 DeepSeek), one GPU instead of two, and the weights are loading off /mnt/c which is slow. So its not a clean refutation, just a data point that the backend pick isnt wrong everywhere.

The part that looks different between our boxes is the CPU side, not the backend pick. Yours benched 56.2 GB/s and delivered ~2 GB/s serving. Ours benched 58.1 GB/s for bf16 and the serving speed lines up with it, no collapse. Might be worth checking whether the container is capping CPU or if its a NUMA thing across those 46 cores, since thats where the 28x gap is showing up rather than in the backend selection itself.