Four launch-path fixes found benchmarking on a real node

[webdevtodayjason]

I set out to measure decode against context depth and couldn't get the model loaded onto a second node. Four separate things were wrong. None of them show up unless you launch through the real path onto a box that isn't a clean room, which is a decent argument for the dogfood rule on its own.

Missing engine image

Engine images have been per-model since 0.5.4, so a node can be asked for an image it has never run. Nothing waited for that first pull. Docker started one implicitly, the launch confirmation timed out underneath it, and the caller got a bare {"error": "Failed to launch engine"} with no mention of an image anywhere. The entrypoint probe degraded at the same time, because docker inspect on a missing image returns nothing and we quietly fall back to a default argv prefix.

ensure_image() pulls it first now, with an hour of headroom since these images run about 20 GB, and refuses the launch if it can't get it.

Port collision

allocate_port() only skipped ports held by our own instances. It had no idea about anything else on the host. On a node where an unrelated service had owned 8000 for three weeks it handed out 8000 anyway, and vLLM died with OSError: [Errno 98] Address already in use after loading the whole model first. It socket-probes the host now, and the probe can be turned off for callers that don't want it.

1 MB request bodies

The API server never set client_max_size, so aiohttp's 1 MB default was in force. That caps a 262k-context model at roughly 190k tokens of prompt. The proxy 413s the request before the engine ever sees it, and nothing in the error tells you which hop refused. Hitting vLLM directly works fine, which is what makes it confusing.

That is how I found it. A 200k-token prompt is 1.01 MB. Default is 64 MB now, sized for a 1M-token context plus base64 image and video parts on the multimodal models, and configurable through max_request_mb.

extra_env

This one is a feature rather than a bug. Some engine settings have no CLI flag at all. The b12x FP4 kernel path on GB10 is selected purely by environment variables. Without a way to pass env per model, those models can only be reached by hand-rolling a container, which is exactly what the launch path exists to prevent.

extra_env carries them through NodeConfig, ModelInfo, catalog_recipe, and the load API, mirroring how extra_vllm_args already works. It merges over the computed NCCL env on purpose: our own default forces VLLM_NVFP4_GEMM_BACKEND=marlin on the pinned image, so a recipe asking for flashinfer-b12x has to win or b12x is unreachable.

Proof

The tests worth pointing at are the ones that pin the actual contracts. One binds a real squatter socket and asserts we allocate around it. One sets a recipe env value that collides with the computed one and proves the recipe wins. One proves start_solo won't run a container when the image can't be had. 15 new tests, 723 green.

One of these I caused myself. My first cut of the request-ceiling tests built a full create_app() outside a running loop, which bound the module-level download semaphore to the wrong loop and made an unrelated download test flaky. Pulled the arithmetic out into _client_max_bytes() and tested that directly instead. Three full-suite runs clean afterwards.

Not in here

No catalog recipe for b12x yet. extra_env is the plumbing; the recipe waits on measurements. Nothing changes for any model that already launches, and default behavior is unchanged apart from the request ceiling.

🤖 Generated with Claude Code