Retry an engine that dies on the way up during startup replay

[webdevtodayjason]

Problem

Rolling 0.5.4 onto spark-3 lost its model. The startup sweep killed the running engine and replay relaunched immediately, while the driver was still releasing the GPU from the container that had just died. The nvidia hook handed the new container no device and it exited during weight load:

Can't initialize NVML
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Triton is installed but 0 active driver(s) found (expected 1)
Failed to import from vllm._C: ModuleNotFoundError("No module named 'vllm._C'")

Host nvidia-smi was healthy throughout, and a container launched from inside the orchestrator saw the GPU fine. Re-issuing the identical load a few minutes later worked with zero GPU-failure lines, which is what identifies it as a release race rather than a broken node.

Nothing retried, so the node came back advertising nothing and the model stayed missing until a human re-loaded it. That is the worst shape for this failure: a restart looks successful and the model is just gone.

Worth being precise, because the first diagnosis was wrong: this is NOT a failed launch check. The container reached Running, so start_solo() correctly returned True. The engine died minutes later, during weight load. The gap was that nothing watched it after the launch confirmation.

Fix

_ensure_serving() waits for an engine to bind and, if it never does, waits 30 seconds for the GPU to finish releasing and relaunches once. It is applied to both the boot primary and each replayed stacked instance.

The retry is deliberately single. A model that fails twice has a real problem, and a retry loop would hide it behind minutes of silence.

Tests

6 new, covering: no relaunch when the engine binds first time, one relaunch when it never binds, giving up after exactly one retry, a relaunch that fails to start, a relaunch that raises, and that the retry actually waits before asking for the GPU again. 708 pass, ruff clean.

This was only diagnosable because of #62 — dropping docker run --rm left the crashed container as a readable corpse. Before that it was silence, which is how it went unnoticed.