Description
Problem
When the gRPC call to metal-api's BootService fails, pixie logs the error and serves nothing. iPXE has no boot URLs, gives up, and never retries — the machine sits idle indefinitely and never registers. Only an out-of-band power reset recovers it.
Observed in mini-lab (pixie v0.4.1), where two machines PXE-booted simultaneously and one lost:
18:26:19 boot resp {"kernel":"…/metal-kernel","init_ram_disks":["…/metal-hammer-initrd.img.lz4"]} ← machine02, ok
18:26:34 ERROR boot error="rpc error: code = Unavailable desc = error reading from server: EOF" ← machine01
18:26:34 Couldn't get a bootspec for mac=aa:c1:ab:22:0d:8e
metal-api was healthy. The RPC failure was a one-off on the path through the envoy Gateway LB. make power-reset-machine01 booted the machine cleanly on the next attempt and it registered normally.
Impact
A single transient RPC failure permanently strands a machine. Automated recovery using metal-stack tools is not possible, as metal-api never learns the machine exists. In CI this is a hard integration-test failure; in production it is a machine that silently never comes up.
Suggested fix
When the bootspec lookup fails, return an iPXE script that retries instead of returning nothing — e.g. sleep, then chain the bootspec URL again, with a bounded retry count falling back to reboot.
Description
Problem
When the gRPC call to metal-api's BootService fails, pixie logs the error and serves nothing. iPXE has no boot URLs, gives up, and never retries — the machine sits idle indefinitely and never registers. Only an out-of-band power reset recovers it.
Observed in mini-lab (pixie v0.4.1), where two machines PXE-booted simultaneously and one lost:
metal-api was healthy. The RPC failure was a one-off on the path through the envoy Gateway LB.
make power-reset-machine01booted the machine cleanly on the next attempt and it registered normally.Impact
A single transient RPC failure permanently strands a machine. Automated recovery using metal-stack tools is not possible, as metal-api never learns the machine exists. In CI this is a hard integration-test failure; in production it is a machine that silently never comes up.
Suggested fix
When the bootspec lookup fails, return an iPXE script that retries instead of returning nothing — e.g. sleep, then chain the bootspec URL again, with a bounded retry count falling back to reboot.