x86: run-boot-parallel boots that never reach USERSPACE TEST COMPLETE (2/70 at the r1-era control commit, no serial preserved)

[ryanbreen]

Signature

docker/qemu/run-boot-parallel.sh 5 on beast (breenix-x86 Incus container, -smp 1,
testing,external_test_bins profile), 2 boots out of 5 in one round:

  Test 3: FAIL (x86 userspace verdict rejected the run)
    x86 userspace gate: FAIL - USERSPACE TEST COMPLETE was absent; boot did not finish
  Test 4: FAIL (x86 userspace verdict rejected the run)
    x86 userspace gate: FAIL - USERSPACE TEST COMPLETE was absent; boot did not finish

The two boots never reach 🎯 USERSPACE TEST COMPLETE; the round took 235 s of wall clock against
~50 s for every clean round in the same battery, so those boots ran to the runner's marker wait rather
than exiting early. The last kernel line captured in the round log is a truncated
[DEBUG] kernel::syscall::handlers: sys_ — i.e. the boot was still emitting syscall traffic when the
wait expired.

This is not the reader_exit_15 wake-delay shape tracked on #586 (that shape completes the boot
and fails a userspace process); it is a boot that does not finish, and it deserves its own name rather
than riding #586's.

Where it was seen, and where it was not

Observed during the R37 x86 A/B for the #609 branch — alternating 5-boot rounds between the r1-era
commit 8bcc1999 and branch HEAD d28407ae, image rebuilt at every switch:

commit	boots	"USERSPACE TEST COMPLETE was absent"
control 8bcc1999	70	2 (both in the same round)
HEAD d28407ae	35	0

Both occurrences are on the pre-#609-fix side, so this is not the #609 branch's; it is filed
because it is an unattributed x86 red with no issue of its own.

Evidence gap (stated rather than papered over)

No serial was preserved for either boot. run-boot-parallel.sh writes into
/tmp/breenix_boot_N/, which the next round overwrites, and the preservation wrapper was only added
to the following battery. Six further control-only rounds (30 boots) with per-round serial
preservation did not reproduce it, so this issue carries the round log and the rate, not a serial.
Anyone reproducing it should snapshot /tmp/breenix_boot_*/serial_kernel.txt immediately after each
round.

Suggested first moves

1. Make the runner preserve serials for failing boots by itself (it already knows which boot failed),
   so the next occurrence arrives with evidence attached.
2. Have the "boot did not finish" verdict print the last N kernel lines and the wait duration, so a
   hang is separable from a slow boot without re-deriving it from a raw serial.

Provenance

Branch fix/609-early-kthread-dispatch, R37 amend slot (T3-G PR1 r2 review finding R2-B4, control
leg). Recorded in docs/planning/teardown-unification/609-RCA-RETRACTION-2026-08-21.md §6.5.

[ryanbreen]

Main-baseline x86 boot-parallel run (T3-G PR2 round-2 side-quest)

Ran docker/qemu/run-boot-parallel.sh 5 × 4 batches (20 boots total) on beast
(breenix-x86 Incus container, same harness/profile as this issue) at main @ 9602d6d4
(merge of #632, fix/609-early-kthread-dispatch) — no branch changes in play, this is a clean
main-tip baseline.

Result: 18/20 pass, 2 fail (10%) — both failures are this issue's shape (USERSPACE TEST COMPLETE was absent; boot did not finish), one in batch 2 (test 2) and one in batch 3 (test 2).

Unlike the original report, both serials were preserved this time
(beast:/root/pr2-evidence/main-baseline/batch2/breenix_boot_2/,
beast:/root/pr2-evidence/main-baseline/batch3/breenix_boot_2/), so the evidence gap this issue
flagged is now partially closed. Both hung boots carry the identical concrete signature: the
kernel log balloons to ~130-156k lines (vs ~4-4.3k for a clean boot) almost entirely repeated:

[DEBUG] kernel::syscall::handlers: sys_read: fd=140737454784552, buf_ptr=0x1, count=0

fd=140737454784552 = 0x7ffffdffc028, a stack address, not a real fd — a thread spins forever
re-issuing sys_read with garbage args until the 900s gate timeout kills it. This is a livelock,
not a crash: no panic, no page-table fault, no ESR/FAR/ELR anywhere in either log.

So: this shape is confirmed pre-existing on main at 2/20 (10%), reproducing independently of
any branch under test — it is not something introduced by fix/607-576-zero-pc-family (which is
aarch64-only) or any other branch that surfaced it. This is a real, currently-unfixed x86 defect
class (garbage-fd sys_read livelock), not flakiness in the harness.

Evidence: beast:/root/pr2-evidence/main-baseline/{batch1,batch2,batch3,batch4}/, mirrored tally at
/private/tmp/claude-501/-Users-wrb-fun-code-breenix/14b2f8b6-d391-4fe4-b1ff-96563e0f434a/scratchpad/t3g/pr2/pr2-x86-baseline.md
on the originating Mac session.