Symptom
./docker/qemu/run-aarch64-refusal-drain-gate.sh (Phase 5 of run-aarch64-full-test.sh) fails intermittently with:
[TEST:process:kernel_stack_ownership_oracle:FAIL:ownership stress slot allocation/free equality failed]
Rate — pre-existing, measured on main
Six standalone runs of the gate on main @ 102317b4, nothing else in the tree:
| run |
verdict |
| 1 |
FAILED |
| 2-5 |
PASSED |
| 6 |
FAILED |
2 / 6 red on main. Found while gating the #580 branch, where the same signature appeared once in a full-test run and then passed 3/3 standalone. It is not caused by that branch.
The measurement
From the main run-6 serial:
[KSTACK_OWNER_ORACLE:aarch64:creation_rows=1000:creation_owned=1000:one_owner=1000:two_owner=0:zero_owner=0:
fork_rows=1:fork_owned=1:slot_returns_exact_one=1:
slot_alloc_delta=1000:slot_free_delta=1001:slot_balance=-1:
frames_mapped_delta=0:frames_released_delta=0:frame_balance=0:...]
Every ownership measurement is exact. The only divergence is one extra slot free inside the window: slot_alloc_delta=1000, slot_free_delta=1001, slot_balance=-1.
kernel/src/tracing/providers/teardown.rs:5658-5670 asserts slot_alloc_delta == slot_free_delta across the stress loop, and those deltas are taken from global slot counters. The oracle runs with the rest of the system live, so any concurrently reaped thread that returns its kernel-stack slot inside the window lands in slot_free_delta with no matching allocation in the same window — the equality is not a property of the workload the oracle performs, it is a property of the whole system being quiescent, which nothing enforces.
Two readings, both worth checking before repairing
- Measurement defect (likely). The delta pair should be scoped to slots the oracle itself allocates — e.g. tagged/owned slot ids, or a per-oracle counter — rather than the global counters.
slot_balance would then be the assertion and the delta equality would follow.
- A real unpaired free.
-1 is always the sign observed so far; if a slot can be returned twice, the same counters would show it. A repair that only narrows the window would hide that. Worth confirming slot_balance reaches 0 again after the window on a failing boot before choosing (1).
Impact
Phase 5 of the full aarch64 test is a merge gate, so this reddens roughly a third of merge attempts on a signature unrelated to whatever is being merged. It is not a pre-adjudicated tolerance signature and should not become one — the fix is to make the assertion measure what the oracle actually does.
Failing serial preserved with the #580 gate evidence.
Symptom
./docker/qemu/run-aarch64-refusal-drain-gate.sh(Phase 5 ofrun-aarch64-full-test.sh) fails intermittently with:Rate — pre-existing, measured on main
Six standalone runs of the gate on
main@102317b4, nothing else in the tree:2 / 6 red on main. Found while gating the #580 branch, where the same signature appeared once in a full-test run and then passed 3/3 standalone. It is not caused by that branch.
The measurement
From the main run-6 serial:
Every ownership measurement is exact. The only divergence is one extra slot free inside the window:
slot_alloc_delta=1000,slot_free_delta=1001,slot_balance=-1.kernel/src/tracing/providers/teardown.rs:5658-5670assertsslot_alloc_delta == slot_free_deltaacross the stress loop, and those deltas are taken from global slot counters. The oracle runs with the rest of the system live, so any concurrently reaped thread that returns its kernel-stack slot inside the window lands inslot_free_deltawith no matching allocation in the same window — the equality is not a property of the workload the oracle performs, it is a property of the whole system being quiescent, which nothing enforces.Two readings, both worth checking before repairing
slot_balancewould then be the assertion and the delta equality would follow.-1is always the sign observed so far; if a slot can be returned twice, the same counters would show it. A repair that only narrows the window would hide that. Worth confirmingslot_balancereaches 0 again after the window on a failing boot before choosing (1).Impact
Phase 5 of the full aarch64 test is a merge gate, so this reddens roughly a third of merge attempts on a signature unrelated to whatever is being merged. It is not a pre-adjudicated tolerance signature and should not become one — the fix is to make the assertion measure what the oracle actually does.
Failing serial preserved with the #580 gate evidence.