Investigate large hidden-state differences when independent requests are co-packed (Qwen3.6, no-grad)

[bradhilton]

A bounded same-actor no-gradient diagnostic found large hidden-state differences when the same three independent requests were included in a larger pack. This currently blocks claiming numerical qualification for the packing/recovery work in #848 / #870 / #900. It does not establish a regression introduced by #900 or identify an underlying kernel.

Observed September 15, 2026 on one H200, Qwen/Qwen3.6-35B-A3B, candidate trainer runtime SHA256 5df3a1138923faa6705862037cd067d4b8b27cbdb9ef99c1094116efeb237d23, source revision 586fc4a6dbbabc876f6c7a571a86c47d6b064e4a. Two trainable rank-1 LoRA slots and a frozen reference were resident, with restored optimizer state; no backward, optimizer update, inference request or cache-release operation occurred.

The same actor ran A1, A2, B, A3:

• A: original requests 0–2, 32,496 packed/logical rows, one admitted child.
• B: original requests 0–11, 134,591 rows, one admitted child; compare the unchanged first three requests with A.
• A2/A1 and A3/A1 were bit-for-bit identical for all three outputs.
• B/A1 exceeded the unchanged BF16 tolerances (atol=1e-5, rtol=0.016) for all three outputs, with maximum absolute differences 10.9453125, 8.0 and 9.171875. A3/B also failed.
• All twelve B outputs were finite. All nine measured resident adapter/optimizer state digests were equal. Full base-model state was not rehashed by this diagnostic.

The four native stages completed. Original strict offline projection confirms native_completed=true, scalar_complete=true, all_numerical_conditions_passed=false, and zero cache-release calls. Process exit 0 is completion, not numerical acceptance. Exact four owned resource UIDs and two host process groups were independently reconciled absent.

Source investigation: the selected GDN path creates fresh root recurrent/conv state and both packings use variable-length processing. Common requests 1/2 retain their GDN offsets but still differ; request 0 moves to a later offset. Full attention, GDN, MoE, compilation, unmeasured base state and observer effects remain possible locations. No cause has been isolated. A marker-based CPU check of the original packing/projection/spool path passed; that is not proof of native numerical correctness.

Next owned work (Schulman/Peirce): a controlled row-count versus segment/boundary-treatment comparison, preserving the common three requests and original token content. Separate baseline/candidate/main-replica backward diagnostics remain in progress. No tolerance relaxation or art.megatron edit is part of this issue intake.

Durable private evidence (payloads are not attached):

• Run root: /var/tmp/art870-aaba-native-v1-20260915-root
• Strict result: offline-result/result.json, SHA256 97d922966396785661cbac6cf247e5e4081a59ef38216b069599eabc66a4e770
• Root closure: gpu-root-closure.json, SHA256 4421c8d875889813a2cf1248e924cb3e8cb178f4d15248f6e5872de5398154b3
• Full raw terminal log SHA256 a70d1185edaf6277ea10b221d1eb87bf23cd35f57bd22ddcf695e26ed7bc292f; 172 replayed actor occurrences join 86 unique actual records, each appearing twice.
• Source/decision matrix: /var/tmp/art870-packing-boundary-20260915-physical-tiipnfq4/, manifest 742797acbdd718a1d2d3ecd447ba3192f5cab66bb41eb578b9247316d83c88ab.
• Independent local readback: /var/tmp/art870-aaba-local-readback-20260915-physical-3qcco7wi/, manifest f6227149e2c58c40c4ffd358eb93e9c33bec2333f0a462fec380e5bef148ceb5.