Investigate output and gradient variability between matched same-source Qwen TrainerRank actors

[bradhilton]

September 17 follow-up: within-actor repeatability remains unresolved.

A frozen eight-sequence, three-backward diagnostic again completed without OOM but failed the original comparison: 33 gradient tensors failed cold-to-warm and 40 failed warm-to-warm. The original exit code is 1. A strict offline reader completed successfully and preserved that failed scientific result; all four owned Kubernetes resources and local process groups are independently cleaned up.

We observed one parameter’s incoming and accumulated gradients during both warm passes. Its parameter values stayed identical. Within each pass, incoming and accumulated gradients were byte-for-byte equal; across passes, 3,300 of 4,096 elements changed. However, this parameter (index 4, layer-0 output-projection LoRA A) passed the original tolerance: maximum tolerance ratio 0.28871, relative L2 0.005226. This localizes variation at that leaf to its incoming contribution; it does not explain the other failing gradients or prove a global accumulation/kernel cause. The blocking CPU snapshots can perturb execution.

The next bounded diagnostic targets index 182, layer-11 query-projection LoRA A, which failed warm-to-warm criteria in three retained runs. It will keep the full 660-gradient checks and original tolerances unchanged, with 4 KiB snapshots per vector. Current work is source/CPU qualification; no native success is implied.

Evidence: /var/tmp/art848-warm-leaf-image-20260917-root/root-readout-acceptance.json (5b281455); detailed vector analysis /var/tmp/art848-warm-leaf-readout-source-20260917-forward-jy5folvo/leaf-analysis.json; target selection /var/tmp/art848-warm-leaf-next-target-20260917-forward-suzvu5vm/manifest.json (944f30fc). This uses an older frozen runtime, not the current #898 head. Cross-run state parity and a common cause with #901 remain unproved.

---

The #848/#870 qualification work produced a numerical failure between two independently loaded actors using the same ART main source, checkpoint, inputs and measured execution context. This is separate from the packing-dependent no-gradient case in #901; a common cause is unproved.

Observed September 15, 2026, Qwen/Qwen3.6-35B-A3B, one H200 per actor, rank-1 LoRA. Both baseline actors use ART main 54ebf50f and the same complete tracked ART source namespace. A third actor uses #898 candidate bec279427003452697a73de09ca35e200ed77cda. All ran the unchanged three-stage warmup/warm1/warm2 diagnostic with 6,330 positions, 224 masked positions, three backwards and no optimizer update.

For all three actors, the pre-run full registered tensor inventory completed: 22,264 tensors / 69,556,700,544 logical bytes. Normalized loaded-state SHA256 1bf758dd9af4309998b980762ab4b8b7467ab466979d99af0bfd66b64a2ee6e9 and stable measured context SHA256 b9b1fbb8e408a50fb5b37b37b4c386effbc4fc6f922a7644875d3a5e069e32c5 match. Complete publication/source/descriptor/closure validation succeeded; current strict readout reports no common-condition mismatches. The inventory also checks registration/version/storage/source/RNG/context stability across capture; it is not a full post-training value rehash or proof of identical unmeasured compiler/kernel state.

Strict warm1 comparisons (unchanged atol1e-5, rtol0.016, relative-L2 limit0.03):

• Main versus independent main replica: output max absolute difference 1.14703369140625, relative L2 0.043881414172583; 2,290/6,330 output elements exceed the elementwise tolerance. All 660 gradient tensors fail at least one declared condition; maximum per-tensor relative L2 1.0555278532309724.
• Main versus Account for grouped checkpoint, head, and enclosing forward buffers #898 candidate: all 6,330 forward outputs are bit-for-bit identical. 442/660 gradient tensors pass; 218 fail. Maximum gradient absolute difference 0.000244140625, maximum per-tensor relative L2 0.053877846678882096.
• All three original within-actor repeatability checks also failed and remain authoritative. All three completed their three backward/clear stages; no OOM was observed in this small case. This is not near-cap memory qualification.

Consequences: neither candidate regression nor candidate numerical equivalence is established. Same-source baseline variability confounds attribution; successful offline reader exits preserve negative scientific results. Do not relax tolerances or promote a failed run to acceptance based on these observations.

Owned follow-up (Schulman): compare the retained exact device/native-library/compiler/runtime-context evidence to identify a meaningful controlled next test. Keep near-cap memory-admission testing moving separately. No art.megatron patch or underlying kernel cause is established.

Durable evidence:

• /var/tmp/art898-loaded-{main,candidate,main-replica}-native-v3-20260915-root/ — exact source plans, original raw logs, projections and independent closure receipts.
• /var/tmp/art898-loaded-v3-readout-20260915-root/main-main-replica-result.json and main-candidate-result.json — original strict complete 661-item comparisons.
• /var/tmp/art898-v3-gpu-local-audit-20260915-backward-hf8vvzuy/, manifest 5972ecd97e59410d29fd54b867909744880bf2d5b5a1909adf44294393fd21c8 — source/actor/terminal joins and preserved original failures.
• All twelve exact GPU job/configmap/secret/pod resource IDs and six local process groups were independently confirmed absent after these runs. No new live run is implied by this issue.