trainer_rank: exclude pending native CUDA blocks from memory credit

[bradhilton]

The native CUDA allocator can report a tensor as no longer allocated while its block is still waiting for work on another stream. Counting reserved - allocated as reusable memory credits those pending blocks before the allocator can reuse them.

Use one native memory_stats snapshot and credit reserved - active when allocated/active/reserved counters are consistent nonnegative integers. Missing or inconsistent native counters grant no cache credit; real CUDA API errors still propagate. Keep the existing physical free-memory read, 3% reserve, test-only allocation ceiling and collective ordering. Other allocator backends retain their previous policy, explicitly outside this fix's qualification.

Partial progress on #870: this excludes pending bytes only. It does not resolve private-pool ownership, split-block compatibility, contiguous allocation or external-library headroom. Completed events that the allocator has not yet collected can cause conservative refusal or splitting. No snapshot, synchronization, event query, cache release, public API, art.megatron, dependency or loss changes. Held #888 is not included. This PR is independent of #893/#894.

Validation: 26 targeted CPU tests pass on this exact standalone main-based tree, plus 34 independent source/CPU checks using retained native-layout evidence and API/error/ceiling/reduction boundaries. Ruff, format and diff checks pass. Retained native evidence establishes counter semantics; this candidate has not yet been exercised in a new live GPU workload. Actual prior failures and completed test-resource receipts remain preserved.

Exact head 97df0e071fdae9006096dc1004813c4fd1dc7ce0, base 2ebfc1c2a38dbeeaeefac998323b476989602942. Request explicit behavioral-eligibility assessment under Brad's standing conditions before any merge.

Durable evidence: /home/brad/.local/share/schulman/art870-active-credit-main-20260914-physical-headroom/ (read REPORT-CORRECTION.md alongside the original report) and /home/brad/.local/share/schulman/art870-credit-review-20260914-forward-7zvh91s1/REPORT.md.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. _available_memory_bytes credits reserved - active from one native memory_stats snapshot instead of reserved - allocated, so CUDA blocks awaiting cross-stream events are not counted as reusable cache. Malformed counters grant no credit; other backends unchanged. Partial progress on #870.

Review. Taravangian, Minsky and McCarthy reviewed the merged head 97df0e0 against the pinned PyTorch allocator source and each returned source-correctness CLEAR, but held automatic merge eligibility because the stricter credit intentionally changes refusal/splitting behavior. Evidence: owner's 26 targeted CPU tests plus 34 controls (inspected, not rerun); Minsky reran both test files (38 passed; negative control on main failed the 19 new cases); ruff/format/diff clean; Prek and two-H200 CI still running during review.

Deferred / follow-ups.

• TrainerRank reusable-cache admission can leave insufficient physical CUDA memory for backward library allocations #870 remains partially open: private-pool ownership, split-block compatibility, contiguous allocation and external-library headroom.
• Share budgeted CUDA cache recovery with gradient handoff #888, trainer_rank: preserve memory estimates and account for overlapping expert buffers #893, trainer_rank: retain completed split-forward memory peaks #894 remain separately held; not cleared by this merge.
• Not yet exercised in a live GPU workload; Minsky suggested telemetry if the counter-consistency check fires.