Refresh final TrainerRank admission and propagate counter failures - #899
Conversation
GitHub Actions
Error
GitHub Actions
Error
GitHub Actions
Active
* Bound native cache recovery by measured forward work Reparent the reviewed runtime and corrected budget fixture onto merged #899. No tree or runtime changes from a8691cd. * Ensure checkpoint slots before DP-local recovery retries * Type the scalar recovery-plan test double explicitly * Fix cache recovery test metadata and CI routing
|
Consolidated review record (automated agent review traffic removed 2026-09-17) Change. The microbatch selector could return a cached memory budget after a later check saw less memory. It rechecks the selected demand before returning, refusing when the fresh budget is insufficient; local memory-counter failures propagate through the availability reduction, preserving the original exception. Only Review. Faraday, McCarthy and Minsky cleared source correctness at merged head Findings addressed during review.
Deferred / follow-ups.
|
2 participants
The microbatch selector could return a cached memory budget after a later check observed less available memory. Recheck the selected demand immediately before returning it, retain that final admission evidence, and refuse when the fresh budget is insufficient.
A local memory-counter failure now participates in the existing distributed availability reduction, preserving the original exception object and explicit cause while healthy peers stop. The smallest-wave fallback also reaches its existing world agreement before propagating a subgroup failure. Preserving an earlier implicit exception context is outside this change's guarantee.
Only
art.trainer_rankand its tests change. Availability arithmetic and cache-release policy are unchanged. The final refresh adds one MAX/MIN pair per distributed selection and can refuse work earlier; fresh availability remains a sample, not a reservation. API andart.megatronare unchanged.Validation at
0f2134cd304a930f2323206cdad15788e91da41f: quality checks and hosted two-H200 validation pass. Retained focused evidence includes 75 maintained cases and six cases on each of two CPU/Gloo ranks; the test-only successors additionally passed their affected suites. Faraday, McCarthy and Minsky clear source correctness at the exact head. Reviewers reserve the behavioral merge decision for Brad. These checks do not establish full-model memory feasibility or a complete backward peak bound.Part of #848 and #870; neither issue is fully resolved by this change. The separate adaptive cache policy is in draft #900.