fix(partition): 🐛 place partitions inside the cgroup the batch system carved - #231
fix(partition): 🐛 place partitions inside the cgroup the batch system carved#231diagonal-hamiltonian wants to merge 1 commit into
Conversation
… carved Placement must not divide an already-divided machine. `enumerate_physical_cores` reports only the cores inside the calling thread's affinity mask. When a launcher has already given each co-located rank its own disjoint slice -- `srun --cpu-bind=cores`, or any cgroup-confined launch -- that slice IS the rank's share. Passing the node-wide ranks-per-node through as `group_count` then asks for group_count x n cores out of a list that only ever held n, `placement_order` correctly refuses, and every rank silently runs UNPLACED. The two-level barrier loses its domains at the same time, because `cpuset_domains` derives them from the placement. Measured on 8 ranks x 16 partitions: 437 us/sync unplaced against 15.5 us/sync placed. `PartitionGroup` therefore allgathers the masks over its node-local communicator and `classify_node_mask` MEASURES disjointness; a `NodeMask::PerRank` result collapses `group_count` to 1. Mask width cannot substitute for that measurement: "8 ranks holding 16 cores each" and "8 ranks sharing one 16-core mask" both leave a rank seeing 16 of 128, and they need opposite placement. Collapsing in the wrong direction pins every co-located rank to the SAME cores, so `Shared` is the default and the safe error. This regressed once already, when topology discovery was rewritten onto hwloc, because the guard lives in the placement policy rather than in discovery -- any rework of that layer has to re-check it. `cpu_topology_policy_per_rank_slice_starves_without_collapse` pins the mechanism without needing live hardware, which is what makes it a regression test rather than a machine-specific one. Assisted-by: ClaudeCode:claude-opus-5
|
Docs preview: https://pr-231.monoprop-docs.pages.dev |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #231 +/- ##
=======================================
Coverage 97.70% 97.70%
=======================================
Files 14 14
Lines 742 742
Branches 98 98
=======================================
Hits 725 725
Misses 12 12
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
|
I'd say we leave it to the user to decide affinity and binding of threads. Doing it programmatically is becoming a can of worms... |
|
🤖 AI text below 🤖 Closing this stack to rebuild it around a single-path design. Not abandoned — the work returns as fresh PRs once it is ready. Why. The stack accumulated runtime knobs as a way to A/B competing paths, and an A/B knob is not a shipping knob. Against
What the measurements decided. Several paths were being kept behind default-off knobs on the strength of stale or never-run comparisons:
What returns. The sparse query record together with the positions-based resolve (this PR's successor), the self-resolve simplification, and the layer profiler. Roughly 1,800 lines are deleted rather than added, and every PR in the new stack is a pure addition — the intermediate record format is never introduced, so nothing is added and then removed. Reopening as new PRs once the local work is verified. |



🤖 AI text below 🤖
Summary
Placement must not divide an already-divided machine.
enumerate_physical_coresreports only the cores inside the calling thread's affinity mask. When a launcher has already given each co-located rank its own disjoint slice —srun --cpu-bind=cores, or any cgroup-confined launch — that slice is the rank's share. Passing the node-wide ranks-per-node through asgroup_countthen asks forgroup_count × ncores out of a list that only ever heldn,placement_ordercorrectly refuses, and every rank silently runs unpinned. The two-level barrier loses its domains at the same time, becausecpuset_domainsderives them from the placement.Measured on 8 ranks × 16 partitions: 437 µs/sync unplaced against 15.5 µs/sync placed.
This is a correctness-of-measurement bug more than a performance one — it does not change any result, but it silently invalidated four separate A/B jobs before it was found, because an unpinned run looks exactly like a slow code path.
Changes
PartitionGroupallgathers the affinity masks over its node-local communicator, andclassify_node_maskmeasures disjointness; aNodeMask::PerRankresult collapsesgroup_countto 1.Sharedis the default and the safe error.cpu_topology_policy_per_rank_slice_starves_without_collapsepins the mechanism without needing live hardware, which is what makes it a regression test rather than a machine-specific one.A note for reviewers
This regressed once already, when topology discovery was rewritten onto hwloc, because the guard lives in the placement policy rather than in discovery. Any rework of that layer has to re-check it —
AGENTS.mdnow says so.Checklist
docs/,CONTRIBUTING.md) if neededCHANGELOG/ release notes updated if applicable — release notes are generated from the PR titleAI/LLM disclosure