Skip to content

fix(partition): 🐛 place partitions inside the cgroup the batch system carved - #231

Closed
diagonal-hamiltonian wants to merge 1 commit into
mainfrom
pr/partition-cgroup-placement
Closed

fix(partition): 🐛 place partitions inside the cgroup the batch system carved#231
diagonal-hamiltonian wants to merge 1 commit into
mainfrom
pr/partition-cgroup-placement

Conversation

@diagonal-hamiltonian

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Summary

Placement must not divide an already-divided machine.

enumerate_physical_cores reports only the cores inside the calling thread's affinity mask. When a launcher has already given each co-located rank its own disjoint slice — srun --cpu-bind=cores, or any cgroup-confined launch — that slice is the rank's share. Passing the node-wide ranks-per-node through as group_count then asks for group_count × n cores out of a list that only ever held n, placement_order correctly refuses, and every rank silently runs unpinned. The two-level barrier loses its domains at the same time, because cpuset_domains derives them from the placement.

Measured on 8 ranks × 16 partitions: 437 µs/sync unplaced against 15.5 µs/sync placed.

This is a correctness-of-measurement bug more than a performance one — it does not change any result, but it silently invalidated four separate A/B jobs before it was found, because an unpinned run looks exactly like a slow code path.

Changes

  • PartitionGroup allgathers the affinity masks over its node-local communicator, and classify_node_mask measures disjointness; a NodeMask::PerRank result collapses group_count to 1.
  • Mask width deliberately cannot substitute for that measurement. "8 ranks holding 16 cores each" and "8 ranks sharing one 16-core mask" both leave a rank seeing 16 of 128, and they need opposite placement. Collapsing in the wrong direction pins every co-located rank to the same cores, so Shared is the default and the safe error.
  • cpu_topology_policy_per_rank_slice_starves_without_collapse pins the mechanism without needing live hardware, which is what makes it a regression test rather than a machine-specific one.

A note for reviewers

This regressed once already, when topology discovery was rewritten onto hwloc, because the guard lives in the placement policy rather than in discovery. Any rework of that layer has to re-check it — AGENTS.md now says so.

Checklist

  • Tests added or updated to cover the changes
  • Documentation updated (docstrings, docs/, CONTRIBUTING.md) if needed
  • CHANGELOG / release notes updated if applicable — release notes are generated from the PR title

AI/LLM disclosure

  • I used the following tool to help write this PR description: Claude Code (claude-opus-5)
  • I used the following tool to generate or modify code: Claude Code (claude-opus-5)

… carved

Placement must not divide an already-divided machine.

`enumerate_physical_cores` reports only the cores inside the calling
thread's affinity mask. When a launcher has already given each
co-located rank its own disjoint slice -- `srun --cpu-bind=cores`, or
any cgroup-confined launch -- that slice IS the rank's share. Passing
the node-wide ranks-per-node through as `group_count` then asks for
group_count x n cores out of a list that only ever held n,
`placement_order` correctly refuses, and every rank silently runs
UNPLACED. The two-level barrier loses its domains at the same time,
because `cpuset_domains` derives them from the placement.

Measured on 8 ranks x 16 partitions: 437 us/sync unplaced against
15.5 us/sync placed.

`PartitionGroup` therefore allgathers the masks over its node-local
communicator and `classify_node_mask` MEASURES disjointness; a
`NodeMask::PerRank` result collapses `group_count` to 1. Mask width
cannot substitute for that measurement: "8 ranks holding 16 cores each"
and "8 ranks sharing one 16-core mask" both leave a rank seeing 16 of
128, and they need opposite placement. Collapsing in the wrong
direction pins every co-located rank to the SAME cores, so `Shared` is
the default and the safe error.

This regressed once already, when topology discovery was rewritten onto
hwloc, because the guard lives in the placement policy rather than in
discovery -- any rework of that layer has to re-check it.
`cpu_topology_policy_per_rank_slice_starves_without_collapse` pins the
mechanism without needing live hardware, which is what makes it a
regression test rather than a machine-specific one.

Assisted-by: ClaudeCode:claude-opus-5
@github-actions github-actions Bot added documentation Improvements or additions to documentation cpp labels Aug 15, 2026
@diagonal-hamiltonian diagonal-hamiltonian added the test-in-draft Run CI even in Draft mode label Aug 15, 2026
@github-actions

Copy link
Copy Markdown

Docs preview: https://pr-231.monoprop-docs.pages.dev

@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (6abd839) to head (c51dcd9).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #231   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           
Flag Coverage Δ
cpp 97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@sonarqubecloud

Copy link
Copy Markdown

@robertodr

Copy link
Copy Markdown
Member

I'd say we leave it to the user to decide affinity and binding of threads. Doing it programmatically is becoming a can of worms...

@diagonal-hamiltonian

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

Closing this stack to rebuild it around a single-path design. Not abandoned — the work returns as fresh PRs once it is ready.

Why. The stack accumulated runtime knobs as a way to A/B competing paths, and an A/B knob is not a shipping knob. Against main's 3 environment knobs the branch reached 15. The codebase already states the rule this violates, in EnvConfig.h:

monoprop_COMPACT_RECORD, monoprop_SPARSE_RESOLVE and monoprop_INSERT_PREFETCH were removed once they had been measured … A knob that has no losing arm left is dead configuration surface.

What the measurements decided. Several paths were being kept behind default-off knobs on the strength of stale or never-run comparisons:

  • Sparse emit + the positions encode measure ~4% slower in emit_s than the dense path at both P=1 and P=8 (6/6 paired reps, p=0.031), with a Pauli structural null confirming the effect is real rather than code layout. The encode optimisation was genuine — 43 ns/push — but it optimised a path that loses by more than it recovers. Both are deleted.
  • Zobrist routing's recorded 2.8% emit_s regression turned out to be stale: it was measured on an emit path that has since been rewritten, and now reads flat. Its disposition is under a P sweep.
  • The query-dump capture and replay tooling chose the wire format and is not load-bearing for any shipped behaviour, so it does not ship.

What returns. The sparse query record together with the positions-based resolve (this PR's successor), the self-resolve simplification, and the layer profiler. Roughly 1,800 lines are deleted rather than added, and every PR in the new stack is a pure addition — the intermediate record format is never introduced, so nothing is added and then removed.

Reopening as new PRs once the local work is verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpp documentation Improvements or additions to documentation test-in-draft Run CI even in Draft mode

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants