Skip to content

Add cuda.coop Exchange and Shuffle with Numba-CUDA-MLIR - #11208

Open
tpn wants to merge 10 commits into
codex/cuda-coop-numba-block-load-store-algorithmsfrom
codex/cuda-coop-numba-exchange-shuffle
Open

tpn wants to merge 10 commits into
codex/cuda-coop-numba-block-load-store-algorithmsfrom
codex/cuda-coop-numba-exchange-shuffle

Conversation

@tpn

@tpn tpn commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Why this is needed

Kernels often need to rearrange per-thread values after loading them. This
adds common Exchange and Shuffle APIs with their Numba-CUDA-MLIR providers,
using the group, storage, and synchronization contracts from #11200.

What changes

  • Common exchange converts blocked and striped layouts for Block, physical
    Warp, and logical Warp groups. The backend API also supports Block-only
    warp-striped, scatter, guarded/flagged, ranks/flags, and warp-time-slicing
    forms.
  • Common Block shuffle provides unit up and down operations. The backend
    API adds scalar offset and rotate forms.
  • Both operations return fresh payloads and preserve inputs, including ranks
    that CUB uses internally. Mode selectors are lowercase strings.
  • Storage-consuming operations use per-group scratch and the corresponding
    Block or exact-mask Warp reuse barrier. Payload shapes, dtypes, ranks,
    flags, selectors, and runtime distance ABIs are checked before compilation.
blocked = coop.exchange(
    coop.this_block(), striped, mode="striped_to_blocked"
)
shifted = coop.shuffle(coop.this_block(), blocked, mode="down")

Compile tests verify barrier placement and masks at all six Warp widths.
Runtime tests repeatedly reuse scratch while sibling groups execute different
iteration counts, with each group converged. Opt-in racecheck tests compare
production barriers with a test-only control that removes the reuse barrier.

Warp scatter and Shuffle boundary-output projections remain unsupported.
Invalid runtime rotate bounds or offset values outside the signed-32-bit
provider ABI trap the device; failure tests use isolated processes.

Stacked on #11200.

Validation

Fresh installed-wheel checks at 0ce85aab9385:

  • Host contracts and packaging (full suites): 516 passed on Python 3.14.
  • Numba backend unit/compiler tests (focused, GPUs hidden): 92 passed on Python 3.14.
  • GPU runtime tests (focused): 11 passed on Python 3.14.

Wheel validation and source, installed-package, and bundled-header comparisons passed.

Remote CI passed at 0ce85aab9385: GitHub Actions run. All exact-head checks and statuses, including pre-commit, are successful or conditionally skipped.

@copy-pr-bot

copy-pr-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@cccl-authenticator-app cccl-authenticator-app Bot moved this from Todo to In Progress in CCCL Sep 4, 2026
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch 2 times, most recently from a1439f9 to 8ac30b9 Compare September 4, 2026 18:05
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 7587282 to 03793c5 Compare September 4, 2026 18:10
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from 8ac30b9 to 74015ae Compare September 6, 2026 06:42
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch 3 times, most recently from 7f12cca to e886b91 Compare September 9, 2026 20:58
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from 20f5740 to 80046aa Compare September 9, 2026 20:58
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from e886b91 to a7bfc09 Compare September 9, 2026 21:29
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch 2 times, most recently from 9ed3305 to deecba2 Compare September 9, 2026 22:02
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from a7bfc09 to e8e866c Compare September 9, 2026 22:02
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from deecba2 to f123aa7 Compare September 10, 2026 21:18
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch 2 times, most recently from 34c77ae to 7156b39 Compare September 14, 2026 17:11
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from f123aa7 to 5b06211 Compare September 14, 2026 17:11
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 7156b39 to be57fc5 Compare September 14, 2026 18:41
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from 5b06211 to c45cdeb Compare September 14, 2026 18:41
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from be57fc5 to bdfb35d Compare September 15, 2026 01:57
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch 2 times, most recently from 8e455be to 72fe594 Compare September 15, 2026 18:01
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch 2 times, most recently from 41dc5e8 to 69d0af5 Compare September 15, 2026 23:45
@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from 72fe594 to 49c5c1d Compare September 15, 2026 23:45
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 69d0af5 to f38048f Compare September 16, 2026 03:34
@tpn

tpn commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Regarding the three suppressed comments in this review:

  • Warp scatter is outside this API's supported surface. Physical and logical Warp groups accept only blocked_to_striped and striped_to_blocked; _mode_token() rejects scatter before provider selection. The WarpGroup overload also requires ranks=None, and the existing physical/logical rejection test checks this boundary. The lowerer's rank-copy branch does not make Warp scatter publicly supported, so no runtime case or API expansion is needed here.
  • Added the Exchange/Shuffle implementation modules, stubs, and Block Exchange, Block Shuffle, and Warp Exchange headers to the standalone wheel validator. The wheel build now checks these files directly.
  • Kept scalar Shuffle dtype inference unchanged. The frontend passes an ir.Var; GroupPlanningContext.dtype() follows its definitions, and _dtype_definition() handles ir.Const through _python_scalar_dtype. NumPy scalar constructors such as np.int32(4) use the existing scalar-cast path. The claimed missing constant-dtype path is therefore already implemented.

@github-actions

This comment has been minimized.

@tpn
tpn force-pushed the codex/cuda-coop-numba-logical-warp-load-store branch from 120a402 to cab96ed Compare September 18, 2026 19:15
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 2e1232c to 21b0b9d Compare September 18, 2026 19:15
@github-actions

This comment has been minimized.

@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 21b0b9d to 0ced14d Compare September 18, 2026 21:53
@tpn
tpn requested a review from a team as a code owner September 18, 2026 21:53
@tpn tpn changed the title cuda.coop: add Numba Exchange and Shuffle Add cuda.coop Exchange and Shuffle with Numba-CUDA-MLIR Sep 18, 2026
@tpn
tpn changed the base branch from codex/cuda-coop-numba-logical-warp-load-store to codex/cuda-coop-numba-block-load-store-algorithms September 18, 2026 21:56
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

tpn added 10 commits September 19, 2026 13:18
Define backend-neutral CUB BlockExchange, WarpExchange, and
BlockShuffle specializations. Add portable group contracts for
out-of-place results, storage, synchronization, and preconditions.

Signed-off-by: Trent Nelson <trent@trent.me>
Register family-owned inference and provider selection for block and
Warp Exchange plus block Shuffle. Lower storage-bearing CUB calls with
scope-aware reuse barriers, input preservation, checked runtime Rotate
distances, and exact logical-Warp masks.

Signed-off-by: Trent Nelson <trent@trent.me>
Exercise installed-wheel exports, strict typing, metadata, and package
contents for both portable and qualified APIs. Cover invalid selectors,
payloads, ranks, flags, and scalar Shuffle calls.

Signed-off-by: Trent Nelson <trent@trent.me>
Describe supported layouts, scatter preconditions, storage ownership,
barriers, Shuffle boundaries, and runtime Rotate traps. Update API
coverage without promising deferred boundary results.

Signed-off-by: Trent Nelson <trent@trent.me>
State the signed int32 provider ABI for Offset and distinguish ABI
overflow traps from ordinary out-of-block source behavior. Exercise both
overflow directions in isolated subprocesses.

Signed-off-by: Trent Nelson <trent@trent.me>
Require string selectors across portable and qualified APIs, including
planner, rewrite, and provider validation. Reuse the portable Exchange
mode alias and keep qualified extensions block-only so Warp retains the
shared layout contract.

Signed-off-by: Trent Nelson <trent@trent.me>
Accept structural read-only payloads for Exchange and Shuffle inputs,
including Exchange ranks and valid flags. Keep mutable result typing for
both portable and qualified APIs.

Reject enum-backed modes and distance compatibility objects before
provider selection so runtime validation matches literal-only stubs.

Signed-off-by: Trent Nelson <trent@trent.me>
Mirror portable input and result annotations in qualified markers.

Keep Block Exchange ranks and flags as qualified-only controls.

Treat time slicing the same way and assert the shared runtime contract.

Signed-off-by: Trent Nelson <trent@trent.me>
Signed-off-by: Trent Nelson <trent@trent.me>
Check production barrier ordering and masks at every logical warp width.
Exercise repeated exchanges with independently progressing groups, and
require racecheck to detect the test-only missing-barrier control.

Signed-off-by: Trent Nelson <trent@trent.me>
(cherry picked from commit ae5ec63eb17220ce8a1b46369f7a8470e040e271)
@tpn
tpn force-pushed the codex/cuda-coop-numba-block-load-store-algorithms branch from fadf5c0 to ecec1c5 Compare September 20, 2026 02:56
@tpn
tpn force-pushed the codex/cuda-coop-numba-exchange-shuffle branch from 0ced14d to 0ce85aa Compare September 20, 2026 02:56
@github-actions

Copy link
Copy Markdown
Contributor

🥳 CI Workflow Results

🟩 Finished in 19m 17s: Pass: 100%/8 | Total: 1h 03m | Max: 18m 40s

See results here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

2 participants