test(cuda.core): synchronize IPC buffer initialization - #2657
Open
Andy-Jost wants to merge 1 commit into
Open
Conversation
Ensure the exporting process completes asynchronous allocation and initialization before an importing child accesses the shared buffer.
Andy-Jost
added
bug
|
rwgk
reviewed
Aug 19, 2026
| @@ -82,6 +82,8 @@ def test_main(self, ipc_mempool_device_x2, grant_access_in_parent): | |||
| buffer = mr.allocate(NBYTES, stream=dev1.default_stream) | |||
Contributor
There was a problem hiding this comment.
codex gpt-5.6-sol ultra found:
This allocates asynchronously on dev1.default_stream, while PatternGen(dev1, ...) creates a separate nonblocking stream and initializes the buffer there. The later dev1.sync() waits for both streams, but does not establish allocation-before-copy ordering. CUDA defines that cross-stream access as undefined unless explicitly ordered. CUDA documentation (https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/stream-ordered-memory-allocation.html).
I’d use one stream throughout:
stream = dev1.default_stream
buffer = mr.allocate(NBYTES, stream=stream)
pgen = PatternGen(dev1, NBYTES, stream=stream)
pgen.fill_buffer(buffer, seed=False)
stream.sync()That also avoids a device-wide barrier.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #1308.
Synchronize the exporting device after asynchronously initializing the IPC buffer and before spawning the importing process. CUDA IPC does not carry the exporter's stream ordering into the importer, so the previous test could access the allocation before its initialization completed and compare stale data.
Keep the existing flaky reruns because they cover general multiprocessing timeouts on busy CI runners, independently of this ordering race. The targeted test requires two peer-accessible, IPC-capable GPUs and is left to CI for validation.
Checklist