Fix Ethernet receiver teardown race

[Changho Hwang (chhwang)]

Summary

• Fix the close-vs-blocked-recv race where closing the receive socket before its worker exited could expose recv to EBADF or a reused file descriptor.
• Publish the stop state, shut down both sockets, join the receiver, and only then close both descriptors.
• Preserve exact-read framing, distinguish clean EOF from truncated frames and local shutdown, and publish genuine receiver failures to connection operations.
• Add focused host-only coverage for blocked receiver wakeup, bounded shutdown, repeated teardown, exact reads, EOF handling, and descriptor reservation/reuse.

Validation

• A100: full unit suite 40 passed / 6 skipped; 2,500 socket teardown cycles; 50x 8-rank Ethernet pingpong.
• MI300X: full unit suite 29 passed / 9 skipped; 2,500 socket teardown cycles; 3x 8-rank non-IB aggregate runs; 10x Ethernet sequences.
• No EBADF failures or hangs observed.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Changho Hwang (chhwang)]

/azp run mscclpp-ut

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Fixes Ethernet receiver teardown races by separating socket shutdown from descriptor closure.

Changes:

• Adds exact-read outcomes and non-closing socket shutdown.
• Publishes receiver failures and safely joins before closing descriptors.
• Adds teardown/EOF tests and migrates socket logging.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/core/bootstrap/socket.cc	Implements shutdown and exact-read handling.
src/core/connection.cc	Reorders teardown and propagates receiver errors.
src/core/include/connection.hpp	Adds receiver lifecycle/error state.
src/core/include/socket.h	Declares receive results and shutdown API.
test/unit/socket_tests.cc	Adds socket framing and teardown tests.

Suppressed comments (1)

test/unit/socket_tests.cc:153

• descriptorReused is effectively guaranteed whenever /dev/null opens successfully: dup2 succeeds even when receiverFd is still open because it silently closes the target first. As a result, this assertion cannot verify that close() released the receiver descriptor; check for EBADF directly instead.

    int reuseFd = ::open("/dev/null", O_RDONLY);
    const bool descriptorReused = reuseFd == receiverFd || (reuseFd >= 0 && ::dup2(reuseFd, receiverFd) == receiverFd);
    if (reuseFd >= 0 && reuseFd != receiverFd) ::close(reuseFd);
    if (descriptorReused) ::close(receiverFd);

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Binyang Li (Binyang2014)]

We will check the exception at the begin and the end of the function. Can we use a warper to make it simple?
For example:

template <typename Operation>
void runWithReceiverErrorCheck(Operation&& operation) {
  rethrowReceiverError();  // Failure already published before operation.
  std::forward<Operation>(operation)();
  rethrowReceiverError();  // Failure published while operation was running.
}

void EthernetConnection::flush(int64_t timeout) {
  runWithReceiverErrorCheck([&] {
    // Existing flush implementation.
  });
}

Or other splution

[Changho Hwang (chhwang)]

Added runWithReceiverErrorCheck and wrapped write, updateAndSync, and flush so each operation checks the receiver error before and after execution.

[Changho Hwang (chhwang)]

/azp run mscclpp-ut

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).