New RPCConnectionManager: Single in-enclave manager, no ringbuffer, OpenSSL-owned sockets#8007
Draft
eddyashton wants to merge 32 commits into
Draft
New RPCConnectionManager: Single in-enclave manager, no ringbuffer, OpenSSL-owned sockets#8007eddyashton wants to merge 32 commits into
RPCConnectionManager: Single in-enclave manager, no ringbuffer, OpenSSL-owned sockets#8007eddyashton wants to merge 32 commits into
Conversation
…ransport cert-deferred listening + ALPN + outbound client, RPCConnectionManager (AbstractRPCSessions). Not yet wired into enclave/run.cpp.
…wire RPCConnectionManager into enclave.h/run.cpp, delete RPCSessions/rpc_connections/tls_session. Full build green, 53/53 unit tests pass.
…:Cert::use) and request client cert on inbound for caller auth; add peer-cert capture test. Full build green, unit tests pass.
…xing localhost/[::1] interfaces (cpp, cpp_cose_only, common_ipv6 e2e). Add localhost/IPv6 binding tests.
… large response queued just before close_socket() is not truncated (fixes cpp/cpp_cose_only receipt 'server disconnected'). Add truncation test.
…e (ERR_clear_error) before each SSL op so a stale error from one connection cannot poison SSL_get_error for another (root cause of cpp/cpp_cose_only 'server disconnected'). Add persistent-connection + peer-cert tests.
… handler; branch udp interfaces to listen_udp. Clearly marked QUIC extension points (substrate for OpenSSL >=3.5 native QUIC). e2e_logging udp echo passes; full suite green.
…ection_manager
…ned quic_session.h/src/quic, udp.h + udp/msg_types.h + UDPImpl vestiges in run.cpp, dead RPC ringbuffer message enums (keep tcp::ConnID). Drop old-implementation comments. Fix build after cert.h use->configure_ssl rename + commit-callback include.
…ection_manager
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors CCF’s RPC/session stack to remove the ringbuffer-based TCP/UDP transport plumbing, introduce a SessionWriter seam for protocol sessions, and add a host-side OpenSSL-native epoll-driven transport (plus a minimal UDP datagram server) with a new RPCConnectionManager implementing AbstractRPCSessions.
Changes:
- Introduces
SessionWriter,PlaintextSession, andAbstractRPCSessionsto decouple protocol sessions from transport details and support source-address-aware inbound data. - Replaces the old host RPC connection plumbing with an OpenSSL-native server/session bridge and a new
RPCConnectionManager(including cert-deferred listening and outbound client creation). - Adds a focused
openssl_server_testunit test suite and removes legacy UDP/QUIC/ringbuffer transport code.
Custom instructions used
.github/copilot-instructions.md.github/instructions/reviewing.instructions.md
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/udp/msg_types.h | Removes UDP ringbuffer message types and sockaddr encode/decode helpers. |
| src/tcp/msg_types.h | Removes TCP ringbuffer message types; retains ConnID type. |
| src/quic/test/main.cpp | Deletes obsolete QUIC/OpenSSL call test. |
| src/quic/quic_session.h | Removes legacy QUIC session implementation tied to UDP ringbuffer. |
| src/node/node_state.h | Switches NodeState dependency from RPCSessions to AbstractRPCSessions; adds TLS includes. |
| src/node/jwt_key_auto_refresh.h | Updates to use AbstractRPCSessions and adds TLS includes. |
| src/node/http_node_client.h | Adds missing HTTP RPC context include. |
| src/http/http2_session.h | Migrates HTTP/2 session types from encrypted sessions to PlaintextSession + SessionWriter. |
| src/http/http_session.h | Migrates HTTP/1 session types from encrypted/unencrypted sessions to PlaintextSession + SessionWriter; removes TLS connect override. |
| src/http/http_proc.h | Removes enclave TLS-session include; uses internal logging header instead. |
| src/http/http_parser.h | Removes enclave TLS-session include; uses internal logging header instead. |
| src/host/udp.h | Removes legacy libuv-based UDP implementation. |
| src/host/tls/openssl_session_manager.h | Adds host-side bridge from OpenSSL connections to ccf::Session via SessionWriter. |
| src/host/tls/openssl_server.h | Adds epoll-driven OpenSSL-native TLS/TCP server with backpressure and cross-thread send/close/connect. |
| src/host/test/openssl_server_test.cpp | Adds unit tests covering handshake, round-trips, large transfers/backpressure, concurrency, and the session bridge. |
| src/host/run.cpp | Removes legacy RPC wiring and writes RPC addresses reported by the enclave; leaves a placeholder stub. |
| src/host/rpc_connections.h | Removes legacy ringbuffer/libuv RPC connections manager. |
| src/host/rpc_connection_manager.h | Adds RPCConnectionManager implementing AbstractRPCSessions, owning per-interface OpenSSL transports and UDP datagram servers. |
| src/host/datagram_server.h | Adds minimal epoll-based UDP datagram server (echo path, QUIC extension point). |
| src/enclave/tls_session.h | Removes enclave-side TLS session implementation. |
| src/enclave/session.h | Replaces Encrypted/Unencrypted sessions with PlaintextSession over SessionWriter. |
| src/enclave/session_writer.h | Adds SessionWriter abstraction for thread-safe outbound writes/close requests. |
| src/enclave/rpc_sessions.h | Removes legacy enclave-side RPCSessions and its ringbuffer handlers. |
| src/enclave/no_more_sessions.h | Adds soft-cap wrapper session that returns 503 and closes. |
| src/enclave/main.cpp | Extends enclave entry to return resolved RPC addresses to the host. |
| src/enclave/entry_points.h | Extends enclave entry point signature to include RPC addresses buffer. |
| src/enclave/enclave.h | Switches enclave to use RPCConnectionManager; binds/resolves RPC interfaces and reports addresses back to host. |
| src/enclave/client_session.h | Reworks outbound connect to use an injected connect callback rather than ringbuffer messages. |
| src/enclave/abstract_rpc_sessions.h | Introduces backend-agnostic RPC sessions interface for node components. |
| include/ccf/node/session.h | Extends session inbound API to include a source sockaddr (for datagrams). |
| CMakeLists.txt | Registers openssl_server_test as a unit test. |
…Config (so the enclave-side RPC manager receives it); track per-connection last_active in OpenSSLServer and sweep idle connections off the epoll timeout. Plumbed manager->bridge->server. idletimeout e2e passes.
…ection_manager
…ection_manager
…ection_manager
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the old split RPC networking path with an OpenSSL-native connection layer. Previously, socket handling lived in host-side libuv code while TLS and protocol sessions were driven through enclave-side ringbuffer messages and memory BIOs. With CCF now running as a single process, that split is no longer useful, so RPC sockets, TLS, protocol session creation, and per-interface policy now live behind a single RPC connection manager.
TLS now terminates at the connection layer. Protocol sessions receive plaintext and write responses through a
SessionWriter, so HTTP, HTTP/2, and custom protocols no longer own TLS state directly. This removes the RPC ringbuffer message path, the memory-BIO TLS session layer, and the old libuv RPC connection containers.The new transport is built around non-blocking sockets owned by OpenSSL, with epoll driving handshakes, reads, writes, buffered output, graceful close, outbound client connections, deferred certificate loading, and idle connection cleanup. Per-interface behavior such as certificates, session caps, metrics, HTTP parser settings, custom protocol dispatch, and outbound client creation is centralized in
RPCConnectionManager.UDP remains as a small datagram transport. The current QUIC/UDP echo behavior is represented as a datagram session, while custom UDP protocols are routed to per-peer sessions. Native QUIC is still future work (needs OpenSSL >= 3.5) rather than implemented here.
Structural Breakdown
OpenSSLServeris the low-level connection transport. It owns the socket fd,SSLobjects, epoll registrations, read/write buffers, handshake state, graceful-close state, outbound connect requests, certificate reload requests, and idle-timeout sweeping.OpenSSLSessionManageris the bridge from transport connections toccf::Session. It lazily creates sessions for inbound connections, registers outbound sessions, forwards plaintext bytes into sessions, implementsSessionWriter, and reports session closure back to the owner.RPCConnectionManageris the higher-level RPC owner. It replaces the old RPC session container and owns one transport bridge per TCP interface, plus UDP interface state. It applies per-interface caps, certificates, parser settings, app protocol selection, session metrics, custom protocol routing, UDP peer demux, and outbound client creation.SessionWriter,Session, andPlaintextSessionform the new session boundary. Sessions no longer encrypt/decrypt; they parse plaintext and emit plaintext responses to their writer. HTTP/1 and HTTP/2 sessions were moved onto this boundary.CustomProtocolSubsystemInterfacenow creates sessions from(ConnID, SessionWriter&)rather than a TLS context. This matches the new layering: custom protocols see plaintext and write through the transport-neutral writer.DatagramServeris the UDP socket/epoll transport.DatagramEchoSessioncontains the temporary echo behavior, andRPCConnectionManagermaps UDP peers to sessions for custom datagram protocols.Startup wiring moved accordingly. The enclave creates and owns the RPC manager, binds RPC interfaces, resolves actual bound addresses including ephemeral ports, and reports those addresses back through the enclave entry point so the host can write the RPC addresses file.
The removed files are the old RPC transport stack:
RPCSessions,TLSSession, host RPC connections, legacy UDP plumbing, the old QUIC session, and the TCP/UDP ringbuffer message types that were specific to the split RPC path. Ledger, consensus, and node-to-node uses of ringbuffer/libuv are not part of this change.