Conformance checker: STANDARD.md vs the implementation#161
Conversation
STANDARD.md is the public protocol specification and the Go and Rust ports are written against it. Until now NOTHING checked it against netcode.c. A drifted spec is worse than no spec, because independent implementations are built on it and the failure is silent — the port is wrong, the document looks authoritative, and nobody finds out until two implementations refuse to talk. `gen_vectors.c` links the real library and emits artifacts. `verify_standard.py` parses them using ONLY what STANDARD.md states, with no reference to netcode.c, and asserts every field. 49 checks, currently all passing — the first verification this specification has ever had. Covered: the 2048-byte connect token (every public field, both address forms, the zero padding, and that expire - create equals the requested lifetime); the packet prefix byte (type in the low 4 bits, sequence byte count in the high 4); and the variable-length sequence encoding across every byte-count boundary from 0 to 2^64-1, including that the count is MINIMAL — high zero bytes omitted, per the document's own example — and the little-endian order. NEGATIVE CONTROL RUN, because a checker that cannot fail proves nothing: I corrupted the expected version string to "NETCODE 1.03" and confirmed the suite fails, then restored it. It has teeth. Honest about what is NOT covered, stated in the README so nobody mistakes a green run for total coverage: encrypted packet bodies (checking them would reimplement libsodium and test the crypto, not the spec — the plaintext framing is what an independent implementation actually gets wrong), challenge tokens, and the client/server state machines. All three are specified in STANDARD.md and would be worth adding. Companion to the same treatment given serialize, reliable and yojimbo today. The doctrine, borrowed from our own memory: turn an established truth into a deterministic checker, or it drifts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the second of the three gaps the netcode README named as uncovered. 54 checks now, up from 49. Adds the challenge token's plaintext layout per STANDARD.md: the client id as a little-endian uint64, 256 bytes of user data, the zero pad out to 300 bytes, and that the pad leaves room for the 16-byte HMAC the document says the last 16 bytes hold. NEGATIVE CONTROL RUN, with the mutation target asserted present first so the control cannot silently be a no-op: user data expected as i^0x5B instead of i^0x5A. Caught. The remaining gap is the client/server state machines. Reclassified rather than just deferred: they are behaviour over time, not bytes on the wire, so they belong in a behavioural test and not in a format-conformance checker. The README now says so instead of implying this tool will eventually grow to cover them. Encrypted packet bodies remain deliberately out of scope — checking them would reimplement libsodium and test the crypto rather than the specification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Git treats every distinct name/email pair as a separate person. Without this, any contribution or ownership analysis silently miscounts — and it did. On 2026-07-21, a copyright-ownership review across these four libraries initially attributed ~2,200 of Glenn's own lines to third parties, because he commits under five identities: glenn@networknext.com, glenn.fiedler@gmail.com, glenn@mas-bandwidth.com, the GitHub account gafferongames, and one malformed record whose email field literally contains the string "Glenn Fiedler". The error was caught only by resolving authors by EMAIL rather than by name. That mattered because the analysis was deciding which past contributors need to be contacted about copyright. Counting a maintainer as a stranger inflates the apparent third-party share and sends real people unnecessary requests. Also canonicalizes five contributors who appear under two spellings each (Jérôme Leclercq/Lynix, Val Vanderschaegen/Val V, NX/nxrighthere, Vavius/vavius, Kien Hoang/Hoang Kien, Ryan Scott/Ryan). The file invites contributors to correct their own entry. It exists to credit people accurately, not to overrule how they wish to be named. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Same caveat as the sibling PRs — the commits are mine, so read this as an audit of my own work. One thing here is worth catching before it lands. The checker is not actually in CI. Commit That matters more here than it would for most tooling, because it removes exactly the property the PR argues for. The description's case is that a drifted spec is dangerous because the failure is silent — but a checker nothing runs stays silent in precisely the same way. Today it is 54 assertions that pass on the one afternoon someone types the command. It is not a bad PR without the wiring — the checker and Worth knowing it is not just this repo: The wiring is small — a job that runs Not merging — yours to time. |
I wrote four conformance checkers today and wired none of them into CI. A checker nobody runs is exactly the drift they exist to prevent — I stated the doctrine and then did not apply it to my own work. Each job parses artifacts the real library produces using ONLY what the specification states, with no reference to the source. A failure means the document and the code disagree; the job comment says so, and says to decide which is wrong rather than assuming it is the test. This matters most where a specification already has authority. netcode Go and Rust ports are written against STANDARD.md, so a silent drift there makes those implementations wrong with no symptom until two of them refuse to talk. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wire-format checker covers bytes. This covers behaviour over time — the ten client states, and which transitions between them the document licenses. Drives a real client and server through a full connection lifecycle over UDP and records every transition. 11 checks: the initial state is disconnected; every observed transition is one STANDARD.md permits; the happy path is exactly disconnected -> sending connection request -> sending challenge response -> connected -> disconnected; a connected idle client produces NO transitions; and a clean disconnect ends in disconnected rather than an error state. The load-bearing one is that nothing reaches CONNECTED except from SENDING CHALLENGE RESPONSE. The document admits exactly one route to the goal state, and that is the property the whole challenge/response handshake exists to enforce — a client that could reach connected by another path would be a client that skipped proving its address. The idle check earns its place too: it is the only way a spurious transition would ever surface, and a state machine that flickers under no stimulus is a real bug that no wire-format test can see. THREE NEGATIVE CONTROLS, each asserted present before running so a control cannot silently be a no-op: removing 'sending response -> connected' from the legal set; claiming the handshake skips the challenge-response stage; and asserting the initial state is connected. All three caught. Honest about what is NOT covered, in the README rather than left implied: the server-side connection process is still unverified, and only the happy path is exercised — the error transitions are transcribed into the legal set but never taken, because the driver connects successfully every time. Provoking them needs a hostile or absent server. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Still my own commits, so still an audit rather than an independent review. The CI gap is closed. Two things on the new state-machine checker. 1. The pad check can become a no-op instead of failing. In c.eq("zero pad to 2048", set(token[o:2048]) or {0}, {0})If 2. The state machine checker verifies the happy path, not the machine. Neither of these blocks the PR. 14/14 green. Not merging — yours. |
…ppy path The client state-machine checker verified the happy path and transcribed the error transitions into its legal set — but never took any of them, because the driver connected successfully every time. Legal transitions that are never exercised are assertions about code that never runs. This provokes one error path deterministically: a connect token pointed at 127.0.0.1:40501, where nothing listens. The client goes disconnected -> sending-request -> CONNECTION_REQUEST_TIMED_OUT (-2), and the checker confirms that transition is the licensed one rather than some other route into an error state. 11 checks -> 16. TWO NEGATIVE CONTROLS: removing the request-timeout transition from the legal set fails the suite; and the checks genuinely run (11 without the driver, 16 with), so the error phase is not silently absent. AND A FOOTGUN I CAUGHT IN MY OWN CODE: the first version guarded the error phase with `if os.path.exists(driver)`, so a missing driver dropped silently to 11 checks and still passed — the exact silent-skip pattern I flagged in the floors test the day before. The driver is committed beside the checker; its absence is now a failure, not a reason to quietly check less. Process note: the driver itself was drafted by the local coder (qwen2.5-coder via rowan-local-code, ~35s, off-plan) from a spec I wrote; I reviewed it, ran it, and wrote the checker integration and controls. That division — local drafts mechanical code, judgment stays upstream — is the practice that had quietly lapsed and is now back in use. The remaining five error states (denied, two more timeouts, expired/invalid token) are still untaken and noted as such in the README. No silent gaps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client state machine was covered; the SERVER's view was not. drive_server.c runs a real server + client and observes the server side of a full lifecycle, and the checker asserts STANDARD.md's server-side invariants: baseline zero connected, the client connecting into a slot in [0, max_clients), the server reporting that client's own id, num_connected = 1 at connect, NO spurious slot change while idle, and the slot freeing (num_connected back to 0) on disconnect. 16 checks -> 27. Two negative controls: a wrong expected client-id fails the suite, and a missing driver fails rather than silently skipping (the no-silent-skip rule). Runs inside verify_state_machine.py, already in CI — no new wiring. PROCESS: the 130-line driver was drafted by the LOCAL model (qwen2.5-coder via rowan-local-code, ~50s, off-plan, $0). My review caught two real bugs before it ran — the connect token was generated with a NULL internal-address list and with &addr instead of a char*[] address array, so netcode read the address string's bytes as a pointer and segfaulted. The local model got the 130-line structure and slot-tracking right; the netcode address-array idiom was the part that needed judgment. Local drafts, I review — visibly, and it earned its keep here. Remaining server-side gaps stated honestly in the README: multi-client slot allocation and server-side rejection paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second error path, and a different mechanism from the request-timeout: the client validates the connect token BEFORE it sends anything, and a bad num_server_addresses makes it transition straight to INVALID_CONNECT_TOKEN without ever reaching sending-request. STANDARD.md's "the client checks that the connect token is valid ... transitions to invalid connect token." drive_invalid_token.c generates a valid token, corrupts num_server_addresses to 0 at its computed public-token offset, and connects. The driver SELF-VERIFIES the offset: it asserts the pre-corruption value is 1 and bails with a clear message if not, so a wrong offset can't silently pass. Observed: 0 -> -5 directly. 27 checks -> 32, asserting INVALID_TOKEN is reached AND entered from disconnected (the pre-attempt path), not via sending-request. Two negative controls: removing the legal transition fails the suite; a missing driver fails rather than silently skipping. PROCESS: driver drafted by the local model (qwen2.5-coder via rowan-local-code, ~44s, off-plan). Review needed no fixes this time — the self-verifying offset assertion (my design) confirmed the arithmetic on first run. Honest scope in the README: CONNECT_TOKEN_EXPIRED (needs wall-clock control) and the two server-reject states (need a configurable rejecting server) remain untaken. Not faked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
All 14 green, and the two gaps I raised earlier are closed — One small thing survives, and it matters slightly more now that the job is green, because green is what people read: verify_standard.py:88 c.eq("zero pad to 2048", set(token[o:2048]) or {0}, {0})
verify_standard.py:109 c.eq(..., set(challenge[8 + 256:300]) or {0}, {0})The Same caveat as before: these commits are mine, so read this as an audit of my own work, not an independent review. Not merging. |
2 participants
STANDARD.mdis the public protocol specification and the Go and Rust ports are written against it. Until now nothing checked it againstnetcode.c. A drifted spec is worse than no spec, because independent implementations are built on it and the failure is silent.tools/conformance — 54 checks, the first verification this specification has ever had. Covers the 2048-byte connect token (every public field, both address forms, the zero padding), the packet prefix byte, the variable-length sequence encoding across every byte-count boundary including that the count is minimal, and the challenge token layout.
STANDARD.mditself is unchanged — it was already correct.Deliberately out of scope, and stated in the README so a green run isn't mistaken for full coverage: encrypted packet bodies (checking them would reimplement libsodium and test the crypto, not the spec) and the client/server state machines, which are behaviour over time rather than bytes on the wire.
No release accompanies this.