feat(ccb): a storage set commits which register histories it trusts, not just which nodes - #770
Merged
Merged
Conversation
…not just which nodes A set of bare member ids says only WHICH NODES a vault trusts. A member that lost and rebuilt its register still satisfied that — same identity key, same configured id — and could then assert "no claim here" for a cell the real incarnation once held. Nothing distinguished it, because owning the identity was the entire test. A set entry is now the pair `(member_id, register_incarnation_id)`, so the set id commits to which durable register histories on those nodes. `0x0002` goes to schema 3; entries sort by MEMBER ID alone and a duplicate member id is refused regardless of incarnation, so one member cannot appear twice under two incarnations. An all-zero incarnation — the value a node has before it has established one — is refused as uncommittable. `VaultStateV2` goes 3 -> 4 with it. The registry's own rule is that nesting is by complete CCB including the nested schema version, so a nested bump propagates upward whether or not the enclosing field list moved. Both superseded pairs are burned. Set ids and vault-state commitments therefore change; that is the reprovision. The incarnation is minted by the node itself on first boot, from the OS random source, into a write-once row committed durably, and is deliberately NOT derivable from the node id or the signing key — a node that kept its key but lost its register MUST come back different, because that is the fact a vault needs. It is established and logged unconditionally at startup, before any set is considered: establishing it only when a set exists would be a bootstrap that never starts, since an operator cannot write the member list without the value. Both directions are bound. Reads carry `x-dsm-register-incarnation` on the held AND absent branches — stamping only `held` would leave the dangerous case, a rebuilt member reporting emptiness, unmarked — and the reader counts an answer only when the member id and the incarnation both match. A mismatch is `Unavailable`: never an absence, never a value, and never an invalid-credit verdict, because the node is not lying, it simply is no longer the member this vault committed. Claims need no new machinery: a node derives its own set id from members including its own durable incarnation, so a rebuilt member computes a different id and the existing foreign-set check refuses before any insert. Root-register authority moves accordingly. Core keeps the network's member ids and quorum; a resolver supplies candidate `(member, incarnation)` pairs through `root_register_candidate_set`, and `RootRegisterProfile::derive_set_id` refuses any candidate whose membership is not the network's before computing an id. Endpoints stay transport metadata. Discovery-added nodes get an empty incarnation on purpose, which fails the catalog closed — discovery can carry transport and can never mint set membership. Fleet bring-up is now explicitly two-phase, because no generator can know an incarnation in advance: boot with `[storage_set]` absent (register inactive, every claim refused), collect each node's logged incarnation, write the member list everywhere, restart. A node whose configured incarnation is not the one its database holds refuses to start. Mutation controls, each restored with a verified tree stamp: - duplicate member id no longer refused -> invalid_inputs_are_refused_rather_than_normalized RED - all-zero incarnation no longer refused -> same test RED on its own assertion - the reader stops checking the incarnation half -> only_the_committed_member_in_its_committed_incarnation_counts RED - NodeStorageSet stops refusing a rebuilt incarnation -> a_node_whose_register_was_rebuilt_refuses_to_serve_the_configured_set RED - the absent branch stops carrying the stamp -> every_read_answer_names_the_register_incarnation_serving_it RED Not independently observable, and stated rather than claimed: the sort KEY (member id vs the whole pair) has no mutation, because duplicates are the only input where the two orders differ and the duplicate refusal fires first. Verification (Rust 1.98.0, the version rust-toolchain.toml declares): - root `make lint` exit 0 - workspace board 3982 passed, 0 failed - node board (local-dev,strict / SQLite) 288 passed, 0 failed - node board (default features / Postgres 14) 258 passed, 0 failed - production_safety_checks (incl. TLA+), ci_scan, no_clock_and_no_json, check_forbidden_symbols, codegen_enforce all exit 0
cryptskii
added a commit
that referenced
this pull request
Sep 5, 2026
…ories the real fleet holds (#771) #770 gave a vault's birth commitment `(member_id, register_incarnation_id)` pairs, but the network-wide root register still derived its set id from whatever candidate the catalog offered, constrained only to the right member ids. "Who chose the incarnation values?" answered "the resolved catalog candidate" — and `derive_set_id(&candidate) -> Result<[u8; 32]>` returned the id derived from its own input, which both call sites then used as the expected id. A checker that returns what it derived hands the authority to its input. `RootRegisterProfile` now carries `storage_set_id` as a pinned commitment, and `verify_candidate` replaces `derive_set_id` at every call site: the candidate's membership must be the network's AND the id re-derived from its pairs must equal the pin. A member that rebuilt its register passes the first and fails the second, and is refused (`SetIdIsNotThePinnedOne`) rather than silently resolved to a different register. Endpoints remain transport metadata. The pin is the three real `(member, incarnation)` pairs, not an opaque digest: member list and set id both derive from the one source, so they cannot drift, and an auditor can read which register histories are trusted. `pinned_root_register_members()` is public — a commitment, not a secret — and every fixture builds from it, so nothing holds parallel values that must match. Before the values existed the placeholder was all-zero, which the encoder already refuses; an unprovisioned network fails closed by construction (`ProfileNotDerivable(ZeroRegisterIncarnation)`), not by anyone remembering to check. The values came off the provisioned nodes in the owner's order. Each member: forensic `pg_dump`, `DROP SCHEMA public CASCADE`, boot on merged `f00d1e0c` with `[storage_set]` absent, mint and log the incarnation. Then the collected pairs were written as `[[storage_set.members]]` to every node and to the client config, and every node restarted into the set logging the identical id `E05YS8101EJH33KY2CG625JJE8A0Z4GJNSEM335TX1XVTWM9RR8G` with no refusal. Two tests pin this end to end: the pinned bytes render to exactly the strings each node logged (cross-checked against the raw `register_incarnation` rows), and core's derivation equals the id every node logged. Mutation control: disabling the pin comparison turned `one_member_rebuilding_its_register_changes_the_set_id` red (`got Ok(())` — the rebuilt member was accepted); restored on an identical tree stamp. Not changed, surfaced: no node sets `node.network_id`, so the faucet-ticket register is inactive fleet-wide until it is configured. A fail-open in `main.rs` (optional config + a defaulted `database.url` that quietly points at localhost) is filed separately; it turned a wrong-image build into a crash loop during provisioning rather than a refusal. Verification (Rust 1.98.0, the version rust-toolchain.toml declares): - root `make lint` exit 0 - workspace board 3984 passed, 0 failed - node board (local-dev,strict / SQLite) 288 passed, 0 failed - node board (default features / Postgres 14) 258 passed, 0 failed - production_safety_checks (incl. TLA+), ci_scan, no_clock_and_no_json, check_forbidden_symbols all exit 0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item B of the locked order, implementing both decisions exactly as you specified them:
0x0002schema 3 with paired entries, and the incarnation checked on reads and claim acceptance.The fact being committed
A set of bare node ids says only which nodes a vault trusts. A member that lost and rebuilt its register still satisfies that — it owns the same identity key and the same configured id — and can then assert "no claim here" for a cell the real incarnation once held. Nothing distinguishes it, because owning the identity was the whole test.
A set entry is now the pair:
So the set id commits not to which nodes, but to which durable register histories on those nodes.
Sorting is by
member_idalone, never by the pair — sorting on the whole entry would let one member appear twice under two incarnations and still produce a strictly ascending list, which is the exact ambiguity this removes. There is a named test for that case.Schema bumps, and why two
0x0002goes 2 → 3.0x0001(VaultStateV2) goes 3 → 4, because the registry's own §2.7 rule is that nesting is by complete CCB including the nested schema version, so a nested bump propagates upward whether or not the enclosing field list moved. Both superseded pairs are burned;a_nested_schema_bump_changes_the_enclosing_encodingis the control that would have caught skipping it.Set ids and vault-state commitments therefore change. That is the reprovision you accepted, and it is the point.
Where the incarnation comes from
A node generates it once, from the OS random source, into a write-once row in its own database, committed durably. It is deliberately not derivable from the signing key, the node id, or any seed — a node that kept its identity key but lost its register database must come back different, because that is the fact a vault needs.
Startup refuses when
[[storage_set.members]]lists an incarnation for this node that its database does not hold. That is the rebuild made loud, at the one moment an operator is looking, instead of silent at read time.Both directions bound, as you required
Reads — every answer carries
x-dsm-register-incarnation, on theheldandabsentbranches alike. Stamping onlyheldwould leave the dangerous case — a rebuilt member reporting emptiness — unmarked. The reader's decision is one function with the full matrix as its contract:UnavailableUnavailable(attribution)UnavailableAn incarnation mismatch is
Unavailable— never absence, never a value, and never an invalid-credit verdict. The node is not lying; it simply is no longer the member this vault committed.Claims — the vault names a
storage_set_idthat is now a function of the pairs, and a node derives its own set id from its members including its own durable incarnation. A rebuilt member computes a different id and refuses the claim asforeign-setbefore any insert, so it cannot write into the old vault's register namespace and have the discontinuity surface only later on reads.Root register
Per your ruling: core
dsmstays config-free and keepsBETA_MEMBERS+ quorum as the authority over which members; the catalog supplies each member's incarnation;RootRegisterProfile::derive_set_idre-derives the id from the candidate pairs and refuses any candidate whose membership is not the network's. Endpoints stay transport metadata and are not inputs. BothProvenanceResolverandPeerEvidenceFetchergainedroot_register_candidate_set, named "candidate" so no call site reads it as authority.Discovery-added nodes get an empty incarnation deliberately, which fails the catalog closed — discovery can carry transport and can never mint set membership.
Open assumption
Your answer says the profile "commits
storage_set_id". Core cannot compute that digest without incarnations, and pinning it as a literal needs a real provisioned value I will not invent. So the committed authority is currently membership + quorum in core, incarnations from the catalog, id re-derived and checked. If you want an additionally-pinned digest once the beta fleet is provisioned, that is a value you would hand me.Mutation controls
Each restored with a verified tree stamp (
git diff | shasumidentical before and after).invalid_inputs_are_refused_rather_than_normalized— the "one member under two incarnations" assertiononly_the_committed_member_in_its_committed_incarnation_counts— on the SAME-NODE-REBUILT-REGISTER rowNodeStorageSetstops refusing a rebuilt incarnationa_node_whose_register_was_rebuilt_refuses_to_serve_the_configured_setevery_read_answer_names_the_register_incarnation_serving_itStated rather than claimed: the sort key — member id alone vs the whole pair — has no mutation control, because duplicates are the only input where the two orders differ and the duplicate refusal fires first. The load-bearing gate is that refusal, which is tested above. The sort is documented as belt-and-braces, not asserted as independently load-bearing.
Fleet bring-up is now two-phase
No generator can know an incarnation in advance, so
generate_node_configs.shno longer emits a member list andproduction.tomldocuments the sequence: boot with[storage_set]absent (register inactive, every claim refused — the correct state for a fleet that has not agreed on a set), collect each node's loggedregister incarnation for node <id>: <Base32-Crockford>, write the member list into every node config and the client env config, restart. A node whose configured incarnation is not the one its database holds refuses to start, so a mistake is loud.The incarnation is established and logged unconditionally at startup, before any set is considered — establishing it only when a set exists would be a bootstrap that never starts.
Verification
Rust 1.98.0, the version
rust-toolchain.tomldeclares.make lint— exit 0--release -- --test-threads=1— 3982 passed, 0 failedlocal-dev,strict) — 288 passed, 0 failedproduction_safety_checks.sh(incl. TLA+),ci_scan.sh,no_clock_and_no_json.sh,check_forbidden_symbols.sh,codegen_enforce.sh— all exit 0Dependent sweep, and a self-inflicted regression worth naming. Making
register_incarnationa required config field broke 106dsm_sdktests — every fixture that writes a fleet TOML. The existing CLAUDE.md rule fires on capability removal; a newly-required field reads as an addition and removes exactly as much. A refinement candidate is filed to widen that trigger. A further 11 "failures" after the fix were my own error — I ran with--test-threads=4when CLAUDE.md documents these suites share process-global singletons; serially they pass.