Skip to content

feat(ccb): a storage set commits which register histories it trusts, not just which nodes - #770

Merged
cryptskii merged 1 commit into
mainfrom
feat/register-incarnation-binding
Sep 5, 2026
Merged

feat(ccb): a storage set commits which register histories it trusts, not just which nodes#770
cryptskii merged 1 commit into
mainfrom
feat/register-incarnation-binding

Conversation

@cryptskii

Copy link
Copy Markdown
Collaborator

Item B of the locked order, implementing both decisions exactly as you specified them: 0x0002 schema 3 with paired entries, and the incarnation checked on reads and claim acceptance.

The fact being committed

A set of bare node ids says only which nodes a vault trusts. A member that lost and rebuilt its register still satisfies that — it owns the same identity key and the same configured id — and can then assert "no claim here" for a cell the real incarnation once held. Nothing distinguishes it, because owning the identity was the whole test.

A set entry is now the pair:

entry = CCB(member_id, register_incarnation_id)
entries sorted strictly by member_id
duplicate member_id forbidden, regardless of incarnation_id
storage_set_id = H_dom(DSM/storage-set, CCB(entries))

So the set id commits not to which nodes, but to which durable register histories on those nodes.

Sorting is by member_id alone, never by the pair — sorting on the whole entry would let one member appear twice under two incarnations and still produce a strictly ascending list, which is the exact ambiguity this removes. There is a named test for that case.

Schema bumps, and why two

0x0002 goes 2 → 3. 0x0001 (VaultStateV2) goes 3 → 4, because the registry's own §2.7 rule is that nesting is by complete CCB including the nested schema version, so a nested bump propagates upward whether or not the enclosing field list moved. Both superseded pairs are burned; a_nested_schema_bump_changes_the_enclosing_encoding is the control that would have caught skipping it.

Set ids and vault-state commitments therefore change. That is the reprovision you accepted, and it is the point.

Where the incarnation comes from

A node generates it once, from the OS random source, into a write-once row in its own database, committed durably. It is deliberately not derivable from the signing key, the node id, or any seed — a node that kept its identity key but lost its register database must come back different, because that is the fact a vault needs.

Startup refuses when [[storage_set.members]] lists an incarnation for this node that its database does not hold. That is the rebuild made loud, at the one moment an operator is looking, instead of silent at read time.

Both directions bound, as you required

Reads — every answer carries x-dsm-register-incarnation, on the held and absent branches alike. Stamping only held would leave the dangerous case — a rebuilt member reporting emptiness — unmarked. The reader's decision is one function with the full matrix as its contract:

echoed verdict
member A, incarnation X (committed) may count
member A, incarnation Y Unavailable
member B, incarnation X Unavailable (attribution)
either half missing Unavailable

An incarnation mismatch is Unavailable — never absence, never a value, and never an invalid-credit verdict. The node is not lying; it simply is no longer the member this vault committed.

Claims — the vault names a storage_set_id that is now a function of the pairs, and a node derives its own set id from its members including its own durable incarnation. A rebuilt member computes a different id and refuses the claim as foreign-set before any insert, so it cannot write into the old vault's register namespace and have the discontinuity surface only later on reads.

Root register

Per your ruling: core dsm stays config-free and keeps BETA_MEMBERS + quorum as the authority over which members; the catalog supplies each member's incarnation; RootRegisterProfile::derive_set_id re-derives the id from the candidate pairs and refuses any candidate whose membership is not the network's. Endpoints stay transport metadata and are not inputs. Both ProvenanceResolver and PeerEvidenceFetcher gained root_register_candidate_set, named "candidate" so no call site reads it as authority.

Discovery-added nodes get an empty incarnation deliberately, which fails the catalog closed — discovery can carry transport and can never mint set membership.

Open assumption

Your answer says the profile "commits storage_set_id". Core cannot compute that digest without incarnations, and pinning it as a literal needs a real provisioned value I will not invent. So the committed authority is currently membership + quorum in core, incarnations from the catalog, id re-derived and checked. If you want an additionally-pinned digest once the beta fleet is provisioned, that is a value you would hand me.

Mutation controls

Each restored with a verified tree stamp (git diff | shasum identical before and after).

mutation named test that went red
duplicate member id no longer refused invalid_inputs_are_refused_rather_than_normalized — the "one member under two incarnations" assertion
all-zero incarnation no longer refused same test, its own assertion
the reader stops checking the incarnation half only_the_committed_member_in_its_committed_incarnation_counts — on the SAME-NODE-REBUILT-REGISTER row
NodeStorageSet stops refusing a rebuilt incarnation a_node_whose_register_was_rebuilt_refuses_to_serve_the_configured_set
the absent branch stops carrying the stamp every_read_answer_names_the_register_incarnation_serving_it

Stated rather than claimed: the sort key — member id alone vs the whole pair — has no mutation control, because duplicates are the only input where the two orders differ and the duplicate refusal fires first. The load-bearing gate is that refusal, which is tested above. The sort is documented as belt-and-braces, not asserted as independently load-bearing.

Fleet bring-up is now two-phase

No generator can know an incarnation in advance, so generate_node_configs.sh no longer emits a member list and production.toml documents the sequence: boot with [storage_set] absent (register inactive, every claim refused — the correct state for a fleet that has not agreed on a set), collect each node's logged register incarnation for node <id>: <Base32-Crockford>, write the member list into every node config and the client env config, restart. A node whose configured incarnation is not the one its database holds refuses to start, so a mistake is loud.

The incarnation is established and logged unconditionally at startup, before any set is considered — establishing it only when a set exists would be a bootstrap that never starts.

Verification

Rust 1.98.0, the version rust-toolchain.toml declares.

  • root make lint — exit 0
  • workspace board --release -- --test-threads=13982 passed, 0 failed
  • node board (SQLite, local-dev,strict) — 288 passed, 0 failed
  • node board (Postgres 14, default features) — 258 passed, 0 failed
  • production_safety_checks.sh (incl. TLA+), ci_scan.sh, no_clock_and_no_json.sh, check_forbidden_symbols.sh, codegen_enforce.sh — all exit 0

Dependent sweep, and a self-inflicted regression worth naming. Making register_incarnation a required config field broke 106 dsm_sdk tests — every fixture that writes a fleet TOML. The existing CLAUDE.md rule fires on capability removal; a newly-required field reads as an addition and removes exactly as much. A refinement candidate is filed to widen that trigger. A further 11 "failures" after the fix were my own error — I ran with --test-threads=4 when CLAUDE.md documents these suites share process-global singletons; serially they pass.

…not just which nodes

A set of bare member ids says only WHICH NODES a vault trusts. A member that
lost and rebuilt its register still satisfied that — same identity key, same
configured id — and could then assert "no claim here" for a cell the real
incarnation once held. Nothing distinguished it, because owning the identity
was the entire test.

A set entry is now the pair `(member_id, register_incarnation_id)`, so the set
id commits to which durable register histories on those nodes. `0x0002` goes to
schema 3; entries sort by MEMBER ID alone and a duplicate member id is refused
regardless of incarnation, so one member cannot appear twice under two
incarnations. An all-zero incarnation — the value a node has before it has
established one — is refused as uncommittable.

`VaultStateV2` goes 3 -> 4 with it. The registry's own rule is that nesting is
by complete CCB including the nested schema version, so a nested bump
propagates upward whether or not the enclosing field list moved. Both
superseded pairs are burned. Set ids and vault-state commitments therefore
change; that is the reprovision.

The incarnation is minted by the node itself on first boot, from the OS random
source, into a write-once row committed durably, and is deliberately NOT
derivable from the node id or the signing key — a node that kept its key but
lost its register MUST come back different, because that is the fact a vault
needs. It is established and logged unconditionally at startup, before any set
is considered: establishing it only when a set exists would be a bootstrap that
never starts, since an operator cannot write the member list without the value.

Both directions are bound. Reads carry `x-dsm-register-incarnation` on the held
AND absent branches — stamping only `held` would leave the dangerous case, a
rebuilt member reporting emptiness, unmarked — and the reader counts an answer
only when the member id and the incarnation both match. A mismatch is
`Unavailable`: never an absence, never a value, and never an invalid-credit
verdict, because the node is not lying, it simply is no longer the member this
vault committed. Claims need no new machinery: a node derives its own set id
from members including its own durable incarnation, so a rebuilt member computes
a different id and the existing foreign-set check refuses before any insert.

Root-register authority moves accordingly. Core keeps the network's member ids
and quorum; a resolver supplies candidate `(member, incarnation)` pairs through
`root_register_candidate_set`, and `RootRegisterProfile::derive_set_id` refuses
any candidate whose membership is not the network's before computing an id.
Endpoints stay transport metadata. Discovery-added nodes get an empty
incarnation on purpose, which fails the catalog closed — discovery can carry
transport and can never mint set membership.

Fleet bring-up is now explicitly two-phase, because no generator can know an
incarnation in advance: boot with `[storage_set]` absent (register inactive,
every claim refused), collect each node's logged incarnation, write the member
list everywhere, restart. A node whose configured incarnation is not the one its
database holds refuses to start.

Mutation controls, each restored with a verified tree stamp:
- duplicate member id no longer refused -> invalid_inputs_are_refused_rather_than_normalized RED
- all-zero incarnation no longer refused -> same test RED on its own assertion
- the reader stops checking the incarnation half -> only_the_committed_member_in_its_committed_incarnation_counts RED
- NodeStorageSet stops refusing a rebuilt incarnation -> a_node_whose_register_was_rebuilt_refuses_to_serve_the_configured_set RED
- the absent branch stops carrying the stamp -> every_read_answer_names_the_register_incarnation_serving_it RED

Not independently observable, and stated rather than claimed: the sort KEY
(member id vs the whole pair) has no mutation, because duplicates are the only
input where the two orders differ and the duplicate refusal fires first.

Verification (Rust 1.98.0, the version rust-toolchain.toml declares):
- root `make lint` exit 0
- workspace board 3982 passed, 0 failed
- node board (local-dev,strict / SQLite) 288 passed, 0 failed
- node board (default features / Postgres 14) 258 passed, 0 failed
- production_safety_checks (incl. TLA+), ci_scan, no_clock_and_no_json,
  check_forbidden_symbols, codegen_enforce all exit 0
@cryptskii
cryptskii merged commit f00d1e0 into main Sep 5, 2026
16 checks passed
cryptskii added a commit that referenced this pull request Sep 5, 2026
…ories the real fleet holds (#771)

#770 gave a vault's birth commitment `(member_id, register_incarnation_id)`
pairs, but the network-wide root register still derived its set id from
whatever candidate the catalog offered, constrained only to the right member
ids. "Who chose the incarnation values?" answered "the resolved catalog
candidate" — and `derive_set_id(&candidate) -> Result<[u8; 32]>` returned the
id derived from its own input, which both call sites then used as the
expected id. A checker that returns what it derived hands the authority to
its input.

`RootRegisterProfile` now carries `storage_set_id` as a pinned commitment, and
`verify_candidate` replaces `derive_set_id` at every call site: the candidate's
membership must be the network's AND the id re-derived from its pairs must
equal the pin. A member that rebuilt its register passes the first and fails
the second, and is refused (`SetIdIsNotThePinnedOne`) rather than silently
resolved to a different register. Endpoints remain transport metadata.

The pin is the three real `(member, incarnation)` pairs, not an opaque digest:
member list and set id both derive from the one source, so they cannot drift,
and an auditor can read which register histories are trusted.
`pinned_root_register_members()` is public — a commitment, not a secret — and
every fixture builds from it, so nothing holds parallel values that must
match. Before the values existed the placeholder was all-zero, which the
encoder already refuses; an unprovisioned network fails closed by
construction (`ProfileNotDerivable(ZeroRegisterIncarnation)`), not by anyone
remembering to check.

The values came off the provisioned nodes in the owner's order. Each member:
forensic `pg_dump`, `DROP SCHEMA public CASCADE`, boot on merged `f00d1e0c`
with `[storage_set]` absent, mint and log the incarnation. Then the collected
pairs were written as `[[storage_set.members]]` to every node and to the client
config, and every node restarted into the set logging the identical id
`E05YS8101EJH33KY2CG625JJE8A0Z4GJNSEM335TX1XVTWM9RR8G` with no refusal. Two
tests pin this end to end: the pinned bytes render to exactly the strings each
node logged (cross-checked against the raw `register_incarnation` rows), and
core's derivation equals the id every node logged.

Mutation control: disabling the pin comparison turned
`one_member_rebuilding_its_register_changes_the_set_id` red (`got Ok(())` —
the rebuilt member was accepted); restored on an identical tree stamp.

Not changed, surfaced: no node sets `node.network_id`, so the faucet-ticket
register is inactive fleet-wide until it is configured. A fail-open in
`main.rs` (optional config + a defaulted `database.url` that quietly points
at localhost) is filed separately; it turned a wrong-image build into a
crash loop during provisioning rather than a refusal.

Verification (Rust 1.98.0, the version rust-toolchain.toml declares):
- root `make lint` exit 0
- workspace board 3984 passed, 0 failed
- node board (local-dev,strict / SQLite) 288 passed, 0 failed
- node board (default features / Postgres 14) 258 passed, 0 failed
- production_safety_checks (incl. TLA+), ci_scan, no_clock_and_no_json,
  check_forbidden_symbols all exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant