feat(tbtc/signer)!: add durable witness history and external anchor (ABI 4.2)

[mswilkison]

Based on #4005. 4 commits.

Descriptor-bound stable .store-id, O_NOFOLLOW/openat with inode and path checks, atomic replacement, an append-only PREPARE/COMMIT/ABORT state-witness journal, retained key-package inventory with dkg_share_epoch, and ABI symbols for durable identity, inventory and witness proof, with cross-language vectors.

BREAKING — coordinated merge required. This bumps the signer ABI 3.5 → 4.0 and the state-witness transcript v1 → v2. It cannot merge alone: keep-core #4199 pins requiredTBTCSignerABIMajor = 3 (pkg/frost/signing/native_tbtc_signer_abi_version.go:22) and will reject this library at load, and ci/frost-signer-pin.env must move to the commit that provides ABI 4.0. That file's own comment states the rule — bump it in the same PR as the Go constant.

Why the transcript had to change

durable_store_fingerprint mixed the stable store_id with three volatile inputs: the parent directory's path string + st_dev + st_ino, the filesystem st_dev, and the advisory lock file's name + st_dev + st_ino. That composite was the first field of every state_commitment, and parse_state_witness_journal rejects any record that does not recompute under the current fingerprint.

So deleting the zero-byte lock file (tmp reaper, "stale lock" cleanup, container rebuild), restoring the data directory from backup at the same path (tar/rsync gives new inodes), renaming or moving the directory, or remounting on btrfs/ZFS/overlayfs/NFS/tmpfs made 100% of committed records unverifiable and the signer unstartable.

Worse, the failure is raised inside StateFileLock::acquire, so TBTC_SIGNER_STATE_CORRUPTION_POLICY=quarantine_and_reset never reaches it. The only remaining operator action was deleting .state-witness — which re-genesises at generation 1, exactly the rollback this feature exists to detect — while frost_tbtc_durable_store_identity kept returning a byte-identical fingerprint, so a host pinning that identity got no signal.

Fix: commitments bind only stable content. The path/filesystem/lock fingerprints are still computed at open, still enforced on every access by revalidate_store_entries (symlink/inode/device/nlink/uid/mode checks unchanged), and still reported for diagnostics — they simply no longer enter any committed transcript. A v1 journal is recognised by magic alone (deliberate: the v1 fingerprint may no longer be recomputable, which is the failure mode) and fails closed with an actionable migration error that names both transcript versions and warns against deleting the journal. A v1 journal is never rewritten or repaired; the v1 vectors are retained as rejection-path regressions.

v2 cross-language vectors — for the Go side (#4199)

Constants: schema tbtc-signer-durable-session-store-identity/v2, backend encrypted-file-v1 (unchanged), domains tbtc-signer-durable-session-store-fingerprint-v2\0, tbtc-signer-state-witness-genesis-v2\0, tbtc-signer-state-witness-commitment-v2\0, magic TBTCWITNESSv2\0\0\0.

fingerprint(store_id) = SHA256(fp_domain ‖ u32be(len(schema))‖schema ‖ u32be(len(backend))‖backend ‖ u32be(32)‖store_id)
genesis(fp)           = SHA256(genesis_domain ‖ fp)
commitment(fp,g,p,d)  = SHA256(commit_domain ‖ fp ‖ u64be(g) ‖ p ‖ d)

durable_store_fingerprint(0x11*32) = 8bb8d21c69e78916e8f165b0c861c0d84c5d7af5393f75b0321fe048f772abba
durable_store_fingerprint(0x24*32) = 52fcbfc4b2c6a93645106a32c62113192cac30b934b905e1ad357792c4ce8628
state_witness_genesis(0x11*32)     = 44085b42d29bf25f06207142f9e2db58eaf86f88d92b6e18104161ce59e98a89
state_commitment(0x11*32,42,0x22*32,0x33*32)
                                   = ea5eb04a4776357e59875f683390a2ff4b7dd511ad394e588dfab147f94fa867

chained: fp = fingerprint(0x11*32)
         genesis(fp)                      = 3179b8bc6614b0951b703f9c418b17cf7cd8b7f1bef1f86587385d4c150efab2
         commitment(fp,1,genesis,0x33*32) = 5387626d5314b17b324f9a7df1ab16fcbf10917a137527bf33c71847e1b77da0

Also fixed

Torn genesis was unrecoverable. Creation wrote a 258-byte header+PREPARE+COMMIT blob non-atomically before syncing; a hard kill left a short file that was fatal on next start, again inside acquire() where the corruption policy cannot help. The asymmetry was backwards — a torn append was repaired, a torn genesis was not. Genesis now publishes atomically: complete image → temp entry in the same directory (openat + O_EXCL + O_NOFOLLOW, mode 0600) → fsync → target-absent check → renameat → directory fsync. Same for .store-id.

A short file at the final name is deliberately not "recovered": this build cannot produce one, so it is external damage, and a zero-length or half-written journal is indistinguishable from deliberate truncation — re-creating it would re-genesis the anti-rollback chain over whatever state image is present. Those bands fail closed with errors that say a short file is not a torn write, and to restore rather than delete. Recovery is proven for real crash residue: a planted orphan temp entry, after which the store comes up clean and advances.

Unbounded re-verification. The journal grew ~210 B/persist with no compaction and was fully re-read and re-hashed roughly four times per persist — at ~1e6 lifetime persists, a ~210 MB journal re-hashed ~4x per persist, degrading toward missed signing and DKG deadlines. Now a per-lock verified-prefix cache makes steady state O(1) (stamp + 48-byte header + 105-byte tail). Newly appended bytes are read back from disk before being trusted, any mismatch falls through to the full parse, and the cache is in-memory only so a fresh open always re-parses. Measured: 60 persists → 12,858-byte journal, exactly 1 full parse, 669 bytes/persist, constant in journal length.

Compaction/rotation is not implemented — preserving the anchor chain and anti-rollback across a rewrite is a separate, riskier change. Resident witness_history still grows ~104 B/persist.

Evidence

cargo fmt --check, cargo check, cargo clippy --all-targets --all-features -- -D warnings all clean. cargo test --lib -- --test-threads=1 → 273 passed / 0 failed / 1 ignored (baseline 263; +10 new tests).

Reviewer notes

• Any developer machine with a pre-change store at $TMPDIR/frost_tbtc_engine_state.json will hit the v1 migration error. That is the intended signal, not a bug — but it presents as ~94 failing tests until the stale store is removed.
• st_dev change could not be simulated portably. The property is proven structurally instead: identity.fingerprint == durable_store_fingerprint(store_id) with no other input, plus a whole-store restore into a different directory (new path, new inodes, new lock file) that verifies and advances.
• Residual, documented in code: a same-uid writer who modifies the journal's middle and leaves size/mtime/ctime unchanged (requires clock control, root, or a coarse-granularity filesystem within one tick) is detected on the next fresh open rather than immediately.
• validate_state_image still reads and hashes the whole state image on every revalidate (4x/persist). Bounded by engine-state size rather than persist count, so out of scope here — but it is the next thing on that hot path.
• The cfg(not(unix)) stub is not compile-verified on a non-unix target; rustup was unavailable for a --target cross-check.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 90ee68a5-2e02-42c7-8afb-1460d85966a6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/signer-store-identity-abi

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}