Skip to content

docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan - #550

Open
rickyrombo wants to merge 6 commits into
mainfrom
mjp-genesis-artifact-guard
Open

docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan#550
rickyrombo wants to merge 6 commits into
mainfrom
mjp-genesis-artifact-guard

Conversation

@rickyrombo

@rickyrombo rickyrombo commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What

Rewrites the genesis migration runbook as a sequenced sixteen-step cutover, and reconciles the reference sections with it and with the code.

The runbook

The previous version ordered the work but left the two hardest transitions underspecified.

Plays were going to be split across chains. They're submitted by whichever node serves the audio — never through the relay — so once nodes start migrating, each one writes its plays to whichever chain it happens to be on while the indexer still reads the old one. The fleet migration takes days, so that's days of plays written where nobody is reading. Step 5 routes all plays through two old-network nodes for the duration; step 13 reverts it.

The indexer switch had no procedure. It's now pinned to a height L chosen in the future, with the flusher filtered below it:

  • L in the future so the stop doesn't race the config rollout. Choosing one at or below the current tip means the indexer sails past it, and those blocks get indexed from the old chain and flushed to the new one.
  • A filter, not a halt — enqueue is fire-and-forget, so confirmed_block is only roughly ordered by id; stopping at the first row above L would strand one below it.
  • "Finished flushing" means pending rows at or below L reach zero and stay zero across a settle window, since a straggler can appear after the queue first reads empty.

That yields an exact boundary: the old indexer covers everything at or below L, the new one everything above H, and the ceiling guarantees nothing above L reached the new chain below H. No pause, and no dependence on the two switches happening at the same instant.

Step 13 also records that reverting the play routing loses a small number of plays on either side of L and duplicates none — plays have no unique key, so there is no dedupe path — and that after is the better default because the window is then bounded by the drain.

Reference reconciliation

Six places contradicted the runbook or the code:

  • Validator count. The facts table gave prod as 9 validators. That's the genesis list; the live set is 67 of 72 registered nodes. The runbook's quorum and jailing math is over the live set.
  • BlockInterval. Section 7 said to leave it at the default, which the runbook now overrides with 20,000. Adds why the default is wrong here (no snapshot exists until height 100,000), that it's producer-side so it needs no fleet agreement, and that retention is Keep × BlockInterval so it can't be lowered freely.
  • State-sync truncation. The table listed blobs but not uploads, audio_previews, or qm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes fix(core): scope the state-sync truncate to the tables the snapshot restores #551 makes the row moot.
  • Bootstrap validator power. Section 3 asserted it holds power 100. Its own registration rewrites that to ValidatorVotingPower10 on mainnet — because the writer never seeds core_registered_nodes.
  • The flusher cursor now points at fix(api): retain flushed new-chain queue rows instead of deleting them AudiusProject/api#1018, which is open and must land before flushing is first enabled.
  • Section 11 omitted the chain-aware fallback to core_indexed_blocks. It's only reached on ErrNoRows, so it never runs on a database that has indexed the old chain — which is why the failure is a silent stall rather than an error.

Earlier corrections in this branch

  • The height check was inverted. It said to confirm block 5,819; both verification artifacts end at 5,839, so an operator following it against a healthy chain would diagnose the dead-chain failure mode. Now points at the height the writer reports, since every run against fresh data ends somewhere different.
  • Seal the artifact before pointing anything at it. A node whose binary doesn't embed this artifact's genesis generates a new one and runs the core migrations down. This destroyed the 2026-08-25 artifact. ALLOW_CONNECTIONS false plus a TEMPLATE copy makes the pristine write unreachable by any connection string.
  • The writer's prerequisites were absent--core-dsn, launchpad secrets, mints file, and a destination empty in both Postgres and its CometBFT directory.
  • Parity doesn't come back clean, and three of the five expected divergences are cases where the migration is more correct than the source.

Scope

Documentation only. No code changes. Code prerequisites for steps 11 and 12 are deliberately out of the runbook and will be handled before it's run.

A verify node started against the pristine chain DB with a binary that did not
embed the artifact's genesis took its "generate new genesis" path and ran the
core migrations down, destroying the writer output. OPENAUDIO_ENV=dev loads the
embedded dev.json, so the binary looks normal right up until it drops the
tables.

Document sealing the artifact with ALLOW_CONNECTIONS false and serving from a
TEMPLATE copy, so no connection string can reach the pristine write, and note
that artifact verification belongs after services start, not before.
…s real prerequisites

Five fixes from a read-through:

- The runbook told the operator to confirm block 5,819. Both verification
  artifacts end at 5,839, so the check inverted: a healthy chain reads as the
  dead-chain failure mode. Every prod run ends at a different height, so the
  check now points at the writer's reported height instead of any literal.

- The writer's prerequisites were undocumented. rewards is its FIRST step and
  needs --core-dsn to the old core chain, the launchpad secrets from env, and
  the mints file; the destination database must exist and be empty in both
  Postgres and its CometBFT directory. Anything missing kills a ten-hour run in
  the first second.

- Step 3 had no pass criterion, and parity legitimately reports divergences --
  including several where the migration is more correct than the source. Lists
  the expected set so a reviewer does not flag the migration for being right.

- The ~10h writer estimate is unsourced; verification runs took 3h09m and
  3h36m. Says to re-derive rather than substituting another machine's number.

- Step 13 fell outside the reversibility framing, but it retires the two
  old-chain state-sync servers and so ends the cheap rollback path.
@rickyrombo rickyrombo changed the title docs(genesis-writer): seal the artifact DB before starting any service docs(genesis-writer): fix the rollout runbook's height check, prerequisites, and parity criterion Aug 26, 2026
…plan

The previous runbook ordered the work but left the two hardest transitions
underspecified: how plays avoid being split across chains while the fleet is
migrating, and how the indexer moves chains without duplicating or dropping
writes.

Sixteen steps now, with the indexer switch pinned to a height L chosen in the
future so it does not race the config rollout, and the flusher filtered below L
so the boundary is exact rather than inferred. Plays are routed through two
old-network nodes for the duration, which keeps them on the chain the indexer is
actually reading.

Records what each irreversible step costs: 14 is the point of no return, 15
retires the rollback anchors, and jailing stops entirely once the old network is
down to ~30 active validators -- after which a halt cannot be recovered by
waiting.
@rickyrombo rickyrombo changed the title docs(genesis-writer): fix the rollout runbook's height check, prerequisites, and parity criterion docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan Aug 26, 2026
…d with the code

Six places where the reference either contradicted the runbook or stated
something the code does not do:

- The facts table gave prod's validator count as 9. That is the genesis list;
  the live set is 67 of 72 registered nodes, and the runbook's quorum and
  jailing math is over the live set.
- Section 7 said to leave BlockInterval at its default, which the runbook now
  overrides with 20,000. Records why the default is wrong here, that it is a
  producer-side setting, and that retention is Keep x BlockInterval so it cannot
  be lowered freely.
- The state-sync truncation table listed blobs but not uploads, audio_previews
  or qm_audio_analyses, which are also wiped. Adds them with how each recovers,
  and notes #551 makes the row moot.
- Section 3 asserted the bootstrap validator holds power 100. Its own
  registration rewrites that to ValidatorVotingPower, 10 on mainnet, because the
  writer never seeds core_registered_nodes.
- The cursor discussion now points at api#1018, which is open and must land
  before flushing is first enabled.
- Section 11 omitted the chain-aware fallback to core_indexed_blocks, which is
  only reached on ErrNoRows and so never runs on a database that has indexed the
  old chain. That is why the failure is a silent stall.
…ld block the rollout

The reference suggested refusing to start when the validator key disagrees with
genesis. New-chain genesis lists one validator and every other node joins by
registration, so that check would stop the entire fleet from ever reaching the
new chain -- it is the rollout mechanism, not a fault.

It also named ensureGenesisFiles, which does not exist. The genesis-file
handling is setup.go:114-126 and compares nothing, correctly. The check worth
having is already there: ensurePrivValidator compares the key file against the
key derived from the delegate key and refuses only when prior signing history
makes a mismatch a double-sign risk.

The T7 node that motivated the suggestion was behaving correctly -- a node whose
key is not in genesis block-syncs and does not propose. It read as a dead chain
only because it was the sole node. Replaces the recommendation with a
diagnostic: log when the key is absent from the genesis set and say the node
will not propose until registered.
…e new binary

Reference SS5 said the bootstrap node would dial old-network hosts and should
suppress them with OPENAUDIO_PERSISTENT_PEERS, no release needed. That assumed
the node runs the existing binary with only genesis swapped. Runbook step 3
builds a binary whose ProdPersistentPeers is already the new bootstrap nodes, so
there are no old-network hosts in the list and nothing to suppress.

Outbound dialing of the old network does not happen on the step 3 binary.
Inbound does, and no env var here changes it: old nodes hold this host in their
address books and keep dialing until those entries age out. Those connections
are rejected at the handshake on the network mismatch.

The env override at step 4 is still worth setting, but for the narrower reason
now stated there: the second bootstrap node does not exist until step 7, so the
baked list would otherwise churn against a host that is not up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant