docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan - #550
Open
rickyrombo wants to merge 6 commits into
Open
docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan#550rickyrombo wants to merge 6 commits into
rickyrombo wants to merge 6 commits into
Conversation
A verify node started against the pristine chain DB with a binary that did not embed the artifact's genesis took its "generate new genesis" path and ran the core migrations down, destroying the writer output. OPENAUDIO_ENV=dev loads the embedded dev.json, so the binary looks normal right up until it drops the tables. Document sealing the artifact with ALLOW_CONNECTIONS false and serving from a TEMPLATE copy, so no connection string can reach the pristine write, and note that artifact verification belongs after services start, not before.
…s real prerequisites Five fixes from a read-through: - The runbook told the operator to confirm block 5,819. Both verification artifacts end at 5,839, so the check inverted: a healthy chain reads as the dead-chain failure mode. Every prod run ends at a different height, so the check now points at the writer's reported height instead of any literal. - The writer's prerequisites were undocumented. rewards is its FIRST step and needs --core-dsn to the old core chain, the launchpad secrets from env, and the mints file; the destination database must exist and be empty in both Postgres and its CometBFT directory. Anything missing kills a ten-hour run in the first second. - Step 3 had no pass criterion, and parity legitimately reports divergences -- including several where the migration is more correct than the source. Lists the expected set so a reviewer does not flag the migration for being right. - The ~10h writer estimate is unsourced; verification runs took 3h09m and 3h36m. Says to re-derive rather than substituting another machine's number. - Step 13 fell outside the reversibility framing, but it retires the two old-chain state-sync servers and so ends the cheap rollback path.
…plan The previous runbook ordered the work but left the two hardest transitions underspecified: how plays avoid being split across chains while the fleet is migrating, and how the indexer moves chains without duplicating or dropping writes. Sixteen steps now, with the indexer switch pinned to a height L chosen in the future so it does not race the config rollout, and the flusher filtered below L so the boundary is exact rather than inferred. Plays are routed through two old-network nodes for the duration, which keeps them on the chain the indexer is actually reading. Records what each irreversible step costs: 14 is the point of no return, 15 retires the rollback anchors, and jailing stops entirely once the old network is down to ~30 active validators -- after which a halt cannot be recovered by waiting.
…d with the code Six places where the reference either contradicted the runbook or stated something the code does not do: - The facts table gave prod's validator count as 9. That is the genesis list; the live set is 67 of 72 registered nodes, and the runbook's quorum and jailing math is over the live set. - Section 7 said to leave BlockInterval at its default, which the runbook now overrides with 20,000. Records why the default is wrong here, that it is a producer-side setting, and that retention is Keep x BlockInterval so it cannot be lowered freely. - The state-sync truncation table listed blobs but not uploads, audio_previews or qm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes #551 makes the row moot. - Section 3 asserted the bootstrap validator holds power 100. Its own registration rewrites that to ValidatorVotingPower, 10 on mainnet, because the writer never seeds core_registered_nodes. - The cursor discussion now points at api#1018, which is open and must land before flushing is first enabled. - Section 11 omitted the chain-aware fallback to core_indexed_blocks, which is only reached on ErrNoRows and so never runs on a database that has indexed the old chain. That is why the failure is a silent stall.
…ld block the rollout The reference suggested refusing to start when the validator key disagrees with genesis. New-chain genesis lists one validator and every other node joins by registration, so that check would stop the entire fleet from ever reaching the new chain -- it is the rollout mechanism, not a fault. It also named ensureGenesisFiles, which does not exist. The genesis-file handling is setup.go:114-126 and compares nothing, correctly. The check worth having is already there: ensurePrivValidator compares the key file against the key derived from the delegate key and refuses only when prior signing history makes a mismatch a double-sign risk. The T7 node that motivated the suggestion was behaving correctly -- a node whose key is not in genesis block-syncs and does not propose. It read as a dead chain only because it was the sole node. Replaces the recommendation with a diagnostic: log when the key is absent from the genesis set and say the node will not propose until registered.
…e new binary Reference SS5 said the bootstrap node would dial old-network hosts and should suppress them with OPENAUDIO_PERSISTENT_PEERS, no release needed. That assumed the node runs the existing binary with only genesis swapped. Runbook step 3 builds a binary whose ProdPersistentPeers is already the new bootstrap nodes, so there are no old-network hosts in the list and nothing to suppress. Outbound dialing of the old network does not happen on the step 3 binary. Inbound does, and no env var here changes it: old nodes hold this host in their address books and keep dialing until those entries age out. Those connections are rejected at the handshake on the network mismatch. The env override at step 4 is still worth setting, but for the narrower reason now stated there: the second bootstrap node does not exist until step 7, so the baked list would otherwise churn against a host that is not up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rewrites the genesis migration runbook as a sequenced sixteen-step cutover, and reconciles the reference sections with it and with the code.
The runbook
The previous version ordered the work but left the two hardest transitions underspecified.
Plays were going to be split across chains. They're submitted by whichever node serves the audio — never through the relay — so once nodes start migrating, each one writes its plays to whichever chain it happens to be on while the indexer still reads the old one. The fleet migration takes days, so that's days of plays written where nobody is reading. Step 5 routes all plays through two old-network nodes for the duration; step 13 reverts it.
The indexer switch had no procedure. It's now pinned to a height
Lchosen in the future, with the flusher filtered below it:Lin the future so the stop doesn't race the config rollout. Choosing one at or below the current tip means the indexer sails past it, and those blocks get indexed from the old chain and flushed to the new one.confirmed_blockis only roughly ordered byid; stopping at the first row aboveLwould strand one below it.Lreach zero and stay zero across a settle window, since a straggler can appear after the queue first reads empty.That yields an exact boundary: the old indexer covers everything at or below
L, the new one everything aboveH, and the ceiling guarantees nothing aboveLreached the new chain belowH. No pause, and no dependence on the two switches happening at the same instant.Step 13 also records that reverting the play routing loses a small number of plays on either side of
Land duplicates none — plays have no unique key, so there is no dedupe path — and that after is the better default because the window is then bounded by the drain.Reference reconciliation
Six places contradicted the runbook or the code:
BlockInterval. Section 7 said to leave it at the default, which the runbook now overrides with 20,000. Adds why the default is wrong here (no snapshot exists until height 100,000), that it's producer-side so it needs no fleet agreement, and that retention isKeep × BlockIntervalso it can't be lowered freely.blobsbut notuploads,audio_previews, orqm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes fix(core): scope the state-sync truncate to the tables the snapshot restores #551 makes the row moot.ValidatorVotingPower— 10 on mainnet — because the writer never seedscore_registered_nodes.core_indexed_blocks. It's only reached onErrNoRows, so it never runs on a database that has indexed the old chain — which is why the failure is a silent stall rather than an error.Earlier corrections in this branch
5,819; both verification artifacts end at 5,839, so an operator following it against a healthy chain would diagnose the dead-chain failure mode. Now points at the height the writer reports, since every run against fresh data ends somewhere different.ALLOW_CONNECTIONS falseplus aTEMPLATEcopy makes the pristine write unreachable by any connection string.--core-dsn, launchpad secrets, mints file, and a destination empty in both Postgres and its CometBFT directory.Scope
Documentation only. No code changes. Code prerequisites for steps 11 and 12 are deliberately out of the runbook and will be handled before it's run.