[INF-1722] Speed up durable CDC replay waves - #17
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket\n- https://linear.app/stream/issue/INF-1722\n\n## Summary\nFix replay deletes so composite primary keys always constrain target access through the complete target PK. Bring the catch-up executor in line with the proven crdb-to-pg shape: four scheduling slices per durable wave, key-affine lanes, safe repeated full-row update coalescing, and COPY-to-temp-stage primary-key upserts for large built-in groups.\n\nThe durable claim schema stays at plan version 5. A v65 binary can reconstruct and finish an in-flight new wave from its stored EndLSN, lane count, manifest, generation fence, and exact work receipts.\n\n## Implementation\n- Force single-row and batched deletes through exact target-primary-key predicates.\n- Coalesce only repeated complete-row updates to the same stable primary key inside one lane transaction.\n- Never coalesce across inserts, deletes, unchanged-TOAST/selective updates, PK mutations, cross-key conflicts, or unsafe barriers.\n- COPY groups of at least 64 binary-safe rows into transaction-local typed stages, then apply one set-based PK upsert.\n- Keep every PostgreSQL source transaction indivisible in one target work transaction.\n- Commit lane DML and its exact receipt atomically; advance the public LSN only after the full contiguous wave is complete.\n- Execute safe epochs around unsafe source transactions concurrently while retaining the unsafe transaction as an ordered barrier.\n- Clarify in CLI/UI that replay_batch_* configures one slice and four slices form the default durable wave.\n\n## Safety and compatibility\n- No source writes, source-slot changes, copy restart, index rebuild, restore, or cutover.\n- On-disk replay plan remains v5 and downgrade-resumable.\n- Inserts, deletes, PK changes, custom/unchanged-TOAST payloads, and cross-key-conflicting relations retain their existing exact ordered paths.\n- Target progress and counters continue to commit exactly once behind the generation fence.\n- Existing active claims are reconstructed from their stored plan before fresh-wave behavior is used.\n\n## Validation\n- go test ./...\n- go vet ./...\n- go test -race ./internal/cdc ./internal/cli ./internal/controller\n- go test -tags=integration ./... -count=1\n- crash after coalesced binary-stage DML+receipt, then exact resume without repeated DML\n- legacy plan v2-v5 reconstruction and monotonic generation-fence suite\n- full source/target table digest equality and exact transaction/change counter assertions\n\n## Benchmark\nLocal PostgreSQL 17, 10,000 source transactions / 100,000 changes / 62.3 MiB retained WAL:\n- v65, 8 sessions: 80.9 MiB/s\n- new wave executor, 8 sessions: 96.5 MiB/s\n- 16 and 32 sessions regressed from target contention, so rollout retains replay_workers=8\n\n## Rollout and rollback\n- Deploy one controller pod only after CI.\n- Stop the old applier, retain the PVC/queue and source slot, then start the new image from the existing target LSN.\n- Keep replay_workers=8, replay_batch_bytes=8388608, and replay_batch_changes=32768.\n- Verify the first legacy/new claim boundary, exact target progress, apply/source rates, net lag drain, and queue headroom.\n- Rollback to v65 is safe while retaining the same PVC and configuration because the claim protocol remains v5.\n- Do not cut over.\n\n## Checklist\n- [x] Changed code is covered with unit tests.\n- [x] Controller UI wording is covered by existing UI contract tests.\n- [x] Full Docker-backed integration suite passes.\n