Skip to content

[INF-1722] Speed up durable CDC replay waves - #17

Merged
thesyncim merged 2 commits into
mainfrom
codex/inf-1722-delete-pk
Aug 25, 2026
Merged

[INF-1722] Speed up durable CDC replay waves#17
thesyncim merged 2 commits into
mainfrom
codex/inf-1722-delete-pk

Conversation

@thesyncim

@thesyncim thesyncim commented Aug 25, 2026

Copy link
Copy Markdown
Member

Ticket\n- https://linear.app/stream/issue/INF-1722\n\n## Summary\nFix replay deletes so composite primary keys always constrain target access through the complete target PK. Bring the catch-up executor in line with the proven crdb-to-pg shape: four scheduling slices per durable wave, key-affine lanes, safe repeated full-row update coalescing, and COPY-to-temp-stage primary-key upserts for large built-in groups.\n\nThe durable claim schema stays at plan version 5. A v65 binary can reconstruct and finish an in-flight new wave from its stored EndLSN, lane count, manifest, generation fence, and exact work receipts.\n\n## Implementation\n- Force single-row and batched deletes through exact target-primary-key predicates.\n- Coalesce only repeated complete-row updates to the same stable primary key inside one lane transaction.\n- Never coalesce across inserts, deletes, unchanged-TOAST/selective updates, PK mutations, cross-key conflicts, or unsafe barriers.\n- COPY groups of at least 64 binary-safe rows into transaction-local typed stages, then apply one set-based PK upsert.\n- Keep every PostgreSQL source transaction indivisible in one target work transaction.\n- Commit lane DML and its exact receipt atomically; advance the public LSN only after the full contiguous wave is complete.\n- Execute safe epochs around unsafe source transactions concurrently while retaining the unsafe transaction as an ordered barrier.\n- Clarify in CLI/UI that replay_batch_* configures one slice and four slices form the default durable wave.\n\n## Safety and compatibility\n- No source writes, source-slot changes, copy restart, index rebuild, restore, or cutover.\n- On-disk replay plan remains v5 and downgrade-resumable.\n- Inserts, deletes, PK changes, custom/unchanged-TOAST payloads, and cross-key-conflicting relations retain their existing exact ordered paths.\n- Target progress and counters continue to commit exactly once behind the generation fence.\n- Existing active claims are reconstructed from their stored plan before fresh-wave behavior is used.\n\n## Validation\n- go test ./...\n- go vet ./...\n- go test -race ./internal/cdc ./internal/cli ./internal/controller\n- go test -tags=integration ./... -count=1\n- crash after coalesced binary-stage DML+receipt, then exact resume without repeated DML\n- legacy plan v2-v5 reconstruction and monotonic generation-fence suite\n- full source/target table digest equality and exact transaction/change counter assertions\n\n## Benchmark\nLocal PostgreSQL 17, 10,000 source transactions / 100,000 changes / 62.3 MiB retained WAL:\n- v65, 8 sessions: 80.9 MiB/s\n- new wave executor, 8 sessions: 96.5 MiB/s\n- 16 and 32 sessions regressed from target contention, so rollout retains replay_workers=8\n\n## Rollout and rollback\n- Deploy one controller pod only after CI.\n- Stop the old applier, retain the PVC/queue and source slot, then start the new image from the existing target LSN.\n- Keep replay_workers=8, replay_batch_bytes=8388608, and replay_batch_changes=32768.\n- Verify the first legacy/new claim boundary, exact target progress, apply/source rates, net lag drain, and queue headroom.\n- Rollback to v65 is safe while retaining the same PVC and configuration because the claim protocol remains v5.\n- Do not cut over.\n\n## Checklist\n- [x] Changed code is covered with unit tests.\n- [x] Controller UI wording is covered by existing UI contract tests.\n- [x] Full Docker-backed integration suite passes.\n

@thesyncim thesyncim changed the title [INF-1722] Force replay deletes through target primary key [INF-1722] Speed up durable CDC replay waves Aug 25, 2026
@thesyncim
thesyncim merged commit f7cc09f into main Aug 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant