Skip to content

[INF-1722] Avoid full status scans in CDC metadata reads - #20

Merged
thesyncim merged 1 commit into
mainfrom
codex/inf-1722-capture-metadata
Aug 27, 2026
Merged

[INF-1722] Avoid full status scans in CDC metadata reads#20
thesyncim merged 1 commit into
mainfrom
codex/inf-1722-capture-metadata

Conversation

@thesyncim

@thesyncim thesyncim commented Aug 27, 2026

Copy link
Copy Markdown
Member

Ticket

Summary

The CDC receiver checks the cutover boundary before every replication message. Store.Migration rebuilt the entire status snapshot for that check: a read transaction with ten SELECTs, including inventory aggregates and verification progress. Read the same nine singleton metadata fields directly instead.

One production file changes (29 added / 4 removed lines). No caching, new settings, dependencies, schema changes, WAL/replay-format changes, or changes to cutover/acknowledgement/transaction logic. The full dashboard Snapshot is unchanged.

Evidence

Synthetic c3-sized inventory: 113 tables, 276 parts, 509 indexes, 12 constraints. Median of three 1-second runs, Go 1.26.0, Apple M4 Max:

Metadata read Before After
Time/op 145,941 ns 9,471 ns
Allocations/op 294 47

The metadata read is 15.4x faster; the unchanged full Snapshot remains approximately 146 microseconds. This is not an end-to-end capture-throughput multiplier.

Validation

  • Regression test failed on the original code and passes with the fix: metadata no longer depends on an unrelated apply-progress row; Snapshot still reports missing progress.
  • All nine fields match Snapshot after a separate controller commits the boundary, through writer/read-only handles and after reopening the state directory.
  • Canceled context, missing migration row, and closed-store errors remain visible.
  • Passed: go test ./... -count=1, go vet ./..., and go test -race ./internal/state ./internal/cdc ./internal/cutover -count=1.
  • Passed on isolated native PostgreSQL 18.6: live WAL capture crash/retry; exact lane receipt resume; coalesced-stage resume without repeated DML; atomic serial receipts; pipelined replay ordering/rollback, including arrays and TOAST; target identity rejection; failure clearing only after durable progress; wrong-target cleanup refusal; idle-source marker durability.
  • These nine existing integration tests passed once normally and twice with -race. Their TestPG17... names are historical: the actual runtime was PostgreSQL 18.6, matching production's major version. Docker storage was broken, so a temporary Go -overlay replaced only the test server launcher with fresh loopback PostgreSQL instances. No test assertions or production code were overlaid; the adapter is outside this PR. PostgreSQL 16/17 integration execution is not claimed.

Rollout and risk

The user authorized a progress-preserving c3 deployment after validation. Image v68 adds only the tested Linux binary to the exact running v67 image; the base layers and runtime configuration are unchanged. A credential-free, PVC-free pod smoke test passed and the binary checksum matched. The existing PVC, source slot, capture checkpoints, and target progress/receipts must be preserved; no recopy, index rebuild, source-data changes, or cutover is part of this rollout.

Deployed on 2026-08-27, resumed at 12:16:37 UTC, pinned to sha256:d8876cff3c7a42471ba0af75cae3fc45f887dba8e2f41ce668945f369f34cce2. Saved configuration and snapshot metadata hashes matched before/after. The pre-stop partial claim resumed and finalized at its exact end LSN, adding exactly its 371 transactions and 1,283 changes. The 113 tables, 276 parts, 509 indexes and 12 constraints stayed complete. By 12:22:29 UTC another 1,500,603 changes had been committed beyond the pre-stop checkpoint; no new operation error. Local UI/port-forward verified live.

Measured WAL-position rates: capture averaged 20.0 MiB/s over the 5.5-minute pre-update sample and 130.3 MiB/s over the 5.3-minute post-update sample. In the latter window actual production generated 17.0 MiB/s, while committed target replay advanced about 42.6 MiB/s. These are live observations, not a controlled workload benchmark or a guarantee. Around 192 GiB of end-to-end lag remained; faster capture moves some backlog into the local queue. The UI's “source” rate is capture, not production. Existing retention warnings remain; no whole-dataset parity or cutover-readiness claim.

Checklist

  • Focused regression and benchmark coverage
  • Unit tests, vet, targeted race tests
  • No API/configuration changes requiring documentation
  • Isolated PostgreSQL 18 restart/marker integration tests completed
  • Capture improvement and independent target/source rates measured after authorized deployment

@thesyncim
thesyncim marked this pull request as ready for review August 27, 2026 12:14
@thesyncim
thesyncim merged commit 616a7c8 into main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant