[INF-1722] Verify source rows and audit deferred CDC progress

[thesyncim]

Summary

Keep verification source → target only, defer CDC mismatches once for one minute, confirm replay has passed a fresh source snapshot, and require target convergence or advancement. A source change is not an automatic skip. There is no retry/defer loop.

Add optional, explicitly audited verification-only app exclusions via --verify-ignore-apps and the controller settings. No app IDs are hardcoded and replication is unchanged.

All observed mismatches and final decisions are retained in a durable audit log, including mismatches accepted as progressing.

CDC approach

1. Read recorded CDC keys on source and target. Snapshot presence, row hash, and xmin; immediately audit every mismatch. Initially absent source keys are ignored, including target-only rows and recorded deletes. Reinserted source keys are checked normally.
2. Retain the mismatching keys and their original source/target snapshots. Wait at least one minute while other tables continue scanning.
3. Read the source, emit a WAL marker, and wait for replay to pass it before reading the target. Read the source again to detect changes during the check. Elapsed time alone is not proof of replay progress. Marker/wait hooks are omitted when no live applier is expected.
4. Make a final decision for each candidate:

Source observation	Target observation	Decision
Stable across the fresh bracketed reads, whether changed since the initial read or not	Matches source, even if the target did not change again	Pass as converged; equality is checked first
Unchanged	Does not match, but advanced since the original target snapshot	Pass as advanced, reported separately from equality
Changed, including a no-op update or change-and-revert; no stable match	Advanced since the original target snapshot	Pass as advanced, reported separately from equality
Changed; no stable match	Did not advance	Fail as target_stalled; do not skip because the source changed
Unchanged	Neither matches nor advances	Fail as missing/different
Replay confirmation timeout, read timeout, cancellation, or execution error	Decision could not finish	Incomplete/error, never silently pass or classify unconfirmed lag as divergence

Check equality before advancement. Replay can catch up between the initial source read and the initial target read. At the deferred check, both rows can already match even though the target has not changed since its first snapshot. A stable fresh match passes without requiring a redundant target write. Source changes during the bracketed reads still prevent a verified-equal verdict.

Advancement means a target row appeared or its hash/xmin changed relative to its original target snapshot. Transaction IDs are compared only within the same database, never source xmin versus target xmin. A target deletion counts as advancement only when the source also disappeared; losing a target row still required by the source is not advancement. Advancement is required only when a stable match was not established.

An advancing target is accepted by this live CDC check even if it is not yet equal to the source. That outcome is explicit in the audit and summary, not mislabeled as a matched row.

Exactly one deferred recheck

Each initial mismatch gets one recheck after the one-minute delay. Once classified, it is finished: advancing rows are accepted, stalled rows fail, and nothing is requeued. There is no retry cap or indefinite loop, including when table timeouts are disabled.

The delay and queue time are outside --verify-table-timeout; active reads and replay confirmation share the table's remaining budget. Confirmation is also bounded by --verify-converge-timeout (default one minute), even with table timeouts disabled. If replay does not reach the marker, the check ends incomplete with the pending keys audited. This is one bounded confirmation, not another retry/defer loop. A dedicated worker uses at most one additional source/target connection pair. Heap-sample verification retains its existing WAL-marker retry rule.

Verification app scope

• --verify-ignore-apps 7,42, or controller verify_ignore_apps, excludes mismatches owned by those source app_pk values from verification verdicts. Empty clears the setting. IDs must be positive bigint values; whitespace and duplicates are normalized for verification.
• Works whether app_pk is in the key or an ordinary column. Tables without app_pk, NULL source app values, and other apps are still checked. Target ownership cannot exempt a source row. Fresh source ownership is checked again during rechecks.
• Samples are still read and compared. Excluded mismatches produce ignored_app audit records with app_id and snapshots. Run headers record the exclusion list; result summaries report ignored observations and the controller shows the configured scope. Heap/CDC may observe the same key, so ignored_rows is an observation count, not a unique-row count.
• Settings are persisted without secrets, can be changed only while controller workers are idle, and survive restart. Copy, CDC capture, replay, and database contents are unchanged. A clean verification result applies only to the non-excluded scope.

Durable audit

Append to <dir>/log/verify-audit.jsonl, separate from the controller's bounded output buffer:

• Every initial heap and CDC mismatch, including heap candidates beyond the recheck threshold.
• Heap recheck observations and final CDC outcomes: converged, advanced, unresolved, ignored_app, or incomplete, with a separate source_changed flag and replay confirmation boundary when established.
• Run ID, UTC timestamp, table/key, mismatch kind, presence, hashes, and row versions. CDC outcomes retain original source/target metadata and both later source observations. Incomplete outcomes retain the original source metadata; earlier records retain the preceding observations.
• Run start/end records, including any configured ignored apps. A crashed process may leave a start without an end.

Batches are synced before proceeding; an audit-write failure aborts verification. The file is created with mode 0600 and contains sensitive keys/comparison metadata, not full row contents. Retain it on durable storage; it is not automatically truncated or rotated.

The dashboard distinguishes pending checks, confirmed failures, and incomplete checks. Pending CDC work does not hide existing heap failures. Summaries separate matched rows, target advancement, source changes, and pending candidates.

Validation

• go test ./...
• go vet ./...
• go test -race ./...
• go test -race -tags=integration ./internal/verify -count=1 — PostgreSQL 17
• node --test internal/controller/ui_test.mjs — nine dashboard tests, also added to CI

Tests cover target supersets; missing/different rows; one-minute default; delayed convergence; source updates, deletion, no-op writes, reverts and reinserts still mismatched without target advancement failing; simultaneous source/target advancement; fresh matching reads after both change; already-matching unchanged targets; replay catching up between the initial source and target reads; source changes during target reads; target advancement accepted after exactly one check; stalled targets; mixed matched/advancing/stalled keys; cancellation/timeouts; audit failures; all-mismatch logging; and concurrent append-only audit records across runs. The equality-precedence regression cases reproduced target_stalled before the correction. Tests shorten the delay internally; the application uses one minute.

Additional PostgreSQL tests cover convergence/advancement only after replay confirmation, confirmed stalls, bounded confirmation timeout without a table timeout, marker/wait errors and cancellation, source app exclusions in both key layouts and both strata, NULL/missing app columns, target ownership isolation, source ownership changes during rechecks, and audit failure for excluded mismatches. Config/CLI/controller tests cover validation, clearing, partial updates, persistence and the visible scope banner.

Scope / rollout

This remains sampled verification, not proof of full database equality. Initial target-only rows and unapplied deletes are outside the one-way check. A passing live CDC check may include advancing targets whose equality has not been established.

No CDC replay or database row changes and no deployment manifests in this PR. Verification requires a writable audit log. Back up controller settings before upgrade; rolling back to a binary without the new config field requires restoring the prior settings file or removing that field. No exclusion is enabled by default.