Bound NIP-RS retention and search indexing

[tlongwell-block]

Summary

• bound storage for conforming NIP-RS read-state events (kind 30078, d=read-state:<32 lowercase hex>, exact t=read-state) by hard-deleting superseded payloads instead of retaining soft-deleted history
• preserve NIP-33 ordering and stale-replay protection in a compact per-coordinate watermark, including after a later NIP-09 deletion removes the live head
• migrate existing data safely by seeding watermarks first and physically purging only superseded NIP-RS rows that have a strictly dominating live head
• reduce future full-text-search amplification with a positive search_tsv allowlist for kinds 0, 9, 40002, 45001, and 45003
• give fresh empty databases that allowlist automatically, while leaving populated databases on an explicit out-of-band rewrite path

Runtime behavior

For conforming NIP-RS coordinates, replacement remains serialized by the existing per-coordinate advisory lock. The database trigger advances/checks the durable watermark on every insert—including inserts from pre-PR relay binaries during a rolling deploy—while the application checks both the live event and watermark. An older tuple is rejected (lowest event ID wins equal timestamps), and superseded payloads plus defensive mention rows are physically removed. Legacy soft-delete and NIP-09 paths are also converted to physical deletion by a database trigger, while retaining only the compact ordering watermark.

Other kind-30078 application data and all other NIP-33 kinds keep the existing soft-delete replacement behavior. channel_id remains outside the NIP-33 replacement key.

Deployment behavior

With BUZZ_AUTO_MIGRATE enabled, migration 0007 takes SHARE ROW EXCLUSIVE on events, so reads remain available while event inserts, updates, and deletes wait for commit. It:

1. creates parameterized_event_watermarks and the mention-cleanup index;
2. fails closed if legacy data contains a deleted conforming NIP-RS tuple that outranks its live head;
3. seeds each coordinate's greatest accepted tuple across live and historical rows; and
4. physically deletes only superseded NIP-RS history with a strictly dominating live head.

Migrations 0007 and 0009 are checksum-frozen at their published contents. Additive migration 0009 installs the database triggers that enforce watermark advancement/stale rejection and physical NIP-RS deletion across old and new relay binaries, plus the live-event fence for post-commit mention indexing. Exact duplicate idempotence is enforced at the durable coordinate level: an incoming tuple exactly equal to the watermark is skipped by the insert trigger and reports zero affected rows whether its payload is live or already deleted, eliminating replay/deletion resurrection races without another lock edge. Event replacement and legacy soft-delete cleanup both use the same event-row-before-mentions lock order. Additive migration 0010 replaces only the watermark guard function to make exact equality a zero-row no-op. Databases that already recorded migrations 0001–0009 upgrade directly to 0010 without a sqlx checksum mismatch.

Rows deleted explicitly through NIP-09 and lacking a live head are not purged by 0007. The migration does not rewrite search_tsv or rebuild its GIN index.

Migration 0008 serializes an emptiness check with event writers. A fresh empty database switches to the positive FTS allowlist. A populated database records 0008 as applied without changing its existing denylist expression or rewriting heap/TOAST/GIN storage. Search behavior therefore intentionally differs between fresh and existing installations until maintenance is performed.

Optional maintenance to convert populated databases

The PR is boot-safe without this step: populated databases keep their existing search expression. To convert one to the positive allowlist, run scripts/maintenance/nip_rs_search_allowlist.sql separately in a sized maintenance window. It takes ACCESS EXCLUSIVE, so event reads and writes block; it rewrites the generated search_tsv column and rebuilds the GIN index. Confirm headroom for replacement heap/TOAST/index files plus WAL before running it.

Migration 0007's in-place hard deletes do not by themselves return filesystem space to the volume. The allowlist maintenance script rewrites the generated column and rebuilds the GIN index, replacing those relations, but it needs peak headroom while old and new files coexist and release can be delayed by old snapshots. If cleanup is run without that rewrite—or measured residual bloat remains afterward—use a table repack or VACUUM FULL in a separately sized maintenance window.

Validation

• just ci
• cargo test -p buzz-db
• cargo check -p buzz-db
• Postgres regressions proving replacement hard-deletes the old payload and the watermark rejects replay after simulated NIP-09 deletion
• mixed-version A → old writer B → NIP-09 delete B → replay B and X (A < X < B) regression proving the database advances to B, physically removes old payloads, fences stale post-commit mentions without deadlock, and rejects both exact and intermediate replay
• fresh 0001→0008 migration proving the installed catalog expression is the positive allowlist
• populated-at-0007 migration proving 0008 preserves the legacy expression and leaves all 99 events-family relation file nodes unchanged
• fail-closed rollback test for a deleted tuple outranking its live head
• exact migration/runtime NIP-RS predicate test using a nonconforming three-element t tag
• explicit upgrades from databases with published migrations 0001–0008 recorded to 0009 and 0001–0009 recorded to additive 0010 without checksum mismatch
• all 18 Postgres search integration tests against migrations 0001–0009, including a non-vacuous soft-deletion exclusion tripwire
• local bin/just ci, focused recorded-0001–0009→0010 upgrade/concurrency regression, and Hermit-activated push hooks green at ce10dd1417fcb22430f0e26b9db8d49d911fc42d; GitHub CI pending