Skip to content

test: add crash recovery coverage and nightly storage stability checks - #760

Draft
egolearner wants to merge 1 commit into
mainfrom
test/nightly-storage-stability
Draft

egolearner wants to merge 1 commit into
mainfrom
test/nightly-storage-stability

Conversation

@egolearner

@egolearner egolearner commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add recovery coverage that checks data remains queryable and writable after crashes and storage faults, including later flush, optimize, and reopen operations. The branch includes the merged FTS recovery fix #759.

  • Add 18 deterministic process-crash cases across plain and dual-field FTS schemas: writes, flush, sealing, optimize, rollover, mutations, repeated recovery, and deletion of all rows.
  • Share expected-state verification between checkpoint and storage workers. Check full fields, deleted keys, historical generations, exact L2 distances, small top-k ordering, and field-specific FTS terms with different frequencies and lengths. Check analytical BM25 scores for the pristine single-segment corpus; mutated corpora can retain deleted physical rows in scoring statistics.
  • Verify each recovery-time mutation immediately and retain an update to a recovered document across flush/optimize/reopen.
  • Prepare multiple small segments with cross-segment updates/deletions/upserts. Before optimize fault injection, finish sealing and flush; require the manifest to show multiple input segments replaced by one new segment on success.
  • Add nightly block/inode exhaustion, one-shot/persistent EIO at initial and delayed writes, manifest writes, and compaction forward/vector/FTS files, plus sampled block-write replay. Require evidence of fault hits and successful writes before delayed injection; preserve logs, JUnit, seeds, and failed images.
  • Generate the nightly recovery suite list from CMake registration, so new suites enter the build and runner automatically. New index types and storage paths still require explicit fault workloads.

Validation

On macOS ARM64:

  • All 26 recovery tests passed through the generated suite list (18 checkpoint, 7 write, 1 optimize).
  • Six plain/FTS storage-worker operation/recovery checks passed. The tightened optimize boundary was additionally rerun for both schemas, confirming three input segments and successful compaction.
  • All 10 portable harness tests passed.
  • Clang-tidy, Ruff, actionlint, and whitespace checks passed.

Linux validation uses the default GitHub Actions Ubuntu runner in the fork: https://github.com/egolearner/zvec/actions/runs/35495029003. This run is pending; no full Linux fault-injection pass is claimed yet. The fork-only validation commit copies this workflow to an already registered workflow path, without changing either repository's main branch. The PR does not add a pull_request trigger.

The workflow runs daily at 01:30 Asia/Shanghai after merging into the default branch and supports manual dispatch. Storage faults require sudo and loop mounts; block replay additionally requires dm_log_writes. Missing prerequisites and test failures are reported normally, without skips or expected-failure suppression. Block replay samples persistence states, not physical power cycling. No production storage behavior changes.

Comment thread scripts/stability/test_harness.py Fixed
@egolearner
egolearner force-pushed the test/nightly-storage-stability branch from 2b5f4da to 64fb62f Compare September 20, 2026 05:56
@egolearner
egolearner marked this pull request as ready for review September 20, 2026 05:56
Comment thread scripts/stability/test_harness.py Fixed
@egolearner
egolearner force-pushed the test/nightly-storage-stability branch 3 times, most recently from bb51f0f to 015a1ab Compare September 20, 2026 06:34
Comment thread scripts/stability/test_harness.py Fixed
@egolearner
egolearner force-pushed the test/nightly-storage-stability branch from 015a1ab to 72f63e4 Compare September 20, 2026 06:42
@egolearner
egolearner marked this pull request as draft September 20, 2026 07:07
self.args, self.folder / "db", "flush", False, self.folder
) as process:
raise RuntimeError("parent failure")
self.assertIsNotNone(process)
self.fail("A missing checkpoint acknowledgement must fail")

def test_exception_kills_and_reaps_stopped_worker(self):
process = None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant