fix(txnlog): erase a failed append's partial bytes instead of baking a framing break into the log - #751
Open
kriszyp wants to merge 19 commits into
Open
Conversation
The transaction log's committed watermark (lastCommittedPosition) is in-memory state advanced by commitFinished(); it is never persisted. It was seeded on load from txn.state -- the *flushed* position, i.e. how far RocksDB has already absorbed the log -- which after an unclean exit sits behind the log's recovered tail. Every committed read is bounded by that watermark, so entries that were durable on disk (and that a consumer's boot replay re-applies via readUncommitted) stayed invisible to committed readers until an unrelated later commit advanced the watermark past the whole tail at once. load() already recovers the true end: recoverTail() truncates any torn tail, nextLogPosition is set to the last structurally valid entry, and it is inserted as the write-head sentinel. commitFinished() defines the watermark as the front of uncommittedTransactionPositions, which in a freshly loaded store is exactly nextLogPosition -- so seeding it there only makes load() agree with the rule the store already enforces. Fixes HarperFast/harper#1949 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cross-model review findings: the fixture's `ready` handshake went through an async console.log to a pipe, which the following SIGKILL could drop; a self-kill on Windows surfaces as exit code 1, not a signal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
findPositionByTimestamp()'s indexing loop checked `entryTimestamp == 0` before checking whether the current position was the file header's timestamp slot. A header timestamp of exactly 0 (e.g. an unset/epoch value, as several test fixtures write) was therefore treated as the zero-padding end-of-data marker and truncated this->size back into the header itself -- below TRANSACTION_LOG_FILE_HEADER_SIZE. This is normally invisible: only TransactionLogFile::openFile() on Windows proactively calls findPositionByTimestamp() at open time (to correct for mmap zero-padding), so the corruption only ever surfaced there. It stayed latent until 13d57f7/f385ace8 started seeding lastCommittedPosition from the recovered nextLogPosition on load, which propagated the corrupted size into a value CI asserts on -- failing every Windows runtime (Node 22/24/26, Bun, Deno) on PR #723's new/pre-existing "should return valid lastCommittedPosition after purging earlier log files and reopening" case. Root cause: the header-timestamp branch and the end-of-data check were ordered so the latter could fire on a position that isn't an entry. Reordering to always handle the header slot first (as the timestamp index's designated position-0 entry) restores the invariant that the end-of-data heuristic only ever applies to real entries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the regression test for b10e725. The indexing path is cross-platform even though only Windows openFile() reaches it at open, so the test drives it directly through _findPosition and fails without that fix on any platform. Also tests the committed-watermark seed on logSequenceNumber rather than fullPosition: that union member aliases the two uint32s as a double, so a sequence number >= 0x7ff00000 reads back as NaN or negative and `> 0` would silently skip the seed, falling back to the stale txn.state position this PR exists to fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Windows has no real signals, so Node maps a self-kill to TerminateProcess(h, 1). Accepting code 1 on every platform would let a POSIX run whose kill() threw after the `ready` handshake pass with normal addon teardown, which is the very thing the fixture must rule out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ction Only a batch's final entry carries TRANSACTION_LOG_ENTRY_LAST_FLAG, so a crash partway through a multi-entry transaction leaves whole, well-framed entries that are merely a prefix of it. recoverTail() deliberately keeps those bytes -- they are structurally valid and a readUncommitted replay still wants them -- but seeding the committed watermark at the write head published them to committed readers, exposing a transaction that never closed. The next transaction's flag would then close the phantom group, so replication could see two source transactions merged into one. The scan now reports the offset just past the last flagged entry alongside validEnd, computed in the same walk, and load() seeds there instead. Because writeBatch() writes a batch across multiple log files when it crosses a rotation, the prefix can span files, so the seed walks back through older sequences until one ends on a real transaction boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeding the committed watermark at the last complete transaction keeps a crash mid-batch out of committed reads, but only until the next commit: the leftover entries stay in the file, the watermark jumps past them on the first commitFinished(), and the next batch's TRANSACTION_LOG_ENTRY_LAST_FLAG then closes the phantom group -- the same two-transactions-merged-into-one exposure, one commit later. recoverTail() now drops those entries so the file itself ends on a transaction boundary. Nothing durable depends on them: writeBatch() completes before Transaction::Commit() in every commit path and both commit-thread lanes preserve dispatch order, so an interrupted log write is always the newest thing in the log and its RocksDB commit never ran. The discard is gated on proof that it is one interrupted batch of a flag-setting writer -- a boundary earlier in the same file, plus a single timestamp across the trailing run (writeEntriesV1 stamps every entry of a batch with the batch timestamp, and getMonotonicTimestamp() never repeats). Without that proof the bytes are kept and warned about, which is what protects a batch split across a rotation and a log written before the flag existed; the watermark seed still covers committed readers there. POSIX truncates. Windows overwrites the range with zeros -- its files are pre-extended and a zero timestamp is the end-of-entries marker, so a torn tail needs no repair there but real entries do -- and drops the cached read-only mapping, which is not coherent with WriteFile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rtial failure Zero the end-of-entries marker at `newSize` with its own write before the bulk zeroing loop. A reader stops at the first zero timestamp regardless of what follows, so once that write lands, `newSize` is a safe end-of-log position even if a later chunk fails partway through the rest of the range — the caller can lower `size` to `newSize` unconditionally instead of leaving it at `entriesEnd` while the on-disk marker already sits earlier, which would put the next append past where readers stop. Addresses PR #723 review feedback from cb1kenobi. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request addresses issues with failed appends leaving orphaned bytes on disk, which can cause framing breaks that prevent recovery. It ensures that partial writes are erased on failure, and if the erase itself fails, the file is retired from further appends. Additionally, short header writes during initialization now result in the file being discarded. Feedback was provided regarding the Windows implementation of writeBatchToFile, pointing out that lpNumberOfBytesWritten is unreliable on synchronous WriteFile failures and suggesting the use of SetFilePointer to accurately calculate the number of bytes that landed on disk.
kriszyp
added a commit
that referenced
this pull request
Aug 5, 2026
WriteFile does not promise to set lpNumberOfBytesWritten when a synchronous write fails. It is zero-initialized here, so the value can never be garbage, but it can be left at 0 after a partial write — and the caller erases exactly the range this reports, so under-reporting strands the bytes it missed and reopens the framing break. The file pointer is authoritative: the batch begins by seeking to `size`, so the distance from there is what actually reached the file. Falls back to the accumulated count if the query fails. Addresses PR #751 review feedback from gemini-code-assist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kriszyp
added a commit
that referenced
this pull request
Aug 5, 2026
The previous commit's fallback undid its own reasoning: if SetFilePointerEx failed it dropped back to lpNumberOfBytesWritten, the value it had just established WriteFile does not promise to set. Erasing a range derived from it can strand exactly the bytes the erase exists to remove. An unreportable extent is now TRANSACTION_LOG_BYTES_LANDED_UNKNOWN, and writeEntriesV1 retires the file rather than erasing a range it cannot bound. A figure larger than the bytes handed to the OS is treated the same way — nothing can have landed beyond what was attempted, so a larger number means the platform mis-reported, and an over-large erase would cut into committed entries. Also fail-safe the retirement itself: it is set before the erase and lifted only once the erase has succeeded. The Windows erase allocates, so a throw from inside it would otherwise leave the file appendable with orphaned bytes on disk — the same escape the batch-index restore was moved to avoid. Both branches are Windows-only in production, so a test-only forcedBytesLandedForTests override drives them from the POSIX build, where the deciding code is shared. Both tests fail with the guard removed. Addresses PR #751 review feedback from codex and gemini-code-assist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kriszyp
marked this pull request as ready for review
August 5, 2026 04:43
…ng them A transaction-log append that fails part-way through (ENOSPC, a short write on a full volume) left the bytes it had already written on disk. The fd is opened O_APPEND, so the next successful append landed *after* them: the log ended up with a partial entry embedded mid-file and valid entries on both sides — the one shape recoverTail() deliberately refuses to repair, since truncating there would discard committed transactions. Every reader stops at the break, so entries written after it become unreachable (HarperFast/harper#2016, HarperFast/harper#2063). writeBatchToFile() now reports how many bytes reached the file through a `bytesLanded` out-param instead of collapsing a hard error to -1, and writeEntriesV1() erases that range before throwing. It runs under fileMutex, `size` never advanced over the bytes, and the commit throws, so nothing acknowledged them. Reuses eraseTail(): POSIX truncates back to `size`; Windows zero-fills the range to restore its end-of-entries marker. If the erase itself fails, the resulting break is surfaced as a `log.warn` rather than left silent. The batch's currentEntryIndex is also restored, since none of its entries are on disk. Related hardening in open(): the three unchecked writes that initialize a new file's header are now a single checked write. A header that only partly landed still set `size` to the full header length, so the first append would be framed from the wrong offset for the life of the file — the same defect shape from the same full-disk incident. Fixes #748 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pre-push review (codex + gemini + harper-domain) found the initialization half of the same defect: open() now detects a partial header write, but left the bytes on disk. A file of 0 < size < HEADER_SIZE fails the "too small to be a valid transaction log file" check on every future open, and freeing disk space does not heal it — the segment is un-openable forever. Remove the file before throwing, so the path re-initializes cleanly once the write can complete. removeFile() splits into a fileMutex-acquiring wrapper and a removeFileLocked() body per platform, since open() already holds the lock. Covered by a new ROCKSDB_JS_WRITE test seam (the same pattern as ROCKSDB_JS_WRITEV/ROCKSDB_JS_MADVISE) that caps the header write; on base the test leaves a 5-byte file behind and the reopen throws. Also from review: trim comment narration and note that the Windows bytesLanded accounting relies on synchronous WriteFile, where lpNumberOfBytesWritten is populated even on failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e erased Round-2 pre-push review (codex + gemini + harper-domain) caught the erase failing on its own failure branch: when the append failed AND eraseTail could not remove what landed, the code warned and threw but left the file open O_APPEND. The next successful commit would then land past the orphan and recreate exactly the mid-file break this change exists to prevent — recoverTail() would refuse to repair it and every entry after it would be lost. A file in that state is now retired: writeEntriesV1 returns without writing, which the store already reads as "no progress" and answers by rotating to the next sequence. The orphaned bytes stay the trailing partial that open-time recovery can truncate, so the double fault degrades to a repairable torn tail instead of permanent data loss. Covered by a ROCKSDB_JS_FTRUNCATE seam driving the ENOSPC + EIO double fault; the test asserts the retired file takes no further entries and that the recovery scan still classifies it TruncateTail. Also from review: pid-qualify the test temp paths and drop two comments that narrated adjacent code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round-3 review: the currentEntryIndex restore ran after the erase and warn-emit block, both of which allocate. A bad_alloc there would skip the restore and replace the DBException, so a caller that catches and retries the batch would resume past entries that never reached disk — silently dropping them from the log while their RocksDB commit still runs. The restore has no dependency on the erase, so it moves first. Comment trims, third pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unrelated to this PR's fix, but it blocks its CI: the late-column-family
test asserted db.get() returns a Promise without ever awaiting it. When
dbRunner closes the database first, that in-flight read is aborted
("Database closed during get operation") and the rejection is unhandled,
so vitest exits non-zero with every test passing. Timing-dependent, and
macOS runners lose the race often — it failed the Deno and Node 24 macOS
jobs on this PR while the same commit passed elsewhere.
Pre-existing on main; this is the only unawaited get-promise assertion in
the suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
WriteFile does not promise to set lpNumberOfBytesWritten when a synchronous write fails. It is zero-initialized here, so the value can never be garbage, but it can be left at 0 after a partial write — and the caller erases exactly the range this reports, so under-reporting strands the bytes it missed and reopens the framing break. The file pointer is authoritative: the batch begins by seeking to `size`, so the distance from there is what actually reached the file. Falls back to the accumulated count if the query fails. Addresses PR #751 review feedback from gemini-code-assist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's fallback undid its own reasoning: if SetFilePointerEx failed it dropped back to lpNumberOfBytesWritten, the value it had just established WriteFile does not promise to set. Erasing a range derived from it can strand exactly the bytes the erase exists to remove. An unreportable extent is now TRANSACTION_LOG_BYTES_LANDED_UNKNOWN, and writeEntriesV1 retires the file rather than erasing a range it cannot bound. A figure larger than the bytes handed to the OS is treated the same way — nothing can have landed beyond what was attempted, so a larger number means the platform mis-reported, and an over-large erase would cut into committed entries. Also fail-safe the retirement itself: it is set before the erase and lifted only once the erase has succeeded. The Windows erase allocates, so a throw from inside it would otherwise leave the file appendable with orphaned bytes on disk — the same escape the batch-index restore was moved to avoid. Both branches are Windows-only in production, so a test-only forcedBytesLandedForTests override drives them from the POSIX build, where the deciding code is shared. Both tests fail with the guard removed. Addresses PR #751 review feedback from codex and gemini-code-assist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kriszyp
force-pushed
the
kris/txnlog-append-boundary
branch
from
August 7, 2026 22:42
9e74708 to
8fe614a
Compare
kriszyp
force-pushed
the
kris/txnlog-committed-position-recovery
branch
from
August 10, 2026 23:42
9676862 to
cea3330
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #748.
A transaction-log append that fails part-way through — a full disk, an exhausted quota, a short
write on a failing volume — left the bytes it had already written on disk, unaccounted for. The log
fd is opened
O_APPEND, so the next successful append landed after those bytes: a partial entrypermanently embedded mid-file with valid entries on both sides. That is the one shape open-time
recovery deliberately refuses to repair (truncating there would discard committed, replicated
transactions), so every reader stops at the break and everything written after it becomes
unreachable — the reader-side consequence seen in HarperFast/harper#2016 and HarperFast/harper#2063.
writeBatchToFile()now reports how many bytes reached the file through abytesLandedout-paraminstead of collapsing a hard error to
-1, andwriteEntriesV1()erases that range beforethrowing. Erasing is safe:
writeEntriesV1holdsfileMutexacross the whole append,sizeneveradvanced over the bytes, and the commit throws — nothing acknowledged them. The erase reuses
eraseTail()from #723 (POSIX truncates back tosize; Windows zero-fills the range to restore itsend-of-entries marker).
Three failure edges around that, each from a review round:
appendBoundaryLostflag makeswriteEntriesV1return without writing, which the store alreadyreads as "no progress" and answers by rotating to the next sequence. The orphaned bytes stay the
trailing partial that
recoverTail()can truncate, instead of becoming the mid-file break itmust leave intact. The double fault degrades to a repairable torn tail. Retirement is set before
the erase and lifted only on success, so a throw from inside the erase (the Windows path
allocates) cannot leave the file appendable either.
lpNumberOfBytesWritten, whichWriteFiledoes not promise to set on failure; if that query alsofails, the extent is reported as
TRANSACTION_LOG_BYTES_LANDED_UNKNOWNand the file is retiredrather than erased against a guess. A figure larger than the bytes handed to the OS is treated
the same way — an over-large erase would cut into committed entries.
open()initialized a new file's header with three unchecked writes; aheader that only partly landed still set
sizeto the full header length, framing the firstappend from the wrong offset for the life of the file. It is now one checked write, and a short
one removes the file — a size in
(0, HEADER_SIZE)failsopen()'s "too small" check on everyfuture open, and freeing disk space would not heal it. (
removeFile()splits into afileMutex-acquiring wrapper and a per-platformremoveFileLocked()body, sinceopen()alreadyholds the lock.)
batch.currentEntryIndexis rolled back before the erase and thewarning emit, both of which allocate; a
bad_allocthere must not leave the batch claimingentries that never reached disk.
Stacked on #723
This branches from
kris/txnlog-committed-position-recovery(#723), which introduceseraseTail().Review/merge that one first; the diff here is only this change.
Where to look
writeEntriesV1()'s error branch — the erase range (committedSize..committedSize + bytesLanded) has to match where the bytes actually landed on both platforms: POSIX appends atphysical EOF (which equals
size, hence a plain truncate), Windows seeks tosizebefore writing(hence a zero-fill of exactly that span).
TransactionLogStore::writeBatch— an unchangedsizeis the existing "no progress" signal that triggers rotation, the same mechanism a
max-size file uses. The flag is deliberately sticky for the object's life: it can only
over-retire, never under-retire, and a fresh process rebuilds the object with it clear after
recovery has had its chance at the trailing partial.
bytesLandedaccounting in bothwriteBatchToFile()implementations.Verification
Native GoogleTest, POSIX (
pnpm test:native). Failure injection uses the existing macro-seampattern (
ROCKSDB_JS_WRITEV, plus newROCKSDB_JS_WRITEandROCKSDB_JS_FTRUNCATE), defined onlyin the
rocksdb-js-native-testsgyp target — production builds resolve to the bare syscalls.FailedAppendLeavesTheLogOnAnEntryBoundary— fails on base with exactly the reported shape:file size 43 vs committed 37 (6 orphaned bytes), and after the next append the recovery scan
returns
MidFileCorruptionwith the entry count truncated at the break. Passes with the fix(scan
Clean, both entries readable).UnerasableOrphanRetiresTheFile— ENOSPC + failingftruncate: the retired file takes no furtherentries and the scan still classifies it
TruncateTail.ShortHeaderWriteDiscardsTheFileInsteadOfBrickingIt— fails on base: the 5-byte file survivesand the reopen throws "File is too small to be a valid transaction log file". Passes with the fix.
UnerasableExtent/UnknownandUnerasableExtent/OverReported— both fail on base with theguard removed: the file is either appended past or erased against an untrustworthy range.
AppendThatWritesNothingLeavesTheFileUntouched, plus twoWriteBatchToFiletests forbytesLandedon a hard error and on a nothing-written error.Full gates on macOS:
pnpm test:native120 passed (3 madvise tests skipped — Linux-only),pnpm test728 passed / 1 skipped,pnpm checkclean.Open concerns from the pre-push review
Carried here rather than resolved, so a reviewer does not have to rediscover them:
TransactionLogFile. Nothing exercises the retire → rotate → retry contractthrough
TransactionLogStore::writeBatch; the native-test target does not link the store (itneeds RocksDB), and inducing a real
ENOSPCfrom the JS suite would need a filled volume. Thestore side was verified by reading it (
transaction_log_store.cpp:831rotates on unchanged size),not by a test.
bytesLandedaccounting and zero-fill erase are exercisedonly by fix(txnlog): recover the committed watermark from the log tail on load #723's own
eraseTailcoverage; there is noWriteFileseam, and I cannot build Windowslocally — CI is the only verification for that backend. The decision logic those values feed
(retire-vs-erase) is shared, and a test-only
forcedBytesLandedForTestsoverride drives it fromthe POSIX build.
writeBatch's open-retry loop is unbounded, so apersistently failing volume spins through sequence numbers. A short header write is a new way to
reach that loop, but
openFile()already reached it under the same ENOSPC condition. Worth itsown issue rather than widening this one.
Verified false and dismissed along the way:
removeFileLocked()closes the file handle beforeunlinking (no Windows sharing violation), and
bytesWrittenis zero-initialized before everyWriteFile, so it could never have contained garbage — the repeated "uninitialized stack memory"framing of the Windows finding was wrong, even though the underlying under-reporting risk was real
and is now fixed.
One unrelated commit
test(block-cache): await the second read so teardown cannot abort itfixes a pre-existing flakethat was blocking this PR's CI, not anything introduced here.
test/block-cache.test.tsasserteddb.get()returns a Promise without awaiting it; whendbRunnerclosed the database first, thein-flight read was aborted and the unhandled rejection failed the run with every test passing. It
took out the Deno and Node 24 macOS jobs on this branch while the same commit passed everywhere
else. Identical code is on
main, and it is the only unawaited get-promise assertion in the suite.Reviewed by codex + gemini + harper-domain across four rounds. Generated by Claude Opus 5.
Human-Review-Need: 4 @ 8fe614a