Skip to content

fix(txnlog): erase a failed append's partial bytes instead of baking a framing break into the log - #751

Open
kriszyp wants to merge 19 commits into
kris/txnlog-committed-position-recoveryfrom
kris/txnlog-append-boundary
Open

fix(txnlog): erase a failed append's partial bytes instead of baking a framing break into the log#751
kriszyp wants to merge 19 commits into
kris/txnlog-committed-position-recoveryfrom
kris/txnlog-append-boundary

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 5, 2026

Copy link
Copy Markdown
Member

Fixes #748.

A transaction-log append that fails part-way through — a full disk, an exhausted quota, a short
write on a failing volume — left the bytes it had already written on disk, unaccounted for. The log
fd is opened O_APPEND, so the next successful append landed after those bytes: a partial entry
permanently embedded mid-file with valid entries on both sides. That is the one shape open-time
recovery deliberately refuses to repair (truncating there would discard committed, replicated
transactions), so every reader stops at the break and everything written after it becomes
unreachable — the reader-side consequence seen in HarperFast/harper#2016 and HarperFast/harper#2063.

writeBatchToFile() now reports how many bytes reached the file through a bytesLanded out-param
instead of collapsing a hard error to -1, and writeEntriesV1() erases that range before
throwing. Erasing is safe: writeEntriesV1 holds fileMutex across the whole append, size never
advanced over the bytes, and the commit throws — nothing acknowledged them. The erase reuses
eraseTail() from #723 (POSIX truncates back to size; Windows zero-fills the range to restore its
end-of-entries marker).

Three failure edges around that, each from a review round:

  • The erase itself fails (ENOSPC then EIO). The file is retired rather than warned about: an
    appendBoundaryLost flag makes writeEntriesV1 return without writing, which the store already
    reads as "no progress" and answers by rotating to the next sequence. The orphaned bytes stay the
    trailing partial that recoverTail() can truncate, instead of becoming the mid-file break it
    must leave intact. The double fault degrades to a repairable torn tail. Retirement is set before
    the erase and lifted only on success, so a throw from inside the erase (the Windows path
    allocates) cannot leave the file appendable either.
  • The landed extent is unreportable. Windows takes it from the file pointer rather than
    lpNumberOfBytesWritten, which WriteFile does not promise to set on failure; if that query also
    fails, the extent is reported as TRANSACTION_LOG_BYTES_LANDED_UNKNOWN and the file is retired
    rather than erased against a guess. A figure larger than the bytes handed to the OS is treated
    the same way — an over-large erase would cut into committed entries.
  • A short header write. open() initialized a new file's header with three unchecked writes; a
    header that only partly landed still set size to the full header length, framing the first
    append from the wrong offset for the life of the file. It is now one checked write, and a short
    one removes the file — a size in (0, HEADER_SIZE) fails open()'s "too small" check on every
    future open, and freeing disk space would not heal it. (removeFile() splits into a
    fileMutex-acquiring wrapper and a per-platform removeFileLocked() body, since open() already
    holds the lock.)
  • A throw during cleanup. batch.currentEntryIndex is rolled back before the erase and the
    warning emit, both of which allocate; a bad_alloc there must not leave the batch claiming
    entries that never reached disk.

Stacked on #723

This branches from kris/txnlog-committed-position-recovery (#723), which introduces eraseTail().
Review/merge that one first; the diff here is only this change.

Where to look

  • writeEntriesV1()'s error branch — the erase range (committedSize .. committedSize + bytesLanded) has to match where the bytes actually landed on both platforms: POSIX appends at
    physical EOF (which equals size, hence a plain truncate), Windows seeks to size before writing
    (hence a zero-fill of exactly that span).
  • The retirement path and its contract with TransactionLogStore::writeBatch — an unchanged size
    is the existing "no progress" signal that triggers rotation, the same mechanism a
    max-size file uses. The flag is deliberately sticky for the object's life: it can only
    over-retire, never under-retire, and a fresh process rebuilds the object with it clear after
    recovery has had its chance at the trailing partial.
  • bytesLanded accounting in both writeBatchToFile() implementations.

Verification

Native GoogleTest, POSIX (pnpm test:native). Failure injection uses the existing macro-seam
pattern (ROCKSDB_JS_WRITEV, plus new ROCKSDB_JS_WRITE and ROCKSDB_JS_FTRUNCATE), defined only
in the rocksdb-js-native-tests gyp target — production builds resolve to the bare syscalls.

  • FailedAppendLeavesTheLogOnAnEntryBoundaryfails on base with exactly the reported shape:
    file size 43 vs committed 37 (6 orphaned bytes), and after the next append the recovery scan
    returns MidFileCorruption with the entry count truncated at the break. Passes with the fix
    (scan Clean, both entries readable).
  • UnerasableOrphanRetiresTheFile — ENOSPC + failing ftruncate: the retired file takes no further
    entries and the scan still classifies it TruncateTail.
  • ShortHeaderWriteDiscardsTheFileInsteadOfBrickingItfails on base: the 5-byte file survives
    and the reopen throws "File is too small to be a valid transaction log file". Passes with the fix.
  • UnerasableExtent/Unknown and UnerasableExtent/OverReported — both fail on base with the
    guard removed: the file is either appended past or erased against an untrustworthy range.
  • AppendThatWritesNothingLeavesTheFileUntouched, plus two WriteBatchToFile tests for
    bytesLanded on a hard error and on a nothing-written error.

Full gates on macOS: pnpm test:native 120 passed (3 madvise tests skipped — Linux-only),
pnpm test 728 passed / 1 skipped, pnpm check clean.

Open concerns from the pre-push review

Carried here rather than resolved, so a reviewer does not have to rediscover them:

  • Coverage stops at TransactionLogFile. Nothing exercises the retire → rotate → retry contract
    through TransactionLogStore::writeBatch; the native-test target does not link the store (it
    needs RocksDB), and inducing a real ENOSPC from the JS suite would need a filled volume. The
    store side was verified by reading it (transaction_log_store.cpp:831 rotates on unchanged size),
    not by a test.
  • Windows has no fault injection. Its bytesLanded accounting and zero-fill erase are exercised
    only by fix(txnlog): recover the committed watermark from the log tail on load #723's own eraseTail coverage; there is no WriteFile seam, and I cannot build Windows
    locally — CI is the only verification for that backend. The decision logic those values feed
    (retire-vs-erase) is shared, and a test-only forcedBytesLandedForTests override drives it from
    the POSIX build.
  • Pre-existing, not introduced here: writeBatch's open-retry loop is unbounded, so a
    persistently failing volume spins through sequence numbers. A short header write is a new way to
    reach that loop, but openFile() already reached it under the same ENOSPC condition. Worth its
    own issue rather than widening this one.

Verified false and dismissed along the way: removeFileLocked() closes the file handle before
unlinking (no Windows sharing violation), and bytesWritten is zero-initialized before every
WriteFile, so it could never have contained garbage — the repeated "uninitialized stack memory"
framing of the Windows finding was wrong, even though the underlying under-reporting risk was real
and is now fixed.

One unrelated commit

test(block-cache): await the second read so teardown cannot abort it fixes a pre-existing flake
that was blocking this PR's CI, not anything introduced here. test/block-cache.test.ts asserted
db.get() returns a Promise without awaiting it; when dbRunner closed the database first, the
in-flight read was aborted and the unhandled rejection failed the run with every test passing. It
took out the Deno and Node 24 macOS jobs on this branch while the same commit passed everywhere
else. Identical code is on main, and it is the only unawaited get-promise assertion in the suite.

Reviewed by codex + gemini + harper-domain across four rounds. Generated by Claude Opus 5.

Human-Review-Need: 4 @ 8fe614a

kriszyp and others added 11 commits July 26, 2026 18:32
The transaction log's committed watermark (lastCommittedPosition) is in-memory
state advanced by commitFinished(); it is never persisted. It was seeded on
load from txn.state -- the *flushed* position, i.e. how far RocksDB has already
absorbed the log -- which after an unclean exit sits behind the log's recovered
tail. Every committed read is bounded by that watermark, so entries that were
durable on disk (and that a consumer's boot replay re-applies via
readUncommitted) stayed invisible to committed readers until an unrelated later
commit advanced the watermark past the whole tail at once.

load() already recovers the true end: recoverTail() truncates any torn tail,
nextLogPosition is set to the last structurally valid entry, and it is inserted
as the write-head sentinel. commitFinished() defines the watermark as the front
of uncommittedTransactionPositions, which in a freshly loaded store is exactly
nextLogPosition -- so seeding it there only makes load() agree with the rule the
store already enforces.

Fixes HarperFast/harper#1949

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cross-model review findings: the fixture's `ready` handshake went through an
async console.log to a pipe, which the following SIGKILL could drop; a self-kill
on Windows surfaces as exit code 1, not a signal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
findPositionByTimestamp()'s indexing loop checked `entryTimestamp == 0`
before checking whether the current position was the file header's
timestamp slot. A header timestamp of exactly 0 (e.g. an unset/epoch
value, as several test fixtures write) was therefore treated as the
zero-padding end-of-data marker and truncated this->size back into the
header itself -- below TRANSACTION_LOG_FILE_HEADER_SIZE.

This is normally invisible: only TransactionLogFile::openFile() on
Windows proactively calls findPositionByTimestamp() at open time (to
correct for mmap zero-padding), so the corruption only ever surfaced
there. It stayed latent until 13d57f7/f385ace8 started seeding
lastCommittedPosition from the recovered nextLogPosition on load,
which propagated the corrupted size into a value CI asserts on --
failing every Windows runtime (Node 22/24/26, Bun, Deno) on PR #723's
new/pre-existing "should return valid lastCommittedPosition after
purging earlier log files and reopening" case.

Root cause: the header-timestamp branch and the end-of-data check were
ordered so the latter could fire on a position that isn't an entry.
Reordering to always handle the header slot first (as the timestamp
index's designated position-0 entry) restores the invariant that the
end-of-data heuristic only ever applies to real entries.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the regression test for b10e725. The indexing path is cross-platform
even though only Windows openFile() reaches it at open, so the test drives it
directly through _findPosition and fails without that fix on any platform.

Also tests the committed-watermark seed on logSequenceNumber rather than
fullPosition: that union member aliases the two uint32s as a double, so a
sequence number >= 0x7ff00000 reads back as NaN or negative and `> 0` would
silently skip the seed, falling back to the stale txn.state position this PR
exists to fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Windows has no real signals, so Node maps a self-kill to TerminateProcess(h, 1).
Accepting code 1 on every platform would let a POSIX run whose kill() threw
after the `ready` handshake pass with normal addon teardown, which is the very
thing the fixture must rule out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ction

Only a batch's final entry carries TRANSACTION_LOG_ENTRY_LAST_FLAG, so a crash
partway through a multi-entry transaction leaves whole, well-framed entries that
are merely a prefix of it. recoverTail() deliberately keeps those bytes -- they
are structurally valid and a readUncommitted replay still wants them -- but
seeding the committed watermark at the write head published them to committed
readers, exposing a transaction that never closed. The next transaction's flag
would then close the phantom group, so replication could see two source
transactions merged into one.

The scan now reports the offset just past the last flagged entry alongside
validEnd, computed in the same walk, and load() seeds there instead. Because
writeBatch() writes a batch across multiple log files when it crosses a
rotation, the prefix can span files, so the seed walks back through older
sequences until one ends on a real transaction boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeding the committed watermark at the last complete transaction keeps a crash
mid-batch out of committed reads, but only until the next commit: the leftover
entries stay in the file, the watermark jumps past them on the first
commitFinished(), and the next batch's TRANSACTION_LOG_ENTRY_LAST_FLAG then
closes the phantom group -- the same two-transactions-merged-into-one exposure,
one commit later.

recoverTail() now drops those entries so the file itself ends on a transaction
boundary. Nothing durable depends on them: writeBatch() completes before
Transaction::Commit() in every commit path and both commit-thread lanes preserve
dispatch order, so an interrupted log write is always the newest thing in the log
and its RocksDB commit never ran.

The discard is gated on proof that it is one interrupted batch of a flag-setting
writer -- a boundary earlier in the same file, plus a single timestamp across the
trailing run (writeEntriesV1 stamps every entry of a batch with the batch
timestamp, and getMonotonicTimestamp() never repeats). Without that proof the
bytes are kept and warned about, which is what protects a batch split across a
rotation and a log written before the flag existed; the watermark seed still
covers committed readers there.

POSIX truncates. Windows overwrites the range with zeros -- its files are
pre-extended and a zero timestamp is the end-of-entries marker, so a torn tail
needs no repair there but real entries do -- and drops the cached read-only
mapping, which is not coherent with WriteFile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rtial failure

Zero the end-of-entries marker at `newSize` with its own write before
the bulk zeroing loop. A reader stops at the first zero timestamp
regardless of what follows, so once that write lands, `newSize` is a
safe end-of-log position even if a later chunk fails partway through
the rest of the range — the caller can lower `size` to `newSize`
unconditionally instead of leaving it at `entriesEnd` while the
on-disk marker already sits earlier, which would put the next append
past where readers stop.

Addresses PR #723 review feedback from cb1kenobi.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses issues with failed appends leaving orphaned bytes on disk, which can cause framing breaks that prevent recovery. It ensures that partial writes are erased on failure, and if the erase itself fails, the file is retired from further appends. Additionally, short header writes during initialization now result in the file being discarded. Feedback was provided regarding the Windows implementation of writeBatchToFile, pointing out that lpNumberOfBytesWritten is unreliable on synchronous WriteFile failures and suggesting the use of SetFilePointer to accurately calculate the number of bytes that landed on disk.

Comment thread src/binding/transaction_log/transaction_log_file_windows.cpp Outdated
kriszyp added a commit that referenced this pull request Aug 5, 2026
WriteFile does not promise to set lpNumberOfBytesWritten when a
synchronous write fails. It is zero-initialized here, so the value can
never be garbage, but it can be left at 0 after a partial write — and the
caller erases exactly the range this reports, so under-reporting strands
the bytes it missed and reopens the framing break.

The file pointer is authoritative: the batch begins by seeking to `size`,
so the distance from there is what actually reached the file. Falls back
to the accumulated count if the query fails.

Addresses PR #751 review feedback from gemini-code-assist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 5, 2026
The previous commit's fallback undid its own reasoning: if
SetFilePointerEx failed it dropped back to lpNumberOfBytesWritten, the
value it had just established WriteFile does not promise to set. Erasing
a range derived from it can strand exactly the bytes the erase exists to
remove.

An unreportable extent is now TRANSACTION_LOG_BYTES_LANDED_UNKNOWN, and
writeEntriesV1 retires the file rather than erasing a range it cannot
bound. A figure larger than the bytes handed to the OS is treated the
same way — nothing can have landed beyond what was attempted, so a larger
number means the platform mis-reported, and an over-large erase would cut
into committed entries.

Also fail-safe the retirement itself: it is set before the erase and
lifted only once the erase has succeeded. The Windows erase allocates, so
a throw from inside it would otherwise leave the file appendable with
orphaned bytes on disk — the same escape the batch-index restore was
moved to avoid.

Both branches are Windows-only in production, so a test-only
forcedBytesLandedForTests override drives them from the POSIX build,
where the deciding code is shared. Both tests fail with the guard removed.

Addresses PR #751 review feedback from codex and gemini-code-assist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kriszyp
kriszyp marked this pull request as ready for review August 5, 2026 04:43
cb1kenobi and others added 8 commits August 7, 2026 16:42
…ng them

A transaction-log append that fails part-way through (ENOSPC, a short
write on a full volume) left the bytes it had already written on disk.
The fd is opened O_APPEND, so the next successful append landed *after*
them: the log ended up with a partial entry embedded mid-file and valid
entries on both sides — the one shape recoverTail() deliberately refuses
to repair, since truncating there would discard committed transactions.
Every reader stops at the break, so entries written after it become
unreachable (HarperFast/harper#2016, HarperFast/harper#2063).

writeBatchToFile() now reports how many bytes reached the file through a
`bytesLanded` out-param instead of collapsing a hard error to -1, and
writeEntriesV1() erases that range before throwing. It runs under
fileMutex, `size` never advanced over the bytes, and the commit throws,
so nothing acknowledged them. Reuses eraseTail(): POSIX truncates back
to `size`; Windows zero-fills the range to restore its end-of-entries
marker. If the erase itself fails, the resulting break is surfaced as a
`log.warn` rather than left silent.

The batch's currentEntryIndex is also restored, since none of its
entries are on disk.

Related hardening in open(): the three unchecked writes that initialize
a new file's header are now a single checked write. A header that only
partly landed still set `size` to the full header length, so the first
append would be framed from the wrong offset for the life of the file —
the same defect shape from the same full-disk incident.

Fixes #748

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pre-push review (codex + gemini + harper-domain) found the initialization
half of the same defect: open() now detects a partial header write, but
left the bytes on disk. A file of 0 < size < HEADER_SIZE fails the "too
small to be a valid transaction log file" check on every future open, and
freeing disk space does not heal it — the segment is un-openable forever.
Remove the file before throwing, so the path re-initializes cleanly once
the write can complete.

removeFile() splits into a fileMutex-acquiring wrapper and a
removeFileLocked() body per platform, since open() already holds the lock.

Covered by a new ROCKSDB_JS_WRITE test seam (the same pattern as
ROCKSDB_JS_WRITEV/ROCKSDB_JS_MADVISE) that caps the header write; on base
the test leaves a 5-byte file behind and the reopen throws.

Also from review: trim comment narration and note that the Windows
bytesLanded accounting relies on synchronous WriteFile, where
lpNumberOfBytesWritten is populated even on failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e erased

Round-2 pre-push review (codex + gemini + harper-domain) caught the erase
failing on its own failure branch: when the append failed AND eraseTail
could not remove what landed, the code warned and threw but left the file
open O_APPEND. The next successful commit would then land past the orphan
and recreate exactly the mid-file break this change exists to prevent —
recoverTail() would refuse to repair it and every entry after it would be
lost.

A file in that state is now retired: writeEntriesV1 returns without
writing, which the store already reads as "no progress" and answers by
rotating to the next sequence. The orphaned bytes stay the trailing
partial that open-time recovery can truncate, so the double fault
degrades to a repairable torn tail instead of permanent data loss.

Covered by a ROCKSDB_JS_FTRUNCATE seam driving the ENOSPC + EIO double
fault; the test asserts the retired file takes no further entries and
that the recovery scan still classifies it TruncateTail.

Also from review: pid-qualify the test temp paths and drop two comments
that narrated adjacent code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round-3 review: the currentEntryIndex restore ran after the erase and
warn-emit block, both of which allocate. A bad_alloc there would skip the
restore and replace the DBException, so a caller that catches and retries
the batch would resume past entries that never reached disk — silently
dropping them from the log while their RocksDB commit still runs. The
restore has no dependency on the erase, so it moves first.

Comment trims, third pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unrelated to this PR's fix, but it blocks its CI: the late-column-family
test asserted db.get() returns a Promise without ever awaiting it. When
dbRunner closes the database first, that in-flight read is aborted
("Database closed during get operation") and the rejection is unhandled,
so vitest exits non-zero with every test passing. Timing-dependent, and
macOS runners lose the race often — it failed the Deno and Node 24 macOS
jobs on this PR while the same commit passed elsewhere.

Pre-existing on main; this is the only unawaited get-promise assertion in
the suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
WriteFile does not promise to set lpNumberOfBytesWritten when a
synchronous write fails. It is zero-initialized here, so the value can
never be garbage, but it can be left at 0 after a partial write — and the
caller erases exactly the range this reports, so under-reporting strands
the bytes it missed and reopens the framing break.

The file pointer is authoritative: the batch begins by seeking to `size`,
so the distance from there is what actually reached the file. Falls back
to the accumulated count if the query fails.

Addresses PR #751 review feedback from gemini-code-assist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's fallback undid its own reasoning: if
SetFilePointerEx failed it dropped back to lpNumberOfBytesWritten, the
value it had just established WriteFile does not promise to set. Erasing
a range derived from it can strand exactly the bytes the erase exists to
remove.

An unreportable extent is now TRANSACTION_LOG_BYTES_LANDED_UNKNOWN, and
writeEntriesV1 retires the file rather than erasing a range it cannot
bound. A figure larger than the bytes handed to the OS is treated the
same way — nothing can have landed beyond what was attempted, so a larger
number means the platform mis-reported, and an over-large erase would cut
into committed entries.

Also fail-safe the retirement itself: it is set before the erase and
lifted only once the erase has succeeded. The Windows erase allocates, so
a throw from inside it would otherwise leave the file appendable with
orphaned bytes on disk — the same escape the batch-index restore was
moved to avoid.

Both branches are Windows-only in production, so a test-only
forcedBytesLandedForTests override drives them from the POSIX build,
where the deciding code is shared. Both tests fail with the guard removed.

Addresses PR #751 review feedback from codex and gemini-code-assist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the kris/txnlog-append-boundary branch from 9e74708 to 8fe614a Compare August 7, 2026 22:42
@kriszyp
kriszyp force-pushed the kris/txnlog-committed-position-recovery branch from 9676862 to cea3330 Compare August 10, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants