Skip to content

fix(config): read root config synchronously and bound rename retries by wall clock - #2191

Open
kriszyp wants to merge 34 commits into
mainfrom
kris/win-rename-retry-budget
Open

fix(config): read root config synchronously and bound rename retries by wall clock#2191
kriszyp wants to merge 34 commits into
mainfrom
kris/win-rename-retry-budget

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 17, 2026

Copy link
Copy Markdown
Member

The Windows EPERM: rename harper-config.yaml.tmp was never a budget that was too short: the
worker was blocked against a descriptor its own root-config watcher held. libuv opens a
fsPromises.readFile descriptor on the threadpool but closes it from JS, which cannot run while
the same thread sits in atomicWriteFile's Atomics.wait — so no rename budget could have been
long enough, and the extra 6.4 seconds an earlier head bought was pure event-loop stall on a path
set_configuration reaches from a live request thread.

So the write side goes back to the 3,630 ms window
it had before this branch, and the read side stops holding a handle across a yield: both root-config
watchers read through readConfigFileSync,
and a read that still loses is retried from a timer,
which holds no descriptor either.

Making the read synchronous exposed three states the threadpool round-trip had been hiding, each of
which ends with a worker holding a config nothing will ever correct:

  • A write can land before the watch is armed. ready used to be emitted from inside chokidar's
    initial add; on darwin a write in that window is lost outright (0 ms lost, 5 ms delivered, as
    measured on this PR). ready is now gated on chokidar's own ready plus a
    darwin-only grace,
    and arming always re-reads
    rather than publishing what an earlier read staged.
  • A non-atomic writer truncates before it writes, and the synchronous read is fast enough to land
    in that window — while chokidar throttles away the event carrying the content as a duplicate. An
    empty read now takes the same bounded ladder a locked one does, in both watchers. On Linux this was
    already reproducible: should instantiate and watch the root Harper config file failed 4 of 6 runs
    at this branch's earlier head and 0 of 6 at its merge base.
  • A read can end with no config at all — the ladder spent, the file unparseable, no file to read.
    harper_logger.start() and Scope.ready await those barriers with no timeout, so each outcome now
    boots on defaults and logs what failed
    instead of hanging the worker. It settles carrying nothing rather than {}, because
    updateLogSettings
    reads an empty config as "rotation off, console off" — the one boot that could not read its config
    is the worst one to silently disable logging on.

A config that arrives after a scope has settled is the other half of settling early: nothing
downstream re-runs on it, because componentLoader is long past its await scope.ready. Scope now
answers a repeat ready
the same way it answers remove — by requesting a restart — so a worker that read the file in a
truncate window cannot keep serving the defaults while its siblings serve the operator's config.

Two smaller ones on the way through: a parse failure goes through
parseConfigFile
with logLevel: 'error', so neither yaml's framed error nor its process.emitWarning path can put
config source — credentials included — into a log line; and a chokidar callback queued before
close() is dropped on entry instead of reading a file a shutting-down worker no longer wants.
DESIGN.md carries all of these invariants so neither watcher gets "modernized" back to
fsPromises.readFile.

For the human reviewer

  1. Root-config watchers now block the event loop to read (readFileSync, plus
    up to 500 ms of Atomics.wait on win32). The alternative is to make the writer stop blocking — an async
    rename retry, or moving set_configuration's write off the request thread — which leaves the
    3.63 s synchronous stall this change preserves at the other end. That is the larger API change,
    and reversing this one later means re-touching both watchers and DESIGN.md.
  2. ready is not once-per-watcher, and a repeat now costs a restart. OptionsWatcher emits it
    whenever a scope goes from having no config of its own to having one — the recreated-config-file
    path, and a scope that booted while the file was unreadable — and Scope converts a repeat into
    requestRestart(), on the same terms as its remove listener (a plugin with its own ready
    handler owns the response). The narrower alternative is to leave the arrival inert and accept
    per-worker config divergence until the next restart. Worth a look:
    #applyScopedConfig
    guards on isDeepStrictEqual so a re-read of an unchanged falsy scope value (myPlugin: with no
    body) is not a transition; without that guard every ladder rung of one would request a restart.
  3. The read gates its retry to win32; the rename does not (same three codes, any platform).
    Deliberate: a misclassified read falls through to the timer ladder and still recovers, a rename
    has nothing to fall through to, and process.platform does not answer whether this filesystem
    can replace an open file (WSL drvfs, CIFS/SMB, Docker Desktop bind mounts all report linux).
    The cost is that a genuine Linux EACCES on rename spends the 3.63 s budget before surfacing.
    One-line flip either way. ENOENT is excluded from the read ladder entirely — a missing file is
    not a lock, and OptionsWatcher has always settled it at once as the install window.
  4. A terminal failure in the boot window starts the scope on defaults, rather than failing the
    boot closed. It matches the ENOENT branch beside it, but a worker can now boot a plugin on
    defaults where it previously would have waited — and a Harper running on the wrong config is
    arguably worse than one that refuses to start. That policy call is the same in both watchers by
    design; changing it is a one-place change in each.
  5. The read-retry ladder gives up permanently. After 500 ms of blocking plus a 3.1 s ladder the
    watcher warns and keeps the last valid config, and no further event is coming — the rename
    already fired. A lock outliving ~3.6 s therefore leaves that one worker on the old config until
    the next write or a restart, while its siblings apply the new one, and set_configuration still
    reports success. Strictly better than main's silent swallow; a slow re-arm or a periodic
    reconciliation would close it and is purely additive.
  6. One 500 ms read deadline per path per thread, with a one-budget grace after
    it expires.
    A worker holds 10+ OptionsWatchers over the same root config, so a per-call budget
    would serialize into N × 500 ms of blocked event loop; sharing it means the first watcher spends
    the window and its siblings fail fast into the ladder. What remains per config event is N
    synchronous reads of one small file, which used to run on the threadpool.
  7. The 20 ms darwin arming grace is one measurement on one machine. 0 ms loses the write, 5 ms
    delivers it, and chokidar's own ready alone does not close the race (the reviewer tested it). A
    slower or loaded darwin host silently reopens the window. An observable arming signal would remove
    the timing assumption; chokidar does not expose one.
  8. OptionsWatcher shares the arming re-read, but not the arming barrier. This PR's switch to
    synchronous reads is what opened the unarmed window for the 10+ root-config OptionsWatchers
    componentLoader creates per worker (at the merge base they all read via fsPromises.readFile,
    whose threadpool round-trip deferred past it), so the gate is now
    shared
    rather than left to RootConfigWatcher. What OptionsWatcher takes is the re-read that recovers
    the otherwise-undeliverable write; its ready still goes out on the first read, so it means "the
    config has been read", not "armed". The ordering difference is safe because Scope attaches its
    listeners in its constructor, before any read — the recovered write arrives as a post-ready
    change rather than being lost. Holding ready behind arming too would need every terminal
    outcome to open a second barrier per scope, with a boot hang as the failure mode.
    Still declined and left for a follow-up: the lock ladder stays gated to root-config filenames,
    so an application config.yaml held by AV on Windows goes stale with only a log line. That one is
    genuinely pre-existing — an async read of a locked file failed the same way — and closing it means
    touching every component scope's readiness.
  9. Two more _*ForTests seams ship on OptionsWatcher_refreshForTests() and a read
    counter. They match the class's existing members and are what lets a test tell a ladder rung
    from a chokidar event (chokidar reports the unlocking chmod as a change of its own), but
    _refreshForTests() does expose a synchronous config re-read to any caller.
  10. atomicWriteFile now throws RangeError on non-finite/negative retry options instead of
    clamping. A new throw on the config-write path, whose only non-default callers are tests.
  11. The new writer tests stub renameSync and writeFileSync (Sinon), which AGENTS.md tells new tests not to do.
    Deliberate: neither case can be provoked from a real Linux filesystem — a POSIX rename over an
    open file succeeds, and a part-way ENOSPC write cannot be staged — and an earlier review
    thread asked for a test of the deadline rather than of the constant. The reader-side tests take
    the no-stub route (a real mode-000 file).
  12. A late exhaustion error from the generation that already failed is attributed to its
    replacement.
    Both watchers guard the terminal "the replacement failed too" branch on
    #openCount > 1, which is true the moment the replacement opens — so an ENOSPC still draining
    from generation 1 settles the barrier as a failure even though the polling watcher is healthy.
    The consequence is bounded: the settle is on the defaults, and the replacement's own arming
    re-read then publishes the real config as a change, so it costs an early settle rather than a
    blind scope. Distinguishing them properly means tagging errors with the generation that emitted
    them, in both watchers — worth doing, not worth doing here.
  13. An intentionally empty config costs a 3.1 s boot. An empty read is believed only after the
    full ladder, so an operator who deliberately leaves harper-config.yaml empty delays
    harper_logger.start() and every root scope by the whole budget, once per boot. The alternative
    is distinguishing a truncate window from an empty file by size or mtime, which is a detection
    scheme rather than a constant — cheap to retune, not cheap to replace.
  14. The atomic write is atomic but not durable. writeFileSync + renameSync with no fsync
    on the temp descriptor or the parent directory, so on ext4 with delayed allocation a crash
    shortly after set_configuration can leave a present, zero-length config file. Pre-existing,
    and the new empty-read handling downgrades it from a hang to "boot on defaults with a warning",
    but the operator's configuration is silently not in force until someone rewrites it. Two lines
    in a function this branch already rewrites — deliberately not taken here, because an fsync on
    the set_configuration request thread is its own latency decision.

Verification

  • npm run build, npm run lint, npx prettier --check — pass at this head (lint reports 12
    warnings, all in integrationTests/ files this branch does not touch; they reproduce on the merge
    base).
  • npm run test:unit:main — 4499 passing, 195 pending, 2 failing. Both failures are environmental
    and outside this diff: a JWT RSA-key fixture (unitTests/security/tokenAuthentication.test.js)
    and configValidator's domain-socket path-length warning, which trips on the long worktree path
    this ran in. Neither file, nor anything they import, is touched by this branch.
  • npx mocha "unitTests/components/**/*test*.js" "unitTests/config/**/*.js" — 1774 passing, which
    covers both watchers, Scope, componentLoader, configUtils and the env overlay.
  • npm run test:unit:logging — 127 passing, 12 failing; all 12 fail identically on the merge base
    (11a1c489) in a clean worktree, so they are pre-existing and not this diff.
  • Each new regression test was checked against the un-fixed build: reverting the fix in dist/ makes
    it fail, so none of them pass vacuously. Worth knowing when reading earlier verification claims on
    this branch: #src/* resolves to ./dist/*.js unless NODE_OPTIONS=--conditions=typestrip, so a
    unit run without a preceding npm run build exercises the previous build, not the working tree.
  • npm run test:integration -- integrationTests/apiTests/configuration.test.mjs — 25 passed on an
    earlier head of this branch; the config surface has not changed since.
  • The deadline test fails on origin/main as expected: the attempt-count loop it replaced takes
    ~3.63 s, past the test's upper bound.
  • Linux unit runs do not exercise Windows sharing semantics. The retry cases fabricate a denied
    read with a mode-000 file, which is inert on Windows and ignored by root, so those cases skip
    on exactly the platform the fix targets. The Windows integration job — which reproduced the
    original EPERM exhaustion on an earlier head — is the only end-to-end oracle here.

One CI datum worth a second look, because a Linux run cannot produce it. The first CI pass at
this head failed Integration Tests 2/6 (Windows)describe-metadata-upgrade.test.ts, whose
restart_service probe never became ready inside 120 s. It passed on the re-run and every other
shard was green, so it is a flake by the usual test, but the mechanism is close enough to this
change to name: the Harper log for the failed run carries JavaScript execution has taken too long and is not allowing proper event queue cycling on main/0 covering 17.5 s, starting the
moment Restarting http_workers was logged. The same test passes on main (255 s vs the 383 s
timeout here). 17.5 s is far past any single budget this branch sets (500 ms of read blocking,
3.63 s of rename retry), so if it is this change it would have to be a burst of config events
each taking a fresh blocking window on the main thread rather than one long wait — which is
point 6's shared-deadline behaviour under a rename storm. One green re-run does not settle that;
the next Windows run that stalls should be read with this in mind.

One reviewer nit is partly declined: the in-source rationale for RENAME_RETRY_BUDGET_MS reads as
narration against Harper's zero-new-comment default, but an earlier review thread on this PR asked
for exactly that ("neither budget constant is explained in source any more"), so it stays and the
mechanics live in DESIGN.md.

Refs #2191

Review-Coverage: authored=claude; ran=gemini; blocked=codex(timeout),claude(fallback)(out-of-budget),domain(timeout); declined=cursor-grok,cursor-composer; rounds=23 @ 5eb33f7

Human-Review-Need: 3 @ 5eb33f7

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the maximum retry attempts for renaming files (RENAME_RETRY_MAX_ATTEMPTS) from 12 to 25 in config/configUtils.ts to extend the retry budget to approximately 10 seconds. The corresponding unit test in unitTests/config/configUtils.test.js has been updated to expect 26 total attempts instead of 13. There are no review comments, and I have no additional feedback to provide.

@claude

claude Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@cb1kenobi

Copy link
Copy Markdown
Member

Reviewed ca9c7d12 — no issues found. This PR looks good, nice job!


Generated by Barber AI

Comment thread config/configUtils.ts
Comment thread config/configUtils.ts Outdated
Comment thread unitTests/config/configUtils.test.js Outdated
Comment thread components/OptionsWatcher.ts Outdated
Comment thread config/configUtils.ts Outdated
Comment thread config/readConfigFileSync.ts
Comment thread components/OptionsWatcher.ts Outdated
kriszyp added a commit that referenced this pull request Aug 25, 2026
The Windows rename budget goes back to the 3,630ms window it had before this
branch. The 10-second budget was never the fix: the failing worker was blocked
against a descriptor its own watcher held, so no budget could have been long
enough, and the extra 6.4 seconds was pure event-loop stall on a path
`set_configuration` reaches from a live request thread.

- `readConfigFileSync` retries only on Windows, where these codes mean a writer
  is swapping the file in, and shares one deadline per path across every caller
  on the thread so a 10+ watcher burst costs one budget rather than ten.
- `RootConfigWatcher` retries a read that still loses off the watcher event
  (where it holds no descriptor) instead of sitting on a stale config until the
  next edit, and logs parse and listener failures it previously swallowed.
- `OptionsWatcher`'s async branch now separates read rejections from apply and
  listener throws, so a listener's ENOENT can no longer be read as "the config
  file is gone" and tear the scope down.
- `EBUSY` is retried on Windows only; on POSIX it is a real condition.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
…ladder

Independent review found `OptionsWatcher` short of the recovery
`RootConfigWatcher` had just gained: a lock outliving the read's 500ms budget
emits no new watcher event when it clears, so the root watcher recovered
through its ladder while every scope on the same thread held a stale config
until the next write — two divergent views of one file.

Both now share `ConfigReadRetry`, whose budget is wall clock rather than a
count of attempts. Watcher callbacks and timer callbacks enter through the same
method, so a rename burst delivering several chokidar events in milliseconds
used to spend the whole ladder before the writer had let go.

- `atomicWriteFile` retries `EPERM`/`EACCES` only on Windows, matching the
  classifier added beside it for reads. A POSIX rename over an open file
  succeeds, so those codes are permanent there and the retry only parked the
  calling worker's event loop for 3.6 seconds before failing anyway.
- A config parse failure no longer logs the yaml error message: `prettyErrors`
  frames the offending source lines into it, and config files hold credentials.
  Read and listener failures log through `errorForLog` so the stack survives.
- The new `readConfigFileSync` and watcher-retry tests deny reads with a real
  mode-000 path instead of stubbing `node:fs`, per AGENTS.md, which also makes
  them independent of how `#src` resolves.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
The blocking budget and the timer ladder composed instead of layering: the
per-path deadline retires one budget after it expires, so a rung landing past
that opened a *fresh* 500ms window. One Windows lock outliving a budget stalled
a worker four separate times over ~4s — quadruple what DESIGN.md described, on
a thread serving HTTP, MQTT and replication.

A rung now passes `waitForLock: false` and takes a single attempt: the ladder
already owns the retry, so the blocking budget is spent once, on the first
watcher-driven read, where it still catches a sub-millisecond rename without a
timer round-trip.

- `ConfigReadRetry` derives its backoff from elapsed time rather than from how
  many times it was armed. One atomic rename can deliver add + change + change,
  and each re-armed the ladder further out, so a writer releasing at 150ms could
  leave every scope stale for another ~1.45s with the file readable throughout.
- `atomicWriteFile` goes back to classifying by error code alone. Gating it on
  win32 dropped the retry for a Linux worker whose rootPath is on WSL drvfs, a
  CIFS/SMB mount, or a Docker Desktop bind mount — all report `linux` and all
  return these codes transiently. The reader stays gated because a
  misclassified read falls through to the ladder; a rename has nothing to fall
  through to.
- `#handleUnlink` resets the ladder: the deletion settles what a pending read
  was retrying, and a rung landing after it emitted a second `remove`.
- `OptionsWatcher.#read` returns early once closed, so a chokidar callback
  queued before `close()` can no longer emit into an emitter whose listeners
  have been removed.
- The ladder gets direct unit tests; the watcher case that could not distinguish
  a retry from a watcher event was replaced with what it can actually prove.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
…after close()

Independent review found the two watchers disagreeing about hazards the diff
had already settled on one side.

- `yaml.parse` errors carry the offending source lines in `message` (yaml's
  `prettyErrors` is on by default), and the root config holds credentials.
  `RootConfigWatcher` was taught to log only the code and position; the same
  file read by `OptionsWatcher` emitted the raw error, which `Scope` logs.
  Both now parse through `parseConfigFile`, so neither can frame a credential
  into a log line.
- `RootConfigWatcher.#read` had no `#closed` guard on entry, only in its catch.
  A chokidar callback queued before `close()` could block the shutting-down
  worker in the read budget and then repopulate the config of a closed watcher.
- Comments that narrated the mechanics went back to a pointer at the DESIGN.md
  section that carries the reasoning, per the zero-new-comment default.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
Independent review caught a cycle this branch introduced: `harper_logger`
imports `RootConfigWatcher` at its own bottom to break their dependency, so
building a tagged logger at module scope runs `loggerWithTag()` before
`mainLogger` is initialized — a TDZ `ReferenceError` on the native
type-stripped path, which compiled CommonJS masks. The logger is now built on
first use, the same shape `config/harperConfigEnvVars.ts` uses for this cycle.

The `OptionsWatcher` recovery case asserted only that no error was emitted, so
it stayed green if the watcher never re-read the file at all. It now writes new
contents and locks them in one synchronous block — the queued watcher event is
already denied when it runs — and asserts the options come back current. What
it cannot assert is *which* path delivered them: chokidar reports the unlocking
chmod as a change of its own, so the ladder that covers the no-event case is
proven in `configReadRetry.test.js` instead. The sibling root-watcher case
carried the same claim in a comment; it now says what it proves.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
A terminal read or parse failure emitted only `error`, which settles nothing
that matters at boot: `Scope` logs it and returns, and componentLoader waits on
`Scope.ready` with no timeout. On the platform this branch targets — a Windows
worker whose startup config write collides with its own watcher's first read —
the read burns the blocking budget, spends the ladder, and that plugin's
`handleApplication` is never called. The scope now falls back to the defaults
and emits `ready` before surfacing the error, exactly as the ENOENT branch
beside it already does, and for the reason recorded there.

- `RootConfigWatcher` no longer claims to be "continuing with the previously
  loaded configuration" when the read that failed was the first one.
- `OptionsWatcher` counts read attempts, so a test can tell a ladder rung from a
  chokidar event. The new case locks the file, touches nothing else, and asserts
  the ladder re-reads on its own — the coverage two review rounds asked for, and
  it fails when the rung's callback is stubbed out.
- The test that releases a lock from another process handles its `spawn` failing
  rather than turning a skippable case into an unhandled `error` event.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 25, 2026
`ConfigParseError` was replacing every failure's message, including one thrown
by the parser itself rather than by the document. Only yaml's own parse errors
frame the source lines — and so the credentials — into `message`, so anything
without them now propagates unchanged, where the message is the whole of the
debugging context.

The ladder-wiring test waits for the read count to advance rather than sleeping
a fixed 700ms, so event-loop delay cannot fail a correct implementation.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Comment thread config/RootConfigWatcher.ts Outdated
Comment thread config/configUtils.ts
@kriszyp kriszyp changed the title fix(config): extend the Windows atomic-write rename-retry budget fix(config): read root config synchronously and bound rename retries by wall clock Aug 25, 2026
kriszyp added a commit that referenced this pull request Aug 26, 2026
The synchronous root-config read observes two states the async read never
did, and both end with the thread holding a stale config forever.

`ready` used to be emitted from inside chokidar's initial `add` dispatch,
before the native watch is armed; on darwin a write in that window is lost
outright (review measurement: 0ms lost, 5ms delivered). Gate `ready` on
chokidar's own `ready` plus a darwin-only grace, and stage the first config
rather than emitting a `change` ahead of it.

A non-atomic writer truncates before it writes, and the synchronous read is
fast enough to land in that window. chokidar throttles change events per path
for 50ms and drops the throttled ones, so the event carrying the content is
swallowed as a duplicate of the truncate's and a discarded empty read is the
last read that config gets. Route an empty read through `ConfigReadRetry`,
the same ladder a lock takes, in both watchers.

On Linux this was already reproducible: `should instantiate and watch the
root Harper config file` failed 4 of 6 runs at this branch's head and 0 of 6
at its merge base; it is green 8 of 8 with these fixes.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
Pre-push review, round 1.

A write that lands while the watch is still unarmed produces no event, so
emitting the pre-arming staged config left `ready` carrying a value nothing
would ever correct. Re-read when the gate opens; a failed or empty re-read
falls back to what was staged.

The empty-read guard moves into `OptionsWatcher.#applyContents` so it covers
the asynchronous read too. That path is much less likely to land in a
truncate window, but the consequence there is a spurious `remove` that tears
the scope down rather than a stale value.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
Gating `emit('ready')` on `#config` made a truthy config the barrier's condition,
so a file that parses to nothing — comments only, `---`, an empty document —
never settled it. `harper_logger.start()` awaits that promise with no timeout, so
the worker would hang at boot instead of coming up on the logging defaults. Track
that a read completed (`#configRead`) rather than that it produced a value; it
also stops a null config re-emitting `ready` in place of `change` on every
subsequent read.

`updateLogSettings()` takes the other half: the barrier can now settle with a
null config, so its consumer has to land on the defaults rather than throw out of
the boot path.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
`RootConfigWatcher.ready` and `Scope.ready` are awaited with no timeout, so a
read that ends without a config had to settle them or the worker hangs at boot
instead of failing. Three outcomes did not: a read the ladder could not complete,
a file still empty when the ladder was spent, and a file that would not parse.
Each now boots on defaults and logs what failed, matching what `OptionsWatcher`
already does on its ENOENT and read-failure paths — the two watchers must not
disagree about that policy. A file that becomes readable later still arrives, as
a `change`.

`OptionsWatcher` also stopped falling through a spent empty-read ladder into
`parseConfigFile('')`, which read an empty file as a removed scope.

Also from the pre-push round:
- A darwin arm grace still counting down is cancelled when the watcher falls back
  to polling, so `ready` cannot mean "watching" on a generation that failed.
- The async read path guards its completion handlers on `#closed`: close() cancels
  the retry state and drops the listeners while a read can still be in flight.
- The two new `remove` assertions use a plain listener rather than a sinon spy
  (AGENTS.md forbids new sinon uses).

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
…se the arming gate

The pre-push round's domain leg found that `#resetConfig()` aliased the
module-level `DEFAULT_CONFIG` rather than cloning it. This branch routes three
more paths into that reset, and `#merge` writes an applied config into
`#scopedConfig` in place — so a scope that started on the defaults and was then
configured wrote its own values into the object every later reset hands out, for
the life of the thread. The new test fails without the clone.

The arming gate had the same shape of hole the read paths just closed: chokidar
emits `ready` with no `add` when there is no file to report, so `#markArmed`
skipped its re-read and `ready` stayed pending — the absent-config-file boot hang.
Arming now always re-reads (ENOENT takes the ladder and settles on defaults), and
defers the emit when that read armed a retry so `ready` carries the newer config
rather than the pre-arm one. `close()` settles the barrier too.

`{}` is not "the logging defaults": `updateLogger` reads an absent `rotation` as
rotation off and an absent `console` as console off, so applying a config-less
read would have silently disabled logging on the very boot that could not read
its config. `updateLogSettings` now keeps what `initLogSettings()` established
until a real config arrives.

Also: `OptionsWatcher` tracks whether `ready` has gone out instead of inferring it
from config truthiness — a scope absent from a config that read fine leaves
`#rootConfig` set with no `ready` behind it — and its async `.catch` guards on
`#closed` like both `.then` arms already do.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
The pre-push round's graded leg found the logger fallback ineffective: the
`{}` `RootConfigWatcher` staged for a read that carried no config is an object,
so `updateLogSettings()`'s guard let it through — and `updateLogger` reads an
absent `rotation` as rotation off and an absent `console` as console off, which
silently disabled logging on the very boot that could not read its config. An
empty object is a configuration; "no configuration" has to be spelled as such,
so `#stageBootFallback` settles the barrier carrying nothing and leaves a
previously loaded config in place.

Also from that round: the synchronous read path's outer catch guards on
`#closed` like the asynchronous one already does — a listener of what
`#applyContents` emitted can close the watcher, after which `emit('error')` has
no listener left to reach.

Declined from the same round: that a scope arriving after an unconfigured boot
should be a `change` rather than a second `ready`. `ready` is not
once-per-watcher here — it is how this watcher says the scope has config again,
`Scope` consumes every one with `.on`, and the `remove` → recreated-file path
has emitted it that way all along (`OptionsWatcher.test.js` asserts it). The two
call sites that make that transition now share one `#applyScopedConfig`, and a
test pins the post-fallback arrival.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
…d stop ENOENT taking the ladder

Round 8 of the pre-push review took apart the previous commit's claim that a
repeated `ready` is consumed. It is not: `Scope.#handleOptionsWatcherReady` only
re-emits, `Scope.ready` is a settled `once`, and what recovers the recreated-file
path is the `remove` listener calling `requestRestart()` — which never fires on a
scope that booted with nothing applied. So a worker that read the root config in
a truncate window ran `handleApplication` on the defaults and kept serving them
after the operator's config landed, disagreeing with every worker that read the
file cleanly. A second `ready` now requests a restart, on the same terms as the
`remove` listener: a plugin with its own `ready` handler owns the response.

`RootConfigWatcher` also sent ENOENT into the retry ladder, which
`readConfigFileSync` deliberately does not retry and `OptionsWatcher` settles at
once as the install window. Every boot with no config file — an env-var-only
deployment, an empty mounted rootPath — therefore spent the whole 3.1s budget
inside `harper_logger.start()`. Two watchers, one policy: ENOENT settles
immediately. The config suite drops from 21s to 18s on that alone.

Three narrower ones from the same round:
- `close()` settles the barrier through `#emitReady` rather than duplicating the
  emit bare, so a throwing `ready` listener can no longer skip the teardown under
  it and leave the watcher and its arm timer running.
- A non-exhaustion chokidar scan error is a terminal outcome for arming too:
  chokidar may never reach its own `ready` after one, and nothing else would
  settle the barrier.
- `OptionsWatcher.#surfaceFailure` wraps its `error` emit. On the async path a
  throwing listener had nothing left awaiting it, so it reached Node as an
  unhandled rejection and took the process down over a failed config read.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
…stop a falsy scope restarting

Round 9's adjudicated major is a regression the previous commit introduced.
`once(this, 'ready')` attaches an `error` listener as well as a `ready` one and
drops both when the barrier settles, so settling before reporting — which the
scan-error path now does — is exactly what guarantees the `error` emit has no
listener left. `harper_logger` never registers one. An `error` with no listener
throws synchronously out of chokidar's dispatch, so a worker that had just
decided to survive an EIO/EACCES scan error would instead die of it. It now
reports through the logger when nothing is listening, and `handleError` guards
`#closed` like the read paths do.

Two more from the same round:
- `parseConfigFile` parses with `logLevel: 'error'`. yaml routes warnings
  through `process.emitWarning` rather than a throw, so a framed warning went
  around the scrub entirely and put config source — credentials included — on
  stderr. The `YAMLWarning` branch in `isYamlParseError` was dead code for the
  same reason.
- `#applyScopedConfig` keyed the unconfigured → configured transition on the
  scope value's truthiness, but `myPlugin:` with nothing under it is a
  configured scope whose value is null. Every ladder rung and rename-burst
  re-read of one therefore looked like the transition, and each now costs a
  restart request. A re-read of an unchanged falsy value is not a transition.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Aug 26, 2026
Round 10 found the previous commit reused `#armed` for two things: "chokidar's
scan finished" and "the barrier may settle". Setting it on a scan error made
`#handleArmed` return early when chokidar went on to arm normally, so the
re-read that exists solely to recover a write that landed in the unarmed window
never ran — the exact loss the gate was added for. A terminal outcome now opens
the barrier's own gate and leaves `#armed` to mean what it says. The test asserts
the re-read, not just that `ready` settled.

Also tightens the yaml-warning case to yaml's own warnings, so an unrelated
deprecation emitted in the same window cannot fail it with a misleading message.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
kriszyp and others added 28 commits August 27, 2026 08:16
Prevent same-worker Windows rename self-contention in both root config watchers, retain the exported retry controls, validate the deadline, and avoid speculative EBUSY retries.

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Validate retry timing inputs, retain a finite attempt safety cap, retry Windows sharing violations, preserve the prior POSIX window, and derive synchronous root reads from file identity.

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Use one bounded synchronous reader for both root watchers and keep downstream listener errors out of config-file ENOENT recovery.

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
The Windows rename budget goes back to the 3,630ms window it had before this
branch. The 10-second budget was never the fix: the failing worker was blocked
against a descriptor its own watcher held, so no budget could have been long
enough, and the extra 6.4 seconds was pure event-loop stall on a path
`set_configuration` reaches from a live request thread.

- `readConfigFileSync` retries only on Windows, where these codes mean a writer
  is swapping the file in, and shares one deadline per path across every caller
  on the thread so a 10+ watcher burst costs one budget rather than ten.
- `RootConfigWatcher` retries a read that still loses off the watcher event
  (where it holds no descriptor) instead of sitting on a stale config until the
  next edit, and logs parse and listener failures it previously swallowed.
- `OptionsWatcher`'s async branch now separates read rejections from apply and
  listener throws, so a listener's ENOENT can no longer be read as "the config
  file is gone" and tear the scope down.
- `EBUSY` is retried on Windows only; on POSIX it is a real condition.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…ladder

Independent review found `OptionsWatcher` short of the recovery
`RootConfigWatcher` had just gained: a lock outliving the read's 500ms budget
emits no new watcher event when it clears, so the root watcher recovered
through its ladder while every scope on the same thread held a stale config
until the next write — two divergent views of one file.

Both now share `ConfigReadRetry`, whose budget is wall clock rather than a
count of attempts. Watcher callbacks and timer callbacks enter through the same
method, so a rename burst delivering several chokidar events in milliseconds
used to spend the whole ladder before the writer had let go.

- `atomicWriteFile` retries `EPERM`/`EACCES` only on Windows, matching the
  classifier added beside it for reads. A POSIX rename over an open file
  succeeds, so those codes are permanent there and the retry only parked the
  calling worker's event loop for 3.6 seconds before failing anyway.
- A config parse failure no longer logs the yaml error message: `prettyErrors`
  frames the offending source lines into it, and config files hold credentials.
  Read and listener failures log through `errorForLog` so the stack survives.
- The new `readConfigFileSync` and watcher-retry tests deny reads with a real
  mode-000 path instead of stubbing `node:fs`, per AGENTS.md, which also makes
  them independent of how `#src` resolves.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
The blocking budget and the timer ladder composed instead of layering: the
per-path deadline retires one budget after it expires, so a rung landing past
that opened a *fresh* 500ms window. One Windows lock outliving a budget stalled
a worker four separate times over ~4s — quadruple what DESIGN.md described, on
a thread serving HTTP, MQTT and replication.

A rung now passes `waitForLock: false` and takes a single attempt: the ladder
already owns the retry, so the blocking budget is spent once, on the first
watcher-driven read, where it still catches a sub-millisecond rename without a
timer round-trip.

- `ConfigReadRetry` derives its backoff from elapsed time rather than from how
  many times it was armed. One atomic rename can deliver add + change + change,
  and each re-armed the ladder further out, so a writer releasing at 150ms could
  leave every scope stale for another ~1.45s with the file readable throughout.
- `atomicWriteFile` goes back to classifying by error code alone. Gating it on
  win32 dropped the retry for a Linux worker whose rootPath is on WSL drvfs, a
  CIFS/SMB mount, or a Docker Desktop bind mount — all report `linux` and all
  return these codes transiently. The reader stays gated because a
  misclassified read falls through to the ladder; a rename has nothing to fall
  through to.
- `#handleUnlink` resets the ladder: the deletion settles what a pending read
  was retrying, and a rung landing after it emitted a second `remove`.
- `OptionsWatcher.#read` returns early once closed, so a chokidar callback
  queued before `close()` can no longer emit into an emitter whose listeners
  have been removed.
- The ladder gets direct unit tests; the watcher case that could not distinguish
  a retry from a watcher event was replaced with what it can actually prove.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…after close()

Independent review found the two watchers disagreeing about hazards the diff
had already settled on one side.

- `yaml.parse` errors carry the offending source lines in `message` (yaml's
  `prettyErrors` is on by default), and the root config holds credentials.
  `RootConfigWatcher` was taught to log only the code and position; the same
  file read by `OptionsWatcher` emitted the raw error, which `Scope` logs.
  Both now parse through `parseConfigFile`, so neither can frame a credential
  into a log line.
- `RootConfigWatcher.#read` had no `#closed` guard on entry, only in its catch.
  A chokidar callback queued before `close()` could block the shutting-down
  worker in the read budget and then repopulate the config of a closed watcher.
- Comments that narrated the mechanics went back to a pointer at the DESIGN.md
  section that carries the reasoning, per the zero-new-comment default.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Independent review caught a cycle this branch introduced: `harper_logger`
imports `RootConfigWatcher` at its own bottom to break their dependency, so
building a tagged logger at module scope runs `loggerWithTag()` before
`mainLogger` is initialized — a TDZ `ReferenceError` on the native
type-stripped path, which compiled CommonJS masks. The logger is now built on
first use, the same shape `config/harperConfigEnvVars.ts` uses for this cycle.

The `OptionsWatcher` recovery case asserted only that no error was emitted, so
it stayed green if the watcher never re-read the file at all. It now writes new
contents and locks them in one synchronous block — the queued watcher event is
already denied when it runs — and asserts the options come back current. What
it cannot assert is *which* path delivered them: chokidar reports the unlocking
chmod as a change of its own, so the ladder that covers the no-event case is
proven in `configReadRetry.test.js` instead. The sibling root-watcher case
carried the same claim in a comment; it now says what it proves.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
A terminal read or parse failure emitted only `error`, which settles nothing
that matters at boot: `Scope` logs it and returns, and componentLoader waits on
`Scope.ready` with no timeout. On the platform this branch targets — a Windows
worker whose startup config write collides with its own watcher's first read —
the read burns the blocking budget, spends the ladder, and that plugin's
`handleApplication` is never called. The scope now falls back to the defaults
and emits `ready` before surfacing the error, exactly as the ENOENT branch
beside it already does, and for the reason recorded there.

- `RootConfigWatcher` no longer claims to be "continuing with the previously
  loaded configuration" when the read that failed was the first one.
- `OptionsWatcher` counts read attempts, so a test can tell a ladder rung from a
  chokidar event. The new case locks the file, touches nothing else, and asserts
  the ladder re-reads on its own — the coverage two review rounds asked for, and
  it fails when the rung's callback is stubbed out.
- The test that releases a lock from another process handles its `spawn` failing
  rather than turning a skippable case into an unhandled `error` event.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
`ConfigParseError` was replacing every failure's message, including one thrown
by the parser itself rather than by the document. Only yaml's own parse errors
frame the source lines — and so the credentials — into `message`, so anything
without them now propagates unchanged, where the message is the whole of the
debugging context.

The ladder-wiring test waits for the read count to advance rather than sleeping
a fixed 700ms, so event-loop delay cannot fail a correct implementation.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
The synchronous root-config read observes two states the async read never
did, and both end with the thread holding a stale config forever.

`ready` used to be emitted from inside chokidar's initial `add` dispatch,
before the native watch is armed; on darwin a write in that window is lost
outright (review measurement: 0ms lost, 5ms delivered). Gate `ready` on
chokidar's own `ready` plus a darwin-only grace, and stage the first config
rather than emitting a `change` ahead of it.

A non-atomic writer truncates before it writes, and the synchronous read is
fast enough to land in that window. chokidar throttles change events per path
for 50ms and drops the throttled ones, so the event carrying the content is
swallowed as a duplicate of the truncate's and a discarded empty read is the
last read that config gets. Route an empty read through `ConfigReadRetry`,
the same ladder a lock takes, in both watchers.

On Linux this was already reproducible: `should instantiate and watch the
root Harper config file` failed 4 of 6 runs at this branch's head and 0 of 6
at its merge base; it is green 8 of 8 with these fixes.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Pre-push review, round 1.

A write that lands while the watch is still unarmed produces no event, so
emitting the pre-arming staged config left `ready` carrying a value nothing
would ever correct. Re-read when the gate opens; a failed or empty re-read
falls back to what was staged.

The empty-read guard moves into `OptionsWatcher.#applyContents` so it covers
the asynchronous read too. That path is much less likely to land in a
truncate window, but the consequence there is a spurious `remove` that tears
the scope down rather than a stale value.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Gating `emit('ready')` on `#config` made a truthy config the barrier's condition,
so a file that parses to nothing — comments only, `---`, an empty document —
never settled it. `harper_logger.start()` awaits that promise with no timeout, so
the worker would hang at boot instead of coming up on the logging defaults. Track
that a read completed (`#configRead`) rather than that it produced a value; it
also stops a null config re-emitting `ready` in place of `change` on every
subsequent read.

`updateLogSettings()` takes the other half: the barrier can now settle with a
null config, so its consumer has to land on the defaults rather than throw out of
the boot path.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
`RootConfigWatcher.ready` and `Scope.ready` are awaited with no timeout, so a
read that ends without a config had to settle them or the worker hangs at boot
instead of failing. Three outcomes did not: a read the ladder could not complete,
a file still empty when the ladder was spent, and a file that would not parse.
Each now boots on defaults and logs what failed, matching what `OptionsWatcher`
already does on its ENOENT and read-failure paths — the two watchers must not
disagree about that policy. A file that becomes readable later still arrives, as
a `change`.

`OptionsWatcher` also stopped falling through a spent empty-read ladder into
`parseConfigFile('')`, which read an empty file as a removed scope.

Also from the pre-push round:
- A darwin arm grace still counting down is cancelled when the watcher falls back
  to polling, so `ready` cannot mean "watching" on a generation that failed.
- The async read path guards its completion handlers on `#closed`: close() cancels
  the retry state and drops the listeners while a read can still be in flight.
- The two new `remove` assertions use a plain listener rather than a sinon spy
  (AGENTS.md forbids new sinon uses).

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…se the arming gate

The pre-push round's domain leg found that `#resetConfig()` aliased the
module-level `DEFAULT_CONFIG` rather than cloning it. This branch routes three
more paths into that reset, and `#merge` writes an applied config into
`#scopedConfig` in place — so a scope that started on the defaults and was then
configured wrote its own values into the object every later reset hands out, for
the life of the thread. The new test fails without the clone.

The arming gate had the same shape of hole the read paths just closed: chokidar
emits `ready` with no `add` when there is no file to report, so `#markArmed`
skipped its re-read and `ready` stayed pending — the absent-config-file boot hang.
Arming now always re-reads (ENOENT takes the ladder and settles on defaults), and
defers the emit when that read armed a retry so `ready` carries the newer config
rather than the pre-arm one. `close()` settles the barrier too.

`{}` is not "the logging defaults": `updateLogger` reads an absent `rotation` as
rotation off and an absent `console` as console off, so applying a config-less
read would have silently disabled logging on the very boot that could not read
its config. `updateLogSettings` now keeps what `initLogSettings()` established
until a real config arrives.

Also: `OptionsWatcher` tracks whether `ready` has gone out instead of inferring it
from config truthiness — a scope absent from a config that read fine leaves
`#rootConfig` set with no `ready` behind it — and its async `.catch` guards on
`#closed` like both `.then` arms already do.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
The pre-push round's graded leg found the logger fallback ineffective: the
`{}` `RootConfigWatcher` staged for a read that carried no config is an object,
so `updateLogSettings()`'s guard let it through — and `updateLogger` reads an
absent `rotation` as rotation off and an absent `console` as console off, which
silently disabled logging on the very boot that could not read its config. An
empty object is a configuration; "no configuration" has to be spelled as such,
so `#stageBootFallback` settles the barrier carrying nothing and leaves a
previously loaded config in place.

Also from that round: the synchronous read path's outer catch guards on
`#closed` like the asynchronous one already does — a listener of what
`#applyContents` emitted can close the watcher, after which `emit('error')` has
no listener left to reach.

Declined from the same round: that a scope arriving after an unconfigured boot
should be a `change` rather than a second `ready`. `ready` is not
once-per-watcher here — it is how this watcher says the scope has config again,
`Scope` consumes every one with `.on`, and the `remove` → recreated-file path
has emitted it that way all along (`OptionsWatcher.test.js` asserts it). The two
call sites that make that transition now share one `#applyScopedConfig`, and a
test pins the post-fallback arrival.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…d stop ENOENT taking the ladder

Round 8 of the pre-push review took apart the previous commit's claim that a
repeated `ready` is consumed. It is not: `Scope.#handleOptionsWatcherReady` only
re-emits, `Scope.ready` is a settled `once`, and what recovers the recreated-file
path is the `remove` listener calling `requestRestart()` — which never fires on a
scope that booted with nothing applied. So a worker that read the root config in
a truncate window ran `handleApplication` on the defaults and kept serving them
after the operator's config landed, disagreeing with every worker that read the
file cleanly. A second `ready` now requests a restart, on the same terms as the
`remove` listener: a plugin with its own `ready` handler owns the response.

`RootConfigWatcher` also sent ENOENT into the retry ladder, which
`readConfigFileSync` deliberately does not retry and `OptionsWatcher` settles at
once as the install window. Every boot with no config file — an env-var-only
deployment, an empty mounted rootPath — therefore spent the whole 3.1s budget
inside `harper_logger.start()`. Two watchers, one policy: ENOENT settles
immediately. The config suite drops from 21s to 18s on that alone.

Three narrower ones from the same round:
- `close()` settles the barrier through `#emitReady` rather than duplicating the
  emit bare, so a throwing `ready` listener can no longer skip the teardown under
  it and leave the watcher and its arm timer running.
- A non-exhaustion chokidar scan error is a terminal outcome for arming too:
  chokidar may never reach its own `ready` after one, and nothing else would
  settle the barrier.
- `OptionsWatcher.#surfaceFailure` wraps its `error` emit. On the async path a
  throwing listener had nothing left awaiting it, so it reached Node as an
  unhandled rejection and took the process down over a failed config read.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…stop a falsy scope restarting

Round 9's adjudicated major is a regression the previous commit introduced.
`once(this, 'ready')` attaches an `error` listener as well as a `ready` one and
drops both when the barrier settles, so settling before reporting — which the
scan-error path now does — is exactly what guarantees the `error` emit has no
listener left. `harper_logger` never registers one. An `error` with no listener
throws synchronously out of chokidar's dispatch, so a worker that had just
decided to survive an EIO/EACCES scan error would instead die of it. It now
reports through the logger when nothing is listening, and `handleError` guards
`#closed` like the read paths do.

Two more from the same round:
- `parseConfigFile` parses with `logLevel: 'error'`. yaml routes warnings
  through `process.emitWarning` rather than a throw, so a framed warning went
  around the scrub entirely and put config source — credentials included — on
  stderr. The `YAMLWarning` branch in `isYamlParseError` was dead code for the
  same reason.
- `#applyScopedConfig` keyed the unconfigured → configured transition on the
  scope value's truthiness, but `myPlugin:` with nothing under it is a
  configured scope whose value is null. Every ladder rung and rename-burst
  re-read of one therefore looked like the transition, and each now costs a
  restart request. A re-read of an unchanged falsy value is not a transition.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Round 10 found the previous commit reused `#armed` for two things: "chokidar's
scan finished" and "the barrier may settle". Setting it on a scan error made
`#handleArmed` return early when chokidar went on to arm normally, so the
re-read that exists solely to recover a write that landed in the unarmed window
never ran — the exact loss the gate was added for. A terminal outcome now opens
the barrier's own gate and leaves `#armed` to mean what it says. The test asserts
the re-read, not just that `ready` settled.

Also tightens the yaml-warning case to yaml's own warnings, so an unrelated
deprecation emitted in the same window cannot fail it with a misleading message.

Refs #2191

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…atcher

`componentLoader` gives every root-declared plugin its own `OptionsWatcher` on the
root config, and this branch made those reads synchronous — which removes the
threadpool round-trip that used to defer the first read past darwin's FSEvents
warm-up. A write landing in that window is reported by no event, so the scope held
a stale config until an unrelated write.

The arm gate `RootConfigWatcher` grew for exactly this moves to
`config/watcherArming.ts` and both watchers use it. `OptionsWatcher` shares the
arming re-read, not the barrier: its consumer (`Scope`) attaches its listeners in
its constructor, so the recovered write reaches it as a `change`.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
… terminal watcher outcome

Round-2 review findings on the arming change:

- a truncated write leaves a *prefix* far more often than an empty file, and the
  ladder was released by any nonempty read — so a prefix that failed to parse was
  terminal, with the event carrying the rest of the document already throttled
  away. Both watchers now schedule on a parse failure and release the ladder only
  on a read that parses.
- a replacement watcher that also exhausts never reopens again, so
  `RootConfigWatcher` settles the boot barrier there instead of leaving
  `harper_logger.start()` awaiting it forever.
- `OptionsWatcher`'s own watcher error emitted while `ready` was pending rejected
  it, failing the component load rather than settling it onto the defaults the way
  every read outcome does; and an emit with no listener left threw the read error
  back at us to be logged as a listener fault.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
… they settle

Round-2 review findings:

- `#emitReady` emitted unguarded, and this branch made the first `ready` reachable
  from a ladder timer and from chokidar's error dispatch — a throwing listener
  there is an uncaught exception, not something the caller can absorb.
- the new terminal outcomes reset straight to `DEFAULT_CONFIG`, discarding
  env-var config that is file-independent by construction (#1618); they now take
  the same env fallback the ENOENT path does.
- a polling replacement that also exhausts left `Scope.ready` pending forever —
  the asymmetry with the root watcher's new guard.
- `remove` for a scope declared with no body (`myPlugin:`, i.e. `null`) never
  fired, because presence was tested by truthiness while this branch elsewhere
  made a falsy scope a first-class configured state.
- a watcher generation replaced by the polling fallback re-arms, so its own scan
  gets the reconciling re-read; only `close()` merely cancels the grace.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Round-3 review findings:

- the retry ladder's timer was always unref'd, so a worker whose only remaining
  path to `ready` was that ladder could drain its event loop and exit mid-boot.
  While `ready` is outstanding the timer now holds the thread; afterwards it
  still does not.
- `remove` and the composed-env `error` emitted unguarded, on the same paths
  `#emitError` already had to contain: a throwing consumer there lands in
  chokidar's dispatch or a retry timer, where nothing can absorb it.
- a read that parses fine but carries no block for this scope emitted nothing
  at all, leaving `Scope.ready` pending for a perfectly good config. It now
  settles on the scope's default while keeping the root config that read
  produced — `#settleUnconfigured`'s full reset would discard it.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
`atomicWriteFile` unlinks its temp file on every rename failure but not when
`writeFileSync` throws part-way — ENOSPC above all, which leaves a truncated
temp file that nothing ever collects, since the name carries fresh randomness
on every call.

Round-3 review finding (gemini).

Co-Authored-By: Claude Opus <noreply@anthropic.com>
`#applyEnvOnlyConfig` returned `true` when composing the env-var overlay threw,
which told all three callers "env config was applied". None of them then took
the fallback they take when there is none:

- the ENOENT read path returned without settling the boot barrier at all, so a
  worker booting from env vars alone with a malformed `HARPER_SET_CONFIG` reached
  `componentLoader`'s untimed `await scope.ready` with nothing left to settle it;
- `#settleUnconfigured` emitted `ready` carrying `undefined` instead of the
  defaults, so the scope booted on a config object it would throw reading;
- `#handleUnlink` logged that the env config "remains in effect" and swallowed
  the `remove` for a config file that really was deleted.

It now returns `false` and holds the failure for `#reportEnvComposeFailure`,
which each caller runs after settling — an `error` emitted first settles
`once(this, 'ready')` by rejection instead.

Round-3 review finding (gemini).

Co-Authored-By: Claude Opus <noreply@anthropic.com>
…her.close

chokidar's `close()` returns a promise, and a teardown rejection from it landed
as an unhandled rejection on the one path whose job is to stop caring about the
watcher. The exhaustion-recovery close in the same file and `OptionsWatcher.close`
already contain it.

Round-3 review finding (gemini).

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Round-4 review findings:

- `#setValue` was the one emit in `OptionsWatcher` still going out bare, and
  `#handleUnlink`'s env-only fallback reaches it through `#merge` while inside
  chokidar's own dispatch — so a plugin's `change` handler throwing took the
  worker down on a config deletion the branch had just decided to survive. It
  now reaches consumers as `error`, which is what the existing ENOENT-from-a-
  listener cases pin: a listener fault is not a read fault.
- the ENOENT read path left a ladder deadline armed that an earlier empty read
  had started. Every other terminal path clears it, so an editor that removes
  and rewrites the file (vim's backup-and-write) spent the budget on the
  deletion and then gave the truncate window that follows no retry at all —
  a stale config with no further event, which is what the ladder exists to
  prevent.
- `configReadRetry`'s cases asserted through fixed sleeps against a 100 ms
  timer, the flake shape AGENTS.md names (harper#1138); they use `waitFor`.
- restores the two `// Test-only:` markers the branch had dropped.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the kris/win-rename-retry-budget branch from 5eb33f7 to 94bb680 Compare August 27, 2026 14:26
#openCount: number = 0;
#readCount: number = 0;
#readRetry: ConfigReadRetry = new ConfigReadRetry();
#armGate: ArmGate = new ArmGate();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High: this PR now conflicts with and duplicates already-merged main PR HarperFast/harper#2339 ("Fix intermittent set_configuration 500s on Windows caused by the config write's rename retry"), which shipped an independent fix for the identical root cause

origin/main (current tip c4dd96237) is only 3 commits ahead of this PR's own rebase point (4802d4388), and one of those 3 is 99169ebca (#2339, merged today by the same author). It diagnoses and fixes the exact same problem this PR does — a root-config watcher's async fsPromises.readFile descriptor blocking the atomic-write rename retry on Windows — by making RootConfigWatcher/OptionsWatcher read synchronously. But it's a separate implementation: main uses PartialReadRetry/isPartialReadError from utility/watcherFallback.ts with an attempt-count budget (RENAME_RETRY_MAX_ATTEMPTS = 12, not a wall-clock deadline), calls readFileSync inline with no ArmGate, no arming-grace period, and none of config/watcherArming.ts, config/readConfigFileSync.ts, or config/configReadRetry.ts exist on main at all.

Verified with git merge-tree --write-tree origin/main refs/cb1/pr/2191: real content conflicts (not an evil merge, organic divergence) in exactly components/OptionsWatcher.ts, config/RootConfigWatcher.ts, config/configUtils.ts, and unitTests/config/rootConfigWatcher.test.js — the four files both PRs rewrote. Rebasing onto current main will not be mechanical; it needs a real reconciliation of two different solutions to the same problem, and main's shipped version currently lacks this PR's wall-clock-deadline budget and its darwin arm-gate fix entirely.

Suggested fix: rebase onto current main, and decide deliberately whether to keep main's PartialReadRetry/attempt-count approach and layer the ArmGate fix on top of it, or replace it with this PR's readConfigFileSync/wall-clock-deadline approach — landing both independently is not an option once one is already merged.


Generated by Barber AI

// Every root-declared plugin gets its own root-config watcher and each reads synchronously, so
// each has the unarmed window DESIGN.md's "`ready` means the watcher is armed" describes. The
// write that lands in it is reported by no event, so only this re-read can deliver it.
#handleArmed() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: a config file removed and recreated while ArmGate's darwin grace timer is in flight is not reliably picked back up — reproduces deterministically on macOS, invisible to Linux CI

This PR's own new test, unitTests/components/OptionsWatcher.test.js:457 ("does not write an applied config into the shared defaults"), fails on this macOS host with a 15-second timeout waiting for change after: config removed (→ remove observed correctly), then rewritten. Reproduced 3/3 runs in isolation, and via a standalone instrumented build (temp debug logging, not committed): the ARM_GRACE_MS(darwin)=20ms re-read this #handleArmed schedules can race a real unlink/recreate cycle, and once that happens the recreated file's chokidar event does not reach #handleChange within the window. A raw chokidar-only repro (no OptionsWatcher involved) confirms chokidar itself does deliver add correctly for the same unlink+recreate sequence given time to settle, so the loss is in this class's handling, not chokidar.

Confirmed via the actual CI run (33082213650) that this exact test passes on Linux for Node v22/v24/v26 — consistent with ARM_GRACE_MS being 0 on non-darwin, so the race window this depends on doesn't exist there. This is the same class of gap already called out twice earlier in this PR's own review history (a darwin-only failure Linux CI cannot see), now on the OptionsWatcher side instead of RootConfigWatcher.

Suggested fix: give OptionsWatcher the same re-read-on-arm handling RootConfigWatcher uses for a write that lands mid-grace (its #markArmed/#stageBootFallback settle path), or CI-gate this test to run under a simulated/zeroed grace to be deterministic on every platform rather than only exercising the race by accident.


Generated by Barber AI

Comment thread config/configUtils.ts
// (10+20+40+80+160+320+500*6), so it stays a deadline rather than an attempt count without
// widening the stall: this loop blocks the calling worker's event loop, and `set_configuration`
// reaches it from a live request thread.
const RENAME_RETRY_BUDGET_MS = 3_630;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: RENAME_RETRY_BUDGET_MS's own default value has no test coverage — mutating it to effectively unbounded passes the whole suite unchanged

Mutation-tested this constant (rebuilt with tsc -p tsconfig.build.json, confirmed the mutated value compiled into dist/config/configUtils.js before running): shrinking it to 0 is caught (3 of 95 tests in configUtils.test.js fail, as expected), but raising it to 999_999_999 (effectively unbounded) leaves all 95 passing. Every test in configUtils.test.js that exercises the retry budget passes an explicit retryBudgetMs override (60_000 / Infinity / 50, at lines 254/270/286/300) — none exercises this default constant's own retry-loop behavior at all, so a future accidental widening of the shipped default (e.g. someone "simplifies" this line, or a bad merge) would ship silently.

Suggested fix: add one test that calls atomicWriteFile with no retryBudgetMs override and asserts it gives up at (approximately) the default 3,630 ms window, the way unitTests/config/readConfigFileSync.test.js already does for READ_RETRY_BUDGET_MS.


Generated by Barber AI

// The grace itself is platform-derived (darwin only), so a case can assert when the callback runs
// relative to `arm()` only by waiting for it.
async function waitForArmed(gate) {
for (let waited = 0; waited < 3000 && !gate.armed; waited += 5) await delay(5);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: these tests are deliberately grace-value-agnostic, so they wouldn't catch ARM_GRACE_MS regressing to 0 on darwin

waitForArmed polls for up to 3 seconds regardless of the real grace duration, and the cancel-race test (line 43) adapts its own assertion to whichever timing actually happened (pending.armed ? 1 : 0) rather than asserting a specific darwin behavior. Mutation-tested this: forcing ARM_GRACE_MS = 0 unconditionally leaves this file's 3 tests all green. The darwin grace itself is covered — the same mutation makes unitTests/config/rootConfigWatcher.test.js's "should detect changes written via temp-file + rename (atomic write)" fail with the original Medium's exact timeout — so there's no live gap, just worth noting so a future reader doesn't assume this file pins the darwin timing.


Generated by Barber AI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants