Problem
util/orphanSweep.js keeps its run state in module scope:
let running = false;
let lastRun = null;
Harper components load per worker thread, so this state is per-worker, not per-node. Two consequences, both observed on the kohls prerender cluster on the first real use (2026-08-13):
-
The overlap guard doesn't guard. isOrphanSweepRunning() only knows about the worker answering the request. A second POST that lands on a different worker reports alreadyRunning: false and starts a second concurrent sweep on the same node. Each sweep is a full walk of the target registry (~1.2M rows there), so this doubles a already-expensive scan.
-
The result is unreadable. getLastOrphanSweep() is likewise per-worker, so lastRun reflects only what that worker last did. Observed: e9v returned {"lastRun": null, "alreadyRunning": false} while a sweep was in fact running on another worker, and the completed run's summary was only ever visible in the log. An operator polling the endpoint for the outcome can wait forever on a node that has already finished.
For a destructive sweep this matters more than it does for util/reconcile.js, which has the same pattern: reconcile is periodic and restorative, so a duplicate pass is harmless and its result is not something an operator gates a decision on. Here the operator has to read "how many did it delete, and is it still going" before deciding whether to run again — and right now the answer they get depends on which worker fielded the request.
Current workaround
Read the summary from hdb.log ([prerender] orphan sweep ...) rather than from the API response, and don't re-POST to a node while one is running.
Fix
Move both to node-shared state, the way queue state already does it — the coordination shared buffer, or a small row. Requirements:
running has to be a genuine cross-worker mutex, not an advisory flag, or two workers can still interleave between check and set.
lastRun should be readable from any worker on the node.
- Whatever holds
running must not strand it if the worker that set it dies mid-sweep (an expiry, or ownership tied to something observable), otherwise one crashed sweep locks the node out permanently.
util/reconcile.js should very likely get the same treatment, but it is not urgent for the reason above and can be a follow-up.
Not in scope
Cross-NODE coordination. The sweep is deliberately node-scoped — the in-flight lease check is only authoritative on the owner — so every node running its own sweep is correct and should stay that way.
Introduced in #100 (v0.48.0).
Problem
util/orphanSweep.jskeeps its run state in module scope:Harper components load per worker thread, so this state is per-worker, not per-node. Two consequences, both observed on the kohls prerender cluster on the first real use (2026-08-13):
The overlap guard doesn't guard.
isOrphanSweepRunning()only knows about the worker answering the request. A second POST that lands on a different worker reportsalreadyRunning: falseand starts a second concurrent sweep on the same node. Each sweep is a full walk of the target registry (~1.2M rows there), so this doubles a already-expensive scan.The result is unreadable.
getLastOrphanSweep()is likewise per-worker, solastRunreflects only what that worker last did. Observed: e9v returned{"lastRun": null, "alreadyRunning": false}while a sweep was in fact running on another worker, and the completed run's summary was only ever visible in the log. An operator polling the endpoint for the outcome can wait forever on a node that has already finished.For a destructive sweep this matters more than it does for
util/reconcile.js, which has the same pattern: reconcile is periodic and restorative, so a duplicate pass is harmless and its result is not something an operator gates a decision on. Here the operator has to read "how many did it delete, and is it still going" before deciding whether to run again — and right now the answer they get depends on which worker fielded the request.Current workaround
Read the summary from
hdb.log([prerender] orphan sweep ...) rather than from the API response, and don't re-POST to a node while one is running.Fix
Move both to node-shared state, the way queue state already does it — the coordination shared buffer, or a small row. Requirements:
runninghas to be a genuine cross-worker mutex, not an advisory flag, or two workers can still interleave between check and set.lastRunshould be readable from any worker on the node.runningmust not strand it if the worker that set it dies mid-sweep (an expiry, or ownership tied to something observable), otherwise one crashed sweep locks the node out permanently.util/reconcile.jsshould very likely get the same treatment, but it is not urgent for the reason above and can be a follow-up.Not in scope
Cross-NODE coordination. The sweep is deliberately node-scoped — the in-flight lease check is only authoritative on the owner — so every node running its own sweep is correct and should stay that way.
Introduced in #100 (v0.48.0).