Skip to content

Orphan-sweep run state is per-worker, so the guard doesn't guard and the result is unreadable #102

Description

@harper-joseph

Problem

util/orphanSweep.js keeps its run state in module scope:

let running = false;
let lastRun = null;

Harper components load per worker thread, so this state is per-worker, not per-node. Two consequences, both observed on the kohls prerender cluster on the first real use (2026-08-13):

  1. The overlap guard doesn't guard. isOrphanSweepRunning() only knows about the worker answering the request. A second POST that lands on a different worker reports alreadyRunning: false and starts a second concurrent sweep on the same node. Each sweep is a full walk of the target registry (~1.2M rows there), so this doubles a already-expensive scan.

  2. The result is unreadable. getLastOrphanSweep() is likewise per-worker, so lastRun reflects only what that worker last did. Observed: e9v returned {"lastRun": null, "alreadyRunning": false} while a sweep was in fact running on another worker, and the completed run's summary was only ever visible in the log. An operator polling the endpoint for the outcome can wait forever on a node that has already finished.

For a destructive sweep this matters more than it does for util/reconcile.js, which has the same pattern: reconcile is periodic and restorative, so a duplicate pass is harmless and its result is not something an operator gates a decision on. Here the operator has to read "how many did it delete, and is it still going" before deciding whether to run again — and right now the answer they get depends on which worker fielded the request.

Current workaround

Read the summary from hdb.log ([prerender] orphan sweep ...) rather than from the API response, and don't re-POST to a node while one is running.

Fix

Move both to node-shared state, the way queue state already does it — the coordination shared buffer, or a small row. Requirements:

  • running has to be a genuine cross-worker mutex, not an advisory flag, or two workers can still interleave between check and set.
  • lastRun should be readable from any worker on the node.
  • Whatever holds running must not strand it if the worker that set it dies mid-sweep (an expiry, or ownership tied to something observable), otherwise one crashed sweep locks the node out permanently.

util/reconcile.js should very likely get the same treatment, but it is not urgent for the reason above and can be a follow-up.

Not in scope

Cross-NODE coordination. The sweep is deliberately node-scoped — the in-flight lease check is only authoritative on the owner — so every node running its own sweep is correct and should stay that way.

Introduced in #100 (v0.48.0).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions