docs: sweep patterns, denial churn, and an API ergonomics review

[rcbevans]

Written from a production deployment that got every one of these wrong first. Each entry is a defect that shipped, with the measurement that found it.

New: docs/guides/sweeps.md

The page/fan-out/recurse pattern for enumerating a population — keyset paging over a column the walk does not write, cursor-advance ordering, per-item isolation boundaries, and idempotency by role.

That last table was the expensive one. Dedup is ON CONFLICT (idempotency_scope, idempotency_key) DO NOTHING with no status predicate, and the unique_for preflight only runs when both unique_for and identity_key are present. So:

Role	idempotency_key	unique_for + identity_key
Cron root	no	yes
Successor (self-enqueue)	yes — containing the cursor	no

A successor carrying either one dedups against its own still-running parent and the chain ends at link one. A root without them stacks a fresh chain per tick.

The alternative we shipped first — a capped batch per cron tick — drained 500 rows/hour against a fan-out completing 900/hour, so a 6,000-row population took 12 hours and any failure restarted it from the head.

New: troubleshooting.md entry 19 — unbounded job_events / job_attempts

A denied ConcurrencyReservation writes four durable rows per denial cycle (~1,193 bytes), and two properties make that unbounded:

1. A denial consumes no retry budget. mark_snoozed's SET list contains no attempt assignment; attempt advances only via the dispatch lease, so both sides of attempt < max_attempts move together and the failure gate is unreachable. The only terminal exit is deadline_failed, gated on schedule_to_close — without it, a job can be denied forever. One job reached attempt=1015/1018 over 6h12m.
2. prune cannot reach the rows. A looping job is never terminal and has finished_at reset to NULL on every denial, so the FK cascade never fires. job_events has no archive table and appears in no expiry CTE — while job_attempts is archived.

Measured in production: 2,074,421 denials against 168,963 successes (12.3:1), 2.5 GB across the two tables, 32% of a 7.85 GB database.

The fix is tier separation by what a job occupies: 19.4× oversubscription → 1.60× took 1.06M denials/day → 6,341/day (167×), 1.27 GB/day → ~8 MB/day. The bound that decides it is a blocking rate limiter — an actor that await asyncio.sleep()s inside the job holds its slot and cannot be denied; one that releases and re-queues is what mints denials.

Additions to existing guides

• cron.md — start-if-not-already-running roots
• retries.md — Snooze / RetryAfter(consume_budget=False) as the 429 answer, so nobody writes a custom classifier for it
• workers.md — the workgroup --max-concurrency argv override, which outranks env and is invisible to the other three concurrency layers
• jobs-clients.md, ops.md

docs/review/2026-09-12-api-ergonomics-review.md

The API friction behind these defects, for maintainer triage — not a change request.

---

Docs only; no library code touched.

[rcbevans]

Sequencing note (from a full stack audit): this PR is complementary to the stack (#120 ← #148 ← the third layer) — no duplication — but it touches six files the stack also changed (cron.md, jobs-clients.md, ops.md, troubleshooting.md, workers.md, mkdocs.yml), so textual conflicts are certain, and several passages document pre-fix behavior the stack changes: troubleshooting's denial-churn entry describes exactly what #139 (in flight on the third layer) removes, and jobs-clients.md's 'the collision is silent and successful' text is contradicted by #140's landed dedup observability. Recommendation: hold until the stack merges, rebase onto main, re-verify those passages against post-fix behavior, then land as the docs layer. Also: this PR's review doc is the source of #140–#146; #142–#144 and #146 had not carried their verdicts into the tracker until today (now assigned).

[rcbevans]

Post-stack review complete (two-agent audit against the fully-stacked state, a603982 = #170's head over #148 over #120), plus a reworked branch ready to land.

Mechanical: rebasing this PR onto the post-stack main is conflict-free (rehearsed — both commits replay clean; every hunk pair is disjoint). The risk was never git — it was semantics: this PR's docs describe pre-stack behavior the stack changed.

Every factual contradiction found and reworked (branch sweep-patterns-poststack @ e982481, based on this PR's head):

• cron.md + sweeps.md: the singleton auto-disable trap is gone post-stack — collisions are SUPPRESSED (no strike, no auto-disable); reworded with links to the stack's singleton-and-max_pending section.
• sweeps.md: heartbeat_timeout is now REFUSED at enqueue (ValueError), not silently stored; reclaim latency is TASKQ_LOCK_LEASE only.
• jobs-clients.md + sweeps.md + troubleshooting §15/§18: "the collision is silent" is false post-stack — every idempotency hit logs enqueue_deduplicated with the target's status, WARNING on terminal targets. (Kept the still-true asymmetry: SubJobEnqueuer.enqueue still warns about nothing.)
• troubleshooting §19: retitled and split — job_events now has age-based retention (default 7d) that reaches live-loop rows; job_attempts under non-terminal jobs stays prune-unreachable. The four-rows-per-denial mechanics are reframed as measured pre-contract history (12.3:1, 2.5 GB, 19.4×→1.60× kept — the only place that arithmetic lives), with the shipped contract (counters not rows, no max_attempts raise, MaxAttemptsExceeded terminal exit) linked from upgrading.md.
• The ergonomics review doc gains a dated post-stack addendum (P1's logging half landed; P5's premise inverted by the per-actor partition — its verdict is now stronger; P6's saturation footnote; P8's sweep-count refresh) — body untouched, its point-in-time framing is legitimate.
• Nav collision resolved: this guide is now "Sweep Patterns" (the stack's is "Maintenance Sweeps"), with a disambiguating pointer between them.

Verification: mkdocs build --strict green (three INFO notes are anchors whose targets land with the stack — slugs verified against the stack's actual headings); the docs-contract and accuracy test files pass (43 total).

Landing sequence (per the audit): wait for the in-flight #139 SQL on #170's branch and the stack's merge (this PR's §19 contract framing matches upgrading.md's declared contract either way) → merge post-stack main into this branch (rehearsed conflict-free; a merge keeps the push a fast-forward) → push. The reworked branch is staged locally and ready.