fix(highway): watchdog re-arms at its runtime cap instead of dying silently (6c3y)

amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md

@@ -88,7 +88,7 @@ repo** (a real run left `.amplifier/bin/` behind as untracked pollution).
 | `highway_status.sh BATCH_DIR WIDTH READY` | ONE call reports every lane + watchdog liveness and **computes DEFICIT by code** | "Keep lanes full" stopped being prose the day a run sat at 1 lane with work for 10 |
 | `launch_lane.sh BATCH_DIR LANE REPO GOAL [BASE_REF]` | Worktree + branch + tmux + `/goal` session, idempotent; the ONLY writer of `manifest.tsv` | Hand-written manifests diverged on column count and broke a real batch |
 | `verify_lane.sh BATCH_DIR LANE` | Git-facts probe for one landed lane (DONE.json, ahead-count, three-dot diffstat, uncommitted work) | "Ground truth from git and the filesystem, not from what any session said about itself" |
-| `highway_watchdog.sh BATCH_DIR WIDTH SESSION_ID [INTERVAL] [MAX_HOURS]` | Detached tmux loop that re-wakes THIS session (`amplifier run --resume`) on lane-end / under-width / stale heartbeat | The highway once froze overnight because the manager stopped monitoring the moment it reported status |
+| `highway_watchdog.sh BATCH_DIR WIDTH SESSION_ID [INTERVAL] [MAX_HOURS]` | Detached tmux loop that re-wakes THIS session (`amplifier run --resume`) on lane-end / under-width / stale heartbeat; **re-arms itself at its runtime cap while the batch is open** | The highway once froze overnight because the manager stopped monitoring the moment it reported status — then froze ~34h more when the watchdog hit its own cap |
 | `infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD...` / `infra_ledger.sh BATCH_DIR sweep --all-owners` | Records any infrastructure a lane OR the manager stands up (DTU, gitea instance, container, service, background process) into `infra.tsv` at creation, each with its teardown command; `sweep` runs those commands and exits non-zero until nothing is left standing | A run closed with a DTU and a gitea container still live — nothing the highway stands up should outlive it (Rule 14) |
 
 **`sweep` is the MANAGER's batch-close verb, never a lane's.** It runs EVERY
@@ -113,11 +113,41 @@ perform that teardown. A REAL teardown failure still exits non-zero and leaves
 the row open; the already-gone signature is deliberately narrow, so the signal
 that a teardown genuinely failed is never lost.
 
+**The watchdog's runtime cap re-arms; it does not silently stop.** `MAX_HOURS`
+(default 12) exists so an orphaned watchdog cannot outlive its batch. It used to
+be enforced by exiting — and on 2026-09-03 that exit put the highway to sleep for
+~34 hours, because the only channel a dying watchdog had left was a `wake-needed`
+file read by the thing it was supervising. At the cap it now:
+
+- **re-execs itself** (same arguments, generation counter bumped in
+  `watchdog.log`) while the batch is **open** — `BATCH_DIR/lanes/` exists AND the
+  manager touched `.manager-heartbeat` within `HIGHWAY_ABANDON_MAX` (default
+  21600s / 6h). A re-arm costs you nothing: no wake, no `wake-needed` entry;
+- **winds down** otherwise, so the cap's original purpose survives — an orphan
+  can now outlive its batch by at most one cap period, never indefinitely.
+
+On wind-down it writes the `wake-needed` line **and** dispatches
+`amplifier run --resume` **detached** (`setsid`), never as its own child. That
+detail is load-bearing: in the measured outage the manager answered the death
+notice from inside the watchdog's own tmux session, ran
+`tmux kill-session -t hw-watchdog__<batch>` to clear the "expired" watchdog, and
+killed itself mid-restart before it reached the line that starts the replacement.
+**Never `kill-session` a watchdog that told you it is exiting — it has already
+gone; just start a fresh one.** Escape hatches, all env vars:
+`HIGHWAY_ABANDON_MAX` (how long a silent manager means "abandoned"),
+`HIGHWAY_MAX_SECONDS` (second-resolution cap, overrides `MAX_HOURS`).
+
+The watchdog also touches `BATCH_DIR/.watchdog-heartbeat` every poll, and
+`highway_status.sh` reports `watchdog_hb_age` (`-1` = never ran here) plus
+`SUPERVISION LAPSED <n>s ago` when it is DEAD — so a lapse is read off an
+instrument, never inferred from a wake that never came.
+
 State lives in `BATCH_DIR` (create one per highway, e.g. `~/dev/hw-<name>`):
 `manifest.tsv` (scripts write), `HIGHWAY.md` (you write), `goals/` (pre-composed
 goal files), `lanes/` (worktrees), `.width` (authoritative width), `infra.tsv`
 (the infra ledger), `infra.owners.tsv` (which lane claimed which row),
-`.manager-heartbeat`, `wake-needed`, `watchdog.log`.
+`.manager-heartbeat`, `.watchdog-heartbeat` (the watchdog's own proof of life),
+`wake-needed`, `watchdog.log`.
 
 ## Phase 1 — Intake
 
@@ -308,7 +338,12 @@ The documented failure: the manager reported status and the highway froze
 until morning. Before ANY turn-ending message while lanes run:
 
 1. `highway_status.sh` must show `watchdog=LIVE` — if DEAD, start it (Phase 4
-   command; the session ID is in `<BATCH_DIR>/.session-id`) and re-check.
+   command; the session ID is in `<BATCH_DIR>/.session-id`) and re-check. Read
+   `SUPERVISION LAPSED <n>s ago` in the same output as the size of the blind
+   spot you are re-opening. **Do NOT `tmux kill-session` the dead watchdog
+   first** — if it exited on its own it is already gone, and if you are reading
+   its death notice you are running *inside* that session; killing it kills you
+   (measured, 2026-09-03, ~34h idle). Start the new one directly.
    **Never end a turn with the watchdog dead while lanes are live.**
 2. The todo lane board is current.
 3. The message leads with a state token in the first 100 characters —

amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_status.sh

@@ -77,22 +77,32 @@ done < "$MANIFEST"
 
 WD="hw-watchdog__${BATCH}"
 if tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$WD" 2>/dev/null; then wd_st=LIVE; else wd_st=DEAD; fi
+# SUPERVISION HEARTBEAT (model_performance-6c3y): the watchdog touches
+# .watchdog-heartbeat every poll, so a lapse is reported with a DURATION rather
+# than inferred from a wake that never came. -1 = no heartbeat file at all
+# (watchdog never started here, or predates the heartbeat).
+WD_HB="$BATCH_DIR/.watchdog-heartbeat"
+if [ -f "$WD_HB" ]; then wd_hb_age=$(( now - $(stat -c %Y "$WD_HB") )); else wd_hb_age=-1; fi
 
 open=$(( WIDTH - live )); if [ "$open" -lt 0 ]; then open=0; fi
 if [ "$READY" -lt "$open" ]; then deficit=$READY; else deficit=$open; fi
 
 if [ "$JSON" = 1 ]; then
-  printf '{"ts":"%s","batch":"%s","live":%d,"ended":%d,"done_marker":%d,"stalled":%d,"gone":%d,"width":%d,"width_source":"%s","ready":%d,"deficit":%d,"watchdog":"%s"}\n' \
-    "$(date -u +%FT%TZ)" "$BATCH" "$live" "$ended" "$done_n" "$stalled" "$gone" "$WIDTH" "$width_source" "$READY" "$deficit" "$wd_st"
+  printf '{"ts":"%s","batch":"%s","live":%d,"ended":%d,"done_marker":%d,"stalled":%d,"gone":%d,"width":%d,"width_source":"%s","ready":%d,"deficit":%d,"watchdog":"%s","watchdog_hb_age":%d}\n' \
+    "$(date -u +%FT%TZ)" "$BATCH" "$live" "$ended" "$done_n" "$stalled" "$gone" "$WIDTH" "$width_source" "$READY" "$deficit" "$wd_st" "$wd_hb_age"
   exit 0
 fi
 
 echo
-echo "SUMMARY batch=$BATCH live=$live ended=$ended done_marker=$done_n stalled=$stalled gone=$gone width=$WIDTH width_source=$width_source ready=$READY watchdog=$wd_st"
+echo "SUMMARY batch=$BATCH live=$live ended=$ended done_marker=$done_n stalled=$stalled gone=$gone width=$WIDTH width_source=$width_source ready=$READY watchdog=$wd_st watchdog_hb_age=${wd_hb_age}s"
 echo "DEFICIT=$deficit"
 if [ "$deficit" -gt 0 ]; then
   echo "ACTION: launch $deficit lane(s) NOW - refill before anything else."
 fi
 if [ "$wd_st" = "DEAD" ]; then
-  echo "WARNING: watchdog $WD is not running - do not end the turn until it is."
+  if [ "$wd_hb_age" -ge 0 ]; then
+    echo "WARNING: watchdog $WD is not running - SUPERVISION LAPSED ${wd_hb_age}s ago (last .watchdog-heartbeat) - do not end the turn until it is."
+  else
+    echo "WARNING: watchdog $WD is not running - lapse duration unknown (no .watchdog-heartbeat) - do not end the turn until it is."
+  fi
 fi

amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_watchdog.sh

@@ -13,7 +13,20 @@
 # plus a durable `wake-needed` file in BATCH_DIR in case the resume fails.
 # The orchestrator touches BATCH_DIR/.manager-heartbeat every cycle and deletes
 # wake-needed entries it has processed.
+#
+# AT THE RUNTIME CAP the loop does NOT simply exit (model_performance-6c3y):
+#   batch still open  -> re-exec itself with the same arguments (CAP RE-ARM GUARD)
+#   batch not open    -> deliver a final notice OUT OF BAND and exit (the cap's
+#                        original purpose -- no orphan outliving its batch)
+# It also touches BATCH_DIR/.watchdog-heartbeat every poll (SUPERVISION
+# HEARTBEAT), so highway_status.sh can report not just THAT supervision lapsed
+# but for HOW LONG, without anyone inferring it from a wake that never came.
 set -uo pipefail   # deliberately NOT -e: the watch loop must survive transient failures
+# A FAILED `exec` kills a non-interactive bash outright -- which would turn the
+# re-arm below into the silent stop this whole guard exists to remove. execfail
+# makes exec return instead, so the fallback path can run. (Caught by
+# test_cap_rearms_even_when_the_exec_bit_is_stripped, not by reading the manual.)
+shopt -s execfail
 
 BATCH_DIR=${1:?BATCH_DIR required}
 WIDTH=${2:?WIDTH required}
@@ -26,6 +39,11 @@ LOGF="$BATCH_DIR/watchdog.log"
 STATE="$BATCH_DIR/.watchdog-state"
 HB="$BATCH_DIR/.manager-heartbeat"
 HB_MAX=${HIGHWAY_HEARTBEAT_MAX:-1800}
+# SUPERVISION HEARTBEAT: proof-of-life this process writes for the MANAGER to
+# read. The manager touches $HB so the watchdog can tell it is alive; until now
+# nothing ran the other way, so "the watchdog stopped" was only ever visible to
+# whoever happened to run highway_status.sh.
+WD_HB="$BATCH_DIR/.watchdog-heartbeat"
 WAKE_GAP=${HIGHWAY_WAKE_GAP:-180}
 # If the manager's heartbeat is fresher than this, its foreground turn is still
 # active and will handle refill/merge inline. Waking then spawns a SECOND
@@ -61,7 +79,38 @@ prev_live=-1        # -1 so the very first poll never looks like a non-increase
 ineffective=0       # consecutive ineffective (under-width, non-recovering) polls
 escalate=0          # 1 while the ineffective count is at/over the threshold
 
-deadline=$(( $(date +%s) + MAX_HOURS * 3600 ))
+# CAP: MAX_HOURS is the documented knob. HIGHWAY_MAX_SECONDS overrides it with a
+# second-resolution cap -- the escape hatch for short-lived batches and for the
+# tests, which cannot wait 12h to observe what happens at the boundary.
+MAX_SECONDS=${HIGHWAY_MAX_SECONDS:-$(( MAX_HOURS * 3600 ))}
+deadline=$(( $(date +%s) + MAX_SECONDS ))
+# Which run of this process we are on. Bumped across every re-exec, so the log
+# reads as one continuous supervision record rather than N unexplained starts.
+GENERATION=${HIGHWAY_WATCHDOG_GENERATION:-1}
+# CAP RE-ARM GUARD: "still open" = the lanes directory exists AND the manager
+# has checked in within HIGHWAY_ABANDON_MAX. Deliberately much looser than
+# HB_MAX (a manager idle for 30 min is normal and gets woken; one that has not
+# touched its heartbeat in 6h has gone home). Re-arming ONLY while both hold is
+# what preserves the cap's original purpose -- an orphaned watchdog can now
+# outlive its batch by at most one cap period, never indefinitely.
+LANES_DIR="$BATCH_DIR/lanes"
+ABANDON_MAX=${HIGHWAY_ABANDON_MAX:-21600}
+batch_state=""
+batch_open() {
+  if [ ! -d "$LANES_DIR" ]; then
+    batch_state="no lanes dir at $LANES_DIR"; return 1
+  fi
+  if [ ! -f "$HB" ]; then
+    batch_state="manager heartbeat $HB never created"; return 1
+  fi
+  local age
+  age=$(( $(date +%s) - $(stat -c %Y "$HB") ))
+  if [ "$age" -gt "$ABANDON_MAX" ]; then
+    batch_state="manager heartbeat ancient (${age}s > ${ABANDON_MAX}s)"; return 1
+  fi
+  batch_state="lanes dir present, manager heartbeat ${age}s old"
+  return 0
+}
 START_TS=$(date +%s)
 # Grace after watchdog start during which an ABSENT heartbeat means "manager
 # still warming up in Phase 4" (not "manager dead") -> defer waking. Closes the
@@ -117,15 +166,62 @@ wake() {
   fi
 }
 
-log "watchdog start batch=$BATCH_DIR width=$WIDTH session=$SESSION_ID interval=${INTERVAL}s max=${MAX_HOURS}h hb_max=${HB_MAX}s"
+# The death notice. NOT wake(): it must never be suppressed by WAKE_GAP (the
+# one message that matters most is the one most likely to land inside the gap),
+# and it must NOT run as a child of this process inside this process's own tmux
+# session. On 2026-09-03 it did: the resumed manager was told "restart me",
+# reasonably ran `tmux kill-session -t hw-watchdog__<batch>` -- and killed
+# ITSELF, because it was running inside that very session. It never reached the
+# line that starts the replacement, and the highway sat idle ~34h. So the notice
+# is dispatched detached (setsid/nohup), this process exits immediately, and the
+# prompt says outright that killing the session is unnecessary.
+final_notice() {
+  local reason="$1" batch wd prompt body
+  batch=$(printf '%s' "$(basename "$BATCH_DIR")" | tr -c 'A-Za-z0-9_-' '_')
+  wd="hw-watchdog__${batch}"
+  # Durable signal first: a file survives every delivery failure below.
+  printf '%s\t%s\n' "$(date -u +%FT%TZ)" "$reason" >> "$BATCH_DIR/wake-needed"
+  prompt="HIGHWAY WATCHDOG STOPPED (watchdog): $reason. Supervision of $BATCH_DIR has ENDED - no further wakes will arrive from me. If the highway is still open, start a fresh watchdog (Phase 4 command; session id in $BATCH_DIR/.session-id) and re-check highway_status.sh. The old watchdog process has ALREADY exited: do NOT run 'tmux kill-session -t $wd' first - on 2026-09-03 a responder did exactly that while running inside that session, killed itself mid-restart, and the highway sat idle ~34h."
+  body='if amplifier run --resume "$1" --output-format json "$2" >> "$3" 2>&1; then
+          printf "%s final notice delivered (resume ok)\n" "$(date -u +%FT%TZ)" >> "$3"
+        else
+          printf "%s final notice FAILED (resume rc=%s) - wake-needed is the durable signal\n" "$(date -u +%FT%TZ)" "$?" >> "$3"
+        fi'
+  if command -v setsid >/dev/null 2>&1; then
+    setsid bash -c "$body" _ "$SESSION_ID" "$prompt" "$LOGF" >/dev/null 2>&1 &
+    log "final notice dispatched detached (setsid pid=$!): $reason"
+  else
+    nohup bash -c "$body" _ "$SESSION_ID" "$prompt" "$LOGF" >/dev/null 2>&1 &
+    log "final notice dispatched detached (nohup pid=$!): $reason"
+  fi
+}
+
+log "watchdog start batch=$BATCH_DIR width=$WIDTH session=$SESSION_ID interval=${INTERVAL}s max=${MAX_SECONDS}s generation=$GENERATION hb_max=${HB_MAX}s"
 touch "$STATE"
+touch "$WD_HB"
 
 while :; do
   sleep "$INTERVAL"
+  touch "$WD_HB"   # SUPERVISION HEARTBEAT: written every poll, read by highway_status.sh
 
   if [ "$(date +%s)" -ge "$deadline" ]; then
-    wake "watchdog max runtime (${MAX_HOURS}h) reached - restart me if the highway is still open"
-    log "exit: deadline reached"
+    if batch_open; then
+      # CAP RE-ARM GUARD: supervision continues past the cap while the batch is
+      # open. No wake, no wake-needed line -- a re-arm costs the manager nothing
+      # and needs nothing from it. The re-exec resets the deadline.
+      log "CAP RE-ARM: cap (${MAX_SECONDS}s) reached but batch is still open ($batch_state) - re-exec as generation $(( GENERATION + 1 ))"
+      export HIGHWAY_WATCHDOG_GENERATION=$(( GENERATION + 1 ))
+      exec "$0" "$@"
+      # Reached only if that exec failed (exec bit stripped by a packaging step,
+      # noexec mount). Re-exec through the interpreter already running us before
+      # giving up -- a re-arm that silently degrades to an exit is the very
+      # failure this item exists to close.
+      log "CAP RE-ARM: exec '$0' failed - retrying through ${BASH:-bash}"
+      exec "${BASH:-bash}" "$0" "$@"
+      log "CAP RE-ARM FAILED: could not re-exec '$0' - falling back to the wind-down path"
+    fi
+    final_notice "watchdog max runtime (${MAX_SECONDS}s, generation $GENERATION) reached and the batch is not open ($batch_state) - supervision has stopped"
+    log "exit: deadline reached, batch not open ($batch_state)"
     exit 0
   fi