fix(highway): watchdog re-arms at its runtime cap instead of dying silently (6c3y) by bkrabach · Pull Request #306 · microsoft/amplifier-app-cli

Brian Krabach (bkrabach) · 2026-09-06T03:24:29Z

The bug

The supervisor that catches stalled lanes could stop itself, and nothing caught that. Measured in the batch that filed it (model_performance-6c3y): the watchdog hit its 12h MAX_HOURS cap at 2026-09-03T14:14:05Z and the highway sat idle ~34 hours with four lanes ENDED and unverified.

The mechanism was worse than the ticket said

The item assumed the final message never reached anyone. watchdog.log says otherwise — and says something more useful:

line     1:  2026-09-02T14:04:27Z watchdog start ... max=12h    (generation 1)
line 33456:  2026-09-03T02:10:29Z watchdog start ... max=12h    (generation 2)
line 50060:  2026-09-03T14:14:05Z WAKE: watchdog max runtime (12h) reached - restart me...
line 50274:  2026-09-06T00:26:06Z watchdog start ...            (~34h later)

Between those last two lines there is neither wake delivered (resume ok) nor exit: deadline reached — the watchdog never exited at its cap; it was killed. The wake was delivered (the resumed manager's own reasoning is in the log: "restart the watchdog before doing anything else"), and the manager then ran tmux kill-session -t hw-watchdog__<batch> while running inside that session, as a child of wake()'s synchronous amplifier run --resume. It killed itself mid-restart, before the line that starts the replacement.

So: delivery was never the gap. Where the responder was standing was the gap — and a restart that depends on someone answering a message is one participant away from failing.

The change

highway_watchdog.sh

CAP RE-ARM GUARD — at the cap, exec "$0" "$@" (generation counter bumped) while the batch is open: BATCH_DIR/lanes/ exists AND .manager-heartbeat is younger than HIGHWAY_ABANDON_MAX (default 6h). Costs the manager nothing — no wake, no wake-needed entry.
The cap's purpose is preserved, not traded away. Abandoned batch → still winds down. MAX_HOURS is neither removed nor raised. Residual bound, stated in the docs: an orphan can outlive its batch by at most one cap period, never indefinitely.
shopt -s execfail — a failed exec kills a non-interactive bash outright, silently converting the re-arm back into the stop this fixes. It now falls back to exec "${BASH:-bash}" "$0" "$@". Found by the test, not by reading the manual.
final_notice() on wind-down: writes wake-needed unconditionally (never eaten by WAKE_GAP — the one message with no next poll behind it), then dispatches amplifier run --resume detached (setsid, nohup fallback) and exits. The prompt names the trap: "The old watchdog has ALREADY exited: do NOT run tmux kill-session … first."
SUPERVISION HEARTBEAT — .watchdog-heartbeat touched every poll.

highway_status.sh — reports watchdog_hb_age (JSON, additive) and SUPERVISION LAPSED <n>s ago when DEAD, so a lapse from any cause (cap, crash, kill) is read off an instrument the manager already runs first on every wake, rather than inferred from a wake that never came.

SKILL.md — both branches, both env knobs, the new state file, and the do-NOT-kill-session rule placed in Phase 6 step 1 where a manager actually hits it. Docs ship in the same change (the 2nz lesson), plus a proposed drift-check patch under the lane artifact root covering the docs half too.

Is a manager-side inverse check still needed? No.

Rejected as a distinct mechanism: highway_status.sh is already the manager's mandatory first action on every wake and already printed watchdog=DEAD; a second check adds a thing to remember and no new signal. And it would not have caught this outage — the manager was idle because no wake arrived, so anything that runs "next cycle" is dead code when there is no next cycle. The cheap half is kept (.watchdog-heartbeat + watchdog_hb_age), which is what makes "supervision lapsed, and for how long" a measurement instead of an inference.

Tests — 13 new, subprocess, real outcomes

tests/test_ten_lane_highway_watchdog.py, matching the standard 2nz set (run the real script; prove "nothing ran" with an observable sentinel — here a PATH-stubbed amplifier that records argv):

re-arms while open (asserts generation=3 from one spawned process), and re-arms with the exec bit stripped
winds down with no lanes dir; winds down on an ancient heartbeat; re-arms again when HIGHWAY_ABANDON_MAX is widened (proves the wind-down is the window's doing, not an unrelated refusal)
death notice dispatched detached, names the kill-session trap, and survives a 99999s WAKE_GAP while an ordinary wake in the same run is provably suppressed
.watchdog-heartbeat refreshed per poll; status reports the lapse duration, and reports -1/"unknown" rather than "0s ago" when there is no heartbeat at all
drift-check markers stay greppable (CAP RE-ARM GUARD, SUPERVISION HEARTBEAT)

Tests never touch the default hw tmux socket or a live batch.

Full suite

$ uv run pytest -q
1740 passed, 1 skipped, 13 deselected, 1 xfailed in 27.16s

Baseline re-measured in a clean worktree at 395fa68: 1727 passed, 1 skipped, 13 deselected, 1 xfailed. Delta +13 — exactly the new tests, zero regressions. (The goal file quotes 1682 for that commit; that figure appears stale, so the re-measured baseline is what the delta is taken against.)

Spend

$0.00 against a $0.00 authority (0 runs x 0 arms x $0 / 1.00 = $0.00). Pure shell/source change; no API calls, no DTUs, no infrastructure ledgered.

Lane note: docs/lanes/6c3y-watchdog-selfrestart/DONE-NOTE.md

…lently The supervisor that catches stalled lanes could stop itself and nothing caught THAT. Measured in this batch: the watchdog hit its 12h cap at 2026-09-03T14:14:05Z and the highway sat idle ~34h with four lanes ENDED and unverified. The final wake DID reach the manager -- its reasoning is in watchdog.log ("restart the watchdog before doing anything else") -- and it then ran tmux kill-session on the watchdog's session while running INSIDE it, as a child of the very watchdog it was replacing. It killed itself mid-restart and never reached the line that starts the new one. That is why the log carries neither 'wake delivered' nor 'exit: deadline reached'. - CAP RE-ARM GUARD: at the cap, re-exec with the same args while the batch is open (lanes dir present AND manager heartbeat within HIGHWAY_ABANDON_MAX, default 6h); wind down otherwise, so the cap's original purpose -- no orphan outliving its batch -- is preserved. - shopt -s execfail so a failed exec falls back to \$BASH instead of killing the shell (a silent stop by another name; caught by test). - Wind-down dispatches the death notice DETACHED (setsid/nohup), forced past WAKE_GAP, with a prompt that names the kill-session suicide trap. - SUPERVISION HEARTBEAT: .watchdog-heartbeat every poll; highway_status.sh reports watchdog_hb_age and 'SUPERVISION LAPSED <n>s ago'. - Docs ship in the same change (the 2nz lesson) + a proposed drift-check patch under the lane artifact root. 13 new subprocess tests in tests/test_ten_lane_highway_watchdog.py cover both branches, the exec-bit fallback, gap-proof delivery, and the instrument half.

… just a silent exit

Brian Krabach (bkrabach) · 2026-09-06T03:32:14Z

Manager verification — the diagnosis is excellent and CORRECTS MINE. Held as draft on one gate: macOS CI is red on this PR's own new tests.

First, the part that is better than what I filed

I filed model_performance-6c3y claiming the watchdog "exited at its own cap and left one trace in a file nobody reads." That was wrong, and this lane proved it from the log. Verified against watchdog.log myself:

50060: 2026-09-03T14:14:05Z WAKE: watchdog max runtime (12h) reached ...
50271:   echo "=== killing expired watchdog (12h04m) ==="; tmux -L hw kill-session -t "$WD" ...
50274: 2026-09-06T00:26:06Z watchdog start ...        <- the ~34h gap

The wake was delivered. The resumed manager — running inside the watchdog's own tmux session, as a child of wake()'s synchronous amplifier run --resume — then ran tmux kill-session on that watchdog and killed itself mid-restart, before reaching the line that starts the replacement. As the note puts it: "Delivery was never the gap; where the responder was standing was the gap."

That is a materially better root cause than the one in the item, and it means my proposed "announce your own death via the wake path" fix would not have helped — the announcement already worked.

The test suite matches 2nz's standard (real script via subprocess, PATH-stubbed amplifier recording argv as an observable sentinel, isolated on socket hw-test-6c3y so the live batch is never touched), and shopt -s execfail was found by a test, not by reading the manual — exactly the right way round.

The gate that holds it: macOS is red, on tests this PR adds

pytest (macos-latest, py3.11): FAILURE      pytest (macos-latest, py3.12): FAILURE
pytest (ubuntu-latest, 3.11/3.12): SUCCESS  pytest (windows-latest, 3.11/3.12): SUCCESS

tests/test_ten_lane_highway_watchdog.py does not exist on main, so these are not pre-existing failures — they are this PR's:

tests/test_ten_lane_highway_watchdog.py:196  AssertionError: watchdog did not re-arm past its cap
tests/test_ten_lane_highway_watchdog.py:226  AssertionError: non-executable watchdog did not re-arm
                                             AssertionError: ancient heartbeat: watchdog did not wind down at its cap

So on macOS the re-arm does not happen and the wind-down does not happen — i.e. both branches of the deliverable are unverified there.

Why I am not merging it anyway, even though this highway runs on Linux

The DONE-NOTE already says (line 131) "The script was already GNU/Linux-only (stat -c %Y)" — which is very likely the real explanation, and if so it is a pre-existing platform constraint this PR merely made visible by being the first to test the script.

That is a fine answer. It just has to be the stated one. Right now the PR asserts both branches are tested while CI says they fail on a supported platform, and a red CI merged on a verbal explanation is exactly how a real regression gets waved through later.

Smallest path to merge — pick either:

Skip with a reason, not silently: @pytest.mark.skipif(sys.platform != "linux", reason="highway_watchdog.sh is GNU/Linux-only: stat -c %Y, ..."). This makes the existing constraint explicit and turns CI green honestly.
Make them portable, if the failure is only the stat flavour (stat -f %m on BSD/macOS).

Either way, please say in the DONE-NOTE whether the script is intended to be Linux-only — if it is, that is worth stating once, loudly, rather than being rediscovered by the next lane.

Everything else here is merge-ready; this is one gate, and the fix is small.

Windows CI failed all three tests that EXECUTE the derivation (all at _derived_socket, the bash subprocess call) -- Windows runners have no bash. Same remedy I asked PR #306 to apply to its own macOS failures, so the standard is the same for my PR as for a lane's: skip with a STATED reason rather than silently, and keep everything that can still run running. The eight text assertions read the scripts as data and continue to run on every platform, including the one that catches the defect itself (no script may retain the shared ${HIGHWAY_TMUX_SOCKET:-hw} fallback). The ten-lane-highway scripts are GNU/Linux-only by construction (stat -c %Y, tmux); this makes that constraint explicit instead of leaving it to be rediscovered by the next contributor's red CI.

…ared "hw" (ye80) (#310) * fix(ten-lane-highway): default the tmux socket per batch, not to a shared "hw" A tmux SERVER restart destroys every session on its socket. With every batch defaulting to the shared socket "hw", any batch can annihilate every other batch's lanes AND their watchdogs in one instant. OBSERVED 2026-09-05. A second batch restarted the server on socket "hw": ps -eo pid,lstart,args | grep 'tmux.*-L hw' 1102562 Sat Sep 5 20:05:25 2026 tmux -L hw new-session -d -s hw__converge__... The server was ~4 minutes old while the affected batch had run for hours. It killed three unrelated lanes (three different repos) and that batch's watchdog simultaneously. No lane logged an error, because nothing in a lane went wrong. Ruled out by measurement: OOM (dmesg clean, 47/121 GB), the watchdog's own MAX_HOURS cap (it died at uptime=7929s of a 12h cap), and per-lane failure. WORSE THAN THE LOST WORK: six DTU containers were left RUNNING with open infra-ledger rows -- infrastructure outliving the highway that created it (Rule 14). One of the killed lanes had already finished all five test tiers and committed; only its push and marker were lost. THE CHANGE. Each of the four socket users now derives its default from BATCH_DIR's basename (sanitised with the same tr expression the scripts already use for tmux session names): HIGHWAY_TMUX_SOCKET="${HIGHWAY_TMUX_SOCKET:-hw-$(printf '%s' \ "$(basename "$BATCH_DIR")" | tr -c 'A-Za-z0-9_-' '_')}" Backward compatible: an explicitly exported HIGHWAY_TMUX_SOCKET still wins, so only the DEFAULT changes. Sessions cannot migrate between sockets, so an existing batch cuts over at its next live=0 point -- which is how the batch that hit this incident cut over. TESTS (tests/test_ten_lane_highway_socket_default.py, 11 cases), following tests/test_ten_lane_highway_infra_ledger.py's standard -- exercise the real derivation via bash rather than reimplementing it in Python: * no script retains the ${HIGHWAY_TMUX_SOCKET:-hw} fallback (the defect) * every socket user derives its default from BATCH_DIR * two batches derive two DIFFERENT sockets (the actual point) * an explicit socket still wins (backward compatibility) * a batch name carrying shell metacharacters is sanitised That last test found a defect in its own first draft: the helper interpolated BATCH_DIR into a bash -c string unquoted, so a ';' in a directory name split the command. Fixed by passing it as argv -- which is exactly how the real scripts receive it (BATCH_DIR=${1:?...}), so the test now exercises the real path. * test: guard the bash-executing socket tests on platforms without bash Windows CI failed all three tests that EXECUTE the derivation (all at _derived_socket, the bash subprocess call) -- Windows runners have no bash. Same remedy I asked PR #306 to apply to its own macOS failures, so the standard is the same for my PR as for a lane's: skip with a STATED reason rather than silently, and keep everything that can still run running. The eight text assertions read the scripts as data and continue to run on every platform, including the one that catches the defect itself (no script may retain the shared ${HIGHWAY_TMUX_SOCKET:-hw} fallback). The ten-lane-highway scripts are GNU/Linux-only by construction (stat -c %Y, tmux); this makes that constraint explicit instead of leaving it to be rediscovered by the next contributor's red CI. --------- Co-authored-by: manager ye80 <ye80@localhost>

…on stated macOS CI was red on this PR's own new tests. The cause is not the tests and not the fix: these scripts are GNU/Linux-only BY CONSTRUCTION -- highway_status.sh:70 uses `stat -c %Y`, a GNU coreutils flag with no BSD/macOS equivalent, and the script's own comment at :53 says so: "(stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels.)" The existing guard only caught Windows and missing-bash. macOS HAS bash, so the guard never fired there and the suite ran against a script that cannot work -- reporting a platform limitation as a test failure. This states the constraint instead of rediscovering it: same remedy applied to #310 (Windows/bash) and asked for on #306/#313 by the manager. Linux CI, which is where these scripts run, is unchanged and still green -- nothing is waived silently. If the scripts are ever made portable (`stat -f %m` on BSD), delete this guard and the suite comes back on macOS.

… stated macOS CI was red on this PR's own new tests. The cause is not the tests and not the fix: these scripts are GNU/Linux-only BY CONSTRUCTION -- highway_status.sh:70 uses `stat -c %Y`, a GNU coreutils flag with no BSD/macOS equivalent, and the script's own comment at :53 says so: "(stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels.)" The existing guard only caught Windows and missing-bash. macOS HAS bash, so the guard never fired there and the suite ran against a script that cannot work -- reporting a platform limitation as a test failure. This states the constraint instead of rediscovering it: same remedy applied to #310 (Windows/bash) and asked for on #306/#313 by the manager. Linux CI, which is where these scripts run, is unchanged and still green -- nothing is waived silently. If the scripts are ever made portable (`stat -f %m` on BSD), delete this guard and the suite comes back on macOS.

…80) (#313) * fix(ten-lane-highway): report infra-ledger rows no live lane owns (ye80) Second sub-finding of the 2026-09-05 tmux-server incident. A server restart killed three lanes at once and left SIX DTU containers RUNNING with open infra-ledger rows. Every component worked in isolation: the ledger recorded all six correctly, and highway_status.sh reported the lanes ENDED. Nothing joined the two, so "infrastructure with nothing driving it" (Rule 14) was invisible -- the manager found the containers by running `incus list` by hand. highway_status.sh now joins lane liveness to row ownership and reports `orphan_rows=N owned by: <lane>(n)`, in both the human summary and the JSON line. REPORTING ONLY: nothing is destroyed and no row is flipped. Automatic reaping is deliberately not implemented -- a false positive that prints is a nuisance, a false positive that destroys is another 0rg. A LIVE lane's rows are never counted; that is the load-bearing property, and it is pinned with a live lane and a dead lane holding rows simultaneously, plus a real-tmux test that kills a session mid-test and watches the same batch flip 1 -> 2 orphans. Owner names are resolved exact-first, then by a UNIQUE token-boundary prefix. The two sides of this join genuinely disagree about what a lane is called: in this batch's own files, infra.owners.tsv holds short work-item ids for 5 of 13 owners and long manifest names for the other 8. An exact-match join would have false-alarmed on ~40% of owners, and a report that cries wolf is a report nobody reads. Ambiguous, unresolvable, and unclaimed owners are each reported as such rather than guessed at. 13 tests, following tests/test_ten_lane_highway_infra_ledger.py's standard: run the real script, assert real output, and prove "ran nothing" with an observable `touch <sentinel>` destroy_cmd rather than inferring it from an exit code. Subprocess environments are built explicitly (etuz), since a lane exports HIGHWAY_TMUX_SOCKET. Also ships, as lane artifacts, the fix for the FIRST sub-finding -- the lane_teardown.sh near-miss footgun. That script lives in another repo and is untracked even there, so the patch and its proof harness are shipped under docs/lanes/ rather than committed as code. See the DONE-NOTE. * docs(lane ye80b): record the verify_lane_publication.sh key-scraping finding * test: skip the highway_status.sh suite off GNU/Linux, with the reason stated macOS CI was red on this PR's own new tests. The cause is not the tests and not the fix: these scripts are GNU/Linux-only BY CONSTRUCTION -- highway_status.sh:70 uses `stat -c %Y`, a GNU coreutils flag with no BSD/macOS equivalent, and the script's own comment at :53 says so: "(stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels.)" The existing guard only caught Windows and missing-bash. macOS HAS bash, so the guard never fired there and the suite ran against a script that cannot work -- reporting a platform limitation as a test failure. This states the constraint instead of rediscovering it: same remedy applied to #310 (Windows/bash) and asked for on #306/#313 by the manager. Linux CI, which is where these scripts run, is unchanged and still green -- nothing is waived silently. If the scripts are ever made portable (`stat -f %m` on BSD), delete this guard and the suite comes back on macOS. --------- Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com> Co-authored-by: manager macos-gate <gate@localhost>

Brian Krabach (bkrabach) · 2026-09-06T15:53:37Z

macOS gate: cause found and remedied; CI is not re-triggering on this branch

I pushed the same fix that cleared #313 (merged 4420e14) to this branch as b8e6c1b.

The cause was never the tests or the fix — it is a platform limitation the script documents about itself:

highway_status.sh:53   # (stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels.)

stat -c %Y is a GNU coreutils flag with no BSD/macOS equivalent. The existing guard caught only Windows and missing-bash — but macOS has bash, so it never fired, and the suite ran against a script that cannot work there. Hence live == 0: the liveness sensor was never going to report anything on that platform.

The remedy is the one applied to #310 (Windows/bash) and to #313: skip off GNU/Linux with the reason stated, never waived silently. Linux CI — where these scripts actually run — is unchanged.

Current state, reported honestly rather than left as a surprise: after pushing b8e6c1b and marking this PR ready, zero pytest jobs have been queued across ~12 minutes of polling. #313 queued and went green immediately under identical treatment, so this is specific to this branch/workflow, not to the change. I am not merging on an unverified CI state — the same standard that held this PR in the first place.

The work itself remains verified from my earlier review: the root cause (the resumed manager killed the watchdog session it was standing in, then died mid-restart before reaching the line that starts the replacement) is a better diagnosis than the item I filed, the re-arm/wind-down branches are both tested, and the cap's anti-orphan purpose is preserved. It needs CI to run, not more work.

Amplifier (microsoft-amplifier) added 2 commits September 5, 2026 20:21

docs(lanes): 6c3y DONE-NOTE — root cause was a responder suicide, not…

7db6661

… just a silent exit

Brian Krabach (bkrabach) mentioned this pull request Sep 6, 2026

fix(ten-lane-highway): default the tmux socket per batch, not to a shared "hw" (ye80) #310

Merged

Brian Krabach (bkrabach) mentioned this pull request Sep 6, 2026

fix(ten-lane-highway): report infra-ledger rows no live lane owns (ye80) #313

Merged

Brian Krabach (bkrabach) marked this pull request as ready for review September 6, 2026 15:52

Brian Krabach (bkrabach) mentioned this pull request Sep 6, 2026

feat(ten-lane-highway): ship lane_teardown.sh, the recovery path the skill only named (giwq) #319

Merged

Navigation Menu

fix(highway): watchdog re-arms at its runtime cap instead of dying silently (6c3y) - #306

fix(highway): watchdog re-arms at its runtime cap instead of dying silently (6c3y)#306
Brian Krabach (bkrabach) wants to merge 3 commits into
mainfrom
lane/6c3y-watchdog-selfrestart

Brian Krabach (bkrabach) commented Sep 6, 2026

Uh oh!

Brian Krabach (bkrabach) commented Sep 6, 2026

Uh oh!

Brian Krabach (bkrabach) commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Brian Krabach (bkrabach) commented Sep 6, 2026

The bug

The mechanism was worse than the ticket said

The change

Is a manager-side inverse check still needed? No.

Tests — 13 new, subprocess, real outcomes

Full suite

Spend

Uh oh!

Brian Krabach (bkrabach) commented Sep 6, 2026

Manager verification — the diagnosis is excellent and CORRECTS MINE. Held as draft on one gate: macOS CI is red on this PR's own new tests.

First, the part that is better than what I filed

The gate that holds it: macOS is red, on tests this PR adds

Why I am not merging it anyway, even though this highway runs on Linux

Uh oh!

Brian Krabach (bkrabach) commented Sep 6, 2026

macOS gate: cause found and remedied; CI is not re-triggering on this branch

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants