fix(ten-lane-highway): default the tmux socket per batch, not to a shared "hw" (ye80)

[Brian Krabach (bkrabach)]

The incident

A tmux server restart destroys every session on its socket. Every batch currently defaults to the shared socket hw, so any batch can annihilate every other batch's lanes and their watchdogs in one instant.

Observed 2026-09-05:

$ ps -eo pid,lstart,args | grep 'tmux.*-L hw'
1102562  Sat Sep  5 20:05:25 2026  tmux -L hw new-session -d -s hw__converge__w11-apply-ratified

The server was ~4 minutes old while the affected batch had been running for hours. It killed three unrelated lanes (three different repos) and that batch's watchdog simultaneously. No lane logged an error, because nothing in a lane went wrong.

Ruled out by measurement: OOM (dmesg clean, 47/121 GB used), the watchdog's own MAX_HOURS cap (it died at uptime=7929s of a 12h cap), and per-lane failure — three unrelated lanes do not fail identically at one instant.

Worse than the lost work: six DTU containers were left RUNNING with open infra-ledger rows — infrastructure outliving the highway that created it (Rule 14). Recovered by hand: verified-gone=6 rows-flipped=6 failed=0. One killed lane had already finished all five test tiers and committed; only its push and marker were lost.

The change

Each of the four socket users derives its default from BATCH_DIR's basename, sanitised with the same tr expression the scripts already use for tmux session names:

HIGHWAY_TMUX_SOCKET="${HIGHWAY_TMUX_SOCKET:-hw-$(printf '%s' \
  "$(basename "$BATCH_DIR")" | tr -c 'A-Za-z0-9_-' '_')}"

Backward compatible — an explicitly exported HIGHWAY_TMUX_SOCKET still wins; only the default changes. The variable was already plumbed through all four scripts; only the shared default coupled the batches.

Sessions cannot migrate between sockets, so an existing batch cuts over at its next live=0 point — which is how the batch that hit this incident cut over.

Tests

tests/test_ten_lane_highway_socket_default.py, 11 cases, following test_ten_lane_highway_infra_ledger.py's standard (exercise the real derivation via bash rather than reimplementing it in Python):

• no script retains the ${HIGHWAY_TMUX_SOCKET:-hw} fallback — the defect itself
• every socket user derives its default from BATCH_DIR
• two batches derive two different sockets — the actual point
• an explicit socket still wins (backward compatibility)
• a batch name carrying shell metacharacters is sanitised

That last test found a defect in its own first draft: the helper interpolated BATCH_DIR into a bash -c string unquoted, so a ; in a directory name split the command. Fixed by passing it as argv — which is exactly how the real scripts receive it (BATCH_DIR=${1:?...}), so the test now exercises the real path.

Two smaller findings from the same incident, not fixed here

1. lane_teardown.sh silently no-ops on a near-miss lane name. lane_teardown.sh <batch> drbf teardown --yes reported "lane 'drbf' owns no open rows — nothing to do" while six rows were open under the full name drbf-compaction-notice-ab. On the one path reached for in an emergency, a near-miss name should error listing candidates.
2. Nothing reaps ledger rows whose owning lane is gone. The ledger recorded the six DTUs correctly; nothing notices their lane is dead. Even surfacing "N open rows owned by lanes no longer live" in highway_status.sh would have caught it.

Draft: #306 also edits highway_watchdog.sh (watchdog self-restart). No overlap on the socket lines — verified 0 changed HIGHWAY_TMUX_SOCKET lines there — but they touch the same file, so whichever lands second should rebase.

Tracked as model_performance-ye80.

[Brian Krabach (bkrabach)]

Verification — all 9 checks green, including the Windows jobs that were red.

The three failures were all at _derived_socket — the bash subprocess call; Windows runners have no bash. Fixed with a stated skipif, which is the same remedy I asked #306 to apply to its own macOS failures — so the standard is the same for a manager's PR as for a lane's.

The eight text assertions still run on every platform, including the one that catches the defect itself (no script may retain the shared \${HIGHWAY_TMUX_SOCKET:-hw}` fallback). Only the three that *execute* the derivation are guarded, and the guard names why: the ten-lane-highway scripts are GNU/Linux-only by construction (stat -c %Y`, tmux).

Merging: the incident is reproducible, the fix is one line per script, an explicitly exported HIGHWAY_TMUX_SOCKET still wins, and the batch that hit the incident has already cut over to a per-batch socket successfully.