Skip to content

feat(ten-lane-highway): ship lane_teardown.sh, the recovery path the skill only named (giwq) - #319

Merged
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
lane/giwq-adopt-lane-teardown
Sep 6, 2026
Merged

Brian Krabach (bkrabach) merged 1 commit into
mainfrom
lane/giwq-adopt-lane-teardown

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

The defect

The shipped ten-lane-highway skill's own documented emergency recovery path
pointed at a file the skill does not ship.

$ git -C ~/dev/openai-evals-team-ci ls-files --error-unmatch .amplifier/evaluation/tools/lane_teardown.sh
error: pathspec '.amplifier/evaluation/tools/lane_teardown.sh' did not match any file(s) known to git

infra_ledger.sh's refusal message named that cross-repo path; SKILL.md named
the tool with no path at all. It is the only lane-scoped teardown path — the
alternative, sweep, is the manager's batch-close verb that destroys every
lane's infrastructure. On 2026-09-05 six DTU containers were left running with
open ledger rows after a tmux-server restart killed three lanes; recovering them
required exactly this tool.

What this does

  1. Adopts lane_teardown.sh into
    amplifier_app_cli/data/skills/ten-lane-highway/scripts/, beside its siblings.
  2. Applies ye80b's near-miss fix (fix(ten-lane-highway): report infra-ledger rows no live lane owns (ye80) #313's patch artifact, patch -p4, clean):
    a lane name that almost matches an owner exits 4 naming each candidate
    owner and its open-row count
    , instead of printing success for a teardown
    that would do nothing. On 2026-09-05
    lane_teardown.sh <batch> drbf teardown --yes printed "lane 'drbf' owns no
    open rows — nothing to do"
    and exited 0 while six rows were open under
    drbf-compaction-notice-ab.
  3. Makes every caller name a path that existsinfra_ledger.sh resolves the
    sibling from BASH_SOURCE; SKILL.md uses <skill_directory>/scripts/… and
    gains an instruments-table row; highway_status.sh's orphan-row reclaim hint
    (the same instruction for the same incident) prints a resolved,
    copy-pasteable path instead of a bare name.
  4. Fixes a drifted usage()sed -n '2,72p' was pinned to a header that has
    since grown, silently truncating the help. Now derived from the file.

highway_watchdog.sh is not touched (PR #306 is open against it).

Tests — tests/test_ten_lane_highway_lane_teardown.py, 14 tests

ye80b's 13-assertion shell harness converted (it already parameterised the script
path for exactly this), plus the adoption-safety proofs.

Two standards followed deliberately:

  • Every "ran nothing" claim is observable — a state file that still exists, or
    a touch sentinel that does not. Never an exit code: the buggy path prints a
    success message
    , so an exit code alone cannot tell "refused" from "ran and
    failed" (test_ten_lane_highway_infra_ledger.py's standard).
  • The subprocess env is built explicitlymodel_performance-etuz was this
    same mistake in this same directory: green in CI, red inside every lane
    (569c9b8).

Platform guard sys.platform != "linux" with the reason stated (flock is
util-linux; chmod --reference is a GNU flag with no BSD equivalent) — the
precedent that cleared #310 and #313.

Adoption did not widen what teardown can destroy

Test Proves
test_lane_teardown_has_no_sweep_verb sweep rejected as unknown command; sentinel absent
test_lane_teardown_offers_no_all_owners_escape_hatch --all-owners / --all / --everything each refused; sentinel absent; string appears 0 times in the script
test_the_multi_owner_sweep_refusal_still_refuses 0rg's guard re-run post-adoption: exit 3, both sentinels absent
test_a_live_lanes_rows_are_never_touched protected-untouched=4 and the four live containers still exist — live lane + dead lane holding rows simultaneously
test_reconcile_still_refuses_to_reclaim_a_live_row reconcile runs no destroy command
test_the_sweep_refusal_names_a_lane_teardown_path_that_actually_exists the printed path is stat-ed and resolved — a string assertion would have passed against the original defect

Fail-before / pass-after

Shell harness vs the unpatched upstream copy — ye80b's shape, reproduced:

CASE 1 — the incident: a NEAR-MISS lane name must FAIL LOUDLY
  FAIL  exited 0 on a near-miss name (the footgun: reports success, does nothing)
  FAIL  did not name the candidate lane — an operator cannot act on this
  PASS  destroyed nothing and flipped no row while refusing
…
CASE 5 — the MIRROR near miss: long name typed, rows claimed short
  FAIL  mirror near miss exited 0 without naming a candidate
==================================================
RESULT: FAIL — 3 expectation(s) unmet

New pytest vs that same unpatched copy — the same three, and no others:

FAILED …::test_near_miss_lane_name_refuses_instead_of_reporting_success
FAILED …::test_near_miss_refusal_names_the_candidate_lane_and_its_open_row_count
FAILED …::test_the_mirror_near_miss_long_name_typed_rows_claimed_short
3 failed, 11 passed

New pytest vs the full pre-adoption state (script absent, docs at origin/main):
12 failed, 2 passed.

After: shell harness RESULT: PASS — near-miss guard present, protection intact;
pytest 14 passed.

Full suite

$ uv run pytest -q
1894 passed, 1 skipped, 13 deselected, 1 xfailed in 19.28s

Baseline on untouched origin/main: 1880 passed, 1 skipped, 13 deselected, 1 xfailed.

One caveat, reported rather than dropped: the first full run after venv
creation reported 1 failed
test_truststore_wrap_bio_shim.py::test_real_truststore_is_covered_at_cli_import
with 'skipped: truststore is not importable'. It did not reproduce across
four subsequent full runs (including one with a throwaway extra test file to
perturb collection order), and the origin/main baseline was green.

Exactly one implementation

The shipped copy is authoritative. This lane owns only amplifier-app-cli, so
the evals-repo side is stated, not done:
.amplifier/evaluation/tools/lane_teardown.sh must be replaced by (and
git add-ed over — it is currently untracked) the thin shim at
docs/lanes/giwq-adopt-lane-teardown/PROPOSED-evals-lane-teardown-shim.sh.

A shim rather than a deletion, because live GOAL.md files already handed to
running lanes spell the old path — deleting it turns each into No such file or directory at exactly the moment an operator is mid-incident. It never falls
back to a local implementation
: a fallback is a second implementation, and a
stale one reintroduces precisely this near-miss footgun. Not found ⇒ exit 127
naming the fix.

Lane note and transcripts: docs/lanes/giwq-adopt-lane-teardown/.

…skill only named (giwq)

model_performance-giwq. The skill's own refusal message sent an operator,
mid-incident, to `.amplifier/evaluation/tools/lane_teardown.sh` -- a path in a
DIFFERENT repo, to a file untracked even there:

  $ git -C ~/dev/openai-evals-team-ci ls-files --error-unmatch \
      .amplifier/evaluation/tools/lane_teardown.sh
  error: pathspec ... did not match any file(s) known to git

That is the ONLY lane-scoped teardown path. The alternative, `sweep`, is the
manager's batch-close verb that destroys EVERY lane's infrastructure. On
2026-09-05 six DTU containers were left running with open ledger rows after a
tmux-server restart killed three lanes; recovering them required exactly this
tool.

ADOPTED. lane_teardown.sh now ships beside its siblings in scripts/, carrying
ye80b's near-miss fix (PR #313's patch artifact, applied with `patch -p4`
clean): a lane name that ALMOST matches an owner exits 4 naming each candidate
owner and its open-row count, instead of printing success for a teardown that
would do nothing. On 2026-09-05 `lane_teardown.sh <batch> drbf teardown --yes`
printed "lane 'drbf' owns no open rows - nothing to do" and exited 0 while SIX
rows were open under `drbf-compaction-notice-ab`.

Callers now name a path that EXISTS: infra_ledger.sh's sweep refusal resolves
the sibling from BASH_SOURCE, SKILL.md uses <skill_directory>/scripts/..., and
highway_status.sh's orphan-row reclaim hint -- the same instruction for the same
incident -- prints a resolved, copy-pasteable path instead of a bare name.

Also: usage() derives its help from the file instead of the hardcoded '2,72p'
range, which had already drifted past the end of the header.

TESTS. tests/test_ten_lane_highway_lane_teardown.py, 14 tests: ye80b's
13-assertion shell harness converted (it already parameterised the script path
for exactly this), plus the adoption-safety proofs. Every "ran nothing" claim is
observable -- a state file that still exists or a `touch` sentinel that does not
-- because the buggy path PRINTS a success message and an exit code alone cannot
tell "refused" from "ran and failed". The subprocess env is built explicitly
(etuz, 569c9b8: an inherited env is green in CI and red inside every lane).

Adoption did not widen what teardown can destroy, proven: `sweep` is still not a
verb here, --all-owners/--all/--everything are still refused (and the string
appears 0 times in the script), 0rg's multi-owner sweep refusal still exits 3
with both sentinels absent, protected-untouched=4 still holds with a live lane
and a dead lane holding rows simultaneously, and reconcile still refuses a live
row.

FAIL-BEFORE: shell harness against the unpatched upstream copy reproduces
ye80b's "RESULT: FAIL - 3 expectation(s) unmet"; the new pytest against that
same copy fails those same 3 and no others; against the full pre-adoption state
(script absent, docs at origin/main) it is 12 failed, 2 passed.

Full suite: 1894 passed, 1 skipped, 13 deselected, 1 xfailed.

ONE implementation: the shipped copy is authoritative. This lane owns only
amplifier-app-cli, so the evals-repo side is stated rather than done --
docs/lanes/giwq-adopt-lane-teardown/PROPOSED-evals-lane-teardown-shim.sh must
replace (and be git-added over) the untracked copy there. A shim rather than a
deletion because live GOAL.md files already handed to running lanes spell the
old path; it never falls back to a local implementation, because a fallback is a
second implementation and a stale one reintroduces this very footgun.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
Brian Krabach (bkrabach) marked this pull request as ready for review September 6, 2026 19:32
@bkrabach

Copy link
Copy Markdown
Collaborator Author

Manager verification — all 4 gates re-run by me in a scratch clone. Merging.

Head a2e97e1, base e12bf27. CI 9/9 SUCCESS (ubuntu/macOS/Windows × 3.11/3.12 + both integration).

I read the lane's own "no infrastructure registered, sweep never run" as a reason to check whether the teardown path was ever exercised, not as a pass. It was — with constructed fixtures, which is the right way. That line refers only to the live batch ledger, which the goal explicitly forbade touching.

gate decisive fact
1. fail-before 12 failed, 2 passed at main e12bf2714 passed on branch. Not a no-op-when-empty suite: the failures name real behaviour (test_no_shipped_instruction_sends_an_operator_into_another_repo, test_reconcile_still_refuses_to_reclaim_a_live_row)
2. real rows test_the_exact_lane_name_tears_down_exactly_its_own_six_rows asserts the ledger state transitionDEAD_IDS → swept, LIVE_ROWS → open — against registered rows. test_a_live_lanes_rows_are_never_touched asserts protected-untouched=4
3. ye80b inputs near-miss patch applied and pinned: near-miss exits 4; the mirror case (long name typed, rows claimed short) also exits 4; lane_teardown.sh is now shipped in the skill's own scripts/
4. suite/CI 1893 passed locally with one collection-order failure that passes in isolation on BOTH main and branch (11 passed each); CI's own full-suite run is green 9/9

Adoption did not widen what teardown can destroy — proven, not asserted

This was the deliverable that could have been faked by making everything pass. Four independent tests carry it: no sweep verb (string appears 0 times in the adopted script), no --all-owners/--all/--everything escape hatch, 0rg's multi-owner refusal still exits 3 with both sentinels absent, and protected-untouched=4. A near-miss now refuses; a genuinely idle lane still exits 0 — so the guard is a near-miss detector, not "any protected row is an error".

One implementation, and the boundary stated rather than fudged

The shipped copy is authoritative. This lane cannot touch the evals repo, so it wrote the shim as an artifact and stated precisely what must happen there. That is the honest shape: the drift this item exists to end is now ended on the side that ships.

Two things it reported that it did not have to

  • A transferable finding worth more than the fix: lane_teardown.sh's usage() was pinned to sed -n '2,72p' against its own header, which had since grown — so the help text was silently truncated mid-topic and nothing could notice. "A hardcoded line range into your own source is a comment that lies as soon as anyone edits above it." Derived from the file now.
  • It corrected my goal's premise about the pre-existing failure. I told it test_real_truststore_is_covered_at_cli_import was known-pre-existing; it observed the failure once, could not reproduce it across four subsequent runs, found the origin/main baseline green, and reported it as observed-once-not-reproduced rather than claiming my label. My own check agrees: it passes in isolation on both sides, so it is collection-order dependent — real, but not this PR's and not reliably reproducible.

It also edited highway_status.sh (one echo) beyond the two files I named, because it printed the same reclaim instruction with the same "no path at all" defect. Correct call, and it left highway_watchdog.sh alone since PR #306 is open against it.

@bkrabach
Brian Krabach (bkrabach) merged commit be25afd into main Sep 6, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants