Skip to content

feat(ios-runner): hand physical iOS runners across daemon restarts - #2692

Merged
thymikee merged 4 commits into
mainfrom
apex/2681-physical-runner-handoff
Sep 21, 2026
Merged

thymikee merged 4 commits into
mainfrom
apex/2681-physical-runner-handoff

Conversation

@thymikee

@thymikee thymikee commented Sep 19, 2026

Copy link
Copy Markdown
Member

Summary

Graceful daemon shutdown hands a request-proven Apple runner to the next daemon instead of paying the xcodebuild ramp again. That was simulator-only. Physical iOS now joins it (#2681).

resolveRunnerHandoffTarget names the lanes: every Apple Simulator stays on simulator, physical iOS/iPadOS over CoreDevice joins on physical_coredevice. The macOS host, physical tvOS/visionOS, and the usbmux-only xctest backend stay on kill-and-rebuild with typed refusals, because nothing has exercised a handoff for them.

A handoff is claimed only for a ready session that owes no response and whose last exchange did not report main-thread work draining, so a shutdown during a startup tears that runner down. The runner's stdout/stderr are its own log file, inherited at spawn, so the handoff gives up only this daemon's copy of that descriptor and the log it follows - it never closes a pipe the child is writing to (#2681 review). Adoption reads the same predicate and gives the physical lane a second uptime probe for the cold CoreDevice tunnel lookup, with both probe phases spending the request's own startup budget. Every decline reports a reason and lane, and the shutdown runs inside a daemon diagnostics scope so those reasons reach daemon.log at all.

detachIosSimulatorRunnerSessionsForShutdown -> detachIosRunnerSessionsForShutdown.

Validation

Head 33c5fd1d5e, rebased onto origin/main 0ebd2540a3. Ran: pnpm check:quick, check:layering, check:production-exports, check:fallow, pnpm build, pnpm vitest run --project apple-runner (520), --project unit-core (10550 passed, 1 skipped).

check:production-exports exits 0 and still prints the repo-wide pre-existing report (67 unused exports, 18 suppressed, 0 stale); nothing in it is new to this branch.

Not run: pnpm check:affected --run and pnpm gate replay-ios-device - both need a cabled device, which this phase had none of. docs/evidence/ios-physical-runner-handoff-2026-09-19.md carries the exact commands and the assertions each device step must produce, including the ones only a device can answer: a post-handoff write landing in the runner log with the same PID and elapsed time, a replug/reboot/DDI remount between detach and adopt ending in a rebuild rather than a hang, and a lingering detached runner not blocking devicectl install/uninstall or a concurrent xcodebuild.

Unproven risk: that a physical xcodebuild survives the handoff and answers the next daemon. Tests prove the gates decline and that the daemon stops holding a pipe it could close under the child; only the device lane proves the handoff.

@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.72 MB 4.73 MB +4.8 kB
Package (unpacked) 4.72 MB 4.73 MB +4.8 kB
Package (download) 1.41 MB 1.41 MB +1.5 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 18.3 ms 18.7 ms +0.4 ms
CLI --help 55.2 ms 52.2 ms -2.9 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at f48a55a. The output-release check cannot detect what it claims, and no physical-device run shows that the handoff works.

detachRunnerSessionForShutdown calls session.endOutputObservation() and checks isRunnerProcessAlive on the next synchronous line (runner-session.ts#L752). xcodebuild gets EPIPE/SIGPIPE only on its next write, and no write can happen between those two lines, so runner_died_on_output_release likely never fires. A runner that dies on its next write, for example after a device lock or a replug, still gets a detached lease. The next daemon then spends up to 5.5 s probing under the lease lock before it rebuilds. Task 3 of #2681 (prove the child survives, or restructure so it does) is not met, and runner output after the handoff no longer reaches runner.log. Could xcodebuild write stdout and stderr straight to a runner.log file descriptor set at spawn, with listener-ready read from that log? Then the child never hits EPIPE, and endOutputObservation on LaunchedRunnerProcess and RunnerSession can go. If the pipes stay, please remove the inert check and its reason, and show on a device that xcodebuild survives a write after the release.

No physical run backs this yet (.device-evidence/CHECKLIST.md). For each in-scope backend, #2681 asks for the device, iOS version and SHA, a daemon restart, and the next command served by the adopted runner. It also asks for the 5 s cold-tunnel budget to be measured, and for simulator detach and adoption timings compared with main. On a cabled coredevice iPhone at the head: open -i until ready, stop the daemon gracefully, cause at least one runner write (lock and unlock), and show ps -p $RUNNER_PID with the same process across the restart. The next snapshot's request ndjson should show ios_runner_lease_adopted with lane physical_coredevice, and startupTimings with adopt_detached_runner and without ensure_xctestrun or launch_xcodebuild. Repeat Wi-Fi-only with the cable pulled and record ios_runner_lease_adoption_probe durationMs for both, plus simulator adoption timing before and after.

Is the log-file ownership above the smaller change? It removes the pipe seam and the inert check, and the handoff-lane predicate and lease verdict stay in apple-runner-platform.ts and runner-lease.ts. The listener-ready observer in runner-process-launch.ts would first have to read readiness from the log file. Whether xcodebuild keeps the default SIGPIPE disposition also decides whether the release kills the runner, and only a device run can settle that.

Not blocking: the pipe-release tests only count mock destroy() calls, the cold-tunnel tests mock sendRunnerCommandOnce so the cold route-resolver path never runs, and the committed checklist and redacted devicectl JSON under .device-evidence/ fit better in the PR body or an evidence ref, especially because the JSON says it came from a cabled device while the PR body says none was available.

CI is green on 19 checks, but none of them runs a physical device, so CI says nothing about the handoff route. There are no conflicts with main.

Next: settle who owns the runner output so the child survives the daemon, then post a live coredevice restart-handoff run with the adoption diagnostics and measured probe timings.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at aff4d56. This follows up on the earlier review at f48a55a (#2692 (comment)). The runner now writes to a log it owns, which removes the pipe problem, but two new issues and the device evidence remain.

inFlightCommands in runner-session.ts#L984 goes up before the round-trip and down in finally. So when a client cancels with Ctrl-C, or the connection drops during shutdown, the count returns to 0 while the runner still runs the abandoned command on its main thread. runnerMainThreadBusy only reflects the last completed exchange, so resolveRunnerDetachDecision can hand off a busy runner. The next daemon's adoption probe or first command then hits a runner that is still draining, and it pays for a rebuild. This is the case command_in_flight is meant to catch, per the code comment and step 7 of the evidence doc. Could detach eligibility count every command the runner accepted and has not answered, even when this process no longer waits for it? For example, decrement only on a decoded response, and on abort or transport failure set a marker that only a later successful exchange clears. A test that aborts the signal mid-command and then calls detachIosRunnerSessionsForShutdown, expecting command_in_flight, would cover it. It is also untested whether the runner answers uptime while it drains an abandoned command; that decides whether this ends in a failed probe or in adopting a busy runner.

launchRunnerProcess opens runner.log in append mode and records where the new generation starts, but readRunnerLogTail in runner-io.ts#L150 reads the last 64 KB of the whole file. If a generation exits early after writing less than 64 KB, the tail still holds the previous generation's output. classifyBootFailure and resolveRunnerEarlyExitHint can then match the older run, report a stale reason and hint, and quote xcodebuild output that this launch never produced. Could the tail start at max(startOffset, size - maxBytes), so it holds only the current generation's bytes? A test with a prefilled log that holds an older boot-failure line would confirm it.

docs/evidence/ios-physical-runner-handoff-2026-09-19.md still has every box unchecked and says no device was available. So nothing yet shows that the file-backed runner survives a daemon restart, keeps logging, and is adopted on coredevice, and the cold-tunnel timing is not measured. #2681 task 3 asks for this. On a cabled coredevice iPhone at this head, please record the device, iOS version and SHA, and show:

  • ps -p $RUNNER_PID -o pid,etime with the same PID across daemon stop and the next command,
  • runner.log growing after a replug that follows the handoff,
  • the next snapshot's request ndjson with ios_runner_lease_adopted on lane physical_coredevice, and startupTimings with adopt_detached_runner but without ensure_xctestrun or launch_xcodebuild,
  • ios_runner_lease_adoption_probe durationMs for the tight and cold-tunnel phases, cabled and Wi-Fi-only, plus simulator adoption timing before and after this change.

The failing Smoke check, RunnerTests.testAlertAcceptDoesNotActivateAReplacementWithASharedButton, is a Swift XCTest alert case, and this PR changes nothing under apple/, so it looks unrelated. There are no conflicts with main.

Next: make command_in_flight hold across a cancelled command, limit the early-exit tail to the current launch, and post the coredevice restart-handoff run at this head.

@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from aff4d56 to 33c5fd1 Compare September 19, 2026 19:37
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 33c5fd1. Both blocking findings from the earlier round (aff4d56) are still open in this delta.

session.inFlightCommands is decremented in a finally (https://github.com/callstack/agent-device/blob/33c5fd1/packages/platform-apple/src/runner/runner-session.ts#L1014), so a Ctrl-C cancel or a transport drop resets the count to 0 while the runner keeps executing the abandoned command on its main thread. runnerMainThreadBusy only reflects the last completed exchange, so resolveRunnerDetachDecision (runner-session-types.ts:193) returns detach and shutdown hands off a runner that is still busy. The next daemon then either queues behind that stale work or fails the 500 ms / 5.5 s probe and pays a full xcodebuild rebuild — the exact case command_in_flight exists to catch. Occupancy needs to mean "accepted a command this process hasn't seen answered": decrement only on a decoded response, and set a sticky busy marker on abort or transport failure that only a later successful exchange clears. This has to live on the one send helper so every path to the runner is covered, not at the detach call site. Can we get a test that aborts the signal mid-command and then asserts detachIosRunnerSessionsForShutdown returns command_in_flight?

launchRunnerProcess opens runner.log in append mode and records currentFileSize(outputFd) for the tail follower (https://github.com/callstack/agent-device/blob/33c5fd1/packages/platform-apple/src/runner/runner-io.ts#L161), but readLogTail (runner-process-launch.ts:115) calls readRunnerLogTail, which reads bytes size - min(size, maxBytes) against the whole file, not from the recorded start offset. If a generation exits after writing less than maxBytes, the tail still contains the previous generation's output. classifyBootFailure and resolveRunnerEarlyExitHint can then match an older run and report a boot-failure reason quoting xcodebuild output this launch never produced, sending the user down the wrong recovery path. The fix is to thread the recorded generation start offset through to readRunnerLogTail and read from max(startOffset, size - maxBytes), making the offset part of the handle so every reader of that log is bounded the same way. A test with a prefilled log containing an older boot-failure line should go red today and catch this.

The checklist in docs/evidence/ios-physical-runner-handoff-2026-09-19.md:11 is byte-identical to aff4d56 and every box is still unchecked, so there's still no cabled coredevice run at this head. CI is green across 19 checks, and the Smoke XCTest alert case that failed at aff4d56 is green again, consistent with it being unrelated since this PR touches nothing under apple/, but no job here exercises a physical device, so CI says nothing about the handoff route itself. The PR's central claim — that a physical xcodebuild survives the handoff and the runner answers the next daemon — is unproven, and the 5000 ms cold-tunnel constant is unsized against a real tunnel lookup. Could you run this on a cabled coredevice iPhone at 33c5fd1 and post: the same RUNNER_PID across daemon stop and the next command (ps -p $RUNNER_PID -o pid,etime), runner.log growing after a post-handoff replug, the next snapshot's request ndjson showing ios_runner_lease_adopted on lane physical_coredevice with adopt_detached_runner in startupTimings and neither ensure_xctestrun nor launch_xcodebuild, and ios_runner_lease_adoption_probe durations for both the tight and cold-tunnel phases, cabled and Wi-Fi-only, plus simulator adoption timing before and after?

I did not run the test suites, so the 520 / 10550 counts in the PR description are unverified from here. I don't have a physical iOS device, so whether xcodebuild survives the handoff, and whether the runner answers uptime while draining an abandoned command, can't be settled from the code alone — that's exactly what decides whether a stale detach ends in a failed probe or in adopting a busy runner. This pass covered the logical delta, the two carry-over sites, and the lease/adoption ownership model; the rest of the PR was settled in the earlier rounds.

There are no merge conflicts on this branch. Before this can merge, occupancy tracking needs to survive a cancelled command, the early-exit tail needs to be bounded to the current launch, and the coredevice restart-handoff run needs to be posted at this head.

@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from 33c5fd1 to 1016a60 Compare September 20, 2026 11:45
@thymikee

Copy link
Copy Markdown
Member Author

Both carry-overs are fixed. Rebased on newest main; force-pushed.

1. Occupancy survives a cancelled command

Charging is no longer tied to a live wait. sendRunnerCommandAfterPreflight increments on the single send path and does not release in a finally (runner-session.ts:993); an error from that path marks the charge abandoned (runner-session.ts:1020), so the runner keeps the occupancy it really took. The charge comes down only where a response is decoded: settleRunnerCommandAnswered (runner-session.ts:888 on the success path, :912 via settleRunnerCommandExchange for structured replies, which are answers whatever they report).

The abandoned share is sticky rather than per-wait, because one cancellation resets every live wait this side holds while the runner keeps working: abandonedCommands (runner-session.ts:1040) is forgiven by a later answered exchange (Math.max(0, inFlightCommands - 1 - abandonedCommands), :1031) — the runner is demonstrably serving again and stamps anything still draining onto that reply, which is the runnerMainThreadBusy mirror from #2552. resolveRunnerDetachDecision refuses on either counter (runner-session-types.ts:214), reason command_in_flight.

Test: a command abandoned by a cancelled transport keeps the runner occupied (__tests__/runner-session-lifecycle.test.ts) aborts mid-command, asserts the refusal with "reason":"command_in_flight", then serves one command and asserts the handoff proceeds. Red with the finally decrement restored, green as shipped.

2. The early-exit tail is bounded to its generation

One handle per launch: RunnerLogFile { logPath, startOffset } (runner-io.ts:84), built from the append descriptor the child inherited (createRunnerLogFile, runner-io.ts:94, read off fstat of that fd, so it is exactly where the child's first write lands — runner-process-launch.ts:95). Both readers take it: the listener-ready tail (tailRunnerLogFile, runner-io.ts:117) and the tail an error quotes (readRunnerLogTail, runner-io.ts:187, runner-process-launch.ts:120), so from = max(startOffset, size - maxBytes) and nothing below the generation can reach a hint. A file that shrank since spawn was truncated or replaced, so what is there now postdates this launch and is quoted from 0; a generation that wrote nothing quotes ''.

Tests: a quoted tail never reaches back before its own generation, a generation that has written nothing yet has no tail to quote, plus rotation coverage, in __tests__/runner-io.test.ts; and a failure from an older runner generation is not quoted as this launch output in __tests__/runner-early-exit-diagnosis.test.ts, which prefills the verbatim unprovisioned-device log, writes a different failure above it, and asserts IOS_RUNNER_CONNECT_TIMEOUT with no provisioning text quoted. The three offset assertions fail with the bound removed.

Gates (all on 97ba2ceaeb)

pnpm build, format, check:quick, check:layering, check:fallow (0 findings in changed files — executeRunnerCommandWithSession needed the settlement branch pulled into a named helper rather than a baseline move), check:production-exports exit 0 (67 pre-existing repo-wide unused exports, 18 suppressed, 0 stale — unchanged). apple-runner 526/526 across 51 files; unit-core 10630 passed + 1 skipped across 1353 files. No check:affected and no device work: this phase has no device.

What Phase B still has to show on thymikee-iphone

docs/evidence/ios-physical-runner-handoff-2026-09-19.md, in order. Preconditions first: cabled iOS 17+ device; pnpm install --frozen-lockfile && pnpm build && pnpm build:xcuitest && pnpm clean:daemon; $STATE_DIR from pnpm daemon:state-dir; $DEVICE_ID from device list --json with iosPhysicalDeviceBackend: coredevice; $LEASE at ~/.agent-device/apple-runner/leases/$DEVICE_ID.json; $RUNNER_LOG from jq -r .runnerLogPath $LEASE.

  1. Detachopen <app> --platform ios --udid <UDID> --session p2681 --foreground -i --json to publish ready; record $RUNNER_PID/$RUNNER_PORT/$OWNER_TOKEN; daemon stop (graceful); expect ownerToken matching ^detached-owner-, one ios_runner_session_detached line in daemon.log with "lane":"physical_coredevice" and runnerLogPath = $RUNNER_LOG, and ps -p $RUNNER_PID -o pid,etime,comm alive across the restart.
  2. Output still reaches the log after the handoff — unplug 10 s and replug, then wc -c "$RUNNER_LOG" grew and tail -40 shows new bytes on the same PID; after the eventual close --session p2681 the same file ends with the runner's teardown line. A runner that vanishes here is the SIGPIPE regression this restructure exists to remove.
  3. Adoptionsnapshot -i --json --session p2681 …; in that request's ndjson expect ios_runner_lease_adopted on physical_coredevice with runnerPid = $RUNNER_PID, and ios_runner_session_startup_timings.data.timings carrying adopt_detached_runner with no build_xctestrun and no launch_xcodebuild; compare wall time against a cold start with AGENT_DEVICE_IOS_RUNNER_DETACH=0; then ≥5 round-trips on a fresh session.
  4. Probe budget — after the tunnel has gone idle (or unplug/replug), the request ndjson shows ios_runner_lease_adoption_probe twice, probePhase: "tight" then "cold_tunnel", each with timeoutMs, budgetCapMs, durationMs; record both durations so the 500/5000 ms caps stay measurements. If the cold probe also fails, expect ios_runner_lease_adoption_skipped with probe_failed or probe_budget_exhausted and a fresh runner next — designed, still report the reason. Repeat for Wi-Fi-only.
  5. Sticky refusal + generation-bounded quote (section 7) — Ctrl-C a long replay after the runner accepted it, then daemon stop: expect command_in_flight; daemon stop again with nothing further and expect the same refusal; after a served command on the next daemon, expect a handoff. And keep an $RUNNER_LOG from an older failed launch, force a different early exit on top of it, and confirm the error quotes only this generation's bytes.

Also still unproven without hardware: sections 5 (unplug/reboot/DDI remount while orphaned), 6 (detached runner must not block devicectl install/uninstall or an unrelated xcodebuild), and the rest of 7 (mid-startup runner_never_served_a_command, main_thread_occupied, AGENT_DEVICE_IOS_RUNNER_DETACH=0, simulator_set_redirect, xctest_backend, macos_host, SIGKILL takeover) plus section 8 (pnpm gate replay-ios-device, check:affected --run). Report each unchecked box with the command, the diagnostic phase/reason, and the log path — not from unit coverage.

@thymikee

Copy link
Copy Markdown
Member Author

This is a follow-up on the earlier review at 33c5fd1 (#2692 (comment)). The code-side gaps from that review are fixed. What is still missing is the same thing: proof the handoff works on a real device.

At 97ba2ce, every checkbox in docs/evidence/ios-physical-runner-handoff-2026-09-19.md#L11 is still unchecked, and the delta only adds two more unchecked items for the sticky refusal and the generation-bounded quote. The PR's central claim is that a physical xcodebuild run survives a daemon restart and answers the next daemon, with 500/5000 ms probe caps. No CI job exercises a cabled device, so that claim is still unproven. Please run sections 1-4 and the two new section-7 items on a cabled coredevice iPhone at this head and post the output: the same $RUNNER_PID alive across daemon stop (ps -p $RUNNER_PID -o pid,etime), $RUNNER_LOG growing after a post-handoff replug, the next request's ndjson showing ios_runner_lease_adopted on lane physical_coredevice with adopt_detached_runner in startupTimings and no build_xctestrun/launch_xcodebuild, both ios_runner_lease_adoption_probe durations (tight and cold_tunnel), plus a run where you Ctrl-C a long replay then call daemon stop twice and see command_in_flight both times with a handoff only after one served command, and a run where an early exit sits on top of a prior generation's log and the quote covers only this generation's bytes.

Not blocking: whether settleRunnerCommandExchange/settleRunnerCommandAnswered should forgive abandonedCommands only when the reply actually stamps runnerMainThreadBusy (packages/platform-apple/src/runner/runner-session.ts#L1048), whether enrichRunnerFailureFromLog should read through the current generation's handle instead of a bare log path so an older generation's crash text can't misclassify this generation's failure (packages/platform-apple/src/runner/runner-failure-diagnostics.ts#L50), whether the lifecycle test's ok fixture should stamp runnerMainThreadBusy and add a true case to pin the forgiveness rule it's meant to test (packages/platform-apple/src/runner/tests/runner-session-lifecycle.test.ts#L392), and whether createRunnerLogFile should fstat the append descriptor right after fs.openSync instead of after runCmdBackground returns, so bytes written in that window aren't dropped from the tail (packages/platform-apple/src/runner/runner-process-launch.ts#L95) — these can be taken or left for now.

I did not run any suite here, so the counts and the claimed red-without-the-fix result are reasoned from the diff, not executed, and I have no physical iOS device to confirm whether an adopted runner answers or whether a runner draining abandoned work refuses the next daemon's probe. Whether the daemon can actually make an adopted, possibly older runner return an unstamped INVALID_ARGS is a real question: the Swift side builds that reply off the main thread and unstamped, but I haven't seen a shipped daemon produce one.

CI is green, 19 checks, 0 not passing, and there are no known conflicts with main, but no job here drives a cabled iOS device, so green CI says nothing about the handoff route this PR changes. The remaining step before merge is posting the cabled coredevice run at 97ba2ce covering sections 1-4 and the two new section-7 items; the two carry-over code fixes from the last round hold.

@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from 97ba2ce to bd12076 Compare September 20, 2026 13:55
@thymikee

Copy link
Copy Markdown
Member Author

Rebased and the fstat-window nicety is taken. Code is settled on my side — nothing else in the not-blocking list is touched (settleRunner* forgiveness stamping and enrichRunnerFailureFromLog handle-threading left as-is).

  • head bd120761c6, base fc59620932 (origin/main had moved one commit past 3155aea618; rebase was clean, no conflict, and packages/platform-apple/src/runner/host.ts is again a 0-line diff against origin/main — delegate shim intact, no mirror types).

Generation offset is now taken while the descriptor is fresh

runner-process-launch.ts:58 builds createRunnerLogFile(logPath, outputFd) immediately after fs.openSync(logPath, 'a') and before runCmdBackground, instead of after the spawn call returned. An append descriptor opens at end-of-file, so the fstat size is where this generation's first byte lands — but only until something writes, and a runner that dies instantly writes into precisely that window. Previously those first lines landed below the offset, so the readiness tail never saw a listener marker the runner really printed, and an early-exit error quoted ''.

createRunnerLogFile itself is unchanged (runner-io.ts:95); its doc now states why the descriptor has to be read fresh.

Two tests in __tests__/runner-process-launch.test.ts, both writing from inside the runCmdBackground mock so they land in that window:

  • :97 — a listener marker written during the spawn call wakes startup retries;
  • :121 — a boot failure written during the spawn call is quotable from this generation, and the older generation's text above it stays out of the quote.

Both were run against the previous placement and fail there (startupRetryWake never aborts; readLogTail returns the empty string), green at bd120761c6.

Gates at bd120761c6

pnpm build, format, check:quick (lint + typecheck), check:layering (OK, 1740 files), check:fallow (no issues in 115 changed files) all exit 0. apple-runner 528/528 across 51 files; unit-core 10638 passed + 1 skipped across 1353 files. No check:affected and no device work here.

Remaining before-merge item: Phase B on the cabled coredevice

The code is settled; what is left is the physical restart-handoff run, which this phase cannot do (no device). The artifact the coordinator fills in is docs/evidence/ios-physical-runner-handoff-2026-09-19.md — run it at bd120761c6 on thymikee-iphone, report per box with command, diagnostic phase/reason, and log path, and do not mark a box from unit coverage.

Preconditions: pnpm install --frozen-lockfile && pnpm build && pnpm build:xcuitest && pnpm clean:daemon; STATE_DIR=$(pnpm daemon:state-dir); DEVICE_ID from node bin/agent-device.mjs device list --json with iosPhysicalDeviceBackend: coredevice; LEASE=~/.agent-device/apple-runner/leases/$DEVICE_ID.json; RUNNER_LOG=$(jq -r .runnerLogPath $LEASE).

  1. Detachopen <app> --platform ios --udid <UDID> --session p2681 --foreground -i --json until a snapshot returns nodes; jq '.runnerPid,.port,.ownerToken' $LEASE for $RUNNER_PID; node bin/agent-device.mjs daemon stop; then $LEASE token matches ^detached-owner-, one ios_runner_session_detached line in $STATE_DIR/daemon.log with "lane":"physical_coredevice" and runnerLogPath = $RUNNER_LOG, and ps -p $RUNNER_PID -o pid,etime,comm alive with etime spanning the restart.
  2. Log still grows after the handoff — unplug 10 s, replug, then wc -c "$RUNNER_LOG" grew and tail -40 "$RUNNER_LOG" shows new bytes on the same PID; after close --session p2681 the file ends with the runner's own teardown line. Runner gone instead = the SIGPIPE regression to chase.
  3. Adoptionnode bin/agent-device.mjs snapshot -i --json --session p2681 --platform ios --udid <UDID>; in that request's $STATE_DIR/sessions/p2681/requests/<requestId>.ndjson expect ios_runner_lease_adopted on physical_coredevice with runnerPid = $RUNNER_PID, and ios_runner_session_startup_timings.data.timings carrying adopt_detached_runner with no build_xctestrun and no launch_xcodebuild; compare wall time against AGENT_DEVICE_IOS_RUNNER_DETACH=0 cold start; then ≥5 press/snapshot -i round-trips on a fresh session.
  4. Probe durations — after the tunnel has idled (or unplug/replug), the same ndjson shows ios_runner_lease_adoption_probe twice, probePhase: "tight" then "cold_tunnel", and record each one's timeoutMs, budgetCapMs, durationMs; repeat for Wi-Fi-only. If the cold probe also fails, ios_runner_lease_adoption_skipped with probe_failed or probe_budget_exhausted plus a fresh runner next is the designed outcome — record reason and timings.
  5. Sticky refusal (section 7) — Ctrl-C a long replay after the runner accepted it, then daemon stop: expect "reason":"command_in_flight"; daemon stop a second time with nothing further and expect the same refusal again; then serve one command on the next daemon and stop gracefully — that one must hand off.
  6. Prior-generation log (section 7) — keep a $RUNNER_LOG left by an older failed launch, force a different early exit on top of it, and confirm the raised error quotes only this generation's bytes; older boot-failure text in the quote means the generation offset was lost between open and the error.

Sections 5, 6, the rest of 7 and section 8 (pnpm gate replay-ios-device, check:affected --run) stay open without hardware the same way as before.

@thymikee

Copy link
Copy Markdown
Member Author

This is a follow-up to the earlier review at 97ba2ce (#2692 (comment)). The evidence gap from that review is unchanged: at bd12076, all 49 checkboxes in docs/evidence/ios-physical-runner-handoff-2026-09-19.md#L11 are still unchecked, and no CI job drives a cabled device. This delta moves the generation offset onto the launch path every physical start uses, so the run has to happen at this head, not an earlier one. Without it, the PR's central claim is unproven: that a physical xcodebuild runner survives daemon stop and answers the next daemon within the 500/5000 ms probe caps. If the handoff in fact kills or wedges the physical runner, or the probe budget never clears, that would ship unseen, and users on cabled iPhones would pay a full rebuild or lose the session after a restart.

To close this, run sections 1-4 and the two section-7 items on a cabled coredevice iPhone at bd12076 and paste per-box output: the same $RUNNER_PID across daemon stop (via ps -p $RUNNER_PID -o pid,etime), $RUNNER_LOG growing after a post-handoff replug, the next request's ndjson showing ios_runner_lease_adopted on lane physical_coredevice with adopt_detached_runner in startupTimings and no build_xctestrun/launch_xcodebuild, both ios_runner_lease_adoption_probe durations (tight then cold_tunnel), command_in_flight twice after a Ctrl-C'd replay, and an early exit on top of a prior generation's log. For this delta specifically, also show a real boot failure whose first lines were written during the spawn call, appearing in the raised error's xcodebuild.stderr — that's the half of the generation window this change closes.

Not blocking: the four-line comment at the new runner-process-launch.ts call site (packages/platform-apple/src/runner/runner-process-launch.ts#L54) restates the JSDoc already added to createRunnerLogFile in runner-io.ts (lines 88-92), so the rule has two spellings and one owner; worth shortening the call-site comment to point at runner-io.ts instead, but take or leave it.

I did not run the two new tests against the pre-delta placement; the claim that they would fail there is read from the pre-change ordering and the 2 s waitFor deadline, not from a run. One residual case I looked for but could not turn into a defect: with the offset now taken before the spawn, bytes an older still-live runner appends to the same file during the spawn would be charged to the new generation, but I found no production route where two generations write one log at the same time, so this ordering is strictly better than before.

Smoke Tests was still queued when I looked, so there's nothing to attribute from it yet; the other 18 checks pass. If it does go red, the overlap to check first is runner-process-launch.ts, since every simulator and device runner start goes through it and the macOS/iOS smoke route reaches it.

Before this can merge: post the cabled-coredevice run at bd12076 filling in sections 1-4 and the two section-7 boxes of docs/evidence/ios-physical-runner-handoff-2026-09-19.md, including the boot-failure quote sourced from bytes written during the spawn call, and confirm Smoke Tests finishes green.

@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from bd12076 to 3a31b6a Compare September 20, 2026 16:49
@thymikee

Copy link
Copy Markdown
Member Author

Rebased onto origin/main 921a2cb3ac. New head 3a31b6a691 (force-pushed, --force-with-lease). No code changes — git range-diff shows all seven functional commits byte-identical to bd120761c6; only the two evidence commits differ.

One conflict resolved, docs-only. main now owns .device-evidence/CHECKLIST.md (#2682's coordinator checklist) at the path my first commit used for the #2681 checklist, and my move commit deleted it. Both sides resolved in main's favour: .device-evidence/CHECKLIST.md and DECISION.md are a 0-line diff against origin/main, the #2681 checklist lives only at docs/evidence/ios-physical-runner-handoff-2026-09-19.md, and the .gitignore entry for that scratch root is kept. packages/platform-apple/src/runner/host.ts is again a 0-line diff against origin/main — delegate shim intact, no mirror types.

Gates at 3a31b6a691 (base 921a2cb3ac): pnpm build, format (no changes), check:quick (lint + typecheck), check:layering (OK, 1742 source files), check:fallow (no issues in 112 changed files) all exit 0. Targeted apple-runner — runner-io.test.ts (log tail + generation bounds), runner-early-exit-diagnosis.test.ts, runner-process-launch.test.ts, runner-session-lifecycle.test.ts — 38/38; full apple-runner 534/534 across 52 files; unit-core 10740 passed + 1 skipped across 1362 files. No check:affected, no device access in this phase.

The code is settled; nothing is open on my side. The sole remaining before-merge item is the cabled coredevice restart-handoff run — docs/evidence/ios-physical-runner-handoff-2026-09-19.md sections 1-4 plus the two section-7 items (sticky command_in_flight refusal, and the generation-bounded early-exit quote) — which the coordinator will run on thymikee-iphone at 3a31b6a691. Sections 5, 6, the rest of 7 and section 8 stay open for the same reason. Report per box with command, diagnostic phase/reason, and log path.

@thymikee

Copy link
Copy Markdown
Member Author

Rebase checked at 3a31b6a, no new code review. git range-diff against bd12076 confirms the seven functional commits are byte-identical, and the conflict was resolved the way you describe: .device-evidence/CHECKLIST.md, .device-evidence/devicectl-info-details.json and packages/platform-apple/src/runner/host.ts are each a 0-line diff against origin/main at this head. The code verdict from the bd12076 review stands.

The blocker is unchanged. All 49 checkboxes in docs/evidence/ios-physical-runner-handoff-2026-09-19.md are still unchecked at 3a31b6a, and no CI job drives a cabled device, so the central claim — a physical xcodebuild runner survives daemon stop and answers the next daemon inside the 500/5000 ms probe caps — is still unproven. Since the generation-offset commit sits on the launch path every physical start uses, that run has to happen at this head.

Smoke Tests is still running here; there is nothing to attribute yet.

@thymikee

Copy link
Copy Markdown
Member Author

Phase B — physical handoff captured on thymikee-iphone (coordinator-run)

Head 3a31b6a691, iPhone 17 Pro, iOS 27.0, Xcode 26.2/17C52, cabled (transport: wired).

1. Graceful daemon stop detaches instead of killing

Step Observed
cold prepare ios-runner buildMs 0 / connect 249 / healthCheck 6509, runner PID 50471
daemon stop mode: graceful, clean: false
detach diagnostic ios_runner_session_detachedlane: "physical_coredevice", runnerPid 50471, port 49685, runnerLogPath set
runner after daemon exit PID 50471 alive, etime advanced — detached, not killed
re-prepare (fresh daemon) adopted the same PID; healthCheckMs 3 vs 6509 on the cold path

The runner also survived a later daemon idle-exit unattended (PID 50471 still alive ~16 min in, etime 18:00), i.e. detach is not tied to an explicit daemon stop.

2. The startup-timings assertion, from the request diagnostics record

adopt_detached_runner is recorded — in the per-request diagnostics record (sessions/<session>/requests/<id>.ndjson), not in the --json response body or daemon.log. That distinction matters for how the assertion is written:

{"level":"debug","phase":"ios_runner_lease_adoption_probe","durationMs":8,
 "data":{"lane":"physical_coredevice","probePhase":"tight","budgetCapMs":500,"timeoutMs":500,"answered":true,"port":60106}}
{"level":"info","phase":"ios_runner_lease_adopted",
 "data":{"lane":"physical_coredevice","port":60106,"runnerPid":18260,"deviceId":"<udid>"}}
{"level":"info","phase":"ios_runner_session_startup_timings","durationMs":122,
 "data":{"command":"uptime","ready":true,"timings":{"adopt_detached_runner":122}}}

timings contains only adopt_detached_runnerensure_xctestrun, launch_xcodebuild, allocate_port and cleanup_stale_bundles are absent, which is the shape the review asked to see, and it falls out of the early return in createRunnerSession (runner-session.ts:185).

3. Handoff across a real cable cycle

Cable pulled, held ~13 s, reinserted; prepare ios-runner fired 0.3 s after the phone reappeared on the bus (detected on raw USB enumeration, because devicectl list devices lags the physical state by seconds and reports a stale connected):

  • adopted the same runner PID 18260; whole command finished in ~3 s; healthCheckMs 4.
  • the tight probe answered in 8 ms (budgetCapMs 500, answered: true) on lane physical_coredevice.

So the fast path holds through an unplug/replug on the physical lane: no rebuild, no relaunch, no new runner process.

4. What this run does not cover, and the lever for it

probePhase: "cold_tunnel" was not exercised. The second probe phase only runs when the tight probe fails, and after a 13 s unplug macOS re-established the CoreDevice route essentially immediately (8 ms). To force the fallback you need the route to stay down past the 500 ms tight budget — a much longer unplug (route teardown), not a short cable cycle. If you want that branch proven on hardware, say so and I'll run it with a multi-minute unplug; otherwise the honest status is: tight-path adoption proven on hardware, cold-tunnel budget covered only by unit tests.

Note on reproducing this on hardware

devicectl state lag is the practical obstacle, not the feature: it kept reporting connected for seconds after the cable was pulled, so a devicectl-polled harness misses the edge and fires the command in the wrong window. Poll ioreg -p IOUSB for the bus edge instead.

@thymikee

Copy link
Copy Markdown
Member Author

This is the run I was waiting for at 3a31b6a, and it proves the claim on hardware. The runner survives both a graceful daemon stop and an unattended idle exit as the same PID, the next daemon adopts that same process with healthCheckMs 3 against 6509 on the cold path, the startup timings carry only adopt_detached_runner, and the same PID comes back after a real cable cycle with the tight probe answering in 8 ms. The runner_died_on_output_release check I called inert is gone from this head, so that finding is closed as well.

The one branch still covered only by unit tests is probePhase: "cold_tunnel", which a thirteen-second unplug could not reach. That is your call: run the long unplug before merge, or accept the tight path as the proven one and the cold-tunnel budget as unit-covered. Your tip about polling ioreg -p IOUSB instead of devicectl for the bus edge is worth keeping somewhere reusable.

I applied ready-for-human.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 20, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Rework at f9a021c290 — one probe budget instead of two phases

Taking the "cold-tunnel is your call" option as a request to make the smaller thing, plus a pass for
simpler code. Rebased onto origin/main be0902eaea.

The second probe phase is gone. The tight 500 ms probe and the 5000 ms cold-tunnel retry are now
one probe whose cap is derived from the lane (runner-adoption.ts:187-190): 5000 ms on
physical_coredevice, 500 ms otherwise. RunnerProbePhase, the phase loop and the probePhase
diagnostic field are deleted — grep -rn "cold_tunnel\|probePhase" packages/ is empty.

Why this loses nothing measured: a cap is not a sleep. The hardware run answered in 8 ms; that is
8 ms under either cap, so every success path is byte-identical. Only the refusal path changes, and it
gets strictly faster — 5.0 s instead of 5.5 s. What the phase actually encoded was "a physical probe
may pay a devicectl tunnel lookup first", and a single larger cap expresses that more honestly than
two attempts, especially since sendRunnerCommandOnce already resolves the route inside the same
deadline.

Also, all verified before deleting:

  • claimAdoptableRunnerLease / claimLeasedRunner / LeasedRunnerClaim existed only to host the
    second phase — adoption is linear guard → skip(...) again. runner-adoption.ts 393 → 323 lines
    (main is 230).
  • RunnerOccupancyVerdict's 'drained'/'unreported' states had no consumer that distinguishes
    them; both call sites asked === 'occupied'. One named boolean predicate now.
  • The early-exit stdout/stderr fallback and xcodebuild.stdout / xcodebuild.logPath had no
    producer
    once this branch moved stdio to a file and set captureOutput: false.
  • The log truncate/rotate branch had no rotator, and LaunchedRunnerProcess.runnerLogPath echoed a
    path the caller already had.
  • abandonedCommands was only ever read as > 0hasAbandonedCommands. inFlightCommands stays a
    count: concurrent requests on one session are real, and the two facts are not interchangeable
    because an abandoned command is invisible to runnerMainThreadBusy.
  • The comment claiming a single charging send path was wrong — the uptime readiness preflight is
    uncharged. The comment now says so; no behaviour change.

Tests: the refusal-matrix cell for the physical probe now drives a launched probe past its
deadline and asserts probe_failed, instead of asserting phase shape against a mock that rejects
instantly. Test name records that a launched probe can only report probe_failed, never
probe_budget_exhausted, because the clamp makes them mutually exclusive. Added the two missing
detach-path refusals (physical_non_ios_os, xctest_backend), mirroring macos_host.
apple-runner project: 535 passed. check:quick and check:layering green.

Honest scope note

The line-count win is smaller than the deletion list suggests (~50 net): un-refactoring restored
main's shape rather than deleting lines, so the real gain is that runner-adoption.ts is one linear
function shorter with one fewer concept, not 290 fewer lines. Reporting that rather than quoting the
projection.

Still open, and the two measurements that gate them

  • Physical probe budget is UNMEASURED. docs/evidence/… §4 now says so plainly instead of
    describing a phase. The number needed is the probe diagnostic's durationMs against
    budgetCapMs at this head, plus a wedged-runner refusal timing. Evidence doc boxes remain unchecked:
    the runs so far are recorded in the Phase B comment above, not yet in the artifact.
  • Simulator timings vs main, with a threshold: still missing. First attempt produced
    adopt_detached_runner=0ms with launch_xcodebuild present and no probe diagnostic — the session
    was declined before probing, which is the scoped-simulator-set exclusion working as designed. The
    measurement therefore needs a default-set simulator; that is a real gap, not a measurement artefact.
  • pnpm check:affected --run still has not run.

@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from 3a31b6a to f9a021c Compare September 21, 2026 07:13
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at f9a021c. The one-probe rework looks right. The cap now comes from the lane, the deleted helpers and fields have no remaining producers or consumers, and adoption stays exclusive with the same ordering. The hardware run at 3a31b6a still covers the success path, because the probe answered in 8 ms, well under either cap. The 5000 ms physical cap is still unmeasured on hardware, and the evidence doc now says so.

Compatibility & Provenance fails because of this PR. With the gates inlined, tryAdoptRunnerSessionFromLease (runner-adoption.ts#L85) is at cyclomatic 18 and cognitive 17. Could the identity gates (session id, pid present, alive and same identity) go back into one helper that returns a pid or a refusal? It would run before the probe and again for the recycled-pid check after it. Please do not suppress the gate or add a baseline entry.

The Android Smoke Tests failure (the alert-result test element was not visible after five scrolls) looks unrelated. The Android-reachable changes here run only at daemon shutdown, and the rebase pulled in the #2705 test-app layout change. A rerun would confirm it.

There are no conflicts, and ready-for-human stays on. The complexity gate is the next blocker before merge.

@thymikee

Copy link
Copy Markdown
Member Author

Simulator AC measured; two corrections to what I said earlier

Rebased state unchanged; added fc57f0d39e + 378d4dbfa3 (evidence artifact only, no source).

Correction 1 — my "scoped-simulator-set exclusion" claim was wrong. When I reported that the
simulator lane declined adoption (adopt_detached_runner=0, launch_xcodebuild present, no probe
diagnostic), I attributed it to the redirect exclusion. It was my own instrumentation: ls -t across
session request files was reading the physical device's record (00008150-…) instead of the
simulator's. Selecting by device id instead of mtime showed the simulator adopting normally — same PID
across the restart (96688 on both sides), healthCheckMs 4 adopted vs 3941 cold. Nothing about the
redirect path was measured; --simulator-set-path remains untested.

Correction 2 — the detach kill switch is honoured. I first observed it as "ignored". Both times
that was wrong, and the cause is in the checklist itself: daemon accepts only stop, there is no
daemon start, so a prefixed daemon start never starts anything and the daemon is spawned later by
an unrelated command without the flag. Measured properly, on the spawning command: control run survived
(36315), AGENT_DEVICE_IOS_RUNNER_DETACH=0 killed it (37274). The checklist row now says where the
environment has to go.

Simulator timings vs main, threshold fixed in advance (simulator apex-2682-proto, iOS 26.2, no
--simulator-set-path, runner artifact cached both sides, 3 adopted cycles each):

  • adopted wall (connectAfterBuildMs + healthCheckMs): f9a021c290 29 ms, main be0902eaea
    35 ms.
  • cold total: 4284 ms vs 3833 ms.
  • Threshold, set before reading: adopted within +50 ms and under 1.5×; cold within +25%.
    Verdict: adopted is 6 ms faster, cold is +11.8%. Both pass.

One number I want on the record rather than quietly medianed away: the first cold cycle after
daemon stop --clean at this head measured connectAfterBuildMs 10729 + healthCheckMs 5033 — a
possible regression if it were real. It did not reproduce (1064/3941 and 1030/2533 in the two cycles
that followed), so it is first-cycle variance after a clean. main's first cycle was likewise a cold
start (1110/3082) because the preceding --clean had killed its runner, which is why it is out of the
adopted median.

Gates run here: check:daemon-wire-compat clean (178 declarations, 0 changed/removed/added/moved —
deleting probePhase did not move the wire surface), check:layering OK. check:production-exports
reports 67 issues at this head and 67 at main, with an identical finding set (comm clean both
ways); it is not part of pnpm check, so I left the baselines alone rather than regenerating them.

What this does not close: the physical probe cap in §4 is still unmeasured at this head (the only
hardware probe number, 8 ms, predates the single-cap design), §5's fail-closed rows, and both §8 lanes
including check:affected --run. The artifact is now 9 checked / 42 open, with physical results
recorded at their own SHA and a re-verification box beside each.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 378d4db. The two new commits change only the evidence doc. No source changed since f9a021c, so the code review from the last round stands. The new simulator-lane timings and the corrected note on the scoped-simulator-set claim read consistently. The doc also says plainly that the 5000 ms physical cap and the fail-closed unplug, reboot and remount rows are still unproven on hardware.

Compatibility & Provenance still fails on the complexity of tryAdoptRunnerSessionFromLease (runner-adoption.ts#L85): cyclomatic 18, cognitive 17. This is the same failure from the last round, and this PR causes it. The suggested fix is still to move the identity gates into one helper.

There are no conflicts, and ready-for-human stays on. The complexity gate is the next blocker before merge.

The newer head, af55e27, adds one more evidence-doc commit. This review does not cover it, and it does not touch the complexity finding.

@thymikee

Copy link
Copy Markdown
Member Author

§4 measured, and the number indicts the constant (8c5fb73a03)

The reason the physical budget kept coming back "unmeasured" is that per-request records only exist
with --debug
— without it requests/ is created and left empty. That is also the whole backstory of
my earlier misattributed record: I was picking files by mtime from a set that mostly contained unrelated
requests. Captured by emptying requests/, running exactly one adoption, and requiring one file.

Physical probe at this head, warm tunnel: one probe, budgetCapMs 5000, timeoutMs 5000,
durationMs 6, answered true → ios_runner_lease_adopted PID 80569, prepare 951 ms.

The cold case is the problem. A cold first command on this device cost 36856 ms of health check —
that is the cost class the 5000 ms ceiling was raised to absorb, and it is ~7x larger than the cap. A
tunnel that genuinely needs re-establishing cannot be waited out inside 5 s: the probe refuses while the
thing it waits for takes ~37 s. So the physical cap is not a measured warm number (6 ms would allow far
less) and is not sufficient for the cold case it exists for. That is a decision to make, not a constant
to keep: lower it toward the warm measurement and declare the cold path a rebuild by design, or raise it
an order of magnitude. Recorded as an open box rather than resolved by me unilaterally.

Two results that went against the checklist's expectations, recorded as such:

  • §5 unplug row not satisfied. A 60 s unplug + replug + unlock left the orphan (36543, etime 10:27)
    alive and adoptable — prepare 1172 ms, healthCheckMs 3, same PID. No refusal, no rebuild. Keeping
    tunnel state across a short cable cycle is legitimate adoption, so fail-closed needs the reboot variant.
  • §2 not satisfied by a cable cycle. runner.log went 78516 → 78516 bytes across that cycle. The
    runner survived and was adopted and wrote nothing. This doesn't disprove the SIGPIPE guard, but the
    log-growth claim is unproven, not proven.

§1 and §3 are now fully verified at the merged head (see previous comment). §4's cap decision, §2's
log-growth claim, §5's reboot row and both §8 lanes remain open.

@thymikee

Copy link
Copy Markdown
Member Author

Reboot row closed, and the cap question now has an answer (reboot + SIGSTOP probes)

Fail-closed proven. Reboot after a graceful detach: host runner 89944 had already exited, the next
prepare skipped adoption with typed reason runner_process_dead naming that pid, started a fresh
runner, and succeeded in 20000 ms (healthCheckMs 15887). Refusal is a typed reason on a claim, not text.

The cap's 5 s window is unreachable from the host, which settles its direction. runner.log growth
here is a new runner writing to the session log, so I did not tick section 2 from it. To wedge a live
runner I stopped xcodebuild with SIGSTOP - process state Ts, and the probe answered in 7 ms with
answered: true and adoption succeeded. Not because adoption is fail-open: the runner's port is
device-side (lsof -nP -iTCP:61248 -sTCP:LISTEN finds no Mac listener; we connect through the
tunnel), so the parent I stopped is not the answerer. The only process that can be wedged without being
killed lives on the device, out of reach from here - and anything that reaches it also tears the runner
down, which is the runner_process_dead path above.

So the states where 5000 ms could matter cannot be produced, while the two that can are 6-8 ms (device
present) and instant refusal (device gone). Against the measured 36856 ms cold wake, that leaves one
coherent reading: lower the physical cap toward the warm measurement and declare a cold tunnel a
rebuild by design
, which is what already happens. I have not edited the constant - it is a decision
for you, recorded with the measurements either way.

Also worth flagging from this pass: an earlier "legitimate adoption" note I made for the unplug case was
built on a run where the session behind it had been killed by hand. This pass drove every kill from the
host instead, so runner_process_dead above is the result of a known cause.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at a2a58a7. The new commits change only the evidence doc. No source changed since f9a021c, so the code review from that round stands. The typed reasons and phase names that the new rows quote, such as runner_process_dead for the reboot row, exist in the code at this head. I did not check the quoted hardware output itself.

Compatibility & Provenance still fails on the Fallow complexity gate for tryAdoptRunnerSessionFromLease (runner-adoption.ts#L85): cyclomatic 18, cognitive 17. This is the same failure as the last round, and this PR causes it. Smoke Tests, Repo Guards and Coverage were still running when I looked.

The doc still leaves two of its own rows open: the five post-close interaction round-trips in section 3, and the long-unplug refusal row in section 5. Those look like the remaining evidence gaps before merge.

Non-blocking: in runner-process-launch.test.ts (around line 90), a few negative waitFor checks pass on their first read, before the log tail's first 50 ms poll, so they would not catch a tail that starts at offset 0 or keeps following after endOutputObservation.

There are no conflicts, and ready-for-human stays on. The complexity gate is the next blocker before merge.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 5ee2061. The new commit splits tryAdoptRunnerSessionFromLease into private helpers and keeps its behavior: the refusal reasons, their order, and the lease passed to each skip are the same as at a2a58a7. Compatibility & Provenance now passes.

Smoke Tests fails in the live simulator scenario, where the long-press row was not visible after scrolling (job log). This PR does change simulator runner startup, but the runner started and the run failed later, at a scroll-and-visibility check. The same failure shows on main's iOS run 35589604091 and on several other PRs, so it is likely unrelated.

The PR now has merge conflicts with main. I removed ready-for-human until they are resolved. After the rebase, please rerun Smoke Tests.

@thymikee thymikee removed the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 21, 2026
Detaches a healthy runner on graceful daemon shutdown and adopts it on the next
startup in the physical CoreDevice lane, so a daemon restart stops costing a cold
runner build on cabled iPhones. The runner's own stdio moves to a log file that
survives the handoff, early-exit diagnosis reads that file bounded to one
generation, and adoption is gated on lane, lease identity, artifact fingerprint,
and a lane-capped uptime probe.

Includes the #2681 device-verification checklist with the coordinator's hardware
results recorded against the heads they were taken at.

Closes #2681
@thymikee
thymikee force-pushed the apex/2681-physical-runner-handoff branch from 5ee2061 to a4f8246 Compare September 21, 2026 12:24
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at c366b08. The rebase resolution keeps both sides: main's typed startup failures and device states from #2688 and #2690, and this PR's detach and adoption path. The new commit is correct: the log offset is taken only after the queued build lines land, and the early-exit error now names the file it quotes. It changes no handoff or adoption decision, so the physical evidence recorded before it still covers the handoff.

Typecheck & Package fails because of this PR. It makes inFlightCommands and hasAbandonedCommands required on RunnerSession (runner-session-types.ts#L91), and the rebase missed the fixture that main added in #2690 at runner-startup-transport.test.ts#L270. Please add inFlightCommands: 0, hasAbandonedCommands: false to that literal. It is the only error in the job log.

Not blocking: the flush at runner-session.ts#L302 swallows its 2 s timeout and keeps the offset, so a wedged append could still put a build line in the quote. Could a flush rejection mark the tail as unmeasurable instead? Also, no test would fail if the flush or the logPath precedence were reverted.

There are no conflicts, and I added ready-for-human. The Typecheck fix is the next blocker before merge.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 21, 2026
@thymikee
thymikee merged commit 9b68ab5 into main Sep 21, 2026
19 checks passed
@thymikee
thymikee deleted the apex/2681-physical-runner-handoff branch September 21, 2026 15:14
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-21 15:14 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant