feat(ios-runner): hand physical iOS runners across daemon restarts - #2692
Conversation
Size Report
Startup median (7 runs, lower is better):
|
|
Reviewed at f48a55a. The output-release check cannot detect what it claims, and no physical-device run shows that the handoff works.
No physical run backs this yet (.device-evidence/CHECKLIST.md). For each in-scope backend, #2681 asks for the device, iOS version and SHA, a daemon restart, and the next command served by the adopted runner. It also asks for the 5 s cold-tunnel budget to be measured, and for simulator detach and adoption timings compared with main. On a cabled coredevice iPhone at the head: Is the log-file ownership above the smaller change? It removes the pipe seam and the inert check, and the handoff-lane predicate and lease verdict stay in apple-runner-platform.ts and runner-lease.ts. The listener-ready observer in runner-process-launch.ts would first have to read readiness from the log file. Whether xcodebuild keeps the default SIGPIPE disposition also decides whether the release kills the runner, and only a device run can settle that. Not blocking: the pipe-release tests only count mock CI is green on 19 checks, but none of them runs a physical device, so CI says nothing about the handoff route. There are no conflicts with Next: settle who owns the runner output so the child survives the daemon, then post a live coredevice restart-handoff run with the adoption diagnostics and measured probe timings. |
|
Reviewed at aff4d56. This follows up on the earlier review at f48a55a (#2692 (comment)). The runner now writes to a log it owns, which removes the pipe problem, but two new issues and the device evidence remain.
docs/evidence/ios-physical-runner-handoff-2026-09-19.md still has every box unchecked and says no device was available. So nothing yet shows that the file-backed runner survives a daemon restart, keeps logging, and is adopted on coredevice, and the cold-tunnel timing is not measured. #2681 task 3 asks for this. On a cabled coredevice iPhone at this head, please record the device, iOS version and SHA, and show:
The failing Smoke check, Next: make |
aff4d56 to
33c5fd1
Compare
|
Reviewed at 33c5fd1. Both blocking findings from the earlier round (aff4d56) are still open in this delta.
The checklist in docs/evidence/ios-physical-runner-handoff-2026-09-19.md:11 is byte-identical to aff4d56 and every box is still unchecked, so there's still no cabled coredevice run at this head. CI is green across 19 checks, and the Smoke XCTest alert case that failed at aff4d56 is green again, consistent with it being unrelated since this PR touches nothing under apple/, but no job here exercises a physical device, so CI says nothing about the handoff route itself. The PR's central claim — that a physical xcodebuild survives the handoff and the runner answers the next daemon — is unproven, and the 5000 ms cold-tunnel constant is unsized against a real tunnel lookup. Could you run this on a cabled coredevice iPhone at 33c5fd1 and post: the same RUNNER_PID across I did not run the test suites, so the 520 / 10550 counts in the PR description are unverified from here. I don't have a physical iOS device, so whether xcodebuild survives the handoff, and whether the runner answers There are no merge conflicts on this branch. Before this can merge, occupancy tracking needs to survive a cancelled command, the early-exit tail needs to be bounded to the current launch, and the coredevice restart-handoff run needs to be posted at this head. |
33c5fd1 to
1016a60
Compare
|
Both carry-overs are fixed. Rebased on newest
1. Occupancy survives a cancelled commandCharging is no longer tied to a live wait. The abandoned share is sticky rather than per-wait, because one cancellation resets every live wait this side holds while the runner keeps working: Test: 2. The early-exit tail is bounded to its generationOne handle per launch: Tests: Gates (all on
|
|
This is a follow-up on the earlier review at 33c5fd1 (#2692 (comment)). The code-side gaps from that review are fixed. What is still missing is the same thing: proof the handoff works on a real device. At 97ba2ce, every checkbox in docs/evidence/ios-physical-runner-handoff-2026-09-19.md#L11 is still unchecked, and the delta only adds two more unchecked items for the sticky refusal and the generation-bounded quote. The PR's central claim is that a physical xcodebuild run survives a daemon restart and answers the next daemon, with 500/5000 ms probe caps. No CI job exercises a cabled device, so that claim is still unproven. Please run sections 1-4 and the two new section-7 items on a cabled coredevice iPhone at this head and post the output: the same $RUNNER_PID alive across Not blocking: whether I did not run any suite here, so the counts and the claimed red-without-the-fix result are reasoned from the diff, not executed, and I have no physical iOS device to confirm whether an adopted runner answers or whether a runner draining abandoned work refuses the next daemon's probe. Whether the daemon can actually make an adopted, possibly older runner return an unstamped INVALID_ARGS is a real question: the Swift side builds that reply off the main thread and unstamped, but I haven't seen a shipped daemon produce one. CI is green, 19 checks, 0 not passing, and there are no known conflicts with main, but no job here drives a cabled iOS device, so green CI says nothing about the handoff route this PR changes. The remaining step before merge is posting the cabled coredevice run at 97ba2ce covering sections 1-4 and the two new section-7 items; the two carry-over code fixes from the last round hold. |
97ba2ce to
bd12076
Compare
|
Rebased and the fstat-window nicety is taken. Code is settled on my side — nothing else in the not-blocking list is touched (
Generation offset is now taken while the descriptor is fresh
Two tests in
Both were run against the previous placement and fail there ( Gates at
|
|
This is a follow-up to the earlier review at 97ba2ce (#2692 (comment)). The evidence gap from that review is unchanged: at bd12076, all 49 checkboxes in docs/evidence/ios-physical-runner-handoff-2026-09-19.md#L11 are still unchecked, and no CI job drives a cabled device. This delta moves the generation offset onto the launch path every physical start uses, so the run has to happen at this head, not an earlier one. Without it, the PR's central claim is unproven: that a physical xcodebuild runner survives To close this, run sections 1-4 and the two section-7 items on a cabled coredevice iPhone at bd12076 and paste per-box output: the same $RUNNER_PID across Not blocking: the four-line comment at the new runner-process-launch.ts call site (packages/platform-apple/src/runner/runner-process-launch.ts#L54) restates the JSDoc already added to createRunnerLogFile in runner-io.ts (lines 88-92), so the rule has two spellings and one owner; worth shortening the call-site comment to point at runner-io.ts instead, but take or leave it. I did not run the two new tests against the pre-delta placement; the claim that they would fail there is read from the pre-change ordering and the 2 s waitFor deadline, not from a run. One residual case I looked for but could not turn into a defect: with the offset now taken before the spawn, bytes an older still-live runner appends to the same file during the spawn would be charged to the new generation, but I found no production route where two generations write one log at the same time, so this ordering is strictly better than before. Smoke Tests was still queued when I looked, so there's nothing to attribute from it yet; the other 18 checks pass. If it does go red, the overlap to check first is runner-process-launch.ts, since every simulator and device runner start goes through it and the macOS/iOS smoke route reaches it. Before this can merge: post the cabled-coredevice run at bd12076 filling in sections 1-4 and the two section-7 boxes of docs/evidence/ios-physical-runner-handoff-2026-09-19.md, including the boot-failure quote sourced from bytes written during the spawn call, and confirm Smoke Tests finishes green. |
bd12076 to
3a31b6a
Compare
|
Rebased onto One conflict resolved, docs-only. Gates at The code is settled; nothing is open on my side. The sole remaining before-merge item is the cabled coredevice restart-handoff run — |
|
Rebase checked at The blocker is unchanged. All 49 checkboxes in Smoke Tests is still running here; there is nothing to attribute yet. |
Phase B — physical handoff captured on
|
| Step | Observed |
|---|---|
cold prepare ios-runner |
buildMs 0 / connect 249 / healthCheck 6509, runner PID 50471 |
daemon stop |
mode: graceful, clean: false |
| detach diagnostic | ios_runner_session_detached → lane: "physical_coredevice", runnerPid 50471, port 49685, runnerLogPath set |
| runner after daemon exit | PID 50471 alive, etime advanced — detached, not killed |
re-prepare (fresh daemon) |
adopted the same PID; healthCheckMs 3 vs 6509 on the cold path |
The runner also survived a later daemon idle-exit unattended (PID 50471 still alive ~16 min in, etime 18:00), i.e. detach is not tied to an explicit daemon stop.
2. The startup-timings assertion, from the request diagnostics record
adopt_detached_runner is recorded — in the per-request diagnostics record (sessions/<session>/requests/<id>.ndjson), not in the --json response body or daemon.log. That distinction matters for how the assertion is written:
{"level":"debug","phase":"ios_runner_lease_adoption_probe","durationMs":8,
"data":{"lane":"physical_coredevice","probePhase":"tight","budgetCapMs":500,"timeoutMs":500,"answered":true,"port":60106}}
{"level":"info","phase":"ios_runner_lease_adopted",
"data":{"lane":"physical_coredevice","port":60106,"runnerPid":18260,"deviceId":"<udid>"}}
{"level":"info","phase":"ios_runner_session_startup_timings","durationMs":122,
"data":{"command":"uptime","ready":true,"timings":{"adopt_detached_runner":122}}}timings contains only adopt_detached_runner — ensure_xctestrun, launch_xcodebuild, allocate_port and cleanup_stale_bundles are absent, which is the shape the review asked to see, and it falls out of the early return in createRunnerSession (runner-session.ts:185).
3. Handoff across a real cable cycle
Cable pulled, held ~13 s, reinserted; prepare ios-runner fired 0.3 s after the phone reappeared on the bus (detected on raw USB enumeration, because devicectl list devices lags the physical state by seconds and reports a stale connected):
- adopted the same runner PID 18260; whole command finished in ~3 s;
healthCheckMs 4. - the tight probe answered in 8 ms (
budgetCapMs 500,answered: true) on lanephysical_coredevice.
So the fast path holds through an unplug/replug on the physical lane: no rebuild, no relaunch, no new runner process.
4. What this run does not cover, and the lever for it
probePhase: "cold_tunnel" was not exercised. The second probe phase only runs when the tight probe fails, and after a 13 s unplug macOS re-established the CoreDevice route essentially immediately (8 ms). To force the fallback you need the route to stay down past the 500 ms tight budget — a much longer unplug (route teardown), not a short cable cycle. If you want that branch proven on hardware, say so and I'll run it with a multi-minute unplug; otherwise the honest status is: tight-path adoption proven on hardware, cold-tunnel budget covered only by unit tests.
Note on reproducing this on hardware
devicectl state lag is the practical obstacle, not the feature: it kept reporting connected for seconds after the cable was pulled, so a devicectl-polled harness misses the edge and fires the command in the wrong window. Poll ioreg -p IOUSB for the bus edge instead.
|
This is the run I was waiting for at The one branch still covered only by unit tests is I applied |
Rework at
|
3a31b6a to
f9a021c
Compare
|
Reviewed at f9a021c. The one-probe rework looks right. The cap now comes from the lane, the deleted helpers and fields have no remaining producers or consumers, and adoption stays exclusive with the same ordering. The hardware run at 3a31b6a still covers the success path, because the probe answered in 8 ms, well under either cap. The 5000 ms physical cap is still unmeasured on hardware, and the evidence doc now says so. Compatibility & Provenance fails because of this PR. With the gates inlined, The Android Smoke Tests failure (the alert-result test element was not visible after five scrolls) looks unrelated. The Android-reachable changes here run only at daemon shutdown, and the rebase pulled in the #2705 test-app layout change. A rerun would confirm it. There are no conflicts, and |
Simulator AC measured; two corrections to what I said earlierRebased state unchanged; added Correction 1 — my "scoped-simulator-set exclusion" claim was wrong. When I reported that the Correction 2 — the detach kill switch is honoured. I first observed it as "ignored". Both times Simulator timings vs
One number I want on the record rather than quietly medianed away: the first cold cycle after Gates run here: What this does not close: the physical probe cap in §4 is still unmeasured at this head (the only |
|
Reviewed at 378d4db. The two new commits change only the evidence doc. No source changed since f9a021c, so the code review from the last round stands. The new simulator-lane timings and the corrected note on the scoped-simulator-set claim read consistently. The doc also says plainly that the 5000 ms physical cap and the fail-closed unplug, reboot and remount rows are still unproven on hardware. Compatibility & Provenance still fails on the complexity of There are no conflicts, and The newer head, af55e27, adds one more evidence-doc commit. This review does not cover it, and it does not touch the complexity finding. |
§4 measured, and the number indicts the constant (
|
Reboot row closed, and the cap question now has an answer (
|
|
Reviewed at a2a58a7. The new commits change only the evidence doc. No source changed since f9a021c, so the code review from that round stands. The typed reasons and phase names that the new rows quote, such as Compatibility & Provenance still fails on the Fallow complexity gate for The doc still leaves two of its own rows open: the five post- Non-blocking: in There are no conflicts, and |
|
Reviewed at 5ee2061. The new commit splits Smoke Tests fails in the live simulator scenario, where the long-press row was not visible after scrolling (job log). This PR does change simulator runner startup, but the runner started and the run failed later, at a scroll-and-visibility check. The same failure shows on main's iOS run 35589604091 and on several other PRs, so it is likely unrelated. The PR now has merge conflicts with main. I removed |
Detaches a healthy runner on graceful daemon shutdown and adopts it on the next startup in the physical CoreDevice lane, so a daemon restart stops costing a cold runner build on cabled iPhones. The runner's own stdio moves to a log file that survives the handoff, early-exit diagnosis reads that file bounded to one generation, and adoption is gated on lane, lease identity, artifact fingerprint, and a lane-capped uptime probe. Includes the #2681 device-verification checklist with the coordinator's hardware results recorded against the heads they were taken at. Closes #2681
5ee2061 to
a4f8246
Compare
|
Reviewed at c366b08. The rebase resolution keeps both sides: main's typed startup failures and device states from #2688 and #2690, and this PR's detach and adoption path. The new commit is correct: the log offset is taken only after the queued build lines land, and the early-exit error now names the file it quotes. It changes no handoff or adoption decision, so the physical evidence recorded before it still covers the handoff. Typecheck & Package fails because of this PR. It makes Not blocking: the flush at runner-session.ts#L302 swallows its 2 s timeout and keeps the offset, so a wedged append could still put a build line in the quote. Could a flush rejection mark the tail as unmeasurable instead? Also, no test would fail if the flush or the There are no conflicts, and I added |
|
Summary
Graceful daemon shutdown hands a request-proven Apple runner to the next daemon instead of paying the xcodebuild ramp again. That was simulator-only. Physical iOS now joins it (#2681).
resolveRunnerHandoffTargetnames the lanes: every Apple Simulator stays onsimulator, physical iOS/iPadOS over CoreDevice joins onphysical_coredevice. The macOS host, physical tvOS/visionOS, and the usbmux-onlyxctestbackend stay on kill-and-rebuild with typed refusals, because nothing has exercised a handoff for them.A handoff is claimed only for a
readysession that owes no response and whose last exchange did not report main-thread work draining, so a shutdown during a startup tears that runner down. The runner's stdout/stderr are its own log file, inherited at spawn, so the handoff gives up only this daemon's copy of that descriptor and the log it follows - it never closes a pipe the child is writing to (#2681 review). Adoption reads the same predicate and gives the physical lane a second uptime probe for the cold CoreDevice tunnel lookup, with both probe phases spending the request's own startup budget. Every decline reports a reason and lane, and the shutdown runs inside a daemon diagnostics scope so those reasons reachdaemon.logat all.detachIosSimulatorRunnerSessionsForShutdown->detachIosRunnerSessionsForShutdown.Validation
Head
33c5fd1d5e, rebased ontoorigin/main0ebd2540a3. Ran:pnpm check:quick,check:layering,check:production-exports,check:fallow,pnpm build,pnpm vitest run --project apple-runner(520),--project unit-core(10550 passed, 1 skipped).check:production-exportsexits 0 and still prints the repo-wide pre-existing report (67 unused exports, 18 suppressed, 0 stale); nothing in it is new to this branch.Not run:
pnpm check:affected --runandpnpm gate replay-ios-device- both need a cabled device, which this phase had none of.docs/evidence/ios-physical-runner-handoff-2026-09-19.mdcarries the exact commands and the assertions each device step must produce, including the ones only a device can answer: a post-handoff write landing in the runner log with the same PID and elapsed time, a replug/reboot/DDI remount between detach and adopt ending in a rebuild rather than a hang, and a lingering detached runner not blockingdevicectlinstall/uninstall or a concurrentxcodebuild.Unproven risk: that a physical
xcodebuildsurvives the handoff and answers the next daemon. Tests prove the gates decline and that the daemon stops holding a pipe it could close under the child; only the device lane proves the handoff.