What broke
The iOS Smoke Tests job fails during package verification, before any app runs, because one Apple toolchain probe blew its budget and the shared deadline had nothing left for the retry.
AppError: xcrun timed out after 30000ms
code: 'COMMAND_FAILED',
details: { cmd: 'xcrun', args: [ '--sdk', 'iphonesimulator', '--show-sdk-version' ] }
Package verification failed. Retained /private/var/folders/.../agent-device-package-AMdfST for inspection.
[ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL] @agent-device@… verify-installed-snapshot-bridge
Seen on 95d07b67b: run 35507203662, job 106068926059, 2026-09-20T11:16:45Z — the run died 4m7s in, with no simulator interaction yet. Same shape as the earlier toolchain-timeout family (#2422), on the snapshot-bridge prober rather than the runner's.
Why the retry did not save it
packages/platform-apple/src/snapshot-source/cache-identity.ts already knows about the cold-host problem: readSnapshotSourceToolchain runs xcodebuild -version, sw_vers twice, uname -m, then xcrun --show-sdk-version, each with Math.min(COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, remainingSnapshotSourceMs(...)), and runToolchainProbe retries exactly once on the exec-layer timeout.
The retry is skipped exactly when it is needed. runToolchainProbe re-throws when deadline.clock.remainingMs(...) <= 0 (line 118), so a single stall that consumes the remaining group budget produces this error with no second attempt. #2422 measured the syspolicyd signature scan at roughly 18–19s for the first xcodebuild/xcrun exec of each tool — two cold tools, one budget, one retry that only fires if there is already nothing left to spend.
Three ways this could be right, and they are not equivalent
- Size the group deadline for two cold scans instead of one, so
xcodebuild and xcrun can each pay Apple's first-exec stall.
- Warm both tools before the sequence — one throwaway
xcrun -sdk iphonesimulator -find .../xcodebuild -version at process start, so the identity probes all run against a warm toolchain and keep their tight budgets.
- Stop treating the identity probe as fatal for the job. This data keys a build cache. A host where it times out can build uncached; failing the whole smoke job to protect a cache key inverts that priority, and the retry-once path shows the authors already considered it transient.
Please pick at the owning type rather than adding a fourth budget; COLD_TOOLCHAIN_PROBE_TIMEOUT_MS is already shared between the two probers and should stay that way.
Why it matters now
It is currently indistinguishable from a real device failure at a glance, and it is holding the iOS side of #2704 red on a change that only touches examples/test-app app startup. Anything that reads as "smoke failed" without saying "the toolchain probe stalled on a cold host" costs a full CI round to disprove.
What broke
The iOS Smoke Tests job fails during package verification, before any app runs, because one Apple toolchain probe blew its budget and the shared deadline had nothing left for the retry.
Seen on
95d07b67b: run35507203662, job106068926059,2026-09-20T11:16:45Z— the run died 4m7s in, with no simulator interaction yet. Same shape as the earlier toolchain-timeout family (#2422), on the snapshot-bridge prober rather than the runner's.Why the retry did not save it
packages/platform-apple/src/snapshot-source/cache-identity.tsalready knows about the cold-host problem:readSnapshotSourceToolchainrunsxcodebuild -version,sw_verstwice,uname -m, thenxcrun --show-sdk-version, each withMath.min(COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, remainingSnapshotSourceMs(...)), andrunToolchainProberetries exactly once on the exec-layer timeout.The retry is skipped exactly when it is needed.
runToolchainProbere-throws whendeadline.clock.remainingMs(...) <= 0(line 118), so a single stall that consumes the remaining group budget produces this error with no second attempt. #2422 measured the syspolicyd signature scan at roughly 18–19s for the firstxcodebuild/xcrunexec of each tool — two cold tools, one budget, one retry that only fires if there is already nothing left to spend.Three ways this could be right, and they are not equivalent
xcodebuildandxcruncan each pay Apple's first-exec stall.xcrun -sdk iphonesimulator -find .../xcodebuild -versionat process start, so the identity probes all run against a warm toolchain and keep their tight budgets.Please pick at the owning type rather than adding a fourth budget;
COLD_TOOLCHAIN_PROBE_TIMEOUT_MSis already shared between the two probers and should stay that way.Why it matters now
It is currently indistinguishable from a real device failure at a glance, and it is holding the iOS side of #2704 red on a change that only touches
examples/test-appapp startup. Anything that reads as "smoke failed" without saying "the toolchain probe stalled on a cold host" costs a full CI round to disprove.