diff --git a/.device-evidence/CHECKLIST-runner-failures.md b/.device-evidence/CHECKLIST-runner-failures.md new file mode 100644 index 0000000000..e9e6a1d5e4 --- /dev/null +++ b/.device-evidence/CHECKLIST-runner-failures.md @@ -0,0 +1,148 @@ +# Runner-failure evidence checklist (#2680, #2683) + +Live evidence the coordinator runs serially on the connected iPhone. Each item names the exact +command, the environment it needs, and the rendered error that proves the change. Do not paraphrase +the JSON: paste it. Record the commit SHA the build under test was made from. + +## #2680 — typed build-failure reasons + +Build the CLI first, then stop any warm daemon so the run is on this commit: + +```sh +pnpm build && pnpm clean:daemon +node --experimental-strip-types src/bin.ts daemon stop --all || true +``` + +### 1. Signing with no team configured -> `signing_no_development_team` + +```sh +env -u AGENT_DEVICE_IOS_TEAM_ID -u AGENT_DEVICE_IOS_PROVISIONING_PROFILE \ + node --experimental-strip-types src/bin.ts --json \ + prepare ios-runner --platform ios --device "" +``` + +Expected: exit non-zero, one error object with + +```json +{ + "code": "COMMAND_FAILED", + "message": "xcodebuild build-for-testing failed", + "hint": "Configure signing in Xcode or set AGENT_DEVICE_IOS_TEAM_ID for physical-device runs.", + "details": { "reason": "signing_no_development_team" } +} +``` + +`hint`, `logPath` and `diagnosticId` are top-level, never inside `details`. Also record the Xcode +version (`xcodebuild -version`) so the `signing_no_development_team` fixture in +`packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts` can move from +`shipped-sniff-trigger` to `captured` and its `xcodeVersion` from `unobserved` to that version. +Paste the whole error so the fixture's `output` can become the capture and its `command` can be +recorded. + +### 2. A bundle identifier somebody else already owns -> `bundle_identifier_already_registered` + +```sh +env AGENT_DEVICE_IOS_TEAM_ID="" \ + AGENT_DEVICE_IOS_BUNDLE_ID="com.apple.TestFlight" \ + node --experimental-strip-types src/bin.ts --json \ + prepare ios-runner --platform ios --device "" +``` + +Expected: same envelope shape with +`details.reason: "bundle_identifier_already_registered"` and a hint naming +`AGENT_DEVICE_IOS_BUNDLE_ID`. A registered-but-foreign identifier may surface the `Failed registering +bundle identifier` line or the `App Identifier ... is not available` line; both rules produce this +one reason, so record which line xcodebuild printed. + +### 3. Reasons with no device exposure (host-side or configuration-only) + +These do not need the iPhone, but do need a real xcodebuild run; record output with the Xcode +version so the matching fixture's `provenance` can be upgraded: + +```sh +# devtools_security_developer_mode_disabled (macOS admin state, no device work) +DevToolsSecurity -status + +# signing_provisioning_profile_missing: point the runner build at a profile that is not installed, +# then read the reason off the same prepare command as above. +env AGENT_DEVICE_IOS_TEAM_ID="" \ + AGENT_DEVICE_IOS_PROVISIONING_PROFILE="no-such-profile-installed" \ + node --experimental-strip-types src/bin.ts --json \ + prepare ios-runner --platform ios --device "" +``` + +Expected: `details.reason` is `signing_provisioning_profile_missing`, and the profile rows only fire +when xcodebuild says what is wrong with the profile **on the same line as the profile**: its +`IDEProvisioningErrorDomain` diagnostic naming the profile, `doesn't include ...`, or `has expired` +(#2688 review). Paste the whole error and keep the line breaks — which line carried which phrase is +what promotes the `profile-xcode-signing-error`, `profile-does-not-cover-app-id` and `profile-expired` +fixtures from `invented-shape` to `captured`, and a capture that splits the two phrases across lines +belongs to `profile-note-above-an-expired-certificate` instead. A different reason is worth recording +just as much: say which one and treat the fixtures as unconfirmed rather than editing the rules to +fit. + +### 4. The line that claims no reason yet -> `build_failed_unclassified` + +`xcodebuild` reports a settings mismatch with a line that names a profile ("has conflicting +provisioning settings"). #2680 deliberately publishes `build_failed_unclassified` for it, because no +capture has proved which lever clears it. To reach it, pin a profile while leaving automatic signing +on: + +```sh +env AGENT_DEVICE_IOS_TEAM_ID="" \ + AGENT_DEVICE_IOS_PROVISIONING_PROFILE="match-development" \ + node --experimental-strip-types src/bin.ts --json \ + prepare ios-runner --platform ios --device "" +``` + +Expected: either `signing_provisioning_profile_missing` (xcodebuild complained about the profile and +said what was wrong with it) or `build_failed_unclassified` — which is also what a run that merely +mentions the profile it used gets, since a name is not a complaint (#2688 review). Paste the error and the `xcodebuild -version` either way: a +capture of the conflicting-settings line is what would let a follow-up name the cause, and the +capture must show which build setting disagrees before any hint naming a lever is written. + +## Results — coordinator run, 2026-09-20 + +Built from `15808ae228` on `thymikee-iphone` (iPhone 17 Pro, iOS 27.0, build 24A437), cabled. + +``` +$ xcodebuild -version +Xcode 26.2 +Build version 17C52 +``` + +Section 3 is captured. A device build pointed at a team with no certificate, on a fresh derived +path so no cached artifact short-circuits it, reaches signing and fails with one long `error:` line +per target: + +``` +.../AgentDeviceRunner.xcodeproj: error: No Accounts: Add a new account in Accounts settings. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner') +.../AgentDeviceRunner.xcodeproj: error: No profiles for 'com.callstack.agentdevice.runner' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'com.callstack.agentdevice.runner'. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner') +``` + +`details.reason` is `signing_provisioning_profile_missing` with the profile hint, and the fixture +`no-profiles-for-bundle-id` is now `captured` with this transcript verbatim. This is the run that +answers the wrapping question the rows were held on: the matched phrase arrives inside one `error:` +line, so the sibling rows in the same provisioning family do not split the way a wrapped line would. + +Sections 1 and 2 are blocked on this account, and the mechanism is worth recording because it is the +same for all of them: against a signed-in account with a valid identity, `xcodebuild` is invoked with +`-allowProvisioningUpdates`, so the build either signs successfully or dies earlier than the +diagnostic a row keys on. + +- Unsetting `AGENT_DEVICE_IOS_TEAM_ID` **succeeds** — automatic signing resolves the team from the + installed identity and reuses an installed team profile. So section 1's `signing_no_development_team` + cannot be induced here; it needs an account signed in with no development team. +- `AGENT_DEVICE_IOS_BUNDLE_ID=com.apple.TestFlight` **succeeds** for the same reason, and a bogus + `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` is repaired rather than honoured. So section 2's + `bundle_identifier_already_registered` needs an app id owned by a different team that automatic + signing cannot register. +- The same gating applies to `bundle_identifier_unavailable` (`App Identifier` + `not available`), + `profile-does-not-cover-app-id` (`Provisioning profile` + `doesn't include`) and `profile-expired` + (`Provisioning profile` + `has expired`): each needs a profile or app id already claimed elsewhere, + which this account will not produce. Recorded beside the fixtures in + `runner-startup-failure-fixtures.ts` so the rows read as host-gated, not unexamined. + +Section 4 needs a build that fails for an unrelated reason while naming no signing fact; the +classifier's behaviour there is pinned by `runner-startup-failure-reasons.test.ts` and needs no +device claim to hold. diff --git a/packages/platform-apple/src/runner/__tests__/runner-client.test.ts b/packages/platform-apple/src/runner/__tests__/runner-client.test.ts index d972d872b2..8d039437b0 100644 --- a/packages/platform-apple/src/runner/__tests__/runner-client.test.ts +++ b/packages/platform-apple/src/runner/__tests__/runner-client.test.ts @@ -34,7 +34,6 @@ import type { DeviceInfo } from '@agent-device/kernel/device'; import { isReadOnlyRunnerCommand } from '../runner-command-traits.ts'; import { isRetryableRunnerError, - resolveRunnerBuildFailureHint, resolveRunnerEarlyExitHint, shouldRetryRunnerConnectError, withRunnerCommandId, @@ -437,15 +436,6 @@ test('resolveRunnerEarlyExitHint falls back to runner connect timeout hint', () assert.match(hint, /pnpm clean:xcuitest/i); }); -test('resolveRunnerBuildFailureHint suggests cache cleanup for non-signing failures', () => { - const hint = resolveRunnerBuildFailureHint( - new AppError('COMMAND_FAILED', 'xcodebuild build-for-testing failed'), - ); - - assert.match(hint, /pnpm clean:xcuitest/i); - assert.match(hint, /~\/\.agent-device\/apple-runner\/derived/i); -}); - test('shouldRetryRunnerConnectError does not retry xcodebuild early-exit errors', () => { const err = new AppError( 'COMMAND_FAILED', diff --git a/packages/platform-apple/src/runner/__tests__/runner-dev-tools-security.test.ts b/packages/platform-apple/src/runner/__tests__/runner-dev-tools-security.test.ts new file mode 100644 index 0000000000..5a5f98932e --- /dev/null +++ b/packages/platform-apple/src/runner/__tests__/runner-dev-tools-security.test.ts @@ -0,0 +1,77 @@ +import assert from 'node:assert/strict'; +import { beforeEach, test, vi } from 'vitest'; +import { AppError, normalizeError } from '@agent-device/kernel/errors'; +import { appleRunnerTestHost } from '../test-host.ts'; +import { assertDevToolsSecurityForIosRunner } from '../runner-dev-tools-security.ts'; +import { IOS_DEVICE, IOS_SIMULATOR } from './device-fixtures.ts'; +import { RUNNER_STARTUP_FAILURE_FIXTURES } from './runner-startup-failure-fixtures.ts'; + +/** + * `DevToolsSecurity -status` answers for the Mac, not for the iPhone (#2680). The refusal this probe + * threw used to carry a hint and no reason, so a caller could only match its wording — and that + * wording is nearly the same as the device's own Developer Mode state, which is a different fact on + * a different machine. These cases pin that the host refusal publishes the host's own reason, keyed + * on the status it read rather than on the sentence it printed. + */ + +const HOST_REFUSAL_FIXTURE = RUNNER_STARTUP_FAILURE_FIXTURES.find( + (fixture) => fixture.reason === 'devtools_security_developer_mode_disabled', +); + +const runAppleToolCommand = vi.fn(); + +beforeEach(() => { + runAppleToolCommand.mockReset().mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' }); + appleRunnerTestHost.update({ runAppleToolCommand }); +}); + +test('the host DevToolsSecurity refusal publishes its own typed reason', async () => { + assert.ok(HOST_REFUSAL_FIXTURE); + mockDevToolsSecurityOutput(HOST_REFUSAL_FIXTURE.output); + const expectedStatus = HOST_REFUSAL_FIXTURE.output.trim(); + + await assert.rejects( + () => assertDevToolsSecurityForIosRunner(IOS_DEVICE), + (error: unknown) => { + assert.ok(error instanceof AppError); + assert.equal(error.code, 'COMMAND_FAILED'); + assert.match(error.message, /Developer mode is disabled/); + assert.equal(error.details?.reason, 'devtools_security_developer_mode_disabled'); + assert.equal(error.details?.devToolsSecurityStatus, expectedStatus); + + // What the caller renders: hint at top level, reason in details. + const envelope = normalizeError(error, { diagnosticId: 'diag-devtools-1' }); + assert.match(String(envelope.hint), /DevToolsSecurity -enable/); + assert.equal(envelope.diagnosticId, 'diag-devtools-1'); + assert.equal(envelope.details?.hint, undefined); + assert.equal(envelope.details?.diagnosticId, undefined); + assert.equal(envelope.details?.reason, 'devtools_security_developer_mode_disabled'); + return true; + }, + ); +}); + +test('an enabled host developer mode is not a failure', async () => { + mockDevToolsSecurityOutput('Developer mode is currently enabled for development tools.\n'); + + await assert.doesNotReject(() => assertDevToolsSecurityForIosRunner(IOS_DEVICE)); +}); + +test('a simulator never takes the host probe', async () => { + mockDevToolsSecurityOutput('Developer mode is currently disabled.\n'); + + await assert.doesNotReject(() => assertDevToolsSecurityForIosRunner(IOS_SIMULATOR)); + + assert.equal( + runAppleToolCommand.mock.calls.some((call) => call[0] === 'DevToolsSecurity'), + false, + ); +}); + +function mockDevToolsSecurityOutput(stdout: string): void { + runAppleToolCommand.mockImplementation(async (cmd: string) => ({ + exitCode: 0, + stdout: cmd === 'DevToolsSecurity' ? stdout : '', + stderr: '', + })); +} diff --git a/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts new file mode 100644 index 0000000000..b9685bed7c --- /dev/null +++ b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts @@ -0,0 +1,274 @@ +import { AppError } from '@agent-device/kernel/errors'; +import type { RunnerStartupFailureReason } from '../runner-contract.ts'; + +/** + * Recorded startup failures for {@link classifyRunnerStartupFailure} (#2680). + * + * Provenance is the point of this file, so it is stated per entry and never as a blanket claim: + * + * - `captured` — `output` was pasted from a run, and `command` plus `xcodeVersion` (from + * `xcodebuild -version`) were recorded with it by `.device-evidence/CHECKLIST-runner-failures.md`. + * - `shipped-sniff-trigger` — the substrings a rule matches are the ones shipped in + * `resolveSigningFailureHint` before #2680, which is evidence xcodebuild can emit them. The + * sentence around them is ours, so `command` and `xcodeVersion` stay unrecorded. + * - `invented-shape` — no shipped trigger and no capture. The entry exists to exercise a rule and + * makes no claim about wording xcodebuild prints. + * + * Until Phase B captures the real runs, every entry is `unobserved` for `xcodeVersion` and carries + * no `command`: an invocation we did not run is not provenance. Nothing in the classifier reads + * these fields; they exist so a reason can be traced to an observation instead of to a guess. + * + * Blocked on the host, not unexamined. One Phase B run on `thymikee-iphone` / Xcode 26.2 settled the + * shape question these rows were held on — `No profiles for '' were found` arrives as one long + * `error:` line, not a wrapped one — and `no-profiles-for-bundle-id` below is now `captured`. The + * rest are gated on an Apple account this machine does not have, and they need it for the same + * reason: the build either signs successfully or dies before reaching the diagnostic a row keys on. + * `requires-development-team` and `requires-development-team-message-only` need an account that is + * signed in with no development team; automatic signing resolves the team from any installed + * identity, so unsetting `AGENT_DEVICE_IOS_TEAM_ID` builds successfully. + * `bundle-id-registration-failed`, `app-id-not-available` and the `bundle_identifier_unavailable` + * rule it feeds, plus `profile-does-not-cover-app-id` (`Provisioning profile` + `doesn't include`) + * and `profile-expired` (`Provisioning profile` + `has expired`), all need a profile and an app id + * already claimed by someone else: against a working account `-allowProvisioningUpdates` registers + * or repairs the id, so the conflict text is never printed and the build reaches signing success. + * Each of those rows is therefore uninducible here rather than untested, and none of them should be + * read as waiting on effort this machine can supply. + */ + +export type RunnerStartupFailureSite = 'build-for-testing' | 'host-dev-tools-security'; + +/** + * Whether the text reaches the build catch inside the exec error's `details` (`exec-details`, which + * is how a non-zero `xcodebuild` arrives) or only in the thrown message (`message-only`, which is + * how anything the exec layer raised as a plain `Error` arrives after the catch wraps `String(err)`). + */ +export type RunnerStartupFailureCarrier = 'exec-details' | 'message-only'; + +const UNOBSERVED = 'unobserved'; + +export type RunnerStartupFailureFixture = Readonly<{ + /** Stable name for a focused test or a review comment. */ + id: string; + /** The reason this output must reach the caller with. */ + reason: RunnerStartupFailureReason; + /** Which throw site receives this output. */ + site: RunnerStartupFailureSite; + carrier?: RunnerStartupFailureCarrier; + /** The invocation that produced {@link RunnerStartupFailureFixture.output}, once one is recorded. */ + command?: string; + /** `xcodebuild -version` recorded from that run, or `unobserved`. */ + xcodeVersion: string; + provenance: 'captured' | 'shipped-sniff-trigger' | 'invented-shape'; + /** The tool's own stdout/stderr. */ + output: string; + /** The argv the exec reported, which is never evidence of a cause (#2680). */ + args?: readonly string[]; + /** What the pending capture still has to show, and how to reach it. */ + note?: string; +}>; + +export const RUNNER_STARTUP_FAILURE_FIXTURES: readonly RunnerStartupFailureFixture[] = [ + { + id: 'bundle-id-registration-failed', + reason: 'bundle_identifier_already_registered', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: + "error: Failed registering bundle identifier \"com.yourname.agentdevice.runner\" with the developer portal (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Capture with AGENT_DEVICE_IOS_BUNDLE_ID set to an identifier already registered by another team, and record the `xcodebuild -version` of the machine.', + }, + { + id: 'app-id-not-available', + reason: 'bundle_identifier_already_registered', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: + "error: App Identifier 'com.yourname.agentdevice.runner' is not available (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'The second shape of the same cause: no "failed registering" line, so only the two-part "app identifier" + "not available" trigger can name it. Trimmed to the shipped trigger; the real sentence is still unrecorded.', + }, + { + id: 'requires-development-team', + reason: 'signing_no_development_team', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: + "error: Signing for \"AgentDeviceRunner\" requires a development team (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Capture with AGENT_DEVICE_IOS_TEAM_ID unset on a signed-in-but-team-less account.', + }, + { + id: 'requires-development-team-message-only', + reason: 'signing_no_development_team', + site: 'build-for-testing', + carrier: 'message-only', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: + "error: Signing for \"AgentDeviceRunner\" requires a development team (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')", + note: 'Same text arriving in the thrown message instead of the exec details: the catch wraps a non-AppError with String(err), and the rule still has to see it.', + }, + { + id: 'app-identifier-and-availability-in-different-lines', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: App Identifier 'com.yourname.agentdevice.runner' is invalid (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\nnote: The simulator device is not available for this destination\n** TEST BUILD FAILED **\n", + note: 'The same cross-line hazard the profile rows gave up (#2688 review): one line faults the identifier and another says something is not available, and neither line pairs them. The reason needs both in one sentence, which is what `app-id-not-available` records.', + }, + { + id: 'no-profiles-for-bundle-id', + reason: 'signing_provisioning_profile_missing', + site: 'build-for-testing', + command: + 'agent-device prepare ios-runner --platform ios --device --json # AGENT_DEVICE_IOS_TEAM_ID=ZZZZZZZZZZ, fresh AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH', + xcodeVersion: 'Xcode 26.2 / Build version 17C52', + provenance: 'captured', + output: + "/Users/thymikee/.t3/worktrees/agent-device/apex-2680/apple/runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj: error: No Accounts: Add a new account in Accounts settings. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n/Users/thymikee/.t3/worktrees/agent-device/apex-2680/apple/runner/AgentDeviceRunner/AgentDeviceRunner.xcodeproj: error: No profiles for 'com.callstack.agentdevice.runner' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'com.callstack.agentdevice.runner'. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n", + note: 'Captured on `thymikee-iphone`, iPhone 17 Pro, iOS 27.0. Reached by pointing `AGENT_DEVICE_IOS_TEAM_ID` at a team with no certificate on a machine that is not signed into Xcode, with a fresh derived path so no cached artifact short-circuits the build. One `error:` line per target: the phrase the rule matches is not wrapped, which is the evidence the sibling rows were held for. Note the `No Accounts` line above it names nothing the rule reads — the profile row wins on its own line.', + }, + { + id: 'conflicting-provisioning-settings', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: \"AgentDeviceRunner\" has conflicting provisioning settings (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Names a profile while saying the settings disagree, so the profile row must not win. No reason is claimed until a capture proves which lever clears it.', + }, + { + id: 'code-signing-required', + reason: 'signing_unspecified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: + "error: Code signing is required for product type 'Application' (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Signing is named and nothing above it is: the reason stays unspecified on purpose.', + }, + { + id: 'compile-error', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Any build failure that names no signing fact must keep the cache-recovery hint.', + }, + { + id: 'argv-names-a-provisioning-profile', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + args: [ + 'build-for-testing', + 'PROVISIONING_PROFILE_SPECIFIER=match-development', + 'Provisioning Profile: match-development', + ], + note: 'The argv we were asked to run is not xcodebuild evidence: a caller who pinned a profile still gets cache-recovery advice for a compile error (#2680).', + }, + // Narrowed profile rows (#2688 review): each of these requires the profile AND the complaint Xcode + // attaches to it. The bare phrase alone was the shipped sniffer's trigger and is not evidence, so the + // negative entry below is what keeps those rows honest. + { + id: 'profile-xcode-signing-error', + reason: 'signing_provisioning_profile_missing', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: Provisioning profile \"match-development\" is not a valid provisioning profile (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\nError Domain=IDEProvisioningErrorDomain Code=17 \"Provisioning profile 'match-development' is not a valid provisioning profile.\"\n** TEST BUILD FAILED **\n", + note: "Xcode repeats the profile inside the same line as its IDEProvisioningErrorDomain diagnostics, which is what the row reads: domain on one line and profile on another is two facts, not one complaint. Sentence and domain code are our reconstruction; Phase B capture has to record the real wording and this entry's xcodeVersion.", + }, + { + id: 'profile-does-not-cover-app-id', + reason: 'signing_provisioning_profile_missing', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: Provisioning profile \"match-development\" doesn't include application identifier 'com.yourname.agentdevice.runner' (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'The installed profile that does not cover this app id. Advice is the same lever, so the same reason is published; wording unrecorded.', + }, + { + id: 'profile-expired', + reason: 'signing_provisioning_profile_missing', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "error: Provisioning profile \"match-development\" has expired (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'Reinstalling the same profile clears nothing; "a valid profile" in the hint is the operative word. Wording unrecorded.', + }, + { + id: 'profile-mentioned-while-compiling', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "note: Using provisioning profile \"match-development\" to sign the app bundle (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\nerror: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'The hazard the bare `provisioning profile` trigger carried (#2688 review): a failing build can print the profile it used while the failure is a compile error. A benign mention must keep cache-recovery advice; it also says nothing Xcode calls code signing, which is its own honest row.', + }, + { + id: 'profile-note-above-an-expired-certificate', + reason: 'build_failed_unclassified', + site: 'build-for-testing', + xcodeVersion: UNOBSERVED, + provenance: 'invented-shape', + output: + "note: Using provisioning profile \"match-development\" to sign the app bundle (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\nwarning: The certificate \"Apple Development: Example Dev (ABCD1234)\" has expired.\nerror: cannot find 'AgentDeviceRunnerCommand' in scope (in target 'AgentDeviceRunnerUITests' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + note: 'The cross-line hazard a whole-log AND cannot see (#2688 review): a benign profile note three lines above an unrelated expired-certificate warning. Both phrases are in the captured log and neither qualifies the other, so the profile stays unclassified and the reader keeps cache-recovery advice rather than being sent to replace a profile that is fine.', + }, + { + id: 'devtools-security-disabled', + reason: 'devtools_security_developer_mode_disabled', + site: 'host-dev-tools-security', + command: 'DevToolsSecurity -status', + xcodeVersion: UNOBSERVED, + provenance: 'shipped-sniff-trigger', + output: 'Developer mode is currently disabled for development tools.\n', + note: "Host-side refusal. It says nothing about the device's Developer Mode toggle (#2683 reads that).", + }, +]; + +export function buildForTestingFixtures(): RunnerStartupFailureFixture[] { + return RUNNER_STARTUP_FAILURE_FIXTURES.filter((fixture) => fixture.site === 'build-for-testing'); +} + +export function buildFixtureById(id: string): RunnerStartupFailureFixture { + const fixture = RUNNER_STARTUP_FAILURE_FIXTURES.find((candidate) => candidate.id === id); + if (!fixture) throw new Error(`no startup failure fixture records ${id}`); + return fixture; +} + +/** + * What the exec layer hands the build-failure catch: for `exec-details` a COMMAND_FAILED carrying + * the tool's output and the argv in `details` (`execFailureDetails` shape), and for `message-only` + * the plain `Error` the catch turns into `new AppError('COMMAND_FAILED', String(error))`. + */ +export function buildForTestingExecFailure( + fixture: RunnerStartupFailureFixture, + exitCode = 65, +): unknown { + if ((fixture.carrier ?? 'exec-details') === 'message-only') { + return new Error(`xcodebuild exited with code ${exitCode}: ${fixture.output}`); + } + return new AppError('COMMAND_FAILED', `xcodebuild exited with code ${exitCode}`, { + stdout: fixture.output, + stderr: '', + exitCode, + processExitError: true, + cmd: 'xcodebuild', + args: fixture.args ?? ['build-for-testing'], + }); +} diff --git a/packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts new file mode 100644 index 0000000000..08fb735483 --- /dev/null +++ b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts @@ -0,0 +1,295 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { afterEach, beforeEach, test, vi } from 'vitest'; +import { AppError, normalizeError, type NormalizedError } from '@agent-device/kernel/errors'; +import { resetAllProcessMemosForTests } from '@agent-device/kernel/ttl-memo'; +import { appleRunnerTestHost } from '../test-host.ts'; +import type { ExecResult } from '@agent-device/host-kit/command'; +import { createRunnerPhaseBudget, ensureXctestrunArtifact } from '../runner-xctestrun.ts'; +import { + RUNNER_ERROR_RULES, + classifyRunnerStartupFailure, + RUNNER_STARTUP_FAILURE_REASONS, + RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON, + type RunnerStartupFailureReason, +} from '../runner-contract.ts'; +import { assertDevToolsSecurityForIosRunner } from '../runner-dev-tools-security.ts'; +import { appleToolchainProbeResult } from './apple-toolchain-fixtures.ts'; +import { IOS_DEVICE } from './device-fixtures.ts'; +import { + RUNNER_STARTUP_FAILURE_FIXTURES, + buildFixtureById, + buildForTestingExecFailure, + buildForTestingFixtures, + type RunnerStartupFailureFixture, +} from './runner-startup-failure-fixtures.ts'; +import { mkdtempForTestSync } from './tmp-dir.ts'; + +/** + * A `build-for-testing` failure used to reach the caller as prose only, so every consumer that + * wanted to know *which* signing problem it was had to re-match the same substrings (#2680). These + * cases drive each recorded output through the real build-failure catch and assert on the + * normalized envelope: the code is `COMMAND_FAILED` for all of them, so `details.reason` is the + * assertion, and the hint beside it has to be the hint the rule that named the reason carries. + * + * The envelope assertions are deliberate: `normalizeError` moves `hint`, `logPath` and + * `diagnosticId` out of `details` to the top level, and it is called here with no `logPath` + * fallback — a top-level `logPath` therefore proves the build catch put it there. The negative + * cases matter just as much: argv, our own emitted reason, and wording without a typed fact behind + * it must all stay unclassified. + */ + +const CACHE_RECOVERY_HINT = /clean:xcuitest|apple-runner\/derived/; + +const HINT_FOR_REASON: Record = { + bundle_identifier_already_registered: /AGENT_DEVICE_IOS_BUNDLE_ID/, + signing_no_development_team: /AGENT_DEVICE_IOS_TEAM_ID/, + signing_provisioning_profile_missing: /AGENT_DEVICE_IOS_PROVISIONING_PROFILE/, + signing_unspecified: /Automatic Signing/, + devtools_security_developer_mode_disabled: /DevToolsSecurity -enable/, + build_failed_unclassified: CACHE_RECOVERY_HINT, +}; + +const runCmdSync = vi.fn(); +const runCmdStreaming = vi.fn(); +const runAppleToolCommand = vi.fn(); +const DIAGNOSTIC_ID = 'diag-build-failure-1'; +let projectRoot: string; +let derivedPath: string; +let logPath: string; + +beforeEach(() => { + resetAllProcessMemosForTests(); + projectRoot = mkdtempForTestSync('agent-device-startup-failure-root-'); + // `buildXctestrunArtifact` refuses to start a build without the runner project. + fs.mkdirSync( + path.join(projectRoot, 'apple', 'runner', 'AgentDeviceRunner', 'AgentDeviceRunner.xcodeproj'), + { recursive: true }, + ); + derivedPath = mkdtempForTestSync('agent-device-startup-failure-derived-'); + logPath = path.join(derivedPath, 'runner.log'); + process.env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH = derivedPath; + runCmdSync.mockReset().mockImplementation(appleToolchainProbeResult); + runCmdStreaming.mockReset().mockImplementation(async (): Promise => ({ + exitCode: 0, + stdout: '', + stderr: '', + })); + runAppleToolCommand.mockReset().mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' }); + appleRunnerTestHost.update({ + runCmdSync, + runCmdStreaming, + runAppleToolCommand, + findProjectRoot: () => projectRoot, + readVersion: () => '0.0.0-test', + }); +}); + +afterEach(() => { + delete process.env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH; +}); + +for (const fixture of buildForTestingFixtures()) { + test(`a build-for-testing failure publishes ${fixture.reason} for ${fixture.id}`, async () => { + assertFailureEnvelope(await driveBuildFailure(fixture), fixture); + }); +} + +/** + * Every startup failure reaches a caller through one envelope: the typed reason in `details`, its hint + * and the log path hoisted to top level by `normalizeError`, and the tool output still reachable + * underneath for a human. The envelope is asserted per fixture rather than once because the reason and + * the hint have to travel together for every recorded shape, not just for one of them. + */ +function assertFailureEnvelope( + envelope: NormalizedError, + fixture: RunnerStartupFailureFixture, +): void { + assert.equal(envelope.code, 'COMMAND_FAILED'); + assert.equal(envelope.message, 'xcodebuild build-for-testing failed'); + assert.equal(envelope.details?.reason, fixture.reason); + assert.match(String(envelope.hint), HINT_FOR_REASON[fixture.reason]); + // No `logPath` was handed to `normalizeError`: the top-level value can only be the one the + // build catch wrote into the error it throws. + assert.equal(envelope.logPath, logPath); + assert.equal(envelope.diagnosticId, DIAGNOSTIC_ID); + // normalizeError hoists these out of `details`; a caller must read them at top level. + assert.equal(envelope.details?.hint, undefined); + assert.equal(envelope.details?.logPath, undefined); + assert.equal(envelope.details?.diagnosticId, undefined); + assertToolOutputReachable(envelope, fixture); +} + +/** + * The tool output stays reachable for a human reading the failure, redacted and length-bounded on the + * way out — one more reason the reason is typed: classification happens before the truncation a caller + * sees. A message-only failure carries no tool output to reach, which is exactly why the message is + * part of the haystack. + */ +function assertToolOutputReachable( + envelope: NormalizedError, + fixture: RunnerStartupFailureFixture, +): void { + if ((fixture.carrier ?? 'exec-details') !== 'exec-details') { + assert.equal(envelope.details?.details, undefined); + return; + } + const nestedDetails = envelope.details?.details as Record | undefined; + assert.match(String(nestedDetails?.stdout), /AgentDeviceRunner/); +} + +test('every startup failure reason has a recorded fixture', () => { + const reasonsWithFixtures = new Set(RUNNER_STARTUP_FAILURE_FIXTURES.map((f) => f.reason)); + + assert.equal(reasonsWithFixtures.size, RUNNER_STARTUP_FAILURE_REASONS.length); + for (const reason of RUNNER_STARTUP_FAILURE_REASONS) { + assert.ok(reasonsWithFixtures.has(reason), `no fixture records the ${reason} reason`); + } +}); + +test('every reason the classifier can name is produced by a rule row', () => { + const reasonsFromRules = new Set( + RUNNER_ERROR_RULES.flatMap((rule) => (rule.buildFailure ? [rule.buildFailure.reason] : [])), + ); + + for (const reason of RUNNER_STARTUP_FAILURE_REASONS) { + // The catch-all is the classifier's own answer when no row matched, so it names no row. + if (reason === RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON) continue; + assert.ok(reasonsFromRules.has(reason), `no rule row yields the ${reason} reason`); + } +}); + +test('an argv that names a provisioning profile is not evidence of a signing failure', async () => { + // The exec reports the invocation we asked for in `details.args`. Reading the whole details bag + // would let a caller's own pinned profile name the cause of an unrelated compile error and take + // the cache-recovery hint with it (#2680). + const argvFixture = buildFixtureById('argv-names-a-provisioning-profile'); + + const envelope = await driveBuildFailure(argvFixture); + + assert.equal(envelope.details?.reason, RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON); + assert.match(String(envelope.hint), CACHE_RECOVERY_HINT); + assert.doesNotMatch(String(envelope.hint), /AGENT_DEVICE_IOS_PROVISIONING_PROFILE/); +}); + +test('a signing sentence that arrives only in the thrown message is still classified', async () => { + // The catch wraps a non-AppError as `new AppError('COMMAND_FAILED', String(error))`, so the tool's + // sentence can reach the classifier in the message with no details behind it (#2680). + const messageOnly = buildFixtureById('requires-development-team-message-only'); + + const envelope = await driveBuildFailure(messageOnly); + + assert.equal(envelope.details?.reason, 'signing_no_development_team'); + assert.match(String(envelope.hint), /AGENT_DEVICE_IOS_TEAM_ID/); +}); + +test('the failure the build catch publishes does not classify itself', async () => { + // The wrapper carries `reason` and `hint` in its details. Re-running the classifier over it must + // not read our own verdict back out of the bag the rules scan (#2680). + const signingFixture = buildFixtureById('requires-development-team'); + const published = await runBuildCatch(() => buildForTestingExecFailure(signingFixture)); + + const reclassified = classifyRunnerStartupFailure(published); + + assert.equal(reclassified.reason, RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON); + assert.match(reclassified.hint, CACHE_RECOVERY_HINT); +}); + +test('an identical message without the typed host fact is not read as a DevToolsSecurity refusal', async () => { + // The exact sentence the host probe throws, minus the typed `devToolsSecurityStatus` fact only the + // probe publishes. Text alone must not activate the reason (#2680). + const hostRefusal = await expectHostRefusal(); + const withoutFact = new AppError('COMMAND_FAILED', hostRefusal.message, { + stdout: 'developer mode is disabled\n', + stderr: '', + exitCode: 65, + processExitError: true, + }); + + const envelope = await driveBuildRejection(withoutFact); + + assert.equal(envelope.details?.reason, RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON); + assert.match(String(envelope.hint), CACHE_RECOVERY_HINT); + assert.doesNotMatch(String(envelope.hint), /DevToolsSecurity/); +}); + +test('an app identifier named without the availability fact is not read as a taken bundle id', async () => { + const nearMiss: RunnerStartupFailureFixture = { + ...buildFixtureById('app-id-not-available'), + output: + "error: App Identifier 'com.yourname.agentdevice.runner' is invalid (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\n** TEST BUILD FAILED **\n", + }; + + const envelope = await driveBuildFailure(nearMiss); + + assert.equal(envelope.details?.reason, RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON); + assert.match(String(envelope.hint), CACHE_RECOVERY_HINT); + assert.doesNotMatch(String(envelope.hint), /AGENT_DEVICE_IOS_BUNDLE_ID/); +}); + +test('a conflicting-settings failure is not answered with missing-profile advice', async () => { + // The conflicting-settings line names a profile while explaining that the settings disagree. It + // precedes the profile row and claims no cause of its own (#2680). + const conflict = buildFixtureById('conflicting-provisioning-settings'); + + const envelope = await driveBuildFailure(conflict); + + assert.equal(envelope.details?.reason, RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON); + assert.match(String(envelope.hint), CACHE_RECOVERY_HINT); + assert.doesNotMatch(String(envelope.hint), /AGENT_DEVICE_IOS_PROVISIONING_PROFILE/); +}); + +/** Drives a recorded fixture through the real build catch and normalizes what it threw. */ +async function driveBuildFailure(fixture: RunnerStartupFailureFixture): Promise { + return normalizeThrown(await runBuildCatch(() => buildForTestingExecFailure(fixture))); +} + +/** Drives a hand-built rejection through the same real build catch. */ +async function driveBuildRejection(rejection: unknown): Promise { + return normalizeThrown(await runBuildCatch(() => rejection)); +} + +async function runBuildCatch(buildRejection: () => unknown): Promise { + runCmdStreaming.mockReset().mockImplementation(async () => { + throw buildRejection(); + }); + + let caught: unknown; + await assert.rejects( + () => + ensureXctestrunArtifact(IOS_DEVICE, { + logPath, + budget: createRunnerPhaseBudget(120_000, undefined), + }), + (error: unknown) => { + caught = error; + return true; + }, + ); + assert.ok(caught, 'the build-failure catch must throw'); + return caught; +} + +function normalizeThrown(caught: unknown): NormalizedError { + return normalizeError(caught, { diagnosticId: DIAGNOSTIC_ID }); +} + +async function expectHostRefusal(): Promise { + runAppleToolCommand.mockImplementation(async () => ({ + exitCode: 0, + stdout: 'Developer mode is currently disabled for development tools.\n', + stderr: '', + })); + + let caught: unknown; + await assert.rejects( + () => assertDevToolsSecurityForIosRunner(IOS_DEVICE), + (error: unknown) => { + caught = error; + return true; + }, + ); + assert.ok(caught instanceof AppError); + return caught; +} diff --git a/packages/platform-apple/src/runner/runner-artifact.ts b/packages/platform-apple/src/runner/runner-artifact.ts index dd74f0a12d..ffd0f38e0b 100644 --- a/packages/platform-apple/src/runner/runner-artifact.ts +++ b/packages/platform-apple/src/runner/runner-artifact.ts @@ -12,7 +12,7 @@ import { } from './host.ts'; import type { ExecBackgroundResult } from '@agent-device/host-kit/command'; import type { DeviceInfo } from '@agent-device/kernel/device'; -import { resolveRunnerBuildFailureHint } from './runner-contract.ts'; +import { classifyRunnerStartupFailure } from './runner-contract.ts'; import { logChunk } from './runner-io.ts'; import { withXcodebuildSimulatorSetRedirect } from './runner-device-set.ts'; import { @@ -517,8 +517,11 @@ async function buildRunnerXctestrun( if (isRequestCanceledError(error)) throw error; const appErr = error instanceof AppError ? error : new AppError('COMMAND_FAILED', String(error)); - const hint = resolveRunnerBuildFailureHint(appErr); + // The reason and the hint beside it come from one classifier (#2680), so the reason a caller + // switches on can never disagree with the advice it is handed. + const { reason, hint } = classifyRunnerStartupFailure(appErr); throw new AppError('COMMAND_FAILED', 'xcodebuild build-for-testing failed', { + reason, error: appErr.message, details: appErr.details, logPath: options.logPath, diff --git a/packages/platform-apple/src/runner/runner-contract.ts b/packages/platform-apple/src/runner/runner-contract.ts index cf79fcfa04..367a56da23 100644 --- a/packages/platform-apple/src/runner/runner-contract.ts +++ b/packages/platform-apple/src/runner/runner-contract.ts @@ -172,11 +172,32 @@ type RunnerErrorMatch = { code?: AppErrorCode; /** Every entry must appear in the lowercased message. */ messageIncludesAll?: readonly string[]; + /** + * Every entry must appear in the lowercased {@link runnerToolText}: our message plus the tool's + * own `stdout`/`stderr`. Nothing else in `details` is read, so the argv we were asked to run and + * the verdict this classifier already published can never carry a rule (#2680). + */ + toolTextIncludesAll?: readonly string[]; + /** + * Every entry must appear in the SAME line of the lowercased {@link runnerToolText} (#2688 review). + * {@link RunnerErrorMatch.toolTextIncludesAll} proves only that two phrases exist somewhere in a + * captured log, which is a weaker claim than one phrase qualifying the other: a note about the + * profile the build used, three lines above an unrelated expired-certificate warning, says nothing + * about the profile. A row whose evidence is a noun and its complaint asks for both on one line. + */ + toolTextLineIncludesAll?: readonly string[]; /** Required details evidence beyond code/message. */ details?: RunnerErrorDetailsMatch; }; const hasRetriableFlag: RunnerErrorDetailsMatch = (details) => details.retriable === true; +/** + * The host's own `DevToolsSecurity -status` read, published as typed details by the probe that + * takes it. The build-failure rule below keys on this field and never on the probe's message, so + * an error that merely says developer mode is disabled cannot be read as a host refusal (#2680). + */ +const hasDevToolsSecurityStatus: RunnerErrorDetailsMatch = (details) => + typeof details.devToolsSecurityStatus === 'string'; const hasUsbmuxDeviceUnattached: RunnerErrorDetailsMatch = (details) => details.usbmuxDeviceAttached === false; /** @@ -203,17 +224,75 @@ type RunnerErrorVerdicts = { artifactSuspect?: boolean; }; +/** + * Why the Apple runner could not reach the point of serving a command (#2680). Published in + * `details.reason` on the `COMMAND_FAILED` every one of these paths throws, so a caller branches + * on the reason instead of matching prose; the hint that answers it travels with it in + * {@link RUNNER_ERROR_RULES}. + * + * This is the vocabulary #2683 adds the device-readiness members to (Developer Mode and developer + * disk image state read from the device itself), which is why it is keyed on startup rather than on + * `xcodebuild`: an iPhone that refuses the runner for reasons other than signing stops the runner + * before a build is ever the question. + * + * Placement: here beside the rules that produce it, not in `@agent-device/contracts`. Every member + * names a verdict an Apple runner path reaches, while `contracts` carries shapes several surfaces + * answer with (`InfrastructureBootFailureReason`, which both simulator and device boot use). + * Nothing outside this package publishes or consumes this enum, and one declaration is the only way + * a row and its reason cannot disagree. + */ +export const RUNNER_STARTUP_FAILURE_REASONS = [ + 'bundle_identifier_already_registered', + 'signing_no_development_team', + 'signing_provisioning_profile_missing', + 'signing_unspecified', + 'devtools_security_developer_mode_disabled', + 'build_failed_unclassified', +] as const; + +export type RunnerStartupFailureReason = (typeof RUNNER_STARTUP_FAILURE_REASONS)[number]; + +/** + * The reason a startup failure carries when no rule proves a cause. Its hint is deliberately the + * cache-recovery advice rather than anything about signing: an unclassified build is not evidence of + * a signing problem. + */ +export const RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON: RunnerStartupFailureReason = + 'build_failed_unclassified'; + type RunnerErrorRule = { /** Stable rule name for tests and diagnostics. */ reason: string; match: RunnerErrorMatch; verdicts: RunnerErrorVerdicts; + /** + * Set on the rules that also classify why the runner could not start (#2680). Rules like these + * define no recovery verdicts for a runner that never came up — there is no session to invalidate + * and nothing was sent to resend — so the axes stay empty and the row carries only reason plus + * hint. Several rows may name one reason (bundle-identifier registration fails in two shapes), + * and the classifier takes the first match, which is why specific rows precede generic ones. + */ + buildFailure?: { + reason: RunnerStartupFailureReason; + hint: string; + }; +}; + +/** + * The advice the provisioning-profile rows share (#2688). Named once so the three rows that require a + * different complaint cannot drift into three different fixes for one lever. + */ +const PROFILE_UNUSABLE: RunnerErrorRule['buildFailure'] = { + reason: 'signing_provisioning_profile_missing', + hint: 'Install/select a valid iOS provisioning profile, or set AGENT_DEVICE_IOS_PROVISIONING_PROFILE.', }; /** * The one declaration of runner error classes (#1631), mirroring * RUNNER_COMMAND_TRAIT_MANIFEST's role for commands: every recovery predicate - * below derives from this table instead of keeping its own substring chain. + * below derives from this table instead of keeping its own substring chain, + * and since #2680 so does the one classification of startup failures — a row + * carries recovery verdicts, a `buildFailure` reason and hint, or both. * Per axis, the FIRST matching rule that defines the axis wins — which is why * `flagged_retriable` precedes the denials (an explicitly retriable error * stays retriable whatever its message says), and `usbmux_device_unattached` @@ -304,11 +383,136 @@ export const RUNNER_ERROR_RULES: readonly RunnerErrorRule[] = [ match: { code: 'RUNNER_WEDGED' }, verdicts: { sessionFatalReason: 'runner_main_thread_wedged' }, }, + // ── Startup classification (#2680) ─────────────────────────────────────────────────────────── + // These rows answer "why could the runner not get here at all": `xcodebuild build-for-testing` + // refusing, and the host preflight that runs before it. They carry a reason and a hint for the + // caller and no recovery verdicts, because there is no session to invalidate and nothing was sent + // to resend. Specific rows precede generic ones: the classifier takes the first match. + // + // Why these rows are text matchers while the rows above key on a code or a typed field: + // `runnerToolText` reads xcodebuild's own prose because that prose is the only publication these + // failures have — there is no code and no typed field to key on. Its haystack is deliberately + // narrow: our message plus the tool's stdout/stderr, never the whole details bag, which also + // holds the argv we were asked to run (so a caller's own PROVISIONING_PROFILE_SPECIFIER=… would + // otherwise name a signing cause for an unrelated compile error) and the reason and hint this + // classifier just published (so a re-wrapped failure would match itself). The DevToolsSecurity + // row is the other half: where a probe of ours publishes a typed fact, the row keys on that fact + // alone. `resolveRunnerEarlyExitHint` stays outside this table for the same reason it stays a hint + // builder — it classifies a runner that DID build and then exited early, whose reason axis is the + // `BootFailureReason` `classifyBootFailure` already returns, and a build that never produced a + // binary has no boot to classify. + { + reason: 'bundle_identifier_registration_failed', + match: { toolTextIncludesAll: ['failed registering bundle identifier'] }, + verdicts: {}, + buildFailure: { + reason: 'bundle_identifier_already_registered', + hint: 'Set AGENT_DEVICE_IOS_BUNDLE_ID to a unique reverse-DNS value (for example, com.yourname.agentdevice.runner), then retry.', + }, + }, + { + // The identifier and its availability have to meet in one line: `App Identifier` and `not + // available` are two phrases a captured log can carry for reasons that have nothing to do with + // each other, which is the same hazard the profile rows just gave up (#2688 review). + reason: 'bundle_identifier_unavailable', + match: { toolTextLineIncludesAll: ['app identifier', 'not available'] }, + verdicts: {}, + buildFailure: { + reason: 'bundle_identifier_already_registered', + hint: 'Set AGENT_DEVICE_IOS_BUNDLE_ID to a unique reverse-DNS value (for example, com.yourname.agentdevice.runner), then retry.', + }, + }, + { + reason: 'signing_requires_development_team', + match: { toolTextIncludesAll: ['requires a development team'] }, + verdicts: {}, + buildFailure: { + reason: 'signing_no_development_team', + hint: 'Configure signing in Xcode or set AGENT_DEVICE_IOS_TEAM_ID for physical-device runs.', + }, + }, + { + // "conflicting provisioning settings" names a profile while saying the automatic and manual + // settings disagree, so without this row the profile row below would send the reader to install + // a profile for a problem that is a settings mismatch. No reason is claimed for it: nothing has + // captured this failure or proved which lever clears it, and advice the reader cannot follow is + // worse than the cache-recovery advice the unclassified path already gives (#2680). + reason: 'conflicting_provisioning_settings_unproven', + match: { toolTextIncludesAll: ['conflicting provisioning settings'] }, + verdicts: {}, + buildFailure: { + reason: RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON, + hint: RUNNER_CACHE_RECOVERY_HINT, + }, + }, + { + // "No profiles for 'com.example' were found" names the bundle id and the absence in one sentence, + // so the phrase alone is the complaint and needs no second phrase to qualify it. + reason: 'signing_no_profiles_for_bundle_id', + match: { toolTextIncludesAll: ['no profiles for'] }, + verdicts: {}, + buildFailure: PROFILE_UNUSABLE, + }, + // A profile named in the tool's output is only evidence when the output also says what is wrong with + // that profile, in the same line (#2688 review). One bare `provisioning profile` substring was the + // shipped sniffer's trigger, and it is a phrase a failing build can print while talking about + // something else: the codesign command line, a build-settings dump, a note about the profile that was + // used. Requiring a second phrase somewhere in the same log is no better — a note about the profile + // used above an unrelated `has expired` certificate warning would then name the profile. Each row + // below therefore asks for the profile and Xcode's complaint about it on one line, and a failure that + // merely mentions a profile stays unclassified rather than being sent to install one it already has. + { + // Xcode's own signing-error domain beside the profile it rejected: the machine-readable half of its + // `IDEProvisioningErrorDomain` diagnostics, which accompanies the prose rather than replacing it. + reason: 'signing_provisioning_profile_xcode_error', + match: { toolTextLineIncludesAll: ['provisioning profile', 'ideprovisioningerrordomain'] }, + verdicts: {}, + buildFailure: PROFILE_UNUSABLE, + }, + { + // "Provisioning profile \"X\" doesn't include application identifier ..." — the profile that is + // installed but does not cover this app or capability. + reason: 'signing_provisioning_profile_does_not_cover', + match: { toolTextLineIncludesAll: ['provisioning profile', "doesn't include"] }, + verdicts: {}, + buildFailure: PROFILE_UNUSABLE, + }, + { + // "Provisioning profile \"X\" has expired" — installing it again is not the fix; replacing it is, + // which is what the hint's "valid" is for. The full phrase, on the profile's own line: `expired` + // alone is what an expired certificate, a stale session, or a revoked key writes (#2688 review). + reason: 'signing_provisioning_profile_expired', + match: { toolTextLineIncludesAll: ['provisioning profile', 'has expired'] }, + verdicts: {}, + buildFailure: PROFILE_UNUSABLE, + }, + { + // Signing is involved but nothing above names how: the reason says signing and the hint stays + // the generic one it has always carried, rather than naming a misconfiguration no rule proved. + reason: 'signing_unspecified', + match: { toolTextIncludesAll: ['code signing'] }, + verdicts: {}, + buildFailure: { + reason: 'signing_unspecified', + hint: 'Enable Automatic Signing in Xcode or provide AGENT_DEVICE_IOS_TEAM_ID and optional AGENT_DEVICE_IOS_SIGNING_IDENTITY.', + }, + }, + { + reason: 'devtools_security_refused', + match: { code: 'COMMAND_FAILED', details: hasDevToolsSecurityStatus }, + verdicts: {}, + buildFailure: { + reason: 'devtools_security_developer_mode_disabled', + hint: 'Run `sudo DevToolsSecurity -enable`, then retry the iOS runner. UI test runners start suspended until Xcode/testmanagerd can attach.', + }, + }, ]; function matchesRunnerErrorRule(error: AppError, match: RunnerErrorMatch): boolean { if (match.code !== undefined && error.code !== match.code) return false; if (!matchesRunnerErrorDetails(error, match.details)) return false; + if (!matchesRunnerToolText(error, match.toolTextIncludesAll)) return false; + if (!matchesRunnerToolTextLine(error, match.toolTextLineIncludesAll)) return false; return matchesRunnerErrorMessage(error, match.messageIncludesAll); } @@ -323,6 +527,38 @@ function matchesRunnerErrorMessage(error: AppError, parts: readonly string[] | u return parts.every((part) => message.includes(part)); } +/** + * The only text a startup rule may read: our message plus the tool's own `stdout` and `stderr` + * (#2680). The rest of `details` is deliberately out of reach — `cmd`/`args` describe what we were + * asked to run, and `reason`/`hint` are this classifier's own output, which a re-wrapped failure + * would otherwise find and match again. + */ +function runnerToolText(error: AppError): string { + const details = error.details ?? {}; + return [error.message, details.stdout, details.stderr] + .filter((part): part is string => typeof part === 'string') + .join('\n') + .toLowerCase(); +} + +function matchesRunnerToolText(error: AppError, parts: readonly string[] | undefined): boolean { + if (!parts) return true; + const text = runnerToolText(error); + return parts.every((part) => text.includes(part)); +} + +/** + * The same haystack read one line at a time, so a row can require its phrases to be in one sentence + * rather than merely in one file (#2688 review). A captured build log is thousands of lines long, and + * two unrelated lines can hold any pair of words. + */ +function matchesRunnerToolTextLine(error: AppError, parts: readonly string[] | undefined): boolean { + if (!parts) return true; + return runnerToolText(error) + .split('\n') + .some((line) => parts.every((part) => line.includes(part))); +} + function runnerErrorVerdict( error: unknown, axis: Axis, @@ -599,29 +835,32 @@ export async function buildRunnerEarlyExitError(params: { }); } -function resolveSigningFailureHint(error: AppError): string | undefined { - const details = error.details ? JSON.stringify(error.details) : ''; - const combined = `${error.message}\n${details}`.toLowerCase(); - if ( - combined.includes('failed registering bundle identifier') || - (combined.includes('app identifier') && combined.includes('not available')) - ) { - return 'Set AGENT_DEVICE_IOS_BUNDLE_ID to a unique reverse-DNS value (for example, com.yourname.agentdevice.runner), then retry.'; - } - if (combined.includes('requires a development team')) { - return 'Configure signing in Xcode or set AGENT_DEVICE_IOS_TEAM_ID for physical-device runs.'; - } - if (combined.includes('no profiles for') || combined.includes('provisioning profile')) { - return 'Install/select a valid iOS provisioning profile, or set AGENT_DEVICE_IOS_PROVISIONING_PROFILE.'; - } - if (combined.includes('code signing')) { - return 'Enable Automatic Signing in Xcode or provide AGENT_DEVICE_IOS_TEAM_ID and optional AGENT_DEVICE_IOS_SIGNING_IDENTITY.'; +/** + * The one classifier for "the runner did not reach the point of serving a command" (#2680). Every + * path that stops the runner before it answers a request routes its failure through here, so the + * reason a caller sees is produced by the same rows that produce the hint beside it — a reason is + * never inferred from a hint's wording, and an unproven cause is never claimed. + * + * Callers publish the pair as `details.reason` plus the top-level hint on a `COMMAND_FAILED`; the + * code is `COMMAND_FAILED` for every reason, so the reason is the assertion. + */ +export function classifyRunnerStartupFailure(error: unknown): { + reason: RunnerStartupFailureReason; + hint: string; +} { + if (error instanceof AppError) { + for (const rule of RUNNER_ERROR_RULES) { + const buildFailure = rule.buildFailure; + if (!buildFailure) continue; + if (matchesRunnerErrorRule(error, rule.match)) { + return { reason: buildFailure.reason, hint: buildFailure.hint }; + } + } } - return undefined; -} - -export function resolveRunnerBuildFailureHint(error: AppError): string { - return resolveSigningFailureHint(error) ?? RUNNER_CACHE_RECOVERY_HINT; + return { + reason: RUNNER_STARTUP_FAILURE_UNCLASSIFIED_REASON, + hint: RUNNER_CACHE_RECOVERY_HINT, + }; } export function withRunnerCommandId(command: RunnerCommand): RunnerCommand { diff --git a/packages/platform-apple/src/runner/runner-dev-tools-security.ts b/packages/platform-apple/src/runner/runner-dev-tools-security.ts new file mode 100644 index 0000000000..f3ce67e73b --- /dev/null +++ b/packages/platform-apple/src/runner/runner-dev-tools-security.ts @@ -0,0 +1,45 @@ +import { AppError } from '@agent-device/kernel/errors'; +import { isIosFamily, type DeviceInfo } from '@agent-device/kernel/device'; +import { runAppleToolCommand } from './host.ts'; +import { classifyRunnerStartupFailure } from './runner-contract.ts'; + +const DEV_TOOLS_SECURITY_TIMEOUT_MS = 2_000; + +const DEV_TOOLS_SECURITY_REFUSAL_MESSAGE = 'Developer mode is disabled for Apple development tools'; + +/** + * The host half of "can this Mac run an Apple UI test at all", probed before the runner builds. + * + * `DevToolsSecurity -status` reports the macOS developer-tools security setting that governs + * debugserver on THIS machine. It is not the iPhone's Settings > Privacy & Security > Developer + * Mode toggle, which lives on the device and has no host-visible value here — the two states are + * independent even though the wording is nearly the same, so the failure this throws is labelled + * with the host's own reason and never with a device-side one (#2680). + */ +export async function assertDevToolsSecurityForIosRunner(device: DeviceInfo): Promise { + if (!isIosFamily(device) || device.kind !== 'device') return; + const result = await runAppleToolCommand('DevToolsSecurity', ['-status'], { + allowFailure: true, + timeoutMs: DEV_TOOLS_SECURITY_TIMEOUT_MS, + }); + const output = `${result.stdout}\n${result.stderr}`; + if (!/developer mode is currently disabled/i.test(output)) return; + throw buildDevToolsSecurityRefusal(output.trim()); +} + +/** + * The refusal carries the reason and hint that {@link classifyRunnerStartupFailure} derives from + * the typed `devToolsSecurityStatus` fact, instead of naming either here, so the pair a caller + * receives cannot drift from the rule table that owns it. + */ +function buildDevToolsSecurityRefusal(status: string): AppError { + const observed = new AppError('COMMAND_FAILED', DEV_TOOLS_SECURITY_REFUSAL_MESSAGE, { + devToolsSecurityStatus: status, + }); + const { reason, hint } = classifyRunnerStartupFailure(observed); + return new AppError('COMMAND_FAILED', DEV_TOOLS_SECURITY_REFUSAL_MESSAGE, { + reason, + hint, + devToolsSecurityStatus: status, + }); +} diff --git a/packages/platform-apple/src/runner/runner-session.ts b/packages/platform-apple/src/runner/runner-session.ts index e748e19192..2270d12e2a 100644 --- a/packages/platform-apple/src/runner/runner-session.ts +++ b/packages/platform-apple/src/runner/runner-session.ts @@ -6,11 +6,10 @@ import { emitDiagnostic, withDiagnosticTimer, buildSimctlArgsForDevice, - runAppleToolCommand, runXcrun, } from './host.ts'; import type { ExecResult } from '@agent-device/host-kit/command'; -import { isIosFamily, isApplePlatform, type DeviceInfo } from '@agent-device/kernel/device'; +import { isApplePlatform, type DeviceInfo } from '@agent-device/kernel/device'; import type { RunnerLogicalLeaseContext } from '@agent-device/contracts/runner-lease-context'; import type { AppleRunnerLifecycleOptions } from './runner-provider.ts'; import { getFreePort } from './runner-io.ts'; @@ -187,8 +186,12 @@ async function startRunnerSessionWithLease( await measureRunnerStartupStep(startupTimings, 'ensure_booted', async () => { await ensureBootedIfNeeded(device); }); - await measureRunnerStartupStep(startupTimings, 'verify_developer_mode', async () => { - await verifyDeveloperModeForIosRunner(device); + await measureRunnerStartupStep(startupTimings, 'verify_host_dev_tools_security', async () => { + // Loaded here rather than at the top of the file: the runner subtree sits in the eager import + // closure of the seven Apple facades (eager-closure-budgets), and a preflight only a physical + // device ever needs has no business being evaluated to answer a simulator request. + const { assertDevToolsSecurityForIosRunner } = await import('./runner-dev-tools-security.ts'); + await assertDevToolsSecurityForIosRunner(device); }); if (options.cleanStaleBundles) { await measureRunnerStartupStep(startupTimings, 'cleanup_stale_bundles', async () => { @@ -721,20 +724,6 @@ async function ensureBooted(device: DeviceInfo): Promise { }); } -async function verifyDeveloperModeForIosRunner(device: DeviceInfo): Promise { - if (!isIosFamily(device) || device.kind !== 'device') return; - const result = await runAppleToolCommand('DevToolsSecurity', ['-status'], { - allowFailure: true, - timeoutMs: 2_000, - }); - const output = `${result.stdout}\n${result.stderr}`; - if (!/developer mode is currently disabled/i.test(output)) return; - throw new AppError('COMMAND_FAILED', 'Developer mode is disabled for Apple development tools', { - hint: 'Run `sudo DevToolsSecurity -enable`, then retry the iOS runner. UI test runners start suspended until Xcode/testmanagerd can attach.', - devToolsSecurityStatus: output.trim(), - }); -} - export function validateRunnerDevice(device: DeviceInfo): void { if (!isApplePlatform(device.platform)) { throw new AppError( diff --git a/src/commands/schema/cli-help.ts b/src/commands/schema/cli-help.ts index b2addd9db2..5a6599bf26 100644 --- a/src/commands/schema/cli-help.ts +++ b/src/commands/schema/cli-help.ts @@ -689,6 +689,7 @@ iOS physical-device prerequisites: If Xcode cannot choose a profile, set AGENT_DEVICE_IOS_PROVISIONING_PROFILE to the profile name/specifier, not a file path. AGENT_DEVICE_IOS_SIGNING_IDENTITY is optional; omit it unless xcodebuild asks for a specific identity. The profile/team must allow AGENT_DEVICE_IOS_BUNDLE_ID and .uitests. + A runner build failure names its class in error details.reason rather than only in prose: signing_no_development_team, signing_provisioning_profile_missing, bundle_identifier_already_registered, signing_unspecified, devtools_security_developer_mode_disabled (the Mac's DevToolsSecurity setting, which says nothing about the device's Developer Mode toggle), or build_failed_unclassified when nothing proved a cause. Branch on details.reason and follow hint; the message is for humans. First-run XCTest setup/build can take longer than normal commands; keep the device connected and use --debug to inspect signing/build diagnostics if setup times out. Android physical-device prerequisites: diff --git a/website/docs/docs/installation.md b/website/docs/docs/installation.md index a3e08f66da..62e39676f3 100644 --- a/website/docs/docs/installation.md +++ b/website/docs/docs/installation.md @@ -108,6 +108,7 @@ vega device list - `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` - `AGENT_DEVICE_IOS_BUNDLE_ID` (optional runner bundle-id base override) - Free Apple Developer (Personal Team) accounts can fail with "bundle identifier is not available" for generic IDs; set `AGENT_DEVICE_IOS_BUNDLE_ID` to a unique reverse-DNS value (for example `com.yourname.agentdevice.runner`). +- A runner build failure is typed, not prose: `error.details.reason` is one of `signing_no_development_team`, `signing_provisioning_profile_missing`, `bundle_identifier_already_registered`, `signing_unspecified`, `devtools_security_developer_mode_disabled` (the Mac's `DevToolsSecurity` setting, which says nothing about the device's Developer Mode toggle), or `build_failed_unclassified` when nothing proved a cause. Branch on `details.reason` and follow `hint`; the code stays `COMMAND_FAILED` for every reason. - If device setup is slow, keep the device connected and inspect daemon diagnostics after retrying. - If daemon startup reports stale metadata, remove stale files and retry: - `/daemon.json`