Skip to content

feat(fold): pose a foldable iPhone simulator through Device Hub and verify it with CoreDevice - #2717

Open
thymikee wants to merge 2 commits into
fix/macos-surface-press-dwellfrom
feat/foldable-pose
Open

thymikee wants to merge 2 commits into
fix/macos-surface-press-dwellfrom
feat/foldable-pose

Conversation

@thymikee

@thymikee thymikee commented Sep 20, 2026

Copy link
Copy Markdown
Member

Summary

Adds agent-device fold <closed|half-open|open> and client.command.fold({ pose }) for the iPhone Duo simulator. ADR 0025 concluded no official host API sets the hinge pose and left it to the operator. This PR sets it the way the operator does: the macOS helper presses the pose control in the Xcode Device Hub window through accessibility (the earlier "zero accessibility nodes" finding was System Events seeing the trampolined app as pid 0), and the Apple owner trusts only CoreDevice's read-back of devicectl device motion hinge-angle. closed and open are the hinge end stops; half-open is reported only once two consecutive reads agree the hinge stopped, because the first read after pressing Book caught it at 175.1°.

The helper resolves Device Hub from the process table, reopens its window when a headless boot left none, and selects the simulator through the sidebar row keyed TableRow.Device.<udid>, so twins sharing a name cannot be confused. The response reports the verified pose, the angle, and the lit panel's point size, so an agent sees its refs are stale.

agent-device fold open        # Folded to open (hinge 180°, LCD-1 lit at 669x951pt); refs ... are stale
agent-device fold half-open   # Folded to half-open (hinge 130°, ...)
await client.command.fold({ pose: 'closed' })

setFoldPose is a host-side device-runtime operation (no runner); every other platform and provider states a refusal cell. 84 files on top of #2716; the gross diff exceeds the 1,000-line budget because of the ADR update, tests, and the Swift helper subcommand with its own tested library target.

Stacked on #2716 (both touch the macOS helper); rebased onto main after #2715 landed. The helper awaits the action bar after switching the sidebar selection (a live run caught device-hub-pose-control-missing without it), and its pure decisions are XCTested; that test caught a window-title match that accepted iPhone Duo Lab for iPhone Duo. Review round one is addressed in the comment thread: a refusal never names the requested pose, the pose vocabulary lives in device-rotation.ts so the eager-closure budgets hold, and fold declares its own 150s envelope. Pressing a ref on the unfolded inner panel lands on a transposed point; that is a pre-existing snapshot-geometry defect, filed as #2719, not part of this PR.

Validation

Tested at 3fb3c60: pnpm check:affected --run passes (format, lint, typecheck, layering, di-seams, fallow, mcp-metadata, build, package, integration-node, macos-coverage, vitest-related 698 files, integration-progress, replay-compat, daemon-wire-compat, affected-selector).

Live, through the built CLI on an iOS 27.1 iPhone Duo simulator: open → 180° on LCD-1, screenshot 951x669; half-open → 130°; closed → 0° on LCD, screenshot 466x678; 5–16s each with last-sample hinge reads. Screenshots, JSON output, and a Device Hub recording are in the evidence comment below. Also verified from a headless boot with no Device Hub window, and from a window showing another simulator. An iPhone 17 simulator refuses with UNSUPPORTED_OPERATION (single-panel-device) before anything is pressed. Not exercised: a second Xcode running its own Device Hub, and physical foldables (refused by the leaf fact).

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2717/

Built to branch gh-pages at 2026-09-21 09:30 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.68 MB 4.71 MB +27.1 kB
Package (unpacked) 4.68 MB 4.71 MB +27.1 kB
Package (download) 1.40 MB 1.40 MB +8.2 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.4 ms 28.1 ms -0.3 ms
CLI --help 78.8 ms 79.0 ms +0.2 ms

@thymikee

Copy link
Copy Markdown
Member Author

This looks close, but one refusal path contradicts itself and the coverage gate is failing because of where the pose vocabulary got exported.

At 37e7778, awaitHingePose in packages/platform-apple/src/foldable/pose.ts refuses once the 4-attempt budget runs out, even when the last reading already maps to the requested pose (https://github.com/callstack/agent-device/blob/37e7778/packages/platform-apple/src/foldable/pose.ts#L129). For fold half-open, the loop needs two consecutive reads within 0.5 deg, and the ADR's own live sequence (180 -> 175.1 -> 130 -> 130) spends all 4 attempts to get there, so one extra stale read anywhere in that chain ends the loop with observedPose === requestedPose === 'half-open' and the command still reports COMMAND_FAILED, naming the very angle it asked for. The test in pose.test.ts locks this in: it reads 170/150/120/90 and asserts a refusal at hingeAngleDegrees: 90, which is itself half-open. Can you state the rule directly — a fold refusal must never report an observed pose equal to the requested pose — and either return success when budget exhaustion lands on the requested pose, or give "right pose, still settling" its own reason and message, with the test split to cover both cases?

The Coverage job fails on 11 eager-closure-budget assertions, all tracing to one new edge: packages/contracts/src/facades/device.ts -> packages/contracts/src/fold-pose.ts (https://github.com/callstack/agent-device/blob/37e7778/packages/contracts/src/facades/device.ts). That pulls fold-pose exports onto the import path of packages/platform-android/src/mechanics.ts, provider-limrun, provider-webdriver, session-journal, and src/cli.ts, none of which can pose a hinge. Can the pose vocabulary move to a module those entries already evaluate, or be reached only through the @agent-device/contracts/fold-runtime subpath this PR already adds, so only fold's actual readers pay for it?

Is the Apple-only per-platform seam here the same shape as actionButton, or does fold need something narrower — PlatformRuntimeOperations is a total union, every public command declares a device-runtime use, and the Android/webdriver/limrun one-liners look like the existing totality rule rather than new surface? If so the seam itself isn't the problem, just its placement above.

Not blocking, take or leave: parseHingeAngleSample returns the first Angle: line of the devicectl stream rather than the last, so every read can be about five seconds stale — which is also what consumes the retry budget behind the refusal above (https://github.com/callstack/agent-device/blob/37e7778/packages/platform-apple/src/core/hinge-angle.ts#L65). Smaller ones alongside it: SetFoldPoseInput.execution is copied from the orientation runtime but nothing reads it, appleFoldFact gates on hasAppleActionButton() which only coincides with hinge-capable devices today, fold inherits the generic 90s timeout although its own worst case can sum past it, the Android refusal says emulators "expose no hinge control" when foldable AVDs do have a posture control this project simply does not drive yet, the daemon test for an invalid pose asserts a bare .rejects.toThrow() with no error code, and noInventedPoseCommand still forbids the unfold/unfolded aliases that parseFoldPose now accepts.

The PR body says 70 files touched where the diff shows 78, and describes pnpm check:affected --run as passing at a head where Coverage is failing on the eager-closure edge above; both are worth correcting so the record matches the current state.

I could not run anything on macOS, so the Device Hub press, the trampolined-pid discovery, the reopen, and the sidebar row selection are unverified from the diff, and the 313-line Swift helper has no test covering them. The live angles and timings in the PR body are narrative without an attached run, so I checked the code paths they describe but not the readings themselves. Line references above are pinned to 37e7778.

Before this merges, the fold-pose exports need to move off packages/contracts/src/facades/device.ts so Coverage passes without rebaselining the budget, and the half-open refusal needs to stop firing when the observed angle already matches the requested pose.

@thymikee
thymikee changed the base branch from main to fix/macos-surface-press-dwell September 21, 2026 06:41
@thymikee

Copy link
Copy Markdown
Member Author

Live evidence at 82fc2fa (rebased on #2715, stacked on #2716)

Built CLI against the booted iOS 27.1 iPhone Duo simulator, Safari session, Device Hub window switched to the Duo by the helper through the sidebar row. The recording shows the Device Hub window while the three commands ran; the screenshots are agent-device screenshot after each pose.

$ agent-device fold open --json
{ "action": "fold", "pose": "open", "hingeAngleDegrees": 180,
  "screen": { "display": "LCD-1", "widthPt": 669, "heightPt": 951 },
  "message": "Folded to open (hinge 180°, LCD-1 lit at 669x951pt); refs from before the pose change are stale" }   (11s)

$ agent-device fold half-open --json
{ "action": "fold", "pose": "half-open", "hingeAngleDegrees": 130,
  "screen": { "display": "LCD-1", "widthPt": 669, "heightPt": 951 }, ... }   (16s)

$ agent-device fold closed --json
{ "action": "fold", "pose": "closed", "hingeAngleDegrees": 0,
  "screen": { "display": "LCD", "widthPt": 466, "heightPt": 678 }, ... }   (10s)

Screenshot sizes follow the lit panel: 466x678 before, 951x669 after open and half-open, 466x678 after closed.

One defect this run found and the head fixes: right after the helper switched the window from another simulator to the Duo, the action bar had not rendered yet and the first press reported device-hub-pose-control-missing. The helper now awaits the pose control after a device switch; the rerun above started from a window showing the Duo already, and a rerun from a window showing bench-golden-v1 selected the Duo and folded it on the first try.

Not part of this PR: pressing a ref on the unfolded inner panel lands on the wrong point (the snapshot rect for Safari's address field comes back transposed relative to the screenshot). Filed separately with the geometry.

Before: closed pose, outer panel, screenshot 466x678

After fold open: inner panel lit, screenshot 951x669

After fold half-open: hinge 130°, inner panel still lit

After fold closed: outer panel again, 466x678

fold-device-hub-recording.mov

@thymikee

Copy link
Copy Markdown
Member Author

Addressed at c9e6cde (stacked on #2716 at 0c6a78f; both rebased onto main with #2715 and #2718).

Half-open refusal. The rule is now stated where the loop is: a refusal never names the pose that was asked for. closed and open return on the first read at the stop; half-open returns once two reads agree within 0.5°, and when the four-read budget ends while the hinge still reads half-open, that reading is the answer. Only a last reading that maps to some other pose refuses as fold-pose-unverified. The test that locked in the contradiction (170/150/120/90 → refusal at 90) now asserts success at 90, and a sibling asserts the refusal only when the last read is 180 while half-open was asked. Alongside it, parseHingeAngleSample takes the last sample of the stream rather than the first, so a moving hinge is read where it is now; live, fold open dropped from 11s to 6s and half-open still settles at 130°.

Coverage / eager closure. The pose vocabulary moved into packages/contracts/src/device-rotation.ts, which every entry that reads a device pose already evaluates, and platform-runtime-unavailable.ts states the setFoldPose cell directly instead of importing the fold contract. scripts/__tests__/eager-closure-budgets.test.ts passes at this head with no rebaseline; pnpm check:affected --run is green on both heads.

The seam. Yes, same shape as actionButton: PlatformRuntimeOperations is a total union and every owner states its cell. setFoldPose is a host-side device-runtime operation rather than an Interactor member, since no runner takes part, so the per-owner one-liners are the totality rule and nothing else.

Non-blocking, taken: SetFoldPoseInput.execution removed; appleFoldFact reads the iPhone/iPad leaf itself instead of borrowing the Action Button predicate; fold declares its own 150s envelope (one helper press plus four bounded hinge reads can pass 90s on a wedged host), registered in the reviewed envelope set; the Android hint now says the emulator posture control exists and is not driven yet; the invalid-pose daemon test asserts INVALID_ARGS; the help-conformance forbidden pattern exempts agent-device fold … lines so the unfold/unfolded aliases are allowed.

Swift helper tests. The helper's pure decisions (pose → control description, window-title match, sidebar row identifier) moved into an AgentDeviceMacOSDeviceHub library target with XCTests run by the macos-helper gate. Writing them found a real bug: the title match used a bare-space prefix, so iPhone Duo claimed the window of a simulator named iPhone Duo Lab; it now matches Device Hub's exact separator.

PR body corrected: 84 files on top of #2716, validation at c9e6cde. Live evidence with screenshots and the Device Hub recording is in the comment above; the run at that head repeated open → 179.2° (last sample of a stream that ended just before the stop), half-open → 130°, closed → 0°.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at c9e6cde. Both blockers from the last round are fixed. A fold refusal can no longer name the pose that was asked for: the 170/150/120/90 test now resolves at 90, and a sibling test covers the real refusal. The pose vocabulary now lives in the existing device-rotation.ts, so facades/device.ts adds no new eager edge and no budget changed. The smaller notes are taken too (last Angle: sample, the unused execution field, the fold fact gate, the 150 s envelope, the Android wording, the INVALID_ARGS assertion and the unfold alias).

The live evidence comes from you: full JSON at 82fc2fa, and a rerun reported at c9e6cde without attached output. If that rerun's JSON is still around, please link it to close the gap.

Smoke Tests fails on the iOS long-press scroll check. That looks unrelated. This PR touches no test-app, scroll, snapshot or runner interaction code, and the same failure shows on #2363 and #2688.

There are no conflicts, and I added ready-for-human. This PR is stacked on #2716, so #2716 must land first.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 21, 2026
@thymikee
thymikee added this pull request to stack #2720 September 21, 2026 08:59
thymikee and others added 2 commits September 21, 2026 11:26
…erify it with CoreDevice

Add `agent-device fold <closed|half-open|open>` and `client.command.fold({ pose })`.
ADR 0025 recorded that no official host API sets the hinge pose and left it to the
operator. The operator's own control is the pose action bar in the Xcode Device Hub
window, whose controls are ordinary accessibility buttons; the macOS helper gains a
`device-hub pose` subcommand that finds Device Hub in the process table (LaunchServices
registers the trampolined app with pid -1, which is why System Events saw no nodes),
reopens its window with a reopen event when it shows none, selects the simulator through
the sidebar row keyed `TableRow.Device.<udid>`, and presses the control.

The press is not the evidence. The Apple owner reads the hinge angle back through
`devicectl device motion hinge-angle` until it agrees with the request: closed is 0°, open
is 180°, and half-open is any angle between them reported only once two consecutive reads
agree the hinge stopped moving, because a hinge sweeps through half-open angles on its way
to either stop (the live run read 175.1° one stream after pressing Book, 130° after it
settled). The stream never ends on its own, so each read is bounded by devicectl's smallest
`--timeout` of five seconds. The response carries the verified pose, the angle, and the lit
panel's point size, so an agent sees its refs are stale without another capture.

`setFoldPose` is a host-side device-runtime operation rather than an interactor method: it
needs no XCUITest runner, and the leaf fact admits iPhone/iPad simulators while the
operation itself refuses a single-panel simulator from CoreDevice's display table. Every
other platform and provider states its own refusal cell.

Verified through the built CLI on an iOS 27.1 iPhone Duo simulator: open → 180° on LCD-1
(669x951pt, screenshot 951x669), half-open → 130°, closed → 0° on LCD (466x678pt,
screenshot 466x678), from a Device Hub that showed no window for the device; an iPhone 17
refuses with UNSUPPORTED_OPERATION before anything is pressed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… and conformance tables

The contracts export snapshot names the new `fold-runtime` subpath, the six-platform
coverage declaration and its Android evidence row classify the command, the runtime-binding
conformance table resolves it, and the provider output guard drives it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant