fix(macos-helper): hold synthetic clicks long enough to be delivered

[thymikee]

Summary

press on a macOS session whose surface has no runner (frontmost-app, desktop, menubar) never activated anything. The helper posted mouseMoved, leftMouseDown, leftMouseUp back to back, and AppKit drops a mouse-up posted in the same event tick as its down: on an NSButton probe a zero-dwell click delivered 0 of 15 mouse-ups and activated 0 actions; 20 ms activated 14 of 15; 40 ms and above activated 15 of 15.

Click posting now lives in a new AgentDeviceMacOSInput Swift target as a pure schedule plus a poster, so the dwell and the rising click state are unit-testable without a window server. Default hold is 60 ms with a 40 ms floor. --count N is N independent clicks at click state 1, as on every other platform; --double-tap posts each click as a double-click pair with a rising click state, so the two compose into N double-clicks. The helper's process timeout is derived from the click schedule plus margin and the request's abort signal is forwarded. Every stop the host applies — the deadline, a cancelled request, a dropped client — reaches the helper as SIGTERM first (a new kill policy on ExecOptions, SIGKILL only after a one-second grace), so the helper's release handler runs and a button held mid-stop comes back up. 17 files, no scope beyond the macOS press path and the exec kill seam it needed.

agent-device open --surface frontmost-app --platform macos
agent-device press "label=Bluetooth" --platform macos
agent-device press @e7 --double-tap --platform macos

Validation

Head 14d1006f7 (rebased onto main with #2715 and #2718; review rounds one and two addressed in the comment thread). pnpm check:affected --run on that head: all runnable checks passed. Also pnpm test:macos-helper (6 Swift tests), pnpm test:integration:macos-coverage, pnpm test:integration:progress:check, test/integration/provider-scenarios/macos-desktop.test.ts, pnpm typecheck, pnpm lint, pnpm check:fallow --base origin/main.

Live host evidence, agent-device against an AppKit probe and System Settings: press activated 15/15 (0/15 before); --count 3 gave three activations at clickCount 1; --double-tap gave clickCount 1 then 2, and --double-tap --count 2 two such pairs; --hold-ms 10000 --count 4 completed in 40 s with four activations and a normal click afterwards; press "label=Bluetooth" on a frontmost-app session opened the Bluetooth pane, proven by wait "windowtitle=Bluetooth" (screenshots and recording in the evidence comment). Second live run at 14d1006f7 on a Finder frontmost-app session: --count 2 reads as two clickCount 1 clicks on the AppKit probe and leaves a Finder row unopened at --interval-ms 700 (Finder itself derives a double-click from timing, so the default 120 ms interval still opens it); --double-tap opens the folder; a --hold-ms 3000 press stopped one second in by Ctrl-C and by killing the client leaves the button up within 0.4 s with the helper gone, where before the button stayed down for the rest of the hold.

Two gaps stay open and are not fixed here: longpress on those surfaces is still a silent no-op (longPressPoint is declared runner-only in packages/platform-apple/src/runner-demand.ts), and the replay open grammar accepts no --surface, so no live .ad scenario can reach the helper route. pnpm test:replay:macos also fails on this host at the pre-existing About-pane wait in 01-system-settings.ad, identically on unmodified origin/main.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-21 10:36 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.67 MB	4.68 MB	+11.6 kB
Package (unpacked)	4.67 MB	4.68 MB	+11.6 kB
Package (download)	1.39 MB	1.40 MB	+3.7 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	28.1 ms	27.5 ms	-0.6 ms
CLI --help	80.1 ms	78.2 ms	-1.9 ms

[thymikee]

Reviewed at 929cdfc. The fix is right in direction but two gaps stop this from being safe to merge.

clickState is incremented on every down unconditionally (apple/macos-helper/Sources/AgentDeviceMacOSInput/MouseClickDelivery.swift#L48), and --count N with no --double-tap maps to N clicks in one schedule (packages/platform-apple/src/interactions.ts#L222). So press @e1 --count 2 posts clickCount 1 then 2, which the app reads as a double-click, and --count 3 reads as a triple-click. Every other platform treats count as N independent presses (packages/contracts/src/touch-runtime.ts#L234-L249), so the same flag now means a different gesture on macOS, and it gets worse: --double-tap --count 3 gives one triple-click instead of three double-taps. Can the rising click state be made explicit instead of derived from the repeat count — for example a coalesceAsMultiClick field on MouseClickRequest set only from options.doubleTap, with independent repeats always at clickState 1 and doubleTap+count expanded into that many double-click pairs?

The schedule duration is now hold*clicks + interval*(clicks-1), bounded only by the CLI maxima (count<=200, holdMs<=10000, intervalMs<=10000, src/daemon/touch-runtime.ts#L384-L393), but runMacOsPressAction still calls the helper with a fixed 30s timeout and no abort signal (apple/macos-helper/Sources/... via packages/platform-apple/src/os/macos/helper.ts#L419). --count 200 schedules about 36s and gets killed mid-run; the uninterruptible usleep between down and up (MouseClickDelivery.swift#L36) means the kill can land between mouseDown and mouseUp with no release-on-termination handler, leaving the system mouse button stuck down. What must enumerate here is every path that builds a helper timeout from a click schedule — right now none of them account for the schedule length. Could the timeout be derived from hold*clicks + interval*(clicks-1) plus margin, with options.signal passed through and the button released from a termination handler, so no run can end between down and up? A press with --hold-ms 10000 --count 4 against a live macOS target would show whether the command completes without a stuck button and without the helper being killed mid-schedule.

Not blocking: the Swift tests only exercise mouseClickSteps/mouseClickHoldMs and miss the clickState policy in postMouseClick, TouchPressExtras.holdMs silently means something different on macOS than elsewhere with no doc note, the "Build macOS helper" workflow step name no longer matches the "macOS helper build and tests" gate label, and --interval-ms 0 is indistinguishable from unset on this route while jitterPx is echoed but never applied — any or none of these can be picked up.

Is the schedule type doing more than it needs to? A Swift executableTarget can't be imported by a test target, so pulling the schedule into a library target to make it testable seems justified. But MouseClickStep/MouseClickStepKind and the step array exist only to feed the single loop in postMouseClick — could one file with mouseClickHoldMs(requestedMs:) plus a poster loop (move, then per click: down, sleep, up, sleep) carry the same fix with one pure tested function and no intermediate step type, putting the click-count policy next to the request instead of implying it from array position? That said, the click-count decision above should land first, since it determines what the schedule type needs to carry.

The PR body's 15/15 activation and clickCount results, and the Bluetooth-pane test, come from the author and were not reproduced here; the stuck-button scenario is reasoned from the fixed timeout and the uninterruptible sleep, not from killing a live helper mid-hold. CI is green across 21 checks, including the macos-helper gate this PR changes, and the diff stays confined to macos-helper and the platform-apple press path those checks exercise. No conflicts. The two open questions are the click-count rule (independent clicks for --count, rising state only for --double-tap) and bounding the schedule against the helper timeout with a guaranteed mouse-up on termination; once those are settled this is close.

[thymikee]

Live evidence at 5a46d70 (rebased on main with #2715)

Rebased this branch onto main after #2715 landed; no conflicts, pnpm check:affected --run passes on the stacked head 82fc2fa (#2717 now targets this branch, since both touch the macOS helper).

Built CLI on the macOS host, frontmost-app surface with System Settings frontmost:

$ agent-device open --surface frontmost-app --platform macos
$ agent-device press "label=Bluetooth" --json      → "Tapped label=Bluetooth (472, 382)", selector resolution kind "unique"
$ agent-device wait "windowtitle=Bluetooth"         → ok
$ agent-device press "label=General"                → "Tapped label=General (466, 523)"
$ agent-device snapshot -i | grep -E "About|Software Update"
@e127 [button] "About"
@e128 [button] "Software Update"

Both presses activated their sidebar row on the first click through the dwell-held synthetic click. The recording is the host screen while the sequence ran. One note for anyone reproducing: wait "windowtitle=General" does not prove the General pane, because that pane does not retitle the window; the snapshot does.

macos-press.mov

[thymikee]

Addressed at 0c6a78f (rebased onto main with #2715 and #2718).

Click count. The rising click state is no longer derived from the repeat count. MouseClickRequest carries an explicit doubleClick, set only from --double-tap; --count N is N independent presses at click state 1, and --double-tap --count N is N double-click pairs, each pair separated inside by an 80 ms gap and pairs by --interval-ms. The step array and its kinds are gone: the schedule is one pure mouseClickPresses(clicks:doubleClick:intervalMs:) that the XCTests pin (independent counts, rising state per press, pair gap under the system double-click interval), and postMouseClick is one loop over it.

Timeout and the stuck button. runMacOsPressAction now derives its process timeout from the schedule (macOsClickScheduleMs, the same sum the helper's mouseClickScheduleMs computes, plus the 30 s margin) and forwards the request's abort signal; runMacOsHelper takes timeoutMs. The helper installs SIGTERM/SIGINT/SIGHUP handlers that post the pending mouse-up before exiting, so no run can end between a down and its up. Every path that runs a click schedule goes through runMacOsPressAction, so that is the one place the timeout is built.

Live, on an AppKit probe that logs every mouse-down's clickCount and every button activation:

press --count 3                → down clickCount=1 ×3, activated 3 times
press --double-tap             → down clickCount=1, down clickCount=2, activated twice
press --double-tap --count 2   → 1,2,1,2 — two double-clicks
press --hold-ms 10000 --count 4 → completed in 40 s, holdMs 10000 reported, activated 4 times
press                          → activated normally afterwards (no stuck button)

Non-blocking, taken: an explicit --interval-ms 0 is now carried rather than read as unset; commands.md says what --hold-ms, --count, and --double-tap mean on these surfaces and that --jitter-px is not applied; the workflow step is named after the gate.

pnpm check:affected --run passes on this head and on #2717 stacked above it.

[thymikee]

Reviewed at 0c6a78f. Both earlier points are fixed in code. --count N is now N independent presses at click state 1, and only --double-tap raises the state. The helper timeout now follows the click schedule, and the abort signal reaches the helper.

The new release handler does not run on the route that stops the helper. installMouseReleaseOnTermination traps SIGTERM, SIGINT and SIGHUP (MouseClickDelivery.swift#L54). But the host stops the helper only through killProcessTree, which sends SIGKILL on both timeout and abort (exec.ts#L796-L803). Now that context.signal reaches runMacOsSurfacePress, a cancelled press --hold-ms 10000 or a dropped client kills the helper between mouse down and mouse up, and the button stays down. Before this change, a cancel did not stop the helper mid-hold, so this path is new. Every stop the host can apply while a button is down needs to post the mouse-up. For example, send SIGTERM with a short grace period before SIGKILL for this helper, or let the helper release on its own deadline or on stdin EOF. A test that aborts a helper mid-hold through runAppleToolCommand and checks for the up event would pin it.

The live run at 5a46d70 shows single presses only, and this change alters the click schedule on the host. Please add a run at the new head on the macOS host: press --count 2 acting as two single clicks on a control where that differs from a double-click, and press --double-tap (and --double-tap --count 2) opening a Finder item or table row, with the --json output. After the release fix, add a cancelled press --hold-ms 3000 that leaves the button up.

The Smoke Tests failure is an xcrun timed out after 30000ms in the iOS snapshot-bridge check. This PR does not touch that route, so the failure looks unrelated. There are no conflicts.

[thymikee]

Addressed at 14d1006.

Every host stop now reaches the helper as SIGTERM first. ExecOptions gained a kill policy ({ signal, graceMs }); killProcessTree sends the policy's signal, arms an unref'd escalation timer, and clears it on the child's exit, with the group path and the reaped-pid guard shared by both signals. runMacOsHelper asks for SIGTERM with a one-second grace on every helper run, so the release handler runs on the deadline, on a cancelled request, and on a dropped client. The helper also posts the mouse-up before it clears the held-button record, so a signal landing between the two can no longer exit with the button down.

Pinned by two exec tests that use a trap script standing in for the helper (a cancelled command with a kill policy sees the signal and records its release; a child that ignores the first signal is still ended once the grace passes) and by the helper test asserting the policy the press hands the provider. A CI test cannot post real mouse events, so the up event itself is verified live below.

Live run at 14d1006 on the macOS host (session open --platform macos --surface frontmost-app on Finder; the desktop surface refuses press by policy, and the docs note now says so).

Probe. An AppKit window that logs every mouse down/up with the clickCount AppKit derived:

press 320 696 --count 2 --json               → {"count":2,"targetKind":"point"}
  down clickCount=1 / up 1 / down 1 / up 1
press 320 696 --double-tap --json            → {"doubleTap":true}
  down 1 / up 1 / down 2 / up 2
press 320 696 --double-tap --count 2 --json  → {"count":2,"doubleTap":true}
  down 1 / up 1 / down 2 / up 2 / down 1 / up 1 / down 2 / up 2

Finder list row "Opens On Double-Click". press --double-tap opens the folder (window retitles, marker.txt visible). press --count 2 --interval-ms 700 leaves the row selected and the window on finder-demo. One caveat worth knowing: press --count 2 at the default 120 ms interval also opened the folder, so Finder derives its double-click from timing, not from the click state the events carry; the AppKit probe above honours the click state. Two independent clicks that must not read as a double-click in Finder need an interval past the system double-click time.

Cancelled hold. press 320 696 --hold-ms 3000 with the client stopped one second in, button state read through CGEventSource.buttonState(.combinedSessionState, .left):

stop	1 s into hold	0.4 s after the stop	probe
Ctrl-C (SIGINT to the CLI's process group)	button DOWN, helper running	button up, helper gone	up 1 s after down
client killed (SIGKILL to the CLI's process group)	button DOWN, helper running	button up, helper gone	up 1 s after down

Without the fix the same run left the button down until the helper's own 3 s hold ended; a --hold-ms 10000 would have held it for the full ten seconds.

Recordings: macos-press-count-double-tap-cancel.mov (probe and Finder presses), macos-press-hold-cancelled.mov (both cancelled holds).

🤖 Generated with Claude Code

macos-press-count-double-tap-cancel.mov

macos-press-hold-cancelled.mov

[thymikee]

Reviewed at 14d1006. The earlier finding is fixed. Every host stop now reaches the helper as SIGTERM first, with SIGKILL one second later, and callers that set no kill policy still get an immediate SIGKILL through the same path. The helper also posts the mouse-up before it clears the held-button record. Your live run at 14d1006 covers --count 2, --double-tap, and a hold stopped by Ctrl-C and by killing the client. The deadline stop is covered by the new exec escalation test, which uses the same kill route.

Small notes, not blocking: the new CHANGELOG entry still lists desktop for press and click, while commands.md says desktop only inspects; the comment at helper.test.ts#L175 still says the host stops the helper with SIGKILL on both routes; and could the held record keep a prebuilt up event, so the signal handler only posts it and does not create a new event?

Smoke Tests fails at "wait for Agent Device Tester" after the daemon became unreachable in the iOS simulator run. That looks unrelated: no iOS or simctl caller sets the new kill policy, so that route does not reach the changed code. A rerun should confirm it.

There are no conflicts, and I added ready-for-human.