diff --git a/example/src/Examples/SurfaceExample.tsx b/example/src/Examples/SurfaceExample.tsx index e941c7c843..4a745e341e 100644 --- a/example/src/Examples/SurfaceExample.tsx +++ b/example/src/Examples/SurfaceExample.tsx @@ -53,7 +53,7 @@ const SurfaceExample = () => { return ( - + { - + -realistic-1-t0.02.png`). To a human the change is borderline: noticeable only when flipping between the two images. + +## Web + +Tried once on 2026-09-14 against `expo export --platform web` of the example app, served locally, with agent-device's managed browser (`agent-device web setup`, agent-browser 0.27.1). `open` works and a full-page `screenshot` works (1280x577). `screenshot --crop-on 'id="surface-example-elevated"'` is refused: `UNSUPPORTED_OPERATION`, `CROP_TARGET_NOT_ACCEPTED`, `PENDING_PIXEL_IDENTITY_EVIDENCE`. Transcript in `evidence/web-excerpt.json`, the full-page capture in `evidence/diff-images/web-full-page-screenshot.png`. So web is not blocked, but it is a different loop, full-viewport diffs rather than per-section crops, and was not pursued for a Surface-only PoC. + +## Caveats + +- The crop is not perfectly isolated. Under the gross break a full-width 33 px band at the top of the crop changed: the Appbar is itself a `Surface`, and the break changed every Surface. The realistic break showed no bleed. An Appbar-only change could register against this crop; that is inherent in screenshotting real screens rather than isolated components. +- Dev-client chrome can land in the crop. The Expo dev-client's floating "Tools" button, switched on by a stray dev-menu press, sat inside the Android section and produced a deterministic 2,822 px false FAIL with `src/` clean (`evidence/diff-images/android-devclient-tools-button.png`, `evidence/devclient-excerpt.json`). Turning it off in the dev menu ("Tools button") restored 0. A release build would remove the whole class of dev menu, dev launcher and floating button. +- Not measured: swiftshader (what `ubuntu-latest` renders with), cold simulator/emulator boot, another host or day, runtime or image updates, other components, text-heavy crops, dark theme. + +## Reproducing by hand + +Prerequisites: the example app built and installed on the device pinned in `env.json`, Metro running (`yarn example start`). agent-device sessions are keyed by cwd and bound to one device, so run from one directory and give the second platform its own `--session`. The loop is the same on both platforms; only these values differ: + +| value | iOS | Android | +| -------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| `TARGET` | `--platform ios --udid 2464A356-C17C-4B0D-99DB-CDFBDB98826C` | `--platform android --session android` | +| `DENSITY` | `--pixel-density 3` | (none; Android returns native pixels) | +| overlay after `--relaunch` | dev menu, a few seconds after launch: `press 'label="Close"'` | dev launcher: `press 'label="http://:8081"'`, then wait for the bundle | +| `BASELINE` | `example/visual/__baselines__/ios` | `example/visual/__baselines__/android` | + +```bash +AD="npx agent-device@0.21.0" +$AD open com.callstack.reactnativepaperexample --relaunch $TARGET --json +$AD wait stable 500 10000 $TARGET --json +# dismiss the overlay for this platform (table above); the app restores its last screen, +# so if it is not the Surface example, press Back to the list and press the "Surface" row +$AD wait stable 500 10000 $TARGET --json && sleep 2 +$AD screenshot current.png --crop-on 'id="surface-example-elevated"' $DENSITY $TARGET --json +$AD diff screenshot --baseline $BASELINE/surface-example-elevated.png current.png --out diff.png --threshold 0.02 --json +``` + +Expect `differentPixels: 0`. To see a failure, change `shadow(elevation, …)` to `shadow(elevation === 1 ? 2 : elevation, …)` in `src/components/Surface.tsx` (iOS) or `androidElevationLevels[elevation]` to `androidElevationLevels[elevation === 1 ? 2 : elevation]` (Android), relaunch so the app fetches the bundle, capture, diff: about 10,179 px (iOS) or 9,336 px (Android) at 0.02, and `match: true` at 0.1. `git checkout -- src/components/Surface.tsx` afterwards. + +The same loop as one script for both platforms, with device pinning, overlay handling and exit codes, is `example/visual/run.mjs` on branch `poc/agent-device-visual-runner` (commit e9245dc97). It was removed from this PR only to keep the PoC small. + +## What a real runner has to handle + +Each learned while scripting the loop above; documented in `evidence/issues.md`, implemented on the runner branch: + +- Relaunch before every capture. Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS. +- The example app persists navigation state. A relaunch lands on the last screen, so the runner has to press Back to the list root and pick the Surface row, not the header title. +- Dev-client chrome: the dev menu appears seconds after the app is ready, the Android relaunch lands in the dev launcher, and the floating Tools button can appear inside the crop. +- `snapshot` returns an empty `nodes` array when nothing changed unless `--force-full` is passed. +- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default hides real regressions. +- Pin the device (UDID, runtime, density) and refuse to compare or re-baseline on anything else; derive expected capture size from the baseline PNG, per story. +- A plain `.mjs` entry guarded by `import.meta.main` does nothing on Node 20/22; guard with an `argv[1]` comparison. + +## agent-device issues + +Nineteen findings with commands and evidence in `evidence/issues.md`, none filed upstream. The four that matter most: the default diff threshold misses soft-shadow regressions (4), sessions are bound to one device per cwd (6), dev-client chrome renders into screenshots and the diff cannot tell (18), `--crop-on` is refused on web (19). + +## Evidence + +- `evidence/results.csv`: one row per `diff screenshot` run (72 rows): platform, capture, threshold, total and changed pixels, mismatch %, regions, match, and the name of the raw JSON it came from. The raw per-command JSON is on the runner branch. +- `evidence/diff-images/`: one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, the dev-client Tools-button false FAIL, and the full-page web capture. No 0.1 images exist because `diff screenshot --out` writes nothing on a match (issue 17). +- `evidence/a11y-excerpt.json`, `evidence/devclient-excerpt.json`, `evidence/web-excerpt.json`: the nodes and responses that matter; full trees on the runner branch. +- `env.json`: the pinned devices, versions and thresholds. + +The first-run rows predate the test-id rename and their source files say `surface-elevated` / `surface-flat`; renaming an id changes no pixel. diff --git a/example/visual/__baselines__/android/surface-example-elevated.png b/example/visual/__baselines__/android/surface-example-elevated.png new file mode 100644 index 0000000000..c94e2bc1b1 Binary files /dev/null and b/example/visual/__baselines__/android/surface-example-elevated.png differ diff --git a/example/visual/__baselines__/android/surface-example-flat.png b/example/visual/__baselines__/android/surface-example-flat.png new file mode 100644 index 0000000000..4f688b4e61 Binary files /dev/null and b/example/visual/__baselines__/android/surface-example-flat.png differ diff --git a/example/visual/__baselines__/ios/surface-example-elevated.png b/example/visual/__baselines__/ios/surface-example-elevated.png new file mode 100644 index 0000000000..5eb2aa499f Binary files /dev/null and b/example/visual/__baselines__/ios/surface-example-elevated.png differ diff --git a/example/visual/__baselines__/ios/surface-example-flat.png b/example/visual/__baselines__/ios/surface-example-flat.png new file mode 100644 index 0000000000..b909a990ea Binary files /dev/null and b/example/visual/__baselines__/ios/surface-example-flat.png differ diff --git a/example/visual/env.json b/example/visual/env.json new file mode 100644 index 0000000000..453b33ef14 --- /dev/null +++ b/example/visual/env.json @@ -0,0 +1,47 @@ +{ + "platform": "ios", + "device": "iPhone 17 Pro", + "udid": "2464A356-C17C-4B0D-99DB-CDFBDB98826C", + "iosVersion": "26.5", + "runtime": "com.apple.CoreSimulator.SimRuntime.iOS-26-5", + "scale": 3, + "pixelDensity": 3, + "agentDeviceVersion": "0.21.0", + "reactNativeVersion": "0.85.3", + "branch": "poc/agent-device-visual", + "date": "2026-09-10T09:30:58Z", + "baselines": { + "surface-example-elevated": "__baselines__/ios/surface-example-elevated.png", + "surface-example-flat": "__baselines__/ios/surface-example-flat.png" + }, + "diffThresholds": { + "cliDefault": 0.1, + "usedForShadowRegressions": 0.02 + }, + "android": { + "avd": "Pixel_10_Pro", + "device": "Pixel 10 Pro", + "serial": "emulator-5554", + "apiLevel": 37, + "androidRelease": "17", + "systemImage": "system-images/android-37.0/google_apis_playstore_ps16k/arm64-v8a/", + "density": 480, + "screenPx": "1280x2856", + "emulatorVersion": "36.6.11.0 (build_id 15507667)", + "gpuMode": "auto (hw.gpu.enabled=yes, hw.gpu.mode=auto; swiftshader NOT measured)", + "agentDeviceVersion": "0.21.0", + "agentDeviceSession": "android", + "reactNativeVersion": "0.85.3", + "metroHost": "http://192.168.1.151:8081", + "date": "2026-09-10T09:45:00Z", + "baselines": { + "surface-example-elevated": "__baselines__/android/surface-example-elevated.png", + "surface-example-flat": "__baselines__/android/surface-example-flat.png", + "pixelDensityFlag": "unsupported on Android (screenshots are native device pixels)" + }, + "diffThresholds": { + "cliDefault": 0.1, + "usedForShadowRegressions": 0.02 + } + } +} diff --git a/example/visual/evidence/a11y-excerpt.json b/example/visual/evidence/a11y-excerpt.json new file mode 100644 index 0000000000..d2d253df10 --- /dev/null +++ b/example/visual/evidence/a11y-excerpt.json @@ -0,0 +1,74 @@ +{ + "about": "Excerpts from the XCUITest / UiAutomator trees captured 2026-09-10 with `npx agent-device@0.21.0 snapshot --raw --force-full --json`. The full trees (13,500 lines) are on branch poc/agent-device-visual-runner under example/visual/evidence/a11y/. Test id at the time: surface-elevated (renamed surface-example-elevated later; the rename changes no pixel).", + "ios": { + "with_accessible": { + "total_nodes": 115, + "elevation_labelled_nodes": 2, + "surface_section_node": [ + { + "type": "Other", + "role": "RCTViewComponentView", + "identifier": "surface-elevated", + "label": "Elevated surface, Elevation 0, Elevation 1, Elevation 2, Elevation 3, Elevation 4, Elevation 5, Vertical scroll bar, 3 pages", + "rect": { + "x": 0, + "y": 134, + "width": 402, + "height": 214 + } + } + ] + }, + "testID_only": { + "total_nodes": 177, + "elevation_labelled_nodes": 26, + "surface_section_node": [ + { + "type": "Other", + "role": "RCTViewComponentView", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 134, + "width": 402, + "height": 214 + } + } + ] + } + }, + "android": { + "with_accessible": { + "total_nodes": 217, + "elevation_labelled_nodes": 13, + "surface_section_node": [ + { + "type": "android.view.ViewGroup", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 372, + "width": 1280, + "height": 642 + } + } + ] + }, + "testID_only": { + "total_nodes": 217, + "elevation_labelled_nodes": 13, + "surface_section_node": [ + { + "type": "android.view.ViewGroup", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 372, + "width": 1280, + "height": 642 + } + } + ] + } + } +} \ No newline at end of file diff --git a/example/visual/evidence/devclient-excerpt.json b/example/visual/evidence/devclient-excerpt.json new file mode 100644 index 0000000000..33a0499075 --- /dev/null +++ b/example/visual/evidence/devclient-excerpt.json @@ -0,0 +1,42 @@ +{ + "about": "Excerpts captured 2026-09-11 with `snapshot --force-full --json`; full files on branch poc/agent-device-visual-runner under example/visual/evidence/runner/.", + "ios_after_open_relaunch": { + "immediately": { + "total_nodes": 3, + "labels": [ + "React Native Paper Example", + null, + "Downloading 100%\u2026" + ] + }, + "six_seconds_later": { + "total_nodes": 77, + "dev_menu_close_node": [ + { + "type": "Button", + "role": "SwiftUI.AccessibilityNode", + "identifier": "xmark", + "label": "Close", + "rect": { + "x": 344.06965174129357, + "y": 401.5837479270315, + "width": 34.56716417910445, + "height": 34.56716417910445 + } + } + ] + } + }, + "android_devclient_floating_tools_button": [ + { + "type": "android.widget.ImageView", + "label": "Tools", + "rect": { + "x": 1115, + "y": 243, + "width": 78, + "height": 78 + } + } + ] +} \ No newline at end of file diff --git a/example/visual/evidence/diff-images/android-devclient-tools-button.png b/example/visual/evidence/diff-images/android-devclient-tools-button.png new file mode 100644 index 0000000000..856b41c00b Binary files /dev/null and b/example/visual/evidence/diff-images/android-devclient-tools-button.png differ diff --git a/example/visual/evidence/diff-images/android-gross-t0.02.png b/example/visual/evidence/diff-images/android-gross-t0.02.png new file mode 100644 index 0000000000..a6f157ad1a Binary files /dev/null and b/example/visual/evidence/diff-images/android-gross-t0.02.png differ diff --git a/example/visual/evidence/diff-images/android-realistic-1-t0.02.png b/example/visual/evidence/diff-images/android-realistic-1-t0.02.png new file mode 100644 index 0000000000..5e6290efe8 Binary files /dev/null and b/example/visual/evidence/diff-images/android-realistic-1-t0.02.png differ diff --git a/example/visual/evidence/diff-images/ios-gross-t0.02.png b/example/visual/evidence/diff-images/ios-gross-t0.02.png new file mode 100644 index 0000000000..79e8aa8658 Binary files /dev/null and b/example/visual/evidence/diff-images/ios-gross-t0.02.png differ diff --git a/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png b/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png new file mode 100644 index 0000000000..02219e89f4 Binary files /dev/null and b/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png differ diff --git a/example/visual/evidence/diff-images/web-full-page-screenshot.png b/example/visual/evidence/diff-images/web-full-page-screenshot.png new file mode 100644 index 0000000000..496bc7a782 Binary files /dev/null and b/example/visual/evidence/diff-images/web-full-page-screenshot.png differ diff --git a/example/visual/evidence/issues.md b/example/visual/evidence/issues.md new file mode 100644 index 0000000000..0d09a2be23 --- /dev/null +++ b/example/visual/evidence/issues.md @@ -0,0 +1,43 @@ +# agent-device findings from the PoC, verified + +Nineteen observations were recorded while using agent-device 0.21.0 (`npx agent-device@0.21.0`) as a visual regression tool on the example app, on an iPhone 17 Pro simulator (iOS 26.5) and a `Pixel_10_Pro` emulator (API 37). On 2026-09-15 every one was re-checked against the 0.21.3 docs and source, the emulator, and the upstream issue tracker, with one question: is this agent-device's fault, or ours? + +Result: **none was an unreported agent-device bug.** One was a real ergonomic gap and has since been fixed upstream. Two are documented behaviour worth knowing. The rest were our own build, our runner, React Native, or claims that did not survive re-testing. They are kept below because most of them are things anyone scripting this loop will hit. Original numbers are in brackets so the README references still resolve. + +## What holds up + +**The default diff threshold is too loose for shadows; we use 0.02 [4].** Documented, deterministic behaviour, not a defect: `--threshold` "sets the per-pixel RGB tolerance (default 0.1)", and the implementation maps it to a colour distance of about 44 RGB units. A one-step Material elevation change moves a shadow by a few units per channel, so at 0.1 the realistic Surface regression diffs to 0 changed pixels and `match: true` on both platforms, while 0.02 catches it (10,179 px on iOS, 9,336 px on Android, identical on three captures each, noise floor 0 at every threshold down to 0.01). Full numbers in `results.csv`, images in `diff-images/`. Any suite built on this must set the threshold explicitly and verify noise at it. One small upstream ask survives: the diff result JSON does not record which threshold produced it, so a saved diff cannot say what tolerance it was run at. + +**Implicit sessions were keyed by cwd only [6], fixed upstream.** Driving iOS and Android from one checkout needed a hand-named `--session` on every command. Reported as callstack/agent-device#2580 and fixed: the unreleased changelog entry keys the implicit session by workspace and platform (not yet released as of 0.21.3). Our iOS commands also omitted `--platform`, which parked the session on the platform-less default and made the conflict worse. + +**`diff screenshot --out` writes the image only when pixels differ [17].** On a match it writes nothing and deletes any stale file at that path, so a previous run's diff can never be mistaken for the current one. `diffPath` in the response is the signal, not file existence. The docs say it "writes a diff PNG when `--out` is provided" without the on-mismatch qualifier; that is the only remark. + +**`--crop-on` is refused on web [19].** Documented: accepted on iOS simulators and Android emulators only, refused before any device work (`CROP_TARGET_NOT_ACCEPTED`). `open` and full-page `screenshot` work on web (`web-excerpt.json`). A web leg would be full-viewport diffs, a different loop, with page motion frozen since it cannot be cropped out. + +## Our build and our runner, not the tool + +**Expo dev-client chrome ends up in screenshots [3, 9, 14, 18].** A dev build cold-starts into the Expo dev launcher on Android and shows the dev menu on iOS a few seconds after the bundle loads. Reproduced with plain `adb shell am force-stop` and `monkey`, no agent-device involved. The dev-client's floating "Tools" button, toggled by the dev menu entry "Tools button", sat inside the Android crop and produced a deterministic 2,822 px false FAIL with `src/` clean (`diff-images/android-devclient-tools-button.png`, `devclient-excerpt.json`). agent-device already flags the loading state: `wait stable` returned "Settled on a nearly-empty tree, the app may still be loading. Wait for specific content." A runner must reconnect to the dev server after any relaunch, wait for real content, dismiss the menu, and check for the floating button. A release build removes the whole class. + +**Fast Refresh did not reach the Android app after a relaunch [15].** Metro served the new bundle; the app kept the old one until a Reload. Metro and Expo behaviour, and agent-device documents the remedy we did by hand through the dev menu: `agent-device metro reload`, "before `open --relaunch`". A runner should use that, never Fast Refresh. + +**The example app restores its last route on relaunch [13].** `PERSISTENCE_KEY = 'NAVIGATION_STATE'` in `example/src/index.tsx`. A runner cannot assume the list is at the top after `open --relaunch`; it has to press Back to the root and pick the "Surface" row, not the header title with the same label. + +**Android exposes the same label on a row and its text child [7].** React Native's accessibility output, visible on every example row and on the Expo launcher itself, so `find 'label="Surface"'` is `AMBIGUOUS_MATCH` there and unique on iOS. agent-device documents the disambiguators: `--first`, `--last`, a `role=` qualifier, and the read-only `list` action that returns the candidates without tapping. + +**`scroll bottom` hit its pass limit on the example list [2].** A documented, bounded edge-seek that refuses to swipe forever; the dev menu was on screen absorbing the gestures, which is also why the edge was never reached. Use `scroll down 6 --settle`, or `scroll --until ` on 0.21.1 and later. + +## Withdrawn after re-verification + +These were recorded as agent-device problems and are not. + +- **`find` taps its match [1].** `find` is documented as `find ` with `click` as the default action and a read-only `list` action that never taps. Upstream #1625 raised the same complaint and was closed in August with `list` as the answer. We ran bare `find` and got the documented default. +- **No timing in `screenshot --json` [5].** Per-command durations are in the session event log (`agent-device events`, `request.finished` with `durationMs`); `open` returns `startup.durationMs` as well. +- **Stale refs rejected with a confusing message [8].** The response carries `hint: "Capture a fresh interactive snapshot (snapshot -i) or use a stable selector, then retry."`, a reason code, and the set of refs the frame does accept. The message says exactly what to do. +- **`logs` reports success on an inactive log [10].** Bare `logs` is documented as an info query; `success: true` means the query ran, and `active`, `state` and `sizeBytes` are the answer. Logging is documented as off until `logs start`. +- **`--pixel-density` is iOS-only [11].** Documented in the CLI help; Android returns native pixels anyway. +- **`snapshot` returns empty `nodes` when the tree is unchanged [12].** Not reproducible: two consecutive `snapshot --json` returned the full 57 nodes both times, and the source never empties `nodes`, only adds an `unchanged` metadata field. The text-mode "unchanged" acknowledgement is documented. The empty result we saw most likely came from a `--scope` that matched nothing. +- **Occasional exit 1 with empty stdout and stderr [16].** Two unlogged occurrences on iOS; 0 of 40 rapid calls on Android reproduced it; nothing upstream. Not supportable. + +## Filing history + +Five of these were filed on callstack/agent-device on 2026-09-14 (#2579 to #2583) and withdrawn the same day. One (#2581, the Node client "unreachable" without a dependency) was simply wrong and a maintainer closed it within the hour; it has been removed from this file. #2580 was fixed upstream regardless. Nothing else is filed, and nothing will be without a maintainer of this repo deciding to. diff --git a/example/visual/evidence/results.csv b/example/visual/evidence/results.csv new file mode 100644 index 0000000000..5e8efe206c --- /dev/null +++ b/example/visual/evidence/results.csv @@ -0,0 +1,73 @@ +platform,group,capture,threshold,totalPixels,differentPixels,mismatchPercent,regions,match,source +ios,first-run,a-vs-b (density 1),0.1,86028,0,0,0,True,evidence/ios/26-diff.json +ios,first-run,flat a-vs-b (density 1),0.1,86028,0,0,0,True,evidence/ios/30-diff.json +ios,first-run,warm-1,0.1,774252,0,0,0,True,evidence/ios/37-diff.json +ios,first-run,warm-2,0.1,774252,0,0,0,True,evidence/ios/40-diff.json +ios,first-run,warm-3,0.1,774252,0,0,0,True,evidence/ios/43-diff.json +ios,first-run,relaunch-1,0.1,774252,0,0,0,True,evidence/ios/53-diff.json +ios,first-run,gross,0.1,774252,4277,0.55,1,False,evidence/ios/59-diff.json +ios,first-run,gross,0.1,774252,116292,15.02,3,False,evidence/ios/60-diff.json +ios,first-run,revert-1,0.1,774252,0,0,0,True,evidence/ios/63-diff.json +ios,first-run,realistic,0.02,774252,10179,1.31,3,False,evidence/ios/66-diff-t0.02.json +ios,first-run,realistic,0.1,774252,0,0,0,True,evidence/ios/66-diff-t0.1.json +ios,first-run,revert-2,0.1,774252,0,0,0,True,evidence/ios/69-diff.json +ios,first-run,relaunch-1,0.02,774252,0,0,0,True,evidence/ios/6x-diff-relaunch-1-t002.json +ios,first-run,warm-1,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-1-t002.json +ios,first-run,warm-2,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-2-t002.json +ios,first-run,warm-3,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-3-t002.json +android,first-run,relaunch-1,0.01,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.01.json +android,first-run,relaunch-1,0.02,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.02.json +android,first-run,relaunch-1,0.05,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.05.json +android,first-run,relaunch-1,0.1,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.1.json +android,first-run,gross,0.01,821760,265916,32.36,2,False,evidence/android/108-diff-gross-t0.01.json +android,first-run,gross,0.02,821760,208505,25.37,4,False,evidence/android/108-diff-gross-t0.02.json +android,first-run,gross,0.05,821760,118661,14.44,4,False,evidence/android/108-diff-gross-t0.05.json +android,first-run,gross,0.1,821760,65051,7.92,2,False,evidence/android/108-diff-gross-t0.1.json +android,first-run,revert1,0.01,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.01.json +android,first-run,revert1,0.02,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.02.json +android,first-run,revert1,0.05,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.05.json +android,first-run,revert1,0.1,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.1.json +android,first-run,realistic,0.01,821760,14686,1.79,1,False,evidence/android/114-diff-realistic-t0.01.json +android,first-run,realistic,0.02,821760,9336,1.14,1,False,evidence/android/114-diff-realistic-t0.02.json +android,first-run,realistic,0.05,821760,1830,0.22,1,False,evidence/android/114-diff-realistic-t0.05.json +android,first-run,realistic,0.1,821760,0,0,0,True,evidence/android/114-diff-realistic-t0.1.json +android,first-run,revert2,0.01,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.01.json +android,first-run,revert2,0.02,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.02.json +android,first-run,revert2,0.05,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.05.json +android,first-run,revert2,0.1,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.1.json +android,first-run,warm-1,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.01.json +android,first-run,warm-1,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.02.json +android,first-run,warm-1,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.05.json +android,first-run,warm-1,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.1.json +android,first-run,warm-2,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.01.json +android,first-run,warm-2,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.02.json +android,first-run,warm-2,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.05.json +android,first-run,warm-2,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.1.json +android,first-run,warm-3,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.01.json +android,first-run,warm-3,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.02.json +android,first-run,warm-3,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.05.json +android,first-run,warm-3,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.1.json +ios,re-measured,gross,0.02,774252,116292,15.02,3,False,evidence/sensitivity/ios/gross-diff-t0.02.json +ios,re-measured,gross,0.1,774252,4277,0.55,1,False,evidence/sensitivity/ios/gross-diff-t0.1.json +ios,re-measured,realistic-1,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-1-diff-t0.02.json +ios,re-measured,realistic-1,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-1-diff-t0.1.json +ios,re-measured,realistic-2,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-2-diff-t0.02.json +ios,re-measured,realistic-2,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-2-diff-t0.1.json +ios,re-measured,realistic-3,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-3-diff-t0.02.json +ios,re-measured,realistic-3,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-3-diff-t0.1.json +ios,re-measured,reverted,0.02,774252,0,0,0,True,evidence/sensitivity/ios/reverted-diff-t0.02.json +ios,re-measured,reverted,0.1,774252,0,0,0,True,evidence/sensitivity/ios/reverted-diff-t0.1.json +android,re-measured,gross,0.02,821760,208505,25.37,4,False,evidence/sensitivity/android/gross-diff-t0.02.json +android,re-measured,gross,0.1,821760,65051,7.92,2,False,evidence/sensitivity/android/gross-diff-t0.1.json +android,re-measured,gross-probe,0.02,821760,0,0,0,True,evidence/sensitivity/android/gross-probe-diff-t0.02.json +android,re-measured,gross-probe,0.1,821760,0,0,0,True,evidence/sensitivity/android/gross-probe-diff-t0.1.json +android,re-measured,realistic-1,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-1-diff-t0.02.json +android,re-measured,realistic-1,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-1-diff-t0.1.json +android,re-measured,realistic-2,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-2-diff-t0.02.json +android,re-measured,realistic-2,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-2-diff-t0.1.json +android,re-measured,realistic-3,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-3-diff-t0.02.json +android,re-measured,realistic-3,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-3-diff-t0.1.json +android,re-measured,reverted,0.02,821760,0,0,0,True,evidence/sensitivity/android/reverted-diff-t0.02.json +android,re-measured,reverted,0.1,821760,0,0,0,True,evidence/sensitivity/android/reverted-diff-t0.1.json +android,re-measured,sanity-before-break,0.02,821760,0,0,0,True,evidence/sensitivity/android/sanity-before-break-diff-t0.02.json +android,re-measured,sanity-before-break,0.1,821760,0,0,0,True,evidence/sensitivity/android/sanity-before-break-diff-t0.1.json diff --git a/example/visual/evidence/web-excerpt.json b/example/visual/evidence/web-excerpt.json new file mode 100644 index 0000000000..35f8593766 --- /dev/null +++ b/example/visual/evidence/web-excerpt.json @@ -0,0 +1,33 @@ +{ + "about": "agent-device 0.21.0, --platform web (managed agent-browser 0.27.1, Node 24.18), against `expo export --platform web` of the example app served on 127.0.0.1:4322, 2026-09-14.", + "open": { + "success": true, + "device": "Agent Browser Chrome", + "startupMs": 1576 + }, + "screenshot_crop_on": { + "command": "screenshot web-crop.png --crop-on 'id=\"surface-example-elevated\"' --platform web --session web --json", + "success": false, + "error": { + "code": "UNSUPPORTED_OPERATION", + "message": "screenshot --crop-on is not accepted on web targets", + "hint": "This command is not available for the selected platform/device.", + "diagnosticId": "mu0zgqxl-62c0defe", + "logPath": "/Users/juliankobrynski/.agent-device/sessions/web/requests/4f9d41985f52c500.ndjson", + "diagnosticsRecord": { + "session": "web", + "requestId": "4f9d41985f52c500" + }, + "details": { + "reason": "CROP_TARGET_NOT_ACCEPTED", + "rejectionReason": "PENDING_PIXEL_IDENTITY_EVIDENCE" + } + } + }, + "screenshot_plain": { + "command": "screenshot web-full.png --platform web --session web --json", + "success": true, + "width": 1280, + "height": 577 + } +}