Skip to content

test(example): agent-device visual regression PoC for Surface - #5115

Draft
JKobrynski wants to merge 8 commits into
mainfrom
poc/agent-device-visual
Draft

test(example): agent-device visual regression PoC for Surface#5115
JKobrynski wants to merge 8 commits into
mainfrom
poc/agent-device-visual

Conversation

@JKobrynski

@JKobrynski JKobrynski commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Our component tests assert style props and tree snapshots; they cannot tell whether a component looks right. This is a proof of concept for screenshot-based visual regression tests using agent-device against the existing example-app screens, scoped to Surface since most other components are mid-rework.

It answers one question: does agent-device capture deterministically enough to diff Surface against a baseline, and does that diff catch a realistic regression?

Result: yes on iOS and Android, with one catch. Noise floor is 0 px across warm captures and a relaunch. A one-level shadow regression (elevation 1 rendered as 2) is caught at --threshold 0.02 with the same pixel count on three captures per platform, but at agent-device's default threshold (0.1) it is reported as a perfect match on both. Web: open and full-page screenshots work, --crop-on is refused, so a web leg would be a different loop. Full numbers in example/visual/README.md; every number resolves to a row in example/visual/evidence/results.csv.

Screenshots

Realistic break at 0.02, iOS and Android (red = changed pixels, the ring on the Elevation 1 card):

The one false FAIL of the PoC: the Expo dev-client's floating Tools button inside the Android crop, with the app unchanged:

Web, full-page capture (per-section crop refused):

Changes

  • example/src/Examples/SurfaceExample.tsx: testID on the two List.Sections so screenshot --crop-on can target them. Only app change; no library code touched.
  • example/visual/README.md: results, caveats, one reproduction sequence for both platforms, and what a real runner has to handle.
  • example/visual/env.json: the pinned devices and versions the baselines are valid for.
  • example/visual/__baselines__/{ios,android}/: four PNGs.
  • example/visual/evidence/: results.csv (72 diff rows), the diff images, three small tree/response excerpts, and issues.md with 20 agent-device findings. The raw per-command JSON is on the runner branch.

Scope

A PoC, kept one. The single-script runner for both platforms (device pinning, relaunch, overlay handling, exit codes, tests) exists and is verified, on branch poc/agent-device-visual-runner @ e9245dc. It was removed from this PR to keep it reviewable; the README points at it.

Dogfooding

Twenty agent-device findings with commands and evidence in example/visual/evidence/issues.md. None filed upstream yet.

Test plan

"Reproducing by hand" in the README, verified 2026-09-10 to 14 on the pinned iPhone 17 Pro simulator and Pixel_10_Pro emulator. Expect differentPixels: 0 against the committed baselines; apply the one-line Surface break described there, relaunch, and expect about 10,179 px (iOS) / 9,336 px (Android) at 0.02 and match: true at 0.1.

Out of scope

A web loop, CI, other components, dark theme, cross-platform comparison, the runner script.

@JKobrynski
test(example): add agent-device visual regression PoC for Surface
32391c4 
Proof of concept for screenshot-based visual regression testing using
agent-device against the existing example screens, scoped to Surface.

- example/visual/run.mjs: one command per platform (iOS simulator,
  Android emulator) that opens the example app, navigates to the Surface
  screen, crops the "Elevated surface" / "Flat surface" sections, and
  diffs them against committed baselines. Exits 1 on regression;
  --update re-baselines.
- example/visual/README.md: results. Noise floor is 0 px on both
  platforms across warm captures and a relaunch. agent-device's default
  diff threshold (0.1) misses a realistic one-level shadow regression on
  both platforms; 0.02 catches it with noise still at 0.
- example/visual/env.json: pinned device/OS/density/versions the
  baselines are valid for.
- SurfaceExample: testID + accessible on the two List.Sections so
  `screenshot --crop-on` can target them.

Twelve agent-device dogfooding issues were recorded locally for filing.
@JKobrynski
test(example): address review on the visual regression PoC
e4f0311 
- Drop `accessible` from the two List.Sections. With it, iOS collapsed each
  section into a single accessibility element (XCUITest tree evidence in
  evidence/a11y/). `testID` alone resolves `screenshot --crop-on` on both
  platforms with identical rects, so the example app's accessibility tree
  is unchanged by the hook.
- Rename the test ids to surface-example-{elevated,flat} so they cannot be
  confused with the library defaults removed in #5088 / #5099. Baselines
  renamed to __baselines__/<platform>/<story>.png on both platforms.
- Commit the evidence: raw diff JSON behind every table, break diff images,
  issues.md, the a11y trees, and the final script runs.
- run.mjs: env.json mismatch and crop-size mismatch are hard failures
  (exit 2 / 3) unless --force; --update can create a missing baseline;
  always relaunch so the bundle is fresh (Fast Refresh was seen to stop
  reaching Android); handle the Expo dev menu and dev launcher; because
  the example app persists navigation state, go Back to the list root and
  press the Surface row by position, not the header by label; --force-full
  on every snapshot; summary.json written on every exit; --out.
- Re-measure the realistic break three times per platform at pre-declared
  thresholds: bit-identical (iOS 10,179 px, Android 9,336 px at 0.02;
  0 at 0.1) and identical to the original single captures.
- README: status, evidence links, a11y section, re-measurement section,
  issues 13-18, updated Running-it.
@JKobrynski
test(example): address round-2 review on the visual regression PoC
63e1699 
- Entry point: replace `import.meta.main` (Node >= 24.2 only) with a
  realpath comparison of argv[1] and the module URL. On Node 20/22 the
  script previously printed nothing and exited 0. Verified on 20.20.2.
- `--update` skips the capture-size check and prints the captured
  dimensions, so a new story can get a baseline; regression test added.
- `--force` covers the device check only; a size mismatch in diff mode
  has no override.
- Remove the dev-launcher geometric fallback, the unreachable size-check
  branch, the duplicate iOS version comparison and unused exports.
- List-root detection requires the "Examples" title; argument errors
  write a summary.json too.
- Strip npm_config_* from the spawned environment so the nested
  `npx agent-device` works when the script itself runs under `npx -p`.
- run.test.mjs (node:test, 6 cases) and evidence/summarize.mjs, the
  committed generator behind evidence/summary.md.
- Evidence: drop the 12 wait.json files and two full-tree snapshots;
  add the round-2 runs on Node 24 and Node 20.
- README: Android section no longer credits `accessible`; Node
  requirement, test and generator documented; env.json drops the
  parent-commit pin.
@JKobrynski
test(example): address round-3 review on the visual regression PoC
e9245dc 
- Evidence citations: every path in evidence/issues.md now resolves to a
  committed file. Issue 15 re-captured (evidence/runner/ios-*), issues 4
  and 14 rewritten onto committed evidence; the two lost logs are stated
  as not kept.
- Crop size is checked against the baseline PNG's own header, per story,
  so `--update` on a new story is followed by a passing run; env.json
  drops the per-platform crop pins and totalPixels.
- Drop the arg-error summary machinery; an argument error prints and
  exits 2.
- Device observers are injected; the --force test no longer shells out.
  `yarn example test:visual` runs the seven node:test cases; README says
  it is manual, not Jest/hook/CI. README Node version is 20+ throughout.
- Guardrails triggered for real: exit 2 on a UDID that is not attached,
  exit 3 on an 804x428 capture against the 3x baseline
  (evidence/runs/ios/round3-exit*).
- New: the Expo dev-client's floating "Tools" button inside the crop
  produced a deterministic false FAIL on Android with src clean
  (issue 19). The runner now detects it, disables it via the dev menu,
  or stops with exit 2; evidence in evidence/runner/android-* and
  evidence/runs/android/round3-*.
@JKobrynski
test(example): cut the visual regression PoC back to PoC scope
e59fd2e 
Three review rounds turned the reproduction script into a 1,200-line
runner and the evidence folder into 29,000 lines of accessibility-tree
dumps. Neither is what a proof of concept needs. This commit keeps:

- the results (README), the pinned environment, the four baselines and
  the two test ids;
- the diff JSON every table is computed from, the diff images, and the
  19 agent-device issues with their evidence;
- two small excerpts replacing the full accessibility-tree snapshots,
  each saying how to regenerate the full tree;
- hand-run CLI commands to reproduce, instead of a script.

Removed from this PR: run.mjs, run.test.mjs, evidence/summarize.mjs and
summary.md, evidence/runs, evidence/runner, the full evidence/a11y
trees, and the example/package.json test script. All of it is preserved
unchanged on branch poc/agent-device-visual-runner, and the README
lists what a real runner would have to handle, learned from it.
@JKobrynski
test(example): collapse PoC evidence into one results table
7917bdf 
96 per-threshold diff JSON files, each ~20 lines, become one
evidence/results.csv with 72 rows (platform, capture, threshold,
pixels, regions, match, source file). Keep one diff image per platform
for the realistic and gross breaks plus the Tools-button false FAIL.
Raw JSON and the other images stay on poc/agent-device-visual-runner.
This was referenced Sep 14, 2026
Closed
Closed
Closed
Closed
Closed
@JKobrynski
test(example): validate the PoC against the ask and close the gaps
122a82c 
- Web: actually tried. open and full-page screenshot work; --crop-on is
  refused (CROP_TARGET_NOT_ACCEPTED / PENDING_PIXEL_IDENTITY_EVIDENCE).
  Transcript in evidence/web-excerpt.json, issue 20.
- One reproduction sequence for both platforms with a table of the
  platform-specific values, instead of two diverging command blocks;
  the Android dev-launcher step is now in it. Points at the runner
  branch as the single-script form.
- Five issues filed on callstack/agent-device (#2579-#2583) and linked
  from issues.md.
- No em dashes; three self-certifying phrases removed.
@JKobrynski
test(example): drop references to upstream issues that were withdrawn
cf143f9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JKobrynski