agent-device 0.21.0, macOS, Node 24.18. iPhone 17 Pro simulator (iOS 26.5, 3x) and Pixel_10_Pro emulator (API 37, 480 dpi). Found while dogfooding agent-device as a visual regression tool for react-native-paper (callstack/react-native-paper#5115).
A realistic UI regression, a Material Surface at elevation level 1 rendering the level-2 shadow (one step off), is reported as a perfect match by diff screenshot at the default threshold, on both platforms:
npx agent-device@0.21.0 diff screenshot --baseline baseline.png current.png --out diff.png --threshold 0.1 --json
→ { "totalPixels": 774252, "differentPixels": 0, "mismatchPercentage": 0, "match": true }
The same two images at --threshold 0.02:
| platform |
@ 0.1 |
@ 0.05 |
@ 0.02 |
@ 0.01 |
| iOS (1206x642 crop, 3x) |
0 px, match |
– |
10,179 px (1.31 %), 1 dominant region on the changed card |
– |
| Android (1280x642 crop, native) |
0 px, match |
1,830 px (0.22 %) |
9,336 px (1.14 %), 1 region |
14,686 px (1.79 %) |
The 0.02 result is the same on three independent captures per platform, and the noise floor of the same crop (four captures of an unchanged build, including a relaunch) is 0 px at every threshold down to 0.01. So the miss is the threshold, not capture noise. Diff image at 0.02: https://github.com/callstack/react-native-paper/raw/poc/agent-device-visual/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png
A gross break (every level forced to the level-5 shadow) is also under-reported at 0.1 on iOS: 4,277 px / 0.55 % in one thin band, versus 116,292 px / 15 % at 0.02, for a change that is obvious to the eye across the whole row.
Soft shadows and gradients are low-contrast by design, so a per-pixel colour distance of 0.1 filters them out entirely. Suggestions, any of: lower the default; document that 0.1 is unsuitable for anti-aliased or low-contrast content and that callers should sweep thresholds against a known break before trusting a green run; echo the threshold actually used in the JSON output (it is not today).
agent-device 0.21.0, macOS, Node 24.18. iPhone 17 Pro simulator (iOS 26.5, 3x) and Pixel_10_Pro emulator (API 37, 480 dpi). Found while dogfooding agent-device as a visual regression tool for react-native-paper (callstack/react-native-paper#5115).
A realistic UI regression, a Material
Surfaceat elevation level 1 rendering the level-2 shadow (one step off), is reported as a perfect match bydiff screenshotat the default threshold, on both platforms:The same two images at
--threshold 0.02:The 0.02 result is the same on three independent captures per platform, and the noise floor of the same crop (four captures of an unchanged build, including a relaunch) is 0 px at every threshold down to 0.01. So the miss is the threshold, not capture noise. Diff image at 0.02: https://github.com/callstack/react-native-paper/raw/poc/agent-device-visual/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png
A gross break (every level forced to the level-5 shadow) is also under-reported at 0.1 on iOS: 4,277 px / 0.55 % in one thin band, versus 116,292 px / 15 % at 0.02, for a change that is obvious to the eye across the whole row.
Soft shadows and gradients are low-contrast by design, so a per-pixel colour distance of 0.1 filters them out entirely. Suggestions, any of: lower the default; document that 0.1 is unsuitable for anti-aliased or low-contrast content and that callers should sweep thresholds against a known break before trusting a green run; echo the threshold actually used in the JSON output (it is not today).