agent-device 0.21.0, Pixel_10_Pro emulator (API 37, 480 dpi), Expo 56 dev-client of the react-native-paper example app (callstack/react-native-paper#5115).
A visual regression loop that had passed all morning started failing deterministically, twice in a row, with the app source clean and the baseline byte-identical to the committed one:
surface-example-elevated changed=2822 (0.34%) regions=1 threshold=0.02 → FAIL
region: {x:1080, y:2, w:152, h:74}, top-right, "darker"
The diff image shows the Expo dev-client's floating Tools button inside the cropped section: https://github.com/callstack/react-native-paper/raw/poc/agent-device-visual/example/visual/evidence/diff-images/android-devclient-tools-button.png. In the accessibility tree it is an android.widget.ImageView labelled Tools, rect {x:1115, y:243, w:78, h:78}, under the app's own package. It is toggled by the dev menu entry "Tools button" and had been switched on by some earlier dev-menu interaction; toggling it off made the next run pass at 0 px, and the setting survives relaunch.
Nothing in screenshot or diff screenshot distinguishes dev-client chrome from the app, so a dev build can fail a visual diff for reasons that have nothing to do with the app. Related, also seen in this PoC: open --relaunch restarts the app with the Expo dev menu open (iOS) or in the Expo dev launcher (Android), and the dev menu appears a few seconds after the app tree is otherwise ready, so a single overlay check right after open misses it.
Suggestions: a note in the docs that dev-client overlays (dev menu, dev launcher, floating button) render into screenshots and how to dismiss each; or a screenshot option / open flag that hides or dismisses known dev-client chrome; or at least a diff screenshot hint when a changed region coincides with a node that belongs to a known dev-client overlay.
agent-device 0.21.0, Pixel_10_Pro emulator (API 37, 480 dpi), Expo 56 dev-client of the react-native-paper example app (callstack/react-native-paper#5115).
A visual regression loop that had passed all morning started failing deterministically, twice in a row, with the app source clean and the baseline byte-identical to the committed one:
The diff image shows the Expo dev-client's floating Tools button inside the cropped section: https://github.com/callstack/react-native-paper/raw/poc/agent-device-visual/example/visual/evidence/diff-images/android-devclient-tools-button.png. In the accessibility tree it is an
android.widget.ImageViewlabelledTools, rect{x:1115, y:243, w:78, h:78}, under the app's own package. It is toggled by the dev menu entry "Tools button" and had been switched on by some earlier dev-menu interaction; toggling it off made the next run pass at 0 px, and the setting survives relaunch.Nothing in
screenshotordiff screenshotdistinguishes dev-client chrome from the app, so a dev build can fail a visual diff for reasons that have nothing to do with the app. Related, also seen in this PoC:open --relaunchrestarts the app with the Expo dev menu open (iOS) or in the Expo dev launcher (Android), and the dev menu appears a few seconds after the app tree is otherwise ready, so a single overlay check right afteropenmisses it.Suggestions: a note in the docs that dev-client overlays (dev menu, dev launcher, floating button) render into screenshots and how to dismiss each; or a
screenshotoption /openflag that hides or dismisses known dev-client chrome; or at least adiff screenshothint when a changed region coincides with a node that belongs to a known dev-client overlay.