fix: GTAO checkerboard at alpha-cutout silhouettes (white snow-dots on close leaves)

[proggeramlug]

Symptom

White snow-dots sprinkled over tree canopies whenever the camera gets close to leaves (worst at renderScale 0.5, where TSR scatters the pattern). Reported in the shooter; bisected there with a stationary in-canopy capture: SSAO off = clean, TAA off = raw 50% checkerboard over the whole near leaf card.

Root cause

The GTAO quad phase dither (the fix for the periodic wall-wide AO pulse) splits the 8-direction scan across 4 frames and staggers the phase in a 2x2 pixel quad, relying on the bilateral blur + EMA to absorb the 2x2 noise. At alpha-cutout silhouettes both absorbers fail simultaneously: the blur edge-stops on the leaf-to-background depth delta, and wind sway keeps history reprojecting onto what was sky/another card last frame. The inter-phase disagreement shows through as a stationary checkerboard at every leaf edge.

Fix

Adaptive full scan: when a pixel detects the failure signature — its own two directions disagree by > 0.3 visibility (thin-cutout anisotropy), or it sits in the near field (< 1.5 m, where the screen-radius clamp makes per-direction variance extreme) — it scans all 8 directions this frame. Full-estimate pixels have nothing left to disagree with their quad neighbours, so the checkerboard is gone by construction. Flat walls/ground/grass keep the 2-direction path. The march is factored into scan_direction() with unchanged math.

Verification

• Stationary CAMHOLD captures, 4x zoom crops: edge lattice gone at renderScale 1, snow-dots gone at 0.5 (before/after like-for-like).
• Wall-flicker regression: the quad dither + EMA clamp path is untouched for agreeing-direction pixels (walls/ground take the exact same code path as before).
• Cost in the pathological all-foliage frame at 0.5 scale on the 760M: 38 → 34 fps; normal frames pay only on cutout-edge pixels.

https://claude.ai/code/session_018574BdCfSjdpNK3WLgx1hP

Summary by CodeRabbit

• Performance Improvements
  • Improved ambient occlusion rendering efficiency by reusing sampling calculations.
  • Added adaptive sampling to focus additional detail where visual variation or near-field geometry requires it.
  • Improved horizon-based visibility calculations, including more accurate normalization across sampled directions.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9d7e3c5a-f04b-416f-b4db-0fbfcb4e4346

📥 Commits

Reviewing files that changed from the base of the PR and between 66b1964 and 4b98865.

📒 Files selected for processing (1)

• native/shared/src/renderer/shaders/ao.rs

---

📝 Walkthrough

Walkthrough

Changes

SSAO horizon scan

Layer / File(s)	Summary
Shared horizon scan helper and march state native/shared/src/renderer/shaders/ao.rs	Precomputes per-step mip levels and uses scan_direction to calculate bidirectional horizon visibility from Hi-Z samples.
Adaptive directional reconstruction native/shared/src/renderer/shaders/ao.rs	Tracks sampled visibility, performs full eight-direction scans for anisotropic or near-field pixels, and normalizes by the scanned direction count.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant cs_main
  participant scan_direction
  participant HiZPyramid
  cs_main->>cs_main: Compute mip_per_step
  cs_main->>scan_direction: Scan selected horizon directions
  scan_direction->>HiZPyramid: Sample per-step mip levels
  HiZPyramid-->>scan_direction: Return depth samples
  scan_direction-->>cs_main: Return directional visibility
  cs_main->>scan_direction: Scan remaining directions when adaptive criteria match
  scan_direction-->>cs_main: Return full directional visibility

Loading

Possibly related PRs

• Bloom-Engine/engine#85: Modifies the same SSAO shader’s contact-shadow gating and horizon-scan radius handling.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/gtao-cutout-checkerboard

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}