Skip to content

feat(capture): window_tree observations and authoring projector - #131

Merged
abrichr merged 3 commits into
mainfrom
feat/authoring-window-tree-2026-08-31
Sep 1, 2026
Merged

feat(capture): window_tree observations and authoring projector#131
abrichr merged 3 commits into
mainfrom
feat/authoring-window-tree-2026-08-31

Conversation

@abrichr

@abrichr abrichr commented Aug 31, 2026

Copy link
Copy Markdown
Member

PARKED (2026-09-01). Do not merge until the founder says so. Not Operator. ChatGPT.com and Claude.ai author once on the person's computer; later runs are a compiled program. Credentials never appear in the tree as values.

Reader

A capture reviewer checking that window_tree observations exist, that the authoring projector cannot carry field values / window titles / screenshots, that native AX/UIA/AT-SPI/ARIA roles map onto types ElementRole, and that OS-injected input still does not persist. If any of those sentences stops being true, tests/test_authoring_project.py and tests/test_structural_observation.py should fail.

What this changes

Capture gains a third structural query_kind: window_tree. Native observers walk the top-level accessibility tree (Windows UIA, macOS AX, Linux AT-SPI). The raw tree may persist on disk for compile. Password and secure-field values (AXSecureTextField, UIA IsPassword / PasswordBox, AT-SPI password text) are omitted from that tree. Credentials never appear as value.

openadapt_capture.authoring_project is the fail-closed projector onto openadapt.authoring.observe/v1. That payload is the vendor wire. It is not ComputerState.

Hard omissions on the wire:

  • value, title, screenshot (models forbid the keys; tests walk dumped JSON)
  • names and automation_ids that fail the coach-hint bar after whitespace collapse: length > 80, ://, @, six or more consecutive digits, phone, SSN, email
  • labels that fail the types projected-label grammar
  • unmapped native roles
  • process names that do not match ^[A-Za-z0-9 ._-]{1,64}$

Native role mapping (onto types ElementRole):

  • macOS AXButtonbutton; AXSecureTextField / AXTextFieldtext_input
  • Windows UIA Buttonbutton; Edittext_input
  • Linux AT-SPI push buttonbutton; password texttext_input
  • Playwright/ARIA textboxtext_input
  • Unmapped roles still fail closed to { tree: [], reason: "empty_projection" }

Other projector rules:

  • node_id is n_ + 8 hex of HMAC-SHA256(lease_nonce || provider_runtime_id); tests use a fixed key
  • macOS tree nodes keep AXIdentifier as provider_runtime_id
  • cap 200 nodes / 32 KiB, truncated: true past that
  • empty projection is { tree: [], reason: "empty_projection" } with no raw fallback
  • RDP/Citrix: empty tree, coach_only: true, agent_drive: false
  • Windows native: coach_only: true (tree may still be projected)
  • macOS / unique-title Linux / Playwright web: agent_drive: true when that backend is passed in

No record_injected API. Windows LLMHF_INJECTED still returns None. Injected clicks, moves, and scrolls still do not persist.

Deps

T1 (openadapt-types authoring schemas). Re-pinned tests/fixtures/authoring-observe-v1.json from OpenAdaptAI/openadapt-types#35 (3abf298b, feat/authoring-schemas-2026-08-31). Projector stays fail-closed against that schema.

Verification

  • uv run ruff check on the changed modules: pass
  • uv run pytest tests/ -q --ignore=tests/test_browser_bridge.py -m "not slow": 778 passed, 3 skipped, 13 deselected
  • Covers six-digit / phone / SSN / email / @ / :// name drop; native role mapping; unmapped roles fail closed; password/secure values omitted from the raw tree and never on the wire; value/title/screenshot never on the wire; 200-node and 32 KiB caps; injected events still do not persist; Windows LLMHF_INJECTEDNone

Do not merge.

Opened by an agent session, not the founder.

Add a third structural query kind that walks the top-level accessibility
tree. The raw tree may persist on disk for compile. The projector emits
openadapt.authoring.observe/v1 without values, titles, or screenshots.
OS-injected input still does not persist.
Re-pin tests/fixtures/authoring-observe-v1.json from
OpenAdaptAI/openadapt-types#35 (3abf298b). Keep the projector fail-closed
against that schema: unmapped native roles, missing required node fields,
and invalid process names never reach the vendor wire.
Project AX, UIA, AT-SPI, and ARIA roles onto the types ElementRole enum
so native window_tree observations can reach the vendor wire. Unmapped
roles still fail closed. Password and secure-field values are omitted
from the raw tree. macOS tree nodes keep AXIdentifier as the runtime id.
OS-injected input still does not persist.
@abrichr
abrichr merged commit a6fb0cc into main Sep 1, 2026
12 checks passed
@abrichr
abrichr deleted the feat/authoring-window-tree-2026-08-31 branch September 1, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant