feat(capture): window_tree observations and authoring projector - #131
Merged
Conversation
Add a third structural query kind that walks the top-level accessibility tree. The raw tree may persist on disk for compile. The projector emits openadapt.authoring.observe/v1 without values, titles, or screenshots. OS-injected input still does not persist.
Re-pin tests/fixtures/authoring-observe-v1.json from OpenAdaptAI/openadapt-types#35 (3abf298b). Keep the projector fail-closed against that schema: unmapped native roles, missing required node fields, and invalid process names never reach the vendor wire.
Project AX, UIA, AT-SPI, and ARIA roles onto the types ElementRole enum so native window_tree observations can reach the vendor wire. Unmapped roles still fail closed. Password and secure-field values are omitted from the raw tree. macOS tree nodes keep AXIdentifier as the runtime id. OS-injected input still does not persist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PARKED (2026-09-01). Do not merge until the founder says so. Not Operator. ChatGPT.com and Claude.ai author once on the person's computer; later runs are a compiled program. Credentials never appear in the tree as values.
Reader
A capture reviewer checking that
window_treeobservations exist, that the authoring projector cannot carry field values / window titles / screenshots, that native AX/UIA/AT-SPI/ARIA roles map onto typesElementRole, and that OS-injected input still does not persist. If any of those sentences stops being true,tests/test_authoring_project.pyandtests/test_structural_observation.pyshould fail.What this changes
Capture gains a third structural
query_kind:window_tree. Native observers walk the top-level accessibility tree (Windows UIA, macOS AX, Linux AT-SPI). The raw tree may persist on disk for compile. Password and secure-field values (AXSecureTextField, UIAIsPassword/PasswordBox, AT-SPIpassword text) are omitted from that tree. Credentials never appear asvalue.openadapt_capture.authoring_projectis the fail-closed projector ontoopenadapt.authoring.observe/v1. That payload is the vendor wire. It is notComputerState.Hard omissions on the wire:
value,title,screenshot(models forbid the keys; tests walk dumped JSON)automation_ids that fail the coach-hint bar after whitespace collapse: length > 80,://,@, six or more consecutive digits, phone, SSN, email^[A-Za-z0-9 ._-]{1,64}$Native role mapping (onto types
ElementRole):AXButton→button;AXSecureTextField/AXTextField→text_inputButton→button;Edit→text_inputpush button→button;password text→text_inputtextbox→text_input{ tree: [], reason: "empty_projection" }Other projector rules:
node_idisn_+ 8 hex of HMAC-SHA256(lease_nonce || provider_runtime_id); tests use a fixed keyAXIdentifierasprovider_runtime_idtruncated: truepast that{ tree: [], reason: "empty_projection" }with no raw fallbackcoach_only: true,agent_drive: falsecoach_only: true(tree may still be projected)agent_drive: truewhen that backend is passed inNo
record_injectedAPI. WindowsLLMHF_INJECTEDstill returnsNone. Injected clicks, moves, and scrolls still do not persist.Deps
T1 (openadapt-types authoring schemas). Re-pinned
tests/fixtures/authoring-observe-v1.jsonfrom OpenAdaptAI/openadapt-types#35 (3abf298b,feat/authoring-schemas-2026-08-31). Projector stays fail-closed against that schema.Verification
uv run ruff checkon the changed modules: passuv run pytest tests/ -q --ignore=tests/test_browser_bridge.py -m "not slow": 778 passed, 3 skipped, 13 deselected@/://name drop; native role mapping; unmapped roles fail closed; password/secure values omitted from the raw tree and never on the wire; value/title/screenshot never on the wire; 200-node and 32 KiB caps; injected events still do not persist; WindowsLLMHF_INJECTED→NoneDo not merge.
Opened by an agent session, not the founder.