diff --git a/WHATS_NEW.md b/WHATS_NEW.md index ffc28c4e..c5f19636 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-24) — Native Text Reading via the UIA TextPattern (document / selection / visible) + +Read the text in multiline editors and document controls where ValuePattern returns nothing. Full reference: [`docs/source/Eng/doc/new_features/v182_features_doc.rst`](docs/source/Eng/doc/new_features/v182_features_doc.rst). + +- **`get_control_text` / `get_selected_text` / `get_visible_text`** (`AC_get_control_text`, `AC_get_selected_text`, `AC_get_visible_text`): `control_get_value` reads through UIA ValuePattern, which returns an empty string on multiline edits, RichEdit / document controls and web text areas — exactly the controls whose text you most want. This reads through `TextPattern` instead: `get_control_text` returns the whole `DocumentRange`, `get_selected_text` the current `GetSelection`, `get_visible_text` only the on-screen `GetVisibleRanges`. Dispatched through the injectable `accessibility.backends.get_backend()` seam (headless-testable via a fake backend; real UIA calls in the Windows backend), returning `{text}` from the executor/MCP. No `PySide6`. + ## What's new (2026-06-24) — Extended UIA Control Patterns (Expand / Select / Range / Scroll) Drive tree nodes, list/combo items, sliders and scroll natively, not by pixel guessing. Full reference: [`docs/source/Eng/doc/new_features/v181_features_doc.rst`](docs/source/Eng/doc/new_features/v181_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v182_features_doc.rst b/docs/source/Eng/doc/new_features/v182_features_doc.rst new file mode 100644 index 00000000..fc15006c --- /dev/null +++ b/docs/source/Eng/doc/new_features/v182_features_doc.rst @@ -0,0 +1,46 @@ +Native Text Reading via the UIA TextPattern (document / selection / visible) +============================================================================ + +``control_get_value`` reads a control through UIA ValuePattern, but ValuePattern +returns an **empty string** on multiline edits, RichEdit / document controls and +web text areas — exactly the controls whose text you most want to read. UIA +exposes that text through a different pattern, ``TextPattern``, which models the +control's content as text ranges. ``ax_text`` adds three reads on top of the +existing accessibility backend ABC: + +* :func:`get_control_text` — the whole document's text (``DocumentRange``), +* :func:`get_selected_text` — the currently selected text (``GetSelection``), +* :func:`get_visible_text` — only the on-screen text (``GetVisibleRanges``). + +Each function is a thin dispatch onto the injectable +``accessibility.backends.get_backend()`` seam (the same seam the rest of the +accessibility module uses), so the headless core is unit-testable on any +platform by injecting a fake backend; the real UI Automation calls live in the +Windows backend. Backends that don't implement TextPattern raise +``AccessibilityNotAvailableError``. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import (get_control_text, get_selected_text, + get_visible_text) + + # A multiline editor where control_get_value returns "" : + text = get_control_text(name="Editor", role="document") + selection = get_selected_text(name="Editor") # "" when nothing selected + on_screen = get_visible_text(name="Editor") # skips scrolled-off lines + +All locate the control by ``name`` / ``role`` / ``app_name`` / ``automation_id`` +(same as ``control_get_value`` / ``control_invoke``). Each returns the text as a +``str``, or ``None`` when the control is not found or exposes no TextPattern; +``get_selected_text`` returns ``""`` when the control is found but has no +selection. + +Executor commands +----------------- + +``AC_get_control_text`` / ``AC_get_selected_text`` / ``AC_get_visible_text`` each +return ``{"text": ...}``. They are exposed as the matching read-only ``ac_*`` MCP +tools and as Script Builder commands under **Native UI**. diff --git a/docs/source/Zh/doc/new_features/v182_features_doc.rst b/docs/source/Zh/doc/new_features/v182_features_doc.rst new file mode 100644 index 00000000..518edaa0 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v182_features_doc.rst @@ -0,0 +1,41 @@ +透過 UIA TextPattern 讀取原生文字(文件 / 選取 / 可見) +======================================================= + +``control_get_value`` 透過 UIA ValuePattern 讀取控制項,但 ValuePattern 在多行編輯框、 +RichEdit / 文件控制項與網頁文字區塊上會回傳**空字串**——而這些正是你最想讀取其文字的控制項。 +UIA 透過另一個模式 ``TextPattern`` 提供這些文字,它把控制項內容建模為文字範圍(text range)。 +``ax_text`` 在既有的無障礙後端 ABC 之上補上三種讀取: + +* :func:`get_control_text` ——整份文件的文字(``DocumentRange``), +* :func:`get_selected_text` ——目前選取的文字(``GetSelection``), +* :func:`get_visible_text` ——僅螢幕上可見的文字(``GetVisibleRanges``)。 + +每個函式都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派(與無障礙模組 +其餘部分相同的接縫),因此無頭核心可在任何平台透過注入 fake backend 單元測試;真正的 +UI Automation 呼叫位於 Windows 後端。未實作 TextPattern 的後端會拋出 +``AccessibilityNotAvailableError``。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import (get_control_text, get_selected_text, + get_visible_text) + + # 一個 control_get_value 會回傳 "" 的多行編輯框: + text = get_control_text(name="Editor", role="document") + selection = get_selected_text(name="Editor") # 沒有選取時回傳 "" + on_screen = get_visible_text(name="Editor") # 略過捲動到畫面外的列 + +全部以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位控制項(與 +``control_get_value`` / ``control_invoke`` 相同)。各函式以 ``str`` 回傳文字,找不到控制項或 +控制項未提供 TextPattern 時回傳 ``None``;``get_selected_text`` 在找到控制項但沒有選取時 +回傳 ``""``。 + +執行器指令 +---------- + +``AC_get_control_text`` / ``AC_get_selected_text`` / ``AC_get_visible_text`` 各自回傳 +``{"text": ...}``。皆以對應的唯讀 ``ac_*`` MCP 工具及 Script Builder 指令(位於 **Native UI** +分類下)形式提供。 diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index ebd319fc..692a75c1 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -53,6 +53,10 @@ collapse_control, control_expand_state, control_range, expand_control, scroll_control_into_view, select_control_item, set_control_range, ) +# Native text reads via UIA TextPattern (document / selection / visible) +from je_auto_control.utils.ax_text import ( + get_control_text, get_selected_text, get_visible_text, +) # VLM element locator (headless) from je_auto_control.utils.vision import ( VLMNotAvailableError, click_by_description, locate_by_description, @@ -1617,6 +1621,7 @@ def start_autocontrol_gui(*args, **kwargs): "expand_control", "collapse_control", "control_expand_state", "select_control_item", "control_range", "set_control_range", "scroll_control_into_view", + "get_control_text", "get_selected_text", "get_visible_text", # VLM locator "VLMNotAvailableError", "locate_by_description", "click_by_description", "verify_description", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 7d776cc7..7afed681 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1523,6 +1523,21 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None: fields=fields, description="Scroll a control into view (ScrollItemPattern).", )) + specs.append(CommandSpec( + "AC_get_control_text", "Native UI", "Get Control Text", + fields=fields, + description="Read full text via TextPattern (multiline / document safe).", + )) + specs.append(CommandSpec( + "AC_get_selected_text", "Native UI", "Get Selected Text", + fields=fields, + description="Read the currently selected text via TextPattern.", + )) + specs.append(CommandSpec( + "AC_get_visible_text", "Native UI", "Get Visible Text", + fields=fields, + description="Read only the on-screen text via TextPattern.GetVisibleRanges.", + )) def _add_misc_specs(specs: List[CommandSpec]) -> None: diff --git a/je_auto_control/utils/accessibility/backends/base.py b/je_auto_control/utils/accessibility/backends/base.py index 017666fb..7781c8ec 100644 --- a/je_auto_control/utils/accessibility/backends/base.py +++ b/je_auto_control/utils/accessibility/backends/base.py @@ -100,6 +100,29 @@ def scroll_into_view(self, name: Optional[str] = None, """Scroll the matched control into view (ScrollItemPattern); True on success.""" self._unsupported("scroll_into_view") + # --- text patterns (TextPattern reads) --------------------------------- + + def document_text(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return the matched control's full text (TextPattern), or None. + + Reads multiline / document controls where ValuePattern returns ``""``. + """ + self._unsupported("document_text") + + def selected_text(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return the control's currently selected text (TextPattern), or None.""" + self._unsupported("selected_text") + + def visible_text(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return only the on-screen text of the control (TextPattern), or None.""" + self._unsupported("visible_text") + def _unsupported(self, operation: str): """Raise a clear error for an action this backend can't perform.""" raise AccessibilityNotAvailableError( diff --git a/je_auto_control/utils/accessibility/backends/windows_backend.py b/je_auto_control/utils/accessibility/backends/windows_backend.py index fd3f0657..32142003 100644 --- a/je_auto_control/utils/accessibility/backends/windows_backend.py +++ b/je_auto_control/utils/accessibility/backends/windows_backend.py @@ -29,6 +29,7 @@ _UIA_SELECTIONITEM_PATTERN_ID = 10010 _UIA_RANGEVALUE_PATTERN_ID = 10003 _UIA_SCROLLITEM_PATTERN_ID = 10017 +_UIA_TEXT_PATTERN_ID = 10014 _EXPAND_STATES = {0: "collapsed", 1: "expanded", 2: "partial", 3: "leaf"} @@ -263,6 +264,50 @@ def get_range(self, name=None, role=None, app_name=None, except (OSError, AttributeError, ValueError, TypeError): return None + def _text_pattern(self, name, role, app_name, automation_id): + """Find a control and return its IUIAutomationTextPattern, or None.""" + raw = self._find_raw(name, role, app_name, automation_id) + if not raw: + return None + return self._pattern(raw, _UIA_TEXT_PATTERN_ID, + "IUIAutomationTextPattern") + + def document_text(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[str]: + pattern = self._text_pattern(name, role, app_name, automation_id) + if pattern is None: + return None + try: + return str(pattern.DocumentRange.GetText(-1) or "") + except (OSError, AttributeError): + return None + + def selected_text(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[str]: + pattern = self._text_pattern(name, role, app_name, automation_id) + if pattern is None: + return None + try: + selection = pattern.GetSelection() + if not selection or int(selection.Length or 0) == 0: + return "" + return str(selection.GetElement(0).GetText(-1) or "") + except (OSError, AttributeError): + return None + + def visible_text(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[str]: + pattern = self._text_pattern(name, role, app_name, automation_id) + if pattern is None: + return None + try: + ranges = pattern.GetVisibleRanges() + count = int(ranges.Length or 0) + return "".join(str(ranges.GetElement(i).GetText(-1) or "") + for i in range(count)) + except (OSError, AttributeError): + return None + @staticmethod def _read_row(pattern, row: int, cols: int): """Read one grid row into a list of cell strings.""" diff --git a/je_auto_control/utils/ax_text/__init__.py b/je_auto_control/utils/ax_text/__init__.py new file mode 100644 index 00000000..66ed749b --- /dev/null +++ b/je_auto_control/utils/ax_text/__init__.py @@ -0,0 +1,8 @@ +"""Native text reading via the UI Automation TextPattern (document/selection/visible).""" +from je_auto_control.utils.ax_text.ax_text import ( + get_control_text, get_selected_text, get_visible_text, +) + +__all__ = [ + "get_control_text", "get_selected_text", "get_visible_text", +] diff --git a/je_auto_control/utils/ax_text/ax_text.py b/je_auto_control/utils/ax_text/ax_text.py new file mode 100644 index 00000000..7f04f1dc --- /dev/null +++ b/je_auto_control/utils/ax_text/ax_text.py @@ -0,0 +1,61 @@ +"""Native text reading via the UI Automation TextPattern. + +``control_get_value`` reads a control through ValuePattern, but ValuePattern +returns an **empty string** on multiline edits, RichEdit / document controls and +web text areas — exactly the controls whose text you most want to read. UIA +exposes that text through a different pattern, ``TextPattern``, which models the +control's content as text ranges. ``ax_text`` adds three reads on top of the +existing accessibility backend ABC: + +* :func:`get_control_text` — the whole document's text (``DocumentRange``), +* :func:`get_selected_text` — the currently selected text (``GetSelection``), +* :func:`get_visible_text` — only the on-screen text (``GetVisibleRanges``). + +Each function is a thin dispatch onto the injectable +``accessibility.backends.get_backend()`` seam (the same seam the rest of the +accessibility module uses), so the headless core is unit-testable on any +platform by injecting a fake backend; the real UIA calls live in the Windows +backend. Imports no ``PySide6``. +""" +from typing import Optional + + +def _backend(): + from je_auto_control.utils.accessibility.backends import get_backend + return get_backend() + + +def get_control_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return a control's full text via TextPattern (``None`` if not found). + + Unlike :func:`control_get_value`, this works on multiline edits, RichEdit / + document controls and web text areas where ValuePattern returns ``""``. + """ + return _backend().document_text(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def get_selected_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return the control's currently selected text (TextPattern.GetSelection). + + Empty string when nothing is selected; ``None`` if the control is not found + or exposes no TextPattern. + """ + return _backend().selected_text(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def get_visible_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return only the on-screen text of a control (TextPattern.GetVisibleRanges). + + Useful for scrolled documents where :func:`get_control_text` would return the + whole (possibly huge) buffer. ``None`` if the control is not found. + """ + return _backend().visible_text(name=name, role=role, app_name=app_name, + automation_id=automation_id) diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index de3c3236..3e37c0d2 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -2416,6 +2416,33 @@ def _scroll_control_into_view(name: Optional[str] = None, role: Optional[str] = automation_id=automation_id) +def _get_control_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: read a control's full text via TextPattern (multiline-safe).""" + from je_auto_control.utils.ax_text import get_control_text + return {"text": get_control_text(name=name, role=role, app_name=app_name, + automation_id=automation_id)} + + +def _get_selected_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: read a control's currently selected text (TextPattern).""" + from je_auto_control.utils.ax_text import get_selected_text + return {"text": get_selected_text(name=name, role=role, app_name=app_name, + automation_id=automation_id)} + + +def _get_visible_text(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: read only the on-screen text of a control (TextPattern).""" + from je_auto_control.utils.ax_text import get_visible_text + return {"text": get_visible_text(name=name, role=role, app_name=app_name, + automation_id=automation_id)} + + def _read_table(name: Optional[str] = None, role: Optional[str] = None, app_name: Optional[str] = None, automation_id: Optional[str] = None) -> List[List[str]]: @@ -6092,6 +6119,9 @@ def __init__(self): "AC_control_range": _control_range, "AC_set_control_range": _set_control_range, "AC_scroll_control_into_view": _scroll_control_into_view, + "AC_get_control_text": _get_control_text, + "AC_get_selected_text": _get_selected_text, + "AC_get_visible_text": _get_visible_text, "AC_read_table": _read_table, "AC_watchdog_add": _watchdog_add, "AC_watchdog_start": _watchdog_start, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 68e57b09..f5bd346a 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -1157,6 +1157,31 @@ def a11y_control_tools() -> List[MCPTool]: handler=h.scroll_control_into_view, annotations=DESTRUCTIVE, ), + MCPTool( + name="ac_get_control_text", + description=("Read a control's full text via TextPattern: " + "{text}. Works on multiline edits / RichEdit / document " + "controls where ac_control_get_value returns empty."), + input_schema=schema(dict(_M)), + handler=h.get_control_text, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_get_selected_text", + description=("Read a control's currently selected text via " + "TextPattern: {text} ('' when nothing selected)."), + input_schema=schema(dict(_M)), + handler=h.get_selected_text, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_get_visible_text", + description=("Read only the on-screen text of a control via " + "TextPattern.GetVisibleRanges: {text}."), + input_schema=schema(dict(_M)), + handler=h.get_visible_text, + annotations=READ_ONLY, + ), ] diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 6e109873..3818a3fb 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -790,6 +790,21 @@ def scroll_control_into_view(name=None, role=None, app_name=None, return _scroll_control_into_view(name, role, app_name, automation_id) +def get_control_text(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _get_control_text + return _get_control_text(name, role, app_name, automation_id) + + +def get_selected_text(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _get_selected_text + return _get_selected_text(name, role, app_name, automation_id) + + +def get_visible_text(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _get_visible_text + return _get_visible_text(name, role, app_name, automation_id) + + def watchdog_add(title, action="close", case_sensitive=False, name=None): from je_auto_control.utils.watchdog import default_popup_watchdog default_popup_watchdog.add_window_rule( diff --git a/test/unit_test/headless/test_ax_text_batch.py b/test/unit_test/headless/test_ax_text_batch.py new file mode 100644 index 00000000..617ec9a3 --- /dev/null +++ b/test/unit_test/headless/test_ax_text_batch.py @@ -0,0 +1,97 @@ +"""Headless tests for native TextPattern reads (fake backend via the seam).""" +import je_auto_control as ac +from je_auto_control.utils.accessibility.backends import base as backend_base +from je_auto_control.utils.ax_text import ( + get_control_text, get_selected_text, get_visible_text, +) + + +class _FakeBackend(backend_base.AccessibilityBackend): + name = "fake" + available = True + + def __init__(self): + self.calls = [] + + def document_text(self, name=None, role=None, app_name=None, + automation_id=None): + self.calls.append(("document", {"name": name, "role": role, + "app_name": app_name, + "automation_id": automation_id})) + return "line 1\nline 2\nline 3" + + def selected_text(self, name=None, role=None, app_name=None, + automation_id=None): + self.calls.append(("selected", name)) + return "line 2" + + def visible_text(self, name=None, role=None, app_name=None, + automation_id=None): + self.calls.append(("visible", name)) + return "line 1\nline 2" + + +def _inject(monkeypatch, backend): + import je_auto_control.utils.accessibility.backends as backends + monkeypatch.setattr(backends, "_cached_backend", backend, raising=False) + + +def test_document_text_dispatch(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert get_control_text(name="Editor", role="document") == "line 1\nline 2\nline 3" + assert ("document", {"name": "Editor", "role": "document", "app_name": None, + "automation_id": None}) in fake.calls + + +def test_selected_text(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert get_selected_text(automation_id="edit1") == "line 2" + assert fake.calls[0][0] == "selected" + + +def test_visible_text(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert get_visible_text(name="Editor") == "line 1\nline 2" + assert fake.calls[0][0] == "visible" + + +def test_unsupported_backend_raises(monkeypatch): + from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError) + _inject(monkeypatch, backend_base.AccessibilityBackend()) # all _unsupported + try: + get_control_text(name="x") + raised = False + except AccessibilityNotAvailableError: + raised = True + assert raised is True + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapter_wraps_text(monkeypatch): + _inject(monkeypatch, _FakeBackend()) + from je_auto_control.utils.executor.action_executor import _get_control_text + assert _get_control_text(name="Editor") == {"text": "line 1\nline 2\nline 3"} + + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_get_control_text", "AC_get_selected_text", + "AC_get_visible_text"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_get_control_text", "ac_get_selected_text", + "ac_get_visible_text"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_get_control_text", "AC_get_selected_text", + "AC_get_visible_text"} <= specs + + +def test_facade_exports(): + for name in ("get_control_text", "get_selected_text", "get_visible_text"): + assert hasattr(ac, name) and name in ac.__all__