Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions WHATS_NEW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# What's New — AutoControl

## What's new (2026-06-24) — Drop Files onto a Window (WM_DROPFILES)

Complete a drag-and-drop programmatically — drop files onto a target window. Full reference: [`docs/source/Eng/doc/new_features/v187_features_doc.rst`](docs/source/Eng/doc/new_features/v187_features_doc.rst).

- **`plan_file_drop` / `drop_files`** (`AC_plan_file_drop`, `AC_drop_files`): `clipboard_files` *stages* a file list on the clipboard for `Ctrl+V`; this actively **drops** files onto a target window by posting a `WM_DROPFILES` message. It reuses `clipboard_files.build_dropfiles` to pack the `DROPFILES` blob (shared byte layout, not re-implemented) and dispatches through an injectable driver seam, so the build-and-dispatch logic is unit-testable with a fake driver; the real `GlobalAlloc` + `PostMessage` lives in the default Win32 driver. `plan_file_drop` is a pure dry-run returning `{message, paths, point, wide, blob_size}`. No `PySide6`.

## What's new (2026-06-24) — Clipboard Format Inspection (classify / diff available formats)

See which formats are on the clipboard, and detect when its shape changes. Full reference: [`docs/source/Eng/doc/new_features/v186_features_doc.rst`](docs/source/Eng/doc/new_features/v186_features_doc.rst).

- **`classify_format` / `classify_formats` / `diff_formats` / `list_clipboard_formats` / `clipboard_formats`** (`AC_clipboard_formats`, `AC_classify_formats`, `AC_diff_formats`): the clipboard usually holds the same content in several formats at once (a Word copy = text + HTML + RTF; a file copy = CF_HDROP; a screenshot = CF_DIB). This enumerates the live clipboard (`EnumClipboardFormats`) without consuming anything and classifies each format into a friendly category (text/image/files/html/rtf/csv/audio/…); `diff_formats` is a pure monitor primitive returning `{added, removed, changed}` between two snapshots. The classifier and diff are pure (registered names take priority over dynamic ids); only the live enumeration is Win32. No `PySide6`.

## What's new (2026-06-24) — Rich Clipboard Formats (RTF and CSV/TSV)

Put styled text and tables on the clipboard for cross-app paste into Word and Excel. Full reference: [`docs/source/Eng/doc/new_features/v185_features_doc.rst`](docs/source/Eng/doc/new_features/v185_features_doc.rst).

- **`build_rtf` / `rtf_to_text` / `rows_to_csv` / `csv_to_rows` + `set_clipboard_rtf` / `get_clipboard_rtf` / `set_clipboard_csv` / `get_clipboard_csv`** (`AC_set_clipboard_rtf`, `AC_get_clipboard_rtf`, `AC_set_clipboard_csv`, `AC_get_clipboard_csv`): `rich_clipboard` added CF_HTML, but RTF (the format rich editors accept) and the `Csv` format Excel reads were still missing. This adds both: `build_rtf`/`rtf_to_text` build and strip RTF control words and `\uNNNN` / `\'XX` escapes in pure Python (fully unit-testable round-trip), and `rows_to_csv`/`csv_to_rows` wrap the stdlib `csv` module (delimiter-parametrised, so `\t` gives TSV). The codecs are platform-independent; the Win32 get/set share one generic byte-transfer helper, and the sets seed plain text so plain editors still paste. No `PySide6`.

## What's new (2026-06-24) — Keyboard Focus Order (Tab sequence / WCAG audit / set-focus)

Reason about keyboard navigation: the Tab order, a WCAG focus-order audit, and set-focus. Full reference: [`docs/source/Eng/doc/new_features/v184_features_doc.rst`](docs/source/Eng/doc/new_features/v184_features_doc.rst).
Expand Down
49 changes: 49 additions & 0 deletions docs/source/Eng/doc/new_features/v185_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
Rich Clipboard Formats — RTF and CSV/TSV
========================================

``rich_clipboard`` added ``CF_HTML`` for rich paste into Word / Outlook, but two
other cross-application clipboard formats were still missing:

* **RTF** (``"Rich Text Format"``) — the format almost every rich editor accepts
for styled paste. ``build_rtf`` / ``rtf_to_text`` build and strip RTF control
words and ``\uNNNN`` / ``\'XX`` escapes in pure Python, with a fully
unit-testable round-trip.
* **CSV / TSV** (the registered ``"Csv"`` format Excel reads) — ``rows_to_csv`` /
``csv_to_rows`` are a thin, delimiter-parametrised wrapper over the stdlib
``csv`` module, so a table can be put on / read off the clipboard.

The codecs are platform-independent and headless-testable; only the actual
clipboard I/O is Win32 (raising ``RuntimeError`` elsewhere, like the base
``clipboard`` module), and the byte transfer is a single generic helper shared by
both formats. Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import (build_rtf, rtf_to_text, rows_to_csv,
csv_to_rows, set_clipboard_rtf, set_clipboard_csv)

rtf = build_rtf("Hello\nWorld") # minimal valid RTF document
rtf_to_text(rtf) # -> "Hello\nWorld"

rows_to_csv([["a", "b"], ["1", "2"]]) # 'a,b\r\n1,2\r\n'
csv_to_rows("a,b\r\n1,2\r\n") # [["a", "b"], ["1", "2"]]

set_clipboard_rtf("Paste me as styled text") # Windows
set_clipboard_csv([["Name", "Qty"], ["Pen", "3"]], delimiter="\t") # TSV

``build_rtf`` escapes braces / backslashes, turns newlines into ``\par`` and
non-ASCII characters into ``\uNNNN?`` escapes (the output is pure ASCII).
``set_clipboard_rtf`` / ``set_clipboard_csv`` also seed plain text by default so
plain editors still paste something; ``get_clipboard_rtf`` returns the raw RTF
string (feed it to ``rtf_to_text``) and ``get_clipboard_csv`` returns rows.

Executor commands
-----------------

``AC_set_clipboard_rtf`` / ``AC_get_clipboard_rtf`` / ``AC_set_clipboard_csv`` /
``AC_get_clipboard_csv`` (the sets take ``text`` / ``rows`` + ``delimiter``). They
are exposed as the matching ``ac_*`` MCP tools (the sets side-effect-only, the
gets read-only) and as Script Builder commands under **Data**.
49 changes: 49 additions & 0 deletions docs/source/Eng/doc/new_features/v186_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
Clipboard Format Inspection (classify / diff available formats)
===============================================================

The clipboard usually holds the *same* content in several formats at once — a
copy from Word offers ``CF_UNICODETEXT`` + ``HTML Format`` + ``Rich Text Format``,
a file copy offers ``CF_HDROP``, a screenshot offers ``CF_DIB``. Knowing *which
formats are present* (without consuming any of them) tells an automation what it
can paste, and comparing two snapshots detects when the clipboard's shape
changed. ``clipboard_formats`` adds:

* :func:`classify_format` / :func:`classify_formats` — map standard ``CF_*`` ids
and registered format names to friendly categories (text / image / files /
html / rtf / csv / audio / …),
* :func:`diff_formats` — a pure monitor primitive: ``{added, removed, changed}``
between two snapshots,
* :func:`list_clipboard_formats` / :func:`clipboard_formats` — enumerate the live
clipboard (``EnumClipboardFormats``) and classify it.

The classifier and diff are pure functions (unit-testable on any platform); only
the live enumeration is Win32 (raising ``RuntimeError`` elsewhere). Imports no
``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import (classify_formats, diff_formats,
clipboard_formats)

classify_formats([13, {"id": 49383, "name": "HTML Format"}])
# {"categories": ["html", "text"], "has_text": True, "has_image": False, ...}

diff_formats([13, 1], [13, 15]) # {"added": [files], "removed": [text], ...}

clipboard_formats() # live clipboard summary (Windows)

A descriptor is an id (``13``), an ``{"id": ..., "name": ...}`` dict, or an
``(id, name)`` tuple. A registered ``name`` takes priority over the id, since
registered formats have dynamic ids (``>= 0xC000``). Unrecognised formats are
``"other"``.

Executor commands
-----------------

``AC_clipboard_formats`` (live, Windows), ``AC_classify_formats`` (``formats``)
and ``AC_diff_formats`` (``before`` / ``after``) — the latter two are pure and
run anywhere. They are exposed as read-only ``ac_*`` MCP tools and as Script
Builder commands under **Data**.
42 changes: 42 additions & 0 deletions docs/source/Eng/doc/new_features/v187_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
Drop Files onto a Window (WM_DROPFILES)
=======================================

``clipboard_files`` *stages* a file-drop list on the clipboard so a user can
``Ctrl+V`` it; ``file_drop`` actively **drops** files onto a target window — the
completion of a drag-and-drop — by posting a ``WM_DROPFILES`` message carrying a
``DROPFILES`` blob. It reuses ``clipboard_files.build_dropfiles`` to pack that
blob (the byte layout is shared, not re-implemented) and dispatches it through an
injectable *driver* seam, so the build-and-dispatch logic is unit-testable on any
platform with a fake driver; the real ``GlobalAlloc`` + ``PostMessage`` lives in
the default Win32 driver. Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import plan_file_drop, drop_files

# Pure dry-run — inspect the payload without sending:
plan_file_drop(["C:\\a\\one.txt"], point=(10, 20))
# {"message": 0x233, "paths": [...], "point": [10, 20], "wide": True,
# "blob_size": ...}

# Real drop onto a window handle (Windows):
drop_files(hwnd, ["C:\\a\\one.txt", "C:\\b\\two.png"], point=(10, 20))

# Inject a driver to intercept the send (e.g. in tests):
drop_files(hwnd, ["x.txt"], driver=lambda hwnd, blob, point: True)

``point`` is the drop coordinate in the window's client area. ``drop_files``
returns ``bool``; the default driver posts the real ``WM_DROPFILES`` (the
receiving window then owns and frees the memory via ``DragFinish``) and raises
``RuntimeError`` off Windows.

Executor commands
-----------------

``AC_drop_files`` (``hwnd`` / ``paths`` / ``point``) performs the drop;
``AC_plan_file_drop`` (``paths`` / ``point``) is the pure dry-run. They are
exposed as the matching ``ac_*`` MCP tools (drop side-effect-only, plan
read-only) and as Script Builder commands under **Window**.
44 changes: 44 additions & 0 deletions docs/source/Zh/doc/new_features/v185_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
豐富剪貼簿格式——RTF 與 CSV/TSV
==============================

``rich_clipboard`` 已加入 ``CF_HTML`` 以便把豐富內容貼進 Word / Outlook,但仍缺少另外兩種
跨應用程式的剪貼簿格式:

* **RTF**(``"Rich Text Format"``)——幾乎每個豐富編輯器都接受、用於樣式貼上的格式。
``build_rtf`` / ``rtf_to_text`` 以純 Python 建立與剝除 RTF 控制字與 ``\uNNNN`` / ``\'XX``
轉義,並具備完全可單元測試的往返。
* **CSV / TSV**(Excel 讀取的已註冊 ``"Csv"`` 格式)——``rows_to_csv`` / ``csv_to_rows`` 是對
標準庫 ``csv`` 模組的薄包裝(可指定分隔符),讓表格能放上 / 讀下剪貼簿。

這些編解碼器與平台無關且可無頭測試;只有實際的剪貼簿 I/O 為 Win32(在其他平台拋出
``RuntimeError``,與基礎 ``clipboard`` 模組一致),且位元組傳輸是兩種格式共用的單一泛型輔助
函式。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import (build_rtf, rtf_to_text, rows_to_csv,
csv_to_rows, set_clipboard_rtf, set_clipboard_csv)

rtf = build_rtf("Hello\nWorld") # 最小的有效 RTF 文件
rtf_to_text(rtf) # -> "Hello\nWorld"

rows_to_csv([["a", "b"], ["1", "2"]]) # 'a,b\r\n1,2\r\n'
csv_to_rows("a,b\r\n1,2\r\n") # [["a", "b"], ["1", "2"]]

set_clipboard_rtf("以樣式文字貼上我") # Windows
set_clipboard_csv([["Name", "Qty"], ["Pen", "3"]], delimiter="\t") # TSV

``build_rtf`` 會轉義大括號 / 反斜線,把換行轉為 ``\par``,並把非 ASCII 字元轉為 ``\uNNNN?``
轉義(輸出為純 ASCII)。``set_clipboard_rtf`` / ``set_clipboard_csv`` 預設也會種入純文字,讓
純文字編輯器仍能貼上內容;``get_clipboard_rtf`` 回傳原始 RTF 字串(再餵給 ``rtf_to_text``),
``get_clipboard_csv`` 回傳列。

執行器指令
----------

``AC_set_clipboard_rtf`` / ``AC_get_clipboard_rtf`` / ``AC_set_clipboard_csv`` /
``AC_get_clipboard_csv``(set 取 ``text`` / ``rows`` 加 ``delimiter``)。皆以對應的 ``ac_*``
MCP 工具(set 為僅副作用、get 為唯讀)及 Script Builder 指令(位於 **Data** 分類下)形式提供。
41 changes: 41 additions & 0 deletions docs/source/Zh/doc/new_features/v186_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
剪貼簿格式檢視(分類 / 比較可用格式)
====================================

剪貼簿通常同時以多種格式保存*相同*內容——從 Word 複製會提供 ``CF_UNICODETEXT`` +
``HTML Format`` + ``Rich Text Format``,複製檔案會提供 ``CF_HDROP``,截圖會提供 ``CF_DIB``。
知道*目前有哪些格式*(且不消耗任何一個)能讓自動化判斷可以貼上什麼,比較兩份快照則能偵測
剪貼簿的形態何時改變。``clipboard_formats`` 加入:

* :func:`classify_format` / :func:`classify_formats` ——把標準 ``CF_*`` id 與已註冊格式名稱
對應到友善類別(text / image / files / html / rtf / csv / audio……),
* :func:`diff_formats` ——純粹的監看原語:兩份快照之間的 ``{added, removed, changed}``,
* :func:`list_clipboard_formats` / :func:`clipboard_formats` ——列舉存活的剪貼簿
(``EnumClipboardFormats``)並加以分類。

分類器與比較器為純函式(可在任何平台單元測試);只有存活列舉為 Win32(其他平台拋出
``RuntimeError``)。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import (classify_formats, diff_formats,
clipboard_formats)

classify_formats([13, {"id": 49383, "name": "HTML Format"}])
# {"categories": ["html", "text"], "has_text": True, "has_image": False, ...}

diff_formats([13, 1], [13, 15]) # {"added": [files], "removed": [text], ...}

clipboard_formats() # 存活剪貼簿摘要(Windows)

描述子可為 id(``13``)、``{"id": ..., "name": ...}`` 字典,或 ``(id, name)`` 元組。已註冊的
``name`` 優先於 id,因為已註冊格式的 id 是動態的(``>= 0xC000``)。未辨識的格式為 ``"other"``。

執行器指令
----------

``AC_clipboard_formats``(存活,Windows)、``AC_classify_formats``(``formats``)與
``AC_diff_formats``(``before`` / ``after``)——後兩者為純函式,可在任何平台執行。皆以唯讀
``ac_*`` MCP 工具及 Script Builder 指令(位於 **Data** 分類下)形式提供。
38 changes: 38 additions & 0 deletions docs/source/Zh/doc/new_features/v187_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
將檔案拖放到視窗(WM_DROPFILES)
==============================

``clipboard_files`` 只是把檔案拖放清單*放上*剪貼簿,讓使用者可以 ``Ctrl+V``;``file_drop`` 則
主動把檔案**拖放**到目標視窗——也就是拖放動作的完成——透過送出帶有 ``DROPFILES`` 位元組區塊的
``WM_DROPFILES`` 訊息達成。它重用 ``clipboard_files.build_dropfiles`` 來打包該區塊(位元組配置
共用,不重新實作),並透過可注入的 *driver* 接縫分派,因此「打包 + 分派」邏輯可在任何平台以
假 driver 單元測試;真正的 ``GlobalAlloc`` + ``PostMessage`` 位於預設的 Win32 driver。不匯入
``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import plan_file_drop, drop_files

# 純試跑——檢視 payload 但不送出:
plan_file_drop(["C:\\a\\one.txt"], point=(10, 20))
# {"message": 0x233, "paths": [...], "point": [10, 20], "wide": True,
# "blob_size": ...}

# 對視窗 handle 真正拖放(Windows):
drop_files(hwnd, ["C:\\a\\one.txt", "C:\\b\\two.png"], point=(10, 20))

# 注入 driver 以攔截送出(例如在測試中):
drop_files(hwnd, ["x.txt"], driver=lambda hwnd, blob, point: True)

``point`` 是視窗工作區(client area)內的拖放座標。``drop_files`` 回傳 ``bool``;預設 driver 送出
真正的 ``WM_DROPFILES``(接收視窗隨後擁有該記憶體並透過 ``DragFinish`` 釋放),在非 Windows 平台
拋出 ``RuntimeError``。

執行器指令
----------

``AC_drop_files``(``hwnd`` / ``paths`` / ``point``)執行拖放;``AC_plan_file_drop``
(``paths`` / ``point``)為純試跑。皆以對應的 ``ac_*`` MCP 工具(drop 為僅副作用、plan 為唯讀)
及 Script Builder 指令(位於 **Window** 分類下)形式提供。
18 changes: 18 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,18 @@
from je_auto_control.utils.focus_order import (
audit_focus_order, focus_control, is_interactive_role, tab_order,
)
# Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
from je_auto_control.utils.clipboard_rich_formats import (
build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
rtf_to_text, set_clipboard_csv, set_clipboard_rtf,
)
# Clipboard format inspection (classify / diff available formats)
from je_auto_control.utils.clipboard_formats import (
classify_format, classify_formats, clipboard_formats, diff_formats,
list_clipboard_formats,
)
# Drop files onto a window (WM_DROPFILES sender)
from je_auto_control.utils.file_drop import drop_files, plan_file_drop
# VLM element locator (headless)
from je_auto_control.utils.vision import (
VLMNotAvailableError, click_by_description, locate_by_description,
Expand Down Expand Up @@ -1634,6 +1646,12 @@ def start_autocontrol_gui(*args, **kwargs):
"control_type_name", "humanize_role", "humanize_tree",
"assign_node_paths", "find_by_path",
"is_interactive_role", "tab_order", "audit_focus_order", "focus_control",
"build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
"set_clipboard_rtf", "get_clipboard_rtf",
"set_clipboard_csv", "get_clipboard_csv",
"classify_format", "classify_formats", "diff_formats",
"list_clipboard_formats", "clipboard_formats",
"plan_file_drop", "drop_files",
# VLM locator
"VLMNotAvailableError", "locate_by_description", "click_by_description",
"verify_description",
Expand Down
Loading
Loading