diff --git a/AGENT_INSTALL.md b/AGENT_INSTALL.md index 4bf9b64a..359d4a5c 100644 --- a/AGENT_INSTALL.md +++ b/AGENT_INSTALL.md @@ -1,29 +1,66 @@ # browser-skill — Install Guide for AI Agents -You are an AI agent setting up browser-skill for the user. Done = `bsk doctor` -reports no `fail` checks. Explain any remaining warnings to the user; a warning -about local process identity permits browser use over working IPC. Never use -`sudo`; you cannot install the extension yourself (Step 3 is the user's job). +You are an AI agent setting up browser-skill for the user. Done = the intended +harness can load `browser-skill`, `bsk doctor` reports no `fail` checks, and a +small browser task succeeds and cleans up its session. Doctor alone does not +verify skill installation or discovery: no installed skill is reported as `N/A`. +Explain remaining warnings; a local process identity warning permits browser use +over working IPC. Never use `sudo`; the user installs the browser extension. -## 1. Install the CLI and skill +## 1. Install the CLI -Run the install command from [`README.md`](README.md): +For an existing installation, check `bsk --version` and follow the +[upgrade instructions](README.md#updating) if an update is needed. For a new install: **macOS / Linux:** ```bash curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh -bsk install-skill --yes +export PATH="${BSK_INSTALL_DIR:-$HOME/.local/bin}:$PATH" +bsk --version ``` **Windows (PowerShell):** ```powershell irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex -bsk install-skill --yes +bsk --version ``` -## 2. Run `bsk doctor` +The Unix installer cannot update its parent shell's PATH. Repeat the export in +later shell tool calls if needed, or use the installed binary's absolute path +(`~/.local/bin/bsk` by default; `~/.local/bin/bsk.exe` on Windows). A running agent +may retain its old PATH even after a new terminal picks up the installation. + +## 2. Install for the intended agent harness + +- **DeepSeek Harness (`dsh`):** follow the [plugin setup](README.md#deepseek-harness-plugin) + for the user's profile. The plugin supplies its own skill and native tools; + skip `bsk install-skill`. +- **Other supported harnesses:** inspect the available IDs and paths: + + ```sh + bsk install-skill --list --json + ``` + + Install into the intended harness explicitly. For Cursor, for example: + + ```sh + bsk install-skill --harness cursor --json + ``` + + Replace `cursor` with the ID from the list. Explicit `--harness` works even + when detection is false. `--yes` without `--harness` selects every detected + harness and fails if none are detected; it does not identify the current agent. +- **Unlisted harnesses:** follow the [manual skill instructions](README.md#quick-start) + and use that harness's documented skill directory. + +Check the install result and destination. Existing files are skipped; inspect +them before deciding whether to keep them or restore the bundled skill with +`--force`, which overwrites the file. Doctor explains paused automatic updates; +custom instructions are preserved. Verify discovery in Step 5. + +## 3. Run `bsk doctor` If this environment reaps child processes after every shell command, first follow the [sandbox setup guide](docs/sandboxed-agents.md): arrange a persistent daemon @@ -41,13 +78,16 @@ disabled, an unavailable daemon needs host-side startup using the same directory For a path/permission failure, use the resolved path in the report to check the shared directory and sandbox access rules; do not guess `/home/` or delete daemon files. A fresh install where only `extension connected` fails is expected; -go to Step 3. +go to Step 4. + +## 4. Connect the browser extension -## 3. Open the extension install page +If the intended browser is already connected, continue to Step 5. If its extension +is installed but disconnected, check that the connection is enabled and the local +port matches the daemon. For a server deployment, follow +[remote pairing](docs/remote-extension-connection.md). -If `extension connected` is `FAIL` (`0 browsers connected`), the user likely -has not installed or enabled the browser extension yet. Open the install page -matching the user's browser — +If the extension is not installed, open the page matching the user's browser — [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) for Chrome and other Chromium browsers, [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) @@ -58,5 +98,21 @@ for Microsoft Edge — then tell the user: > (or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) > on Edge), open the popup, and wait until it turns green. Reply when done. -If opening the page fails, give the user the same link. Then run `bsk doctor` -once more. No `fail` checks → tell the user it's ready and explain any warnings. +If opening the page fails, give the user the same link. Wait for the user's reply, +then run `bsk doctor` once more. + +## 5. Verify skill discovery and first use + +Confirm that the intended harness lists or can invoke `browser-skill`. If it needs +a new agent session or profile restart to discover the skill, tell the user how +to do that and report verification as pending until it has been loaded there. + +Use the installed skill to open `https://example.com` and summarize the page. +For CLI harnesses, start with `bsk session start --no-focus --json`, retain the +returned session ID, then navigate and observe using `--session `. Stop that +session with `bsk session stop ` on success or failure. With multiple browsers, +use `bsk browsers` and add `--browser ` when starting the session. +For dsh, use its injected `browser_*` tools instead. + +Report success only after the page is read and the test session is stopped. +If a step remains blocked, report which part is ready and what remains unverified. diff --git a/CHANGELOG.md b/CHANGELOG.md index b0900e71..16ac7d12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,39 @@ Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number. ### Added +- [Remote browser connections](docs/remote-extension-connection.md) with a built-in + server, one-use pairing links, device credential renewal and revocation, and + support for native TLS or a TLS reverse proxy. Remote upload and download are unsupported. +- [Operation audit](docs/operation-audit.md): opt-in task history stored on the + daemon host, with redacted operation metadata, export, deletion and 30-day retention +- [Full-page screenshots](docs/long-screenshot.md) from extension Quick Actions and + the CLI, with streamed PNG output, cancellation and lazy-loaded page capture +- Canvas visual refs, on-demand element screenshots and screenshot-bound point + clicks; observation cursor continuation when an explicit token limit is used - [Scroll-to element primitive](docs/scroll-to.md) across CLI, Extension and DSH Plugin, with ancestor-clipped visible bounds, iframe support and cooperative cancellation +- Native [mouse-wheel input](docs/wheel.md), explicit focus and blur actions +- [Host-managed daemon setup](docs/sandboxed-agents.md) with `BSK_HOME` and + `BSK_AUTO_START=0` for agents whose command sandboxes reap background processes +- Configurable local connection port in the extension popup and Korean localization + +### Changed + +- **Automation settings:** the extension's saved borrow-confirmation and human-help + switches govern existing and new sessions. `--unattended`, `tab borrow --no-confirm` + and `BSK_REQUEST_HELP=off` are deprecated compatibility inputs and cannot override + these switches. Set the browser preferences when upgrading unattended workflows. +- Managed CLI skills update only while their content matches the installed baseline; + custom instructions and local edits are preserved, with recovery guidance in `doctor` +- DSH Plugin: use the native browser sidebar when available, with a floating-panel fallback + +### Fixed + +- Windows installer path handling, verification of the resolved executable, and + replacement of installations using a daemon from another directory +- Browser connection preference recovery and compatibility during staggered component upgrades +- Observation document and frame geometry consistency, Canvas target identity checks, + and screenshot/session cleanup during cancellation or navigation ## [0.2.1] - 2026-09-09 diff --git a/README.md b/README.md index 0984ff5d..e8d471b5 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,7 @@ or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill ```bash curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh +export PATH="${BSK_INSTALL_DIR:-$HOME/.local/bin}:$PATH" ``` **Windows** (PowerShell — installs to `~/.local/bin`): @@ -93,7 +94,11 @@ curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.s irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex ``` -Verify the binary: +The export makes the CLI available in the current Unix shell. A running agent may +need the same PATH setting in each shell call, or the installed binary's absolute +path. Restart the agent if it retains an old PATH after installation. + +Verify the binary in the terminal or agent environment that will use it: ```bash bsk --version @@ -138,6 +143,11 @@ Use Space to select the Agent harness you want to install into, then press Enter to install the skill. Run `bsk install-skill --list` to see internal variants and install paths. +For non-interactive installation, specify the intended harness, for example +`bsk install-skill --harness cursor --json`. Explicit selection also works when +the harness is not detected. `--yes` alone installs into every detected harness +and fails when none are detected. + To install your own instructions, use `bsk install-skill --harness cursor --source ./SKILL.md`. An explicit `--source` stays custom even if its contents match the bundled skill. Existing installations are skipped unless you add `--force`. @@ -165,14 +175,59 @@ Other shell-capable agent harnesses are supported too. Copy `browser-skill/SKILL.md` to install the skill manually. DeepSeek Harness uses a dedicated plugin instead — see [DeepSeek Harness plugin](#deepseek-harness-plugin). +#### 4. Verify the connection + +Run `bsk doctor` and follow its hints. Open the extension popup and confirm it is +connected. Explain any warnings and resolve failures before testing browser use. +Doctor can pass with no skill installed (`N/A`); verify skill discovery separately. + -Start a new Agent session and write a prompt that needs the browser, for example: +Start a new Agent session, confirm `browser-skill` is available in the harness, +and ask it to open `https://example.com` and summarize the page. For harnesses +with slash-command skill invocation, for example: ```text /browser-skill open example.com and summarize what is on the page. ``` +A successful first-use check reads the page and stops its BrowserSkill session. +If the skill is missing, check the target harness and install path before retrying. + +### Updating + +For the default local setup, finish active browser tasks before updating: + +```sh +bsk update --yes +``` + +If Windows reports a staged update, wait for the replacement to finish before +checking `bsk --version`. + +When it installs an update, this command restarts a running daemon with default +startup settings. If you replaced the binary using the installer instead, restart +the existing daemon with `bsk daemon restart` after tasks finish. + +For a custom port, host-managed sandbox daemon, or remote server, stop the daemon +in its owning host or supervisor, run `bsk update --yes --no-restart-daemon`, and +start it there with its original flags and `BSK_HOME`. Set `BSK_AUTO_START=0` +in agent commands while managing it; see the [sandbox](docs/sandboxed-agents.md) +and [remote](docs/remote-extension-connection.md) setup guides. + +Update the extension through its browser store; for an unpacked development build, +rebuild and reload it. Store availability may lag the CLI release. Check the CLI, +daemon and extension versions with `bsk --version` and `bsk status`, then run +`bsk doctor`. New features such as full-page screenshots need matching builds. +Update the [DSH plugin separately](#deepseek-harness-plugin) and restart its profile. +Managed CLI skills synchronize on daemon startup, `session start`, or `doctor`; +local edits and custom skills are preserved. Start a new agent session to load +updated instructions. + +**Upgrading to 0.3.0:** `--unattended`, `tab borrow --no-confirm`, and +`BSK_REQUEST_HELP=off` no longer bypass confirmation or disable help. Choose the +corresponding extension settings described below. See [what changed](CHANGELOG.md). + ### Automation settings The extension popup has two independent **Automation settings**, both enabled by default. diff --git a/README.zh-CN.md b/README.zh-CN.md index a6e59a74..2679b361 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -73,6 +73,7 @@ BrowserSkill 由两个本地运行组件组成：`bsk` CLI/daemon 和浏览器 ```bash curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh +export PATH="${BSK_INSTALL_DIR:-$HOME/.local/bin}:$PATH" ``` **Windows**（PowerShell，安装到 `~/.local/bin`）： @@ -81,7 +82,10 @@ curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.s irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex ``` -验证二进制： +上面的 export 让当前 Unix shell 能找到 CLI。正在运行的 Agent 可能需要在每次 Shell +调用中设置同样的 PATH，或使用安装后二进制的绝对路径。如果 Agent 安装后仍沿用旧 PATH，请重启 Agent。 + +在实际使用工具的终端或 Agent 环境中验证二进制： ```bash bsk --version @@ -123,6 +127,10 @@ bsk install-skill 用 Space 选择需要安装的 Agent harness，然后按 Enter 安装 skill。运行 `bsk install-skill --list` 可查看 internal 变体及安装路径。 +非交互安装时显式指定目标 harness，例如 `bsk install-skill --harness cursor --json`。 +即使未检测到该 harness，也可显式选择。单独使用 `--yes` 会安装到所有检测到的 harness， +一个也未检测到时会报错。 + 安装自定义指令可运行 `bsk install-skill --harness cursor --source ./SKILL.md`。 显式指定 `--source` 的安装始终视为自定义，即使内容与内置 skill 相同。 已有安装默认跳过，添加 `--force` 才会覆盖。 @@ -143,14 +151,51 @@ daemon 启动、`session start` 和 `doctor` 会检查已安装的 skill：只 其他支持 Shell 的 Agent harness 也可使用 BrowserSkill，但需手动将 [`skill/SKILL.md`](skill/SKILL.md) 复制到对应 skills 目录下的 `browser-skill/SKILL.md`。DeepSeek Harness 走独立插件，见 [DeepSeek Harness 插件](#deepseek-harness-插件)。 +#### 4. 验证连接 + +运行 `bsk doctor` 并按提示处理，打开扩展弹窗确认已连接。测试浏览器操作前，说明警告并解决失败项。 +未安装任何 skill 时，doctor 仍可能通过（该项为 `N/A`）；skill 是否被发现需要单独验证。 + -启动一个新的 Agent 会话，写一条需要使用浏览器的 prompt，例如： +启动一个新的 Agent 会话，确认 harness 中可用 `browser-skill`，再让它打开 +`https://example.com` 并总结页面。对于支持斜杠命令调用 skill 的 harness，例如： ```text /browser-skill open example.com and summarize what is on the page. ``` +首次使用验证应成功读取页面，并停止本次 BrowserSkill session。 +如果找不到 skill，先检查目标 harness 和安装路径，再重试。 + +### 升级 + +默认本地配置下，先结束正在执行的浏览器任务，再更新： + +```sh +bsk update --yes +``` + +如果 Windows 提示更新已暂存（staged），请等待替换完成后再检查 `bsk --version`。 + +该命令安装新版本时，会以默认启动配置重启正在运行的 daemon。 +如果通过安装脚本替换了二进制，则在任务结束后运行 `bsk daemon restart`，重启已有 daemon。 + +对于自定义端口、宿主管理的沙盒 daemon 或远程服务器，先在所属宿主环境或进程管理器中停止 daemon， +运行 `bsk update --yes --no-restart-daemon`，再以原有参数和 `BSK_HOME` 在那里启动。 +维护期间，在 Agent 命令中设置 `BSK_AUTO_START=0`；详见[沙盒](docs/sandboxed-agents.md)和 +[远程连接](docs/remote-extension-connection.md)配置说明。 + +通过浏览器商店更新扩展；开发时加载的解压版本需要重新构建并重新加载。 +商店版本可能晚于 CLI 上线。使用 `bsk --version` 和 `bsk status` 核对 CLI、daemon 和扩展版本， +再运行 `bsk doctor`。长截图等新功能需要匹配的版本。 +[DSH 插件需要单独更新](#deepseek-harness-插件)，并重启对应 profile。 +受管理的 CLI skill 会在 daemon 启动、`session start` 或 `doctor` 时同步；本地编辑和自定义 skill 会保留。 +启动新的 Agent 会话以加载更新后的指令。 + +**升级到 0.3.0：** `--unattended`、`tab borrow --no-confirm` 和 `BSK_REQUEST_HELP=off` +不再跳过确认或关闭人工协助。请在扩展中选择下文说明的对应设置。版本变化见[更新日志](CHANGELOG.md)。 + ### 自动化设置与无人值守 插件弹窗提供两个默认开启的独立设置。**用户在插件中保存的设置对所有会话具有最终决定权：** diff --git a/apps/extension/PRIVACY.md b/apps/extension/PRIVACY.md index fd959d07..f337a232 100644 --- a/apps/extension/PRIVACY.md +++ b/apps/extension/PRIVACY.md @@ -1,6 +1,6 @@ # BrowserSkill — Privacy Policy -**Last updated:** September 14, 2026 +**Last updated:** September 15, 2026 This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable. @@ -22,7 +22,7 @@ Depending on the commands the user (via their AI agent) sends to the selected da | Category | What is accessed | Why | |---|---|---| -| **Web page content** | The DOM, accessibility tree, HTML, and visible-tab screenshots of pages opened in the BrowserSkill-controlled "Agent Window," or in user tabs the user explicitly approves for borrowing. | Required so the AI agent can read pages, locate elements, and verify results. | +| **Web page content** | The DOM, accessibility tree, HTML, and screenshots of pages controlled in the "Agent Window," tabs borrowed according to the browser's confirmation setting, or pages selected for user-initiated Quick Actions. | Required to read pages, locate elements, verify results, and capture requested screenshots. | | **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. | | **Tab and window metadata** | Tab IDs, URLs, titles and window IDs, including user tabs listed to select a tab for borrowing. | Required to target automation commands at the correct tab/window. | | **Local extension storage** | A randomly generated 8-character instance ID, an optional user-supplied label, feature preferences including the audit toggle, and optionally a paired endpoint and device credential. | Used to recognize this browser instance and restore user settings. | @@ -41,12 +41,15 @@ Depending on the commands the user (via their AI agent) sends to the selected da The Extension requests the following Chrome permissions. Each is used solely for the single purpose described above. -- **`debugger`** — Attach the Chrome DevTools Protocol to the Agent Window so the agent can observe and interact with pages. Used only on tabs explicitly under BrowserSkill's control. +- **`debugger`** — Attach the Chrome DevTools Protocol to tabs selected for automation or user-initiated capture, so BrowserSkill can observe, interact with, and capture those pages. +- **`activeTab`** — Allow temporary access to the active tab when the user invokes the Extension, for user-initiated Quick Actions. +- **`scripting`** — Inject the full-page screenshot helper into the selected page when it is missing, such as after an extension reload. +- **`webNavigation`** — Track page navigation and frames so captures, recordings, and human-help completion checks follow the correct document. - **`tabs`** — Inspect, create, and close tabs in the Agent Window; query tab metadata. - **`windows`** — Create and manage the dedicated Agent Window that isolates agent activity from the user's normal browsing. - **`alarms`** — Periodically wake the service worker to keep the selected connection alive and renew remote device authorization. - **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the selected WebSocket connection after the machine wakes. No idle data is stored or transmitted. -- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab. +- **`notifications`** — Show a system notification to obtain user approval before borrowing a user-owned tab when browser confirmation is enabled. - **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads. - **`storage`** — Persist a random instance ID, optional label, and feature preferences in `chrome.storage.local`. Remote credentials are kept separately in extension-origin IndexedDB. - **Host permission ``** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. Remote connections also use this permission for the selected endpoint. Remote page-content access requires a task-created or explicitly borrowed tab. @@ -74,7 +77,7 @@ Users can at any time: - Uninstall the Extension from `chrome://extensions`, which removes extension storage. Audit files on the daemon host and exported copies must be deleted separately. - Turn operation audit off in Quick Features to stop collecting new operations while retaining existing history. Previously recorded tasks still receive their final lifecycle status. - Close the Agent Window to stop all agent automation immediately. -- Deny tab-borrow notification prompts to keep their existing tabs off-limits. +- Enable confirmation before borrowing and deny tab-borrow prompts to keep existing tabs off-limits. - Disable the connection, choose Local connection, or stop the selected daemon to disconnect. - Revoke a paired device from the server with `bsk daemon revoke DEVICE_ID`, or use the gateway operator’s revocation controls. diff --git a/apps/extension/PRIVACY.zh-CN.md b/apps/extension/PRIVACY.zh-CN.md index eba74687..e7d6baed 100644 --- a/apps/extension/PRIVACY.zh-CN.md +++ b/apps/extension/PRIVACY.zh-CN.md @@ -1,6 +1,6 @@ # BrowserSkill 隐私政策 -**最后更新：** 2026 年 9 月 14 日 +**最后更新：** 2026 年 9 月 15 日 本隐私政策说明 **BrowserSkill** 浏览器扩展（下称"本扩展"）在您安装和使用过程中如何处理信息。BrowserSkill 作为开源项目 [BrowserSkill](https://github.com/Tencent/BrowserSkill) 的一部分发布，源代码完全公开，可供审计。 @@ -22,11 +22,12 @@ BrowserSkill 让用户指定的 AI 助手通过 `bsk` 命令行工具操作 Chro | 类别 | 访问内容 | 用途 | |---|---|---| -| **网页内容** | 由 BrowserSkill 控制的"代理窗口"中打开的页面，或用户明确授权"借用"的标签页中的 DOM、可访问性树、HTML、可见区域截图。 | 用于让 AI 助手读取页面、定位元素、验证执行结果。 | +| **网页内容** | "代理窗口"中的受控页面、按浏览器确认设置借用的标签页，或用户主动使用快捷功能选择的页面中的 DOM、可访问性树、HTML 和截图。 | 用于读取页面、定位元素、验证执行结果和生成用户请求的截图。 | | **AI 助手模拟的用户输入** | 通过 Chrome DevTools Protocol（CDP）派发的鼠标点击、键盘输入、表单值。 | 用于执行用户委托给 AI 助手的自动化操作。 | -| **标签页与窗口元数据** | 代理窗口及用户明确授权的标签页的 ID、URL、标题、窗口 ID。 | 用于将自动化指令发送到正确的标签页/窗口。 | +| **标签页与窗口元数据** | 标签页 ID、URL、标题和窗口 ID，包括为选择借用目标而列出的用户标签页。 | 用于将自动化指令发送到正确的标签页/窗口。 | | **扩展本地存储** | 一个随机生成的 8 位十六进制实例 ID、可选的用户自定义标签、功能偏好（包括审计开关），以及可选的远程端点和设备凭据。 | 用于识别此浏览器实例并恢复用户设置。 | -| **系统通知** | 当 AI 助手请求"借用"用户已有标签页时显示系统通知。 | 用于在助手访问任何已有标签页之前获取用户的明确同意。 | +| **Agent 请求的文件传输** | 用户明确提供给 `bsk upload` 的本地文件，以及单次 `bsk download` 操作产生的文件。 | 用于将任务文件附加到网页，或将浏览器生成的下载交给发起请求的本地 Agent。 | +| **系统通知** | 当 AI 助手请求"借用"用户已有标签页时显示系统通知的权限。 | 用于在浏览器借用确认开启时获取用户对该标签页的同意。 | ## 4. 数据收集边界 @@ -40,13 +41,18 @@ BrowserSkill 让用户指定的 AI 助手通过 `bsk` 命令行工具操作 Chro 本扩展申请以下 Chrome 权限，每一项都仅用于上述单一用途： -- **`debugger`** —— 将 Chrome DevTools Protocol 附加到代理窗口，使助手能够观察页面并与之交互。仅作用于明确处于 BrowserSkill 控制下的标签页。 +- **`debugger`** —— 将 Chrome DevTools Protocol 附加到选定的自动化或用户主动截图的标签页，使 BrowserSkill 能够观察、操作和截取这些页面。 +- **`activeTab`** —— 用户调用扩展时，临时访问当前标签页，用于用户主动发起的快捷功能。 +- **`scripting`** —— 选定页面缺少长截图辅助脚本时补充注入，例如扩展重新加载后。 +- **`webNavigation`** —— 跟踪页面导航和 frame，让截图、录制与人工协助完成检查对应到正确的文档。 - **`tabs`** —— 在代理窗口内查询、创建、关闭标签页；读取标签页元数据。 - **`windows`** —— 创建并管理独立的代理窗口，将助手的活动与用户的日常浏览隔离开。 - **`alarms`** —— 周期性唤醒 service worker，保持所选 WebSocket 连接并续期远程授权。 -- **`notifications`** —— 在助手请求借用用户标签页时，通过系统通知请求用户授权。 -- **`storage`** —— 在 `chrome.storage.local` 中保存随机实例 ID、可选标签及功能偏好。 -- **主机权限 ``** —— 在助手控制的页面上注入一个小型状态浮层（提示"代理已激活"），并支持用户引导助手访问任意网站。本扩展**不会**读取或传输助手未在控制的网站上的页面内容。 +- **`idle`** —— 检测设备从空闲或锁定状态恢复，以便及时重新建立所选 WebSocket 连接；不保存或传输空闲状态数据。 +- **`notifications`** —— 浏览器借用确认开启时，通过系统通知请求用户授权借用已有标签页。 +- **`downloads`** —— 关联和处理当前 `bsk download` 命令发起的单次下载。如果已认领的传输失败，BrowserSkill 会取消进行中的下载，或删除已完成的浏览器临时文件；不会枚举下载历史或更改未认领的下载。 +- **`storage`** —— 在 `chrome.storage.local` 中保存随机实例 ID、可选标签及功能偏好。远程凭据单独保存在扩展来源的 IndexedDB 中。 +- **主机权限 ``** —— 在助手控制的页面上注入状态浮层（提示"代理已激活"），并支持用户引导助手访问任意网站。远程连接也使用此权限访问用户选择的端点。远程页面内容访问要求标签页由任务创建或明确借用。 ## 6. 数据流向 @@ -58,6 +64,7 @@ BrowserSkill 让用户指定的 AI 助手通过 `bsk` 命令行工具操作 Chro - 实例 ID、可选标签和功能偏好会保存在 `chrome.storage.local` 中，直到用户卸载本扩展或清除扩展存储。 - 页面内容、截图、DOM 快照等观察到的数据，仅作为对指令的响应返回给所选守护进程或网关，录制或截图导出等操作可在扩展中暂存数据，直到相关操作结束或结果释放。 +- 上传和下载字节由本地守护进程暂存在私有、按 session 隔离的目录中。下载暂存文件在复制到指定目标后删除；上传暂存文件保留到 session 结束，以便页面稍后提交表单时仍可读取附件。session 结束、连接断开或守护进程在崩溃后再次启动时，会清理剩余暂存文件。 - 操作审计默认关闭。开启后，所选守护进程会在 `BSK_HOME/audit`（默认用户目录下的 `.bsk/audit`）保存任务与操作元数据，包括时间、工具类型、网站来源、经过基础脱敏的元素名称、状态和错误码；不保存输入值、页面正文、截图、脚本或文件内容。已结束的任务保留 30 天，在服务加载、开启审计、创建任务或查询列表时清理。用户可在审计页导出或删除已结束的记录。导出的副本不参与自动清理。 @@ -70,7 +77,7 @@ BrowserSkill 让用户指定的 AI 助手通过 `bsk` 命令行工具操作 Chro - 在 `chrome://extensions` 中卸载本扩展，从而清除扩展存储；守护进程所在主机的审计文件与自行导出的副本需要另行删除。 - 在快捷功能中关闭操作审计，停止收集新操作并保留已有记录；已记录任务仍会更新最终生命周期状态。 - 关闭代理窗口，立即停止所有助手自动化操作。 -- 拒绝标签页借用通知，将自己已有的标签页隔离在助手访问范围之外。 +- 开启借用前确认，并拒绝借用请求，使已有标签页保持不可访问。 - 关闭连接开关、切回本地连接或停止所选守护进程。 - 在服务器执行 `bsk daemon revoke DEVICE_ID` 撤销设备授权，或使用网关运营者提供的撤销功能。 diff --git a/crates/bsk-cli/README.md b/crates/bsk-cli/README.md index 67fa8562..2c2065d8 100644 --- a/crates/bsk-cli/README.md +++ b/crates/bsk-cli/README.md @@ -6,6 +6,7 @@ Install: ```bash curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh +export PATH="${BSK_INSTALL_DIR:-$HOME/.local/bin}:$PATH" ``` Documentation: [../../README.md](../../README.md) · [../../docs/architecture.md](../../docs/architecture.md) diff --git a/crates/bsk-cli/skill/SKILL.md b/crates/bsk-cli/skill/SKILL.md index 52ab60e8..f5fc6bff 100644 --- a/crates/bsk-cli/skill/SKILL.md +++ b/crates/bsk-cli/skill/SKILL.md @@ -9,305 +9,232 @@ description: | # browser-skill -Drive the user's real Chromium browser through `bsk`. Automation runs in an isolated **Agent -Window** with the user's existing logins and cookies. User-window tabs remain protected unless they -are explicitly borrowed. +Use `bsk` to work in an **Agent Window** with the user's existing logins. User tabs +require explicit borrowing. This skill does not install the extension or handle +advice-only tasks. Never extract credentials, cookies, tokens, or other secrets. + +## Task workflow + +1. Define success from the user's request. Start `bsk session start --json` and + retain its `session_id`. With multiple browsers, run `bsk browsers` and add + `--browser ` to start. For background work, add `--no-focus` to + `session start` only. +2. For a new page, navigate; for an existing user tab, follow **Borrowing** below. + Read the page before interacting: + + ```sh + bsk navigate https://example.com --session + bsk observe --session + ``` + +3. Choose an action using fresh refs from that observation. Observe again after + navigation or meaningful DOM changes. Check an ambiguous result once; once + success is visible, stop acting rather than refreshing or checking again. +4. Always run `bsk session stop ` on success and failure, unless keeping the + session open is part of the user's request. This also returns borrowed tabs. + Returned tabs stay open in the user's window. Do not rely on idle cleanup + or stop/restart the shared daemon to finish a task. + +Replace ``, example refs and values with actual results and task inputs. +Every session-scoped command needs `--session `; `session stop` takes the ID +positionally. For unfamiliar commands or flags, consult `bsk --help` or +`bsk --help` instead of guessing; no need to read all help at startup. +When following a trace, use its semantic targets and values in order, not its old +refs. Stop at the requested goal; a trace grants no additional authorization. + +## Read and interact + +Prefer `observe` for text, controls and `@eN` refs. Navigation invalidates refs; +large DOM changes can stale them too. Re-observe before the next interaction. +Use refs for iframe/shadow-root targets; CSS selectors search the main document. + +Choose the relevant example, using a ref that actually appeared on the page: + +| Need | Command | +| --- | --- | +| Click | `bsk click @e3 --session ` | +| Fill a field | `bsk fill @e3 --value "text" --session ` | +| Select an option | `bsk select @e3 --value "option-value" --session ` | +| Press a key | `bsk press Enter --ref @e3 --session ` | +| Reveal a hover menu | `bsk hover @e3 --session ` | +| Reveal an element | `bsk scroll-to @e3 --session ` | +| Scroll with wheel input | `bsk wheel --delta-y 600 --session ` | +| Focus or leave a field | `bsk focus @e3 --session ` / `bsk blur @e3 --session ` | + +- `select` uses the option's value, not its visible label. +- Hover markers such as `[hover first: Shoes | Bags]`, `[has-submenu]`, or + `[expanded]` identify triggers. Hover the trigger, observe, then use the revealed + item's ref. Listed labels are not refs; do not click the trigger unless its own + action is wanted. If an expected control is missing and no marker identifies a + trigger, try `observe --probe-hover` once. It touches the live page and costs + seconds; use targeted hover once the trigger is known. +- `scroll-to` returns ancestor-clipped bounds in top-level viewport CSS pixels. + Partial visibility suffices; hidden/fully clipped targets fail. It does not test + occlusion. `wheel` sends signed deltas (at least one nonzero), not a guaranteed + scroll distance. An optional target is scrolled into view first; without one, + input lands at the viewport centre. Observe to check the page's response. + +Use `snapshot` for a static accessibility tree, `get-html` for exact markup or +hidden metadata, and `screenshot` for visual content or requested visual evidence. +Do not start with HTML/images just to find ordinary controls; obtain fresh refs +before interacting with controls found that way. + +### Large observations + +There is no default token cap. With `observe --max-tokens `, follow a returned +`next_cursor`/`@more` when relevant content remains: -Do not use this skill for tasks with no browser, for extension installation, or when the user only -wants instructions. Never extract credentials, cookies, tokens, or other secrets from pages. +```sh +bsk observe --cursor --session +``` -## Required lifecycle +Each page replaces the ref map: use its refs before continuing and never reuse +refs from earlier pages. Continuation reads the same capture, without refreshing +or hovering; do not combine it with depth changes or hover probing. New observe/ +snapshot or changed page identity invalidates continuation; then observe afresh. -Every browser task owns a bounded session: +## Borrowing and browser settings -```text -1. bsk session start # retain the printed 4-letter session id -2. bsk ... --session # pass it to every session-scoped command -3. bsk session stop # always run on success and error paths -``` +List before borrowing, and return the tab as soon as the relevant step ends: -Do not rely on the idle timeout for cleanup. Stop the session as soon as the goal is met unless the -user explicitly asks to keep it open. Stopping also returns borrowed tabs. - -By default, browser commands auto-start the daemon when needed. Keep the shared daemon running; -task cleanup is `bsk session stop`, not `bsk daemon stop` or `restart`. - -If the agent environment kills background children when each shell command ends (as reported for -Linux WorkBuddy), arrange a persistent daemon outside that per-command sandbox first. The user -can run `BSK_HOME=/absolute/shared/bsk bsk daemon start` in a normal host terminal. A host-managed -background task can instead run `bsk daemon start --foreground` with the same `BSK_HOME`, using -the host's approved execution path. Do not disable sandbox protection for browser task commands. - -In that environment, pass `BSK_HOME=/absolute/shared/bsk BSK_AUTO_START=0` to **every** `bsk` -command. Replace the example path with one dedicated directory that both sides can access, -including its IPC socket; an `export` in one shell tool call may not persist to the next. If the -daemon is unavailable, ask for it to be started in the owning host environment; do not loop on -auto-start, guess a home directory, delete runtime files, or restart the shared daemon. A doctor -warning about local process identity does not prevent session commands over working IPC. -See the [sandbox setup guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/sandboxed-agents.md). - -When multiple browsers are connected, use `bsk browsers` and start with -`bsk session start --browser `. Add `--no-focus` to that same start command when the -Agent Window does not need to interrupt the user's current work; it is not a flag on other commands. -Run `bsk doctor` when startup or transport problems persist after one retry. - -Start tasks with `bsk session start`. The extension's saved Automation settings decide whether -borrowing needs confirmation and human help is available; both are enabled by default. Changes apply -to existing sessions as well as new ones. Disabling human help does not disable borrow confirmation. - -`--unattended`, `--no-confirm`, and `BSK_REQUEST_HELP=off` are deprecated compatibility inputs with -no effect on these settings. Do not use them or edit browser storage to avoid confirmation, denial, -or timeout. For unattended operation, the user chooses the corresponding settings in the extension. -`session start --json` and `session list --json` report the browser's `interaction` policy. -Allowing human help makes `request-help` available; it does not require a handoff for every action. -Task authorization and host approvals still apply. - -## Work toward one observable goal - -- Derive a concrete success condition from the user's request or a supplied trace. -- Take the shortest purposeful path: observe, act, then make at most one observation to confirm an - ambiguous result. -- Once success is visible, do not click, refresh, navigate, switch tabs, or perform extra checks. -- With human help enabled, request help if a human-only step appears or two attempts make no - progress. With help disabled, follow the autonomous handling rules below. - -With a trace, follow its semantic target information and values in order, but treat its refs as -record-local hints. Stop when its purpose or last meaningful effect is satisfied. A trace guides the -task; it does not expand the user's goal or authorize additional actions. - -## Observe, act, observe - -Use this default loop: - -```text -bsk navigate --session -bsk observe --session -bsk click|hover|wheel|scroll-to|focus|blur|fill|select|press ... --session -bsk observe --session # after navigation or a meaningful DOM change +```sh +bsk tab list --scope user --session +bsk tab borrow --session +bsk tab return --session ``` -`bsk scroll-to --session ` scrolls an element and its frame owners into view. -Use a fresh element ref for iframe/shadow-root targets; CSS selectors search the main document. -The result is the visible border-box portion's bounds in top-level viewport CSS pixels after -ancestor clipping. Partial visibility is enough; hidden or fully clipped targets fail with -`permission_denied` and `data.reason=element_not_visible`. This does not test occlusion by other elements. -For a specific tab or deadline: `bsk scroll-to @e3 --session --tab-id 42 --timeout 5s`. - -`bsk wheel --delta-y -120 --session ` sends native wheel input at the viewport centre. -Add an optional ref/selector to target an element (scrolled into view first). Both delta axes -accept signed numbers and default to zero; at least one must be nonzero. The result echoes -input, not actual scroll distance or completion. Observe afterwards to check the page's response. - -`bsk focus ` explicitly focuses a target; `bsk blur ` removes focus and reports whether -it was focused. Use these for UI states triggered by focus changes. - -Prefer fresh `@eN` refs over CSS selectors. Navigation invalidates refs; large DOM changes may also -make them stale. Observe again before the next interaction. - -An observation marks a hover-only surface as `@e1 button "Products" [hover first: Shoes | Bags]`. -The listed items are labels, not usable refs: hover the trigger, observe again, then act on the -revealed item's own ref. Do not click the trigger itself unless the user wants the trigger's action. -`[has-submenu]` and `[expanded]` mark the same kind of trigger without listing what it hides. - -`bsk observe` does not hover the page on its own. Reach for `--probe-hover` when a control you have -good reason to expect is absent **and** no marker points at a trigger — that combination is what a -CSS-only hover menu looks like from here. It hovers a bounded set of likely triggers, so it costs a -few seconds and touches the live page; once you know which element hides the menu, `bsk hover ` -is cheaper and more precise. - -Escalate page reading only as needed: - -1. `bsk observe` for normal semantic understanding, text, controls, and refs. -2. `bsk observe --probe-hover` once when an expected control is missing and no marker points at a - trigger. -3. `bsk snapshot` when a stricter static accessibility tree is more useful. -4. `bsk get-html` for exact markup or hidden metadata that semantic views cannot provide. -5. `bsk screenshot` for layout, styling, canvas, images, or requested visual evidence. - Use `--full-page` when the user wants a long screenshot of the whole ordinary webpage. - -Do not start with raw HTML or screenshots merely to discover ordinary controls. When interaction is -needed, obtain a fresh observation before acting on screenshot or HTML findings. - -## Canvas and observation continuation - -`observe` may place `@eN canvas [visual:screenshot]` near related page controls. Names are -optional: do not infer a table title or controls inside Canvas from adjacent labels. Visual refs -support `screenshot --ref`; point clicks additionally require its `capture_id` and image coordinates. -They do not support fill/hover or HTML extraction. First observe returns text, -not an image. Use the surrounding semantics to decide whether a Canvas screenshot is needed. -If you cannot receive and understand images in this session, tell the user the Canvas contents -cannot be interpreted and ask them to switch to an image-capable model; continue with available -semantic information. BrowserSkill does not detect the model's capabilities. - -There is no default token cap. With an explicit `--max-tokens` limit, an observation may return -`next_cursor` and an `@more` instruction. Use current refs before calling -`bsk observe --cursor --session `: each response replaces the ref map, so refs from -previous pages must not be reused. A response can contain many Canvas entries. Follow cursors -when relevant content remains, rather than repeatedly reading the same prefix. -Continuation reads the same captured observation; it does not refresh or hover the page. Do not -combine it with depth changes or hover probing. A new observe/snapshot replaces the continuation; -if the page identity changed, observe again. Screenshot execution checks current target identity -and geometry, but permits Canvas repainting and does not freeze pixels. - -A Canvas screenshot can return `capture_id`. To click a point you identified in that image, use -`bsk click eN --capture --image-x --image-y --session `. -Use original PNG pixels (returned width/height), not resized display or viewport coordinates. -Captures are single-use, expire after two minutes, and are invalidated by a newer screenshot of -that ref or observation/continuation. With `capture_unavailable`, view the image but observe and -screenshot again before clicking. Click counts 1/2, buttons and modifiers are supported. -After clicking, observe or screenshot to verify the result; use DOM refs for revealed controls. -A completed click does not prove business success. Canvas repainting is allowed; changed identity, -geometry or hit target is rejected. If `effect_state=unknown`, inspect before retrying with a new -capture. Do not infer cell-editing, IME, drag or hover support from point-click capability. - -## Respect the Agent Window boundary - -Normal page writes affect only Agent Window tabs. To operate a user tab, first list it with -`bsk tab list --scope user --session `, then `bsk tab borrow `. Return it immediately -after the relevant step with `bsk tab return `; never invent a tab id or keep a personal tab -borrowed across unrelated work. - -`tab borrow --timeout 120s` changes the confirmation wait (default 60s), not whether approval is -required. Custom waits need daemon and extension protocol 1.2+. Compatible older peers can still -start sessions and borrow with their default wait. The current CLI's `request-help` needs daemon -protocol 1.3; an unsupported operation does not make the connection unusable. Update all three -components for full browser-setting enforcement; older programs may still end help locally. -Repeating a completed borrow in the same session returns -its existing result. Do not repeat pending requests, denied requests, or confirmation timeouts, -or switch to another browser tool to bypass them. If `reason` is `borrow_outcome_unknown`, inspect -tab and session state before continuing; the tab may already have moved. - -## Ask the human when needed - -When human help is enabled (the default), use `bsk request-help` for login, captcha, OTP, payment -confirmation, consent, or another step the user must complete. Give a precise prompt and pass -fresh `--target` refs/selectors when concrete controls can be highlighted. Use completion criteria -only when the page has a clear stable success signal. - -The result `outcome` is one of `continued`, `completed`, `cancelled`, `timed_out`, or `disabled` -(`navigated` is deprecated — never treat navigation as a completion signal). After a human handoff, -resume only after `continued` or `completed`. Treat `cancelled` as rejection and `timed_out` as a -blocker; do not repeat that request. Observe again after control returns before using refs. - -When help is disabled in the extension, make every -reasonable effort to complete the task autonomously with BrowserSkill. Do not call `request-help`. -If a call returns `disabled`, no human action was confirmed: re-observe and continue working rather -than marking the step blocked merely because help is unavailable. - -Disabling help adds no permission: keep task authorization and host restrictions in force. -Use the current page, existing login state, and authorized credentials or codes to complete the -current step. Where the task authorization and host rules allow, a model with image understanding -may attempt graphical verification through screenshots and supported interactions. Phone-only QR -scans, face verification, and unavailable SMS codes may remain blocked; a text-only model may also -leave an image-only CAPTCHA unresolved. Attempt other authorized steps within available capabilities -and verify the actual result before concluding they cannot be completed. - -After a failed attempt, re-observe and try a different viable approach when available. Do not loop -on identical failures or repeat an action whose outcome is unknown. Report a specific blocker only -when required information or capability is missing, or viable approaches are exhausted; continue -independent work. Do not re-enable help or switch browser backends to work around those limits. - -## Command inventory - -This list of names is complete. Never invent a command outside it; read -`bsk --help` for flags instead of guessing them. - -```text -session start|stop|list browsers status doctor update logs -navigate navigate-back navigate-forward reload wait-for-navigation wait-ms -observe snapshot get-html screenshot console network -click hover wheel scroll-to focus blur fill select press evaluate -tab list|create|close|select|borrow|return window resize emulate -upload download request-help record start|stop -``` +Never invent tab IDs or keep a user tab across unrelated work. Do not repeat +pending, denied or timed-out borrows. For `borrow_outcome_unknown`, inspect tab/ +session state first: the tab may already have moved. Do not bypass an outcome +through another browser backend. `tab borrow --timeout 120s` changes only the +confirmation wait (default 60s); custom waits require daemon and extension protocol 1.2+. -Flags and argument forms that are easy to get wrong: +The extension's saved Automation settings control borrow confirmation and human +help independently; both default on and apply to existing sessions too. Read +`interaction` in `session start --json` or `session list --json` when needed. +Deprecated `--unattended`, `--no-confirm`, and `BSK_REQUEST_HELP=off` cannot override +these settings. Never change browser storage/settings to bypass them. Human-help +availability does not require permission for every action or grant extra authority. +`request-help` requires daemon protocol 1.3; update CLI, daemon and extension for +full settings support. A feature's version error does not disable other operations. -```text -bsk fill --value bsk select --value -bsk screenshot --out bsk emulate --device -bsk upload --file bsk download --out -``` +Remote content reads/actions require task-created or borrowed tabs. Page-opened +popups gain no control automatically; an unowned tab inside the Agent Window +needs the user to move it to a user window before borrowing. Remote upload/download +are unsupported; screenshots work. See the [remote guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/remote-extension-connection.md). -`select` matches an option's `value` attribute, not its visible label. Device preset ids are -lowercase and hyphenated, such as `iphone-14`. +## Human steps and recovery -- `console` and `network` provide bounded, read-only debugging evidence. -- `emulate` applies viewport, user-agent, and touch overrides to one tab; new tabs do not inherit - them. Use `--off` to restore the real environment. -- `evaluate` is a last resort when observe plus normal interactions cannot complete the task. With - `--json`, inspect `.ok`: a JavaScript exception may still have CLI exit code 0 because the RPC - succeeded. Never evaluate credential surfaces to read storage, cookies, or auth data. -- `record` captures a user's actions for later replay. Read `bsk record start --help` before use, - and never record banking, SSO, password-manager, or other sensitive pages. +With help enabled, request help for login, CAPTCHA, OTP, payment confirmation, +consent, or after two attempts make no progress: -## Screenshots +```sh +bsk request-help --session --prompt "Please complete sign-in" --target @e3 +``` + +Use a precise prompt and fresh targets; omit `--target` when no control fits. +Use completion criteria only for a clear, stable success signal. + +| Result | Next step | +| --- | --- | +| Help `continued` / `completed` | Observe again, then resume with fresh refs. | +| Help `cancelled` / `timed_out` | Respect rejection or the blocker; do not repeat the request. | +| Help `disabled` | No human action was confirmed. Re-observe and follow the disabled-help rules below. | +| Stale ref | Observe and retry the intended action once. | +| Unknown tab/session | List current tabs/sessions; never guess IDs or use another task's session. | +| Timeout or unknown effect | Inspect current state before retrying; the action may already have happened. | +| `fill_value_mismatch` | Read the field: formatting may still satisfy the request. Correct only a remaining difference; no blind refill or immediate handoff. | +| Unsupported operation | Use available capabilities; suggest updating only if the missing feature is needed. | + +Navigation alone (including deprecated help outcome `navigated`) is not completion. +For other errors, follow the returned hint and inspect the current state. + +**Help disabled:** do not request help or re-enable it. Use existing login state, +authorized inputs and viable alternatives; disabling help adds no permission and +does not remove borrow confirmation or host restrictions. Where authorized, a +vision-capable model may attempt graphical verification. Phone-only QR scans, +face verification, missing SMS codes or image-only tasks for a text-only model +may remain blocked. Report a specific blocker only when inputs/capabilities are +missing or viable approaches are exhausted; continue independent work. Do not loop +on identical failures, repeat unknown effects or switch backends to bypass limits. +On an unrecoverable failure, report the blocker and stop the owned session. + +## Screenshots and Canvas ```sh bsk screenshot --session --out viewport.png -bsk screenshot --session --ref @e3 --out element.png +bsk screenshot --session --ref @e3 --out element.png --json bsk screenshot --session --full-page --out page.png -bsk screenshot --session --full-page --timeout 5m --out page.png ``` -Without `--ref` or `--full-page`, capture only the visible viewport. `--full-page` and -`--ref` are mutually exclusive. Full-page mode scrolls the document from top to bottom, -follows content loaded during scrolling, and restores the original position and styles. -It is page input: use a selected, session-controlled tab in the Agent Window (create or -borrow first), keep the viewport stable, and respect user interrupts. `--tab-id` targets -a specific tab without selecting it. Chrome internal pages, the Web Store, nested scroll -containers and virtualized lists are not supported by automatic full-page capture. - -Capture and PNG encoding default to two minutes. `--timeout` only applies with -`--full-page`; allow your shell runner enough time for that deadline plus file transfer. -Increase it for longer pages, but do not blindly retry an endlessly growing -page or a cancelled request. Ctrl-C cancels. Failure produces no partial output. -The CLI streams the PNG to disk and returns its path; `--json` also reports dimensions -and byte size. An existing `--out` file is replaced only after a full-page image is -received completely. Omitting `--out` uses a temporary path. No popup preview opens. -Use the matching CLI and extension builds; an unknown full-page RPC indicates an older -extension, not a reason to silently substitute a viewport screenshot. - -## File transfer - -`upload` and `download` stage files through the daemon; the agent never touches browser-internal -paths. Treat upload as disclosure to the website, download as accepting website-controlled bytes. - -Upload has two independent mechanisms — choose explicitly, never rely on automatic fallback: - -- **Default (input mode):** for upload buttons, file-input labels, or "upload from computer" - actions. The command clicks the target and intercepts the native file chooser. -- **`--mode drop`:** for reliably identified attachment-receiving areas — an explicit drop zone, - chat composer, email editor, or form attachment area. Do not target page whitespace, generic - containers, or areas whose attachment ownership is ambiguous. - -Decision sequence when uploading: - -1. Try input mode (the default). -2. If it returns `reason=file_input_not_activated` with `effect_state=none`, re-observe. When a - reliable attachment target exists, try `--mode drop` once against that target. -3. Otherwise request help if enabled; when disabled, follow the autonomous handling rules above. -4. **Never** switch mechanisms or repeat when `effect_state` is `unknown` or `committed` — the - browser may already have applied the file. - -A successful drop means Chrome dispatched the native file-drop event; it does not prove the site -accepted the attachment. Observe the page once after the command. - -Download default-refuses to overwrite; pass `--overwrite` when replacing an existing file is -intended. Read `bsk upload --help` and `bsk download --help` for all flags and error details. - -## Recover without wandering - -- Stale ref: observe again and retry the intended action once. -- Unknown tab or session: list current tabs/sessions; never guess identifiers. -- Timeout: inspect current page state before deciding whether one longer purposeful wait is useful. -- Fill result unconfirmed (`fill_value_mismatch`): observe the field first; the page may have - formatted the value. Continue if the visible result satisfies the user's intent. Otherwise correct - the remaining difference; do not blindly repeat fill or immediately request human help. For other - fill errors, follow the returned hint and inspect current state before retrying. -- Unsupported command: continue with available capabilities; suggest updating only when the missing - command is necessary. -- Unrecoverable failure: report the blocker and stop the session in a finally-style path. - -The CLI's current help and error hints are authoritative for flags, parameters, and recovery -details. +Screenshots return a local PNG path; view the image to interpret it. `--out` +replaces an existing file; omitting it uses a temporary path. `--json` includes +dimensions and byte size. `--ref` and `--full-page` cannot be combined. + +Full-page mode scrolls an ordinary webpage and restores its position/styles. +Use a selected, session-controlled tab and stable viewport; `--tab-id` targets a +tab without selecting it. Internal browser pages, the Web Store, nested scrolling +panels and virtualized lists are unsupported. Capture/encoding defaults to 2m; +`--timeout 5m` extends it only in full-page mode. Allow the shell enough time for +capture plus transfer. Respect cancellation; do not blindly retry endless pages +or substitute a viewport image when an older extension rejects full-page capture. +Use matching CLI/extension builds. Ctrl-C cancels; failed full-page captures save +no partial image. + +For `@eN canvas [visual:screenshot]`, observe returns text, not pixels. Screenshot +that ref when its contents matter; never infer Canvas controls or names from +nearby labels. If images cannot be received/understood, explain the limitation, +ask for an image-capable model when needed, and continue with available semantics. + +To click a point seen in a Canvas image, retain that screenshot's `capture_id`: + +```sh +bsk click @e3 --capture --image-x --image-y --session +``` + +Use ORIGINAL PNG coordinates and dimensions, not resized display/viewport pixels. +Captures are single-use, expire after 2m, and are invalidated by ref replacement +(observe/snapshot/continuation) or a newer screenshot of that ref. With +`capture_unavailable`, the image is view-only: observe and screenshot again before +clicking. Counts 1/2, buttons and modifiers work; Canvas fill, IME, drag, hover +and HTML extraction do not. Repainting is allowed; changed identity/geometry/hit +targets are rejected. Verify the result, using DOM refs for revealed controls; +inspect `effect_state=unknown` before retrying with a new capture. + +## Files and other tools + +```sh +bsk upload @e3 --file ./report.pdf --session +bsk download @e3 --out ./report.pdf --session +``` + +Upload discloses the file to the site; download accepts site-controlled bytes. +Use agent-local paths, not browser-internal staging paths. + +- Default upload clicks an upload button/label and intercepts its file chooser. +- If `reason=file_input_not_activated` and `effect_state=none`, re-observe. Try + `--mode drop` once only on a clear attachment target such as a drop zone or + composer, never whitespace or an ambiguous container. Otherwise follow the + human-help rules. There is no automatic fallback between mechanisms. +- Never retry or switch upload modes for `effect_state=unknown` or `committed`. + A successful drop proves dispatch, not site acceptance; observe the attachment. +- Download refuses overwrite by default; add `--overwrite` only when replacement + is intended. Consult each command's help for other flags. + +Use `console` / `network` for bounded read-only diagnostics; follow returned +sequence cursors. `emulate --device iphone-14` affects one tab; `--off` restores it. +`evaluate` is a last resort: inspect JSON `.ok`, since a script exception can have +CLI exit code 0. Never evaluate secrets. `record start` captures user actions; +read its help first and never record banking, SSO or password-manager pages. +Use `bsk --help` to find navigation/history, tab, wait and window commands. + +## Startup problems + +Commands normally auto-start the daemon. After one failed retry, use `bsk doctor`. +If command sandboxes reap background processes, arrange a persistent daemon in +its owning host environment using the [sandbox guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/sandboxed-agents.md). +Use the same accessible `BSK_HOME` and `BSK_AUTO_START=0` on EVERY sandboxed command; +exports may not persist between shell calls. Keep browser commands sandboxed. +A missing daemon needs host-side startup, not repeated auto-start, guessed home +paths, deleted runtime files or a shared-daemon restart. A local process identity +warning permits browser commands when IPC works. diff --git a/docs/architecture.md b/docs/architecture.md index 8ce92408..ac1a9321 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -5,6 +5,10 @@ Consolidates design §2, §3, §5, and §6. ## System diagram +The diagram shows the default local setup. In [remote mode](remote-extension-connection.md), +the CLI and daemon run on a server; the extension connects from the user's browser +over authenticated WSS. CLI-to-daemon communication remains local IPC on the server. + ```mermaid flowchart TB subgraph harness [Agent harness] @@ -46,7 +50,8 @@ Key modules: ### bsk daemon (same binary: `bsk daemon`) -- Listens on loopback WebSocket (default **52800**, configurable with `bsk daemon start --port`) for extensions. The extension popup saves the matching connection port; saving ends existing sessions and reconnects when enabled. +- In local mode, listens on loopback WebSocket (default **52800**, configurable with `bsk daemon start --port`) for extensions. The extension popup saves the matching connection port; saving ends existing sessions and reconnects when enabled. +- Server mode supports authenticated remote extension connections with device pairing, renewal and revocation. Native TLS or a TLS reverse proxy provides WSS; the deployment supervisor owns server restarts. - Validates `Origin: chrome-extension://…` on handshake. - Maintains `browsers` (connected extensions) and `sessions` (Agent Window bindings). - **Per-session queue** serializes tool calls targeting one session. @@ -101,6 +106,9 @@ mutation for session queueing and user-interruption gating. - **Session stop is mandatory** in agent workflows (`bsk session stop`); idle timeout (default 5 min) is a safety net only. - Multiple sessions on one browser → multiple Agent Windows, fully isolated. +- Remote content reads and actions require task-created or explicitly borrowed tabs. + A page-opened popup or a user tab moved into the Agent Window does not become + controlled automatically; see [remote tab ownership](remote-extension-connection.md#browser-permissions-and-task-lifetime). ### tab_list scopes @@ -144,15 +152,24 @@ flowchart LR CLI <-->|JSON protocol| T ``` -## Security (v1) +## Connection security -- Daemon and WebSocket bind to **loopback** only. +- Local mode binds WebSocket to **loopback**. Server mode permits remote access + through authenticated WSS; plaintext listeners remain on loopback behind TLS + termination or for development. - Extension origin allow-list at WS upgrade. -- No credential storage in bsk — cookies stay in the user's browser profile. +- Website cookies stay in the user's browser profile. Remote device credentials + are stored in extension-origin IndexedDB; the built-in server stores credential + hashes in its private `BSK_HOME`. Pairing and device grants govern remote access. - `evaluate` restricted to Agent Window tabs in sandbox mode. +- Operation audit, when enabled, is stored on the daemon host, including the + server in remote mode. See [operation audit](operation-audit.md). ### File-transfer boundary +Upload and download are supported only for local connections. Remote sessions +return `unsupported`; screenshots and other RPC content results remain available. + - The invoking agent/harness decides whether a transfer is authorized and supplies the task-local source or destination path. - The CLI is the only component that reads an upload source or writes the final download destination. Before browser dispatch it owns rollback of partially staged uploads; after dispatch, ownership moves to the session because a transport timeout cannot prove that Chrome did not attach the file. Download output becomes visible through one atomic commit, and replacement is opt-in without a pre-delete window. The extension never receives either agent-facing path. - The daemon is the authority for storage capabilities and limits. It issues opaque session-scoped transfer IDs, stages bounded chunks in a private runtime directory, and injects only private staged upload paths. For download it mints one relative Chrome directory capability. Only after validating the reported path, file type, symlink boundary, and authoritative byte limit does it take ownership of browser-file cleanup and import the bytes. diff --git a/docs/operation-audit.md b/docs/operation-audit.md index f9358333..87915bd5 100644 --- a/docs/operation-audit.md +++ b/docs/operation-audit.md @@ -1,6 +1,8 @@ # 操作审计 -操作审计帮助个人用户按任务回看 BrowserSkill 代为执行的操作。默认关闭，记录保存在运行本地服务的电脑上，不上传到远端。 +操作审计帮助个人用户按任务回看 BrowserSkill 代为执行的操作，默认关闭。 +记录保存在所选 daemon 所在主机：本地连接时在本机，远程连接时在服务器。 +BrowserSkill 作者不接收这些记录；远程部署的数据流向见[远程浏览器连接](remote-extension-connection.md)。 ## 使用入口 @@ -10,7 +12,7 @@ - 开启后显示最近 5 次任务，包括执行中的任务，并提供「查看全部任务」。 - 点击任务，在独立扩展标签页查看任务摘要、时间线、操作状态和可展开的技术详情；页面每 3 秒更新一次。 - 全部任务页提供分批加载、实际保存目录和「打开文件夹」。详情页支持导出 JSON，以及确认后删除已结束任务。执行中的任务不能删除。 -- 本地服务离线、版本不支持或存储故障时显示明确提示，不能把失败显示成「暂无任务」。离线时仍可保存开关偏好，下次连接时生效。 +- 所选服务离线、版本不支持或存储故障时显示明确提示，不能把失败显示成「暂无任务」。离线时仍可保存开关偏好，下次连接时生效。 ## 任务与结果的含义 @@ -28,13 +30,16 @@ bsk session start --name "整理本周待办" ## 保存位置与保留期限 +以下目录均位于 daemon 所在主机，使用运行 daemon 的系统账户： + | 系统 | 默认目录 | | --- | --- | | macOS | `/Users/<用户名>/.bsk/audit/` | | Windows | `C:\Users\<用户名>\.bsk\audit\` | | Linux | `/home/<用户名>/.bsk/audit/` | -设置 `BSK_HOME` 时保存到该目录下的 `audit` 子目录。以扩展页展示的实际路径为准。开关偏好保存在浏览器配置文件的 `chrome.storage.local` 中；审计正文由本地服务保存，不占用扩展存储配额。卸载扩展不会删除这些本地文件。 +设置 `BSK_HOME` 时保存到该目录下的 `audit` 子目录。以扩展页展示的实际路径为准。开关偏好保存在浏览器配置文件的 `chrome.storage.local` 中；审计正文由所选 daemon 保存，不占用扩展存储配额。卸载扩展或断开远程连接不会删除 daemon 主机上的文件。 +「打开文件夹」也由 daemon 在其主机上执行；远程服务器的目录不会因此在浏览器所在电脑上打开。 每个任务对应一个 `<开始记录时间戳>-.jsonl` 文件，包含逐行追加的事件。macOS/Linux 上目录权限为 `0700`、文件为 `0600`；Windows 使用用户目录继承的权限。保留最近 30 天已结束的任务；服务加载记录、开启审计、创建任务或查询列表时清理过期记录，执行中或仍有待返回调用的任务不会自动删除。导出的文件由用户自行保管，不参与自动清理。 @@ -46,7 +51,7 @@ bsk session start --name "整理本周待办" 不保存截图、页面正文、完整 DOM、输入值、脚本、完整选择器、原始错误消息、上传文件内容或本地文件路径。输入项统一标记为已隐藏；元素名称和任务名截断并进行基础脱敏，仍建议避免在任务名中填写敏感信息。导出仅使用已保存的脱敏数据。 -审计接口仅由扩展页面经后台服务访问，网页内容脚本不能调用。服务按已连接的浏览器实例隔离列表、详情和删除操作，不采用请求参数中的浏览器身份。它沿用现有本地连接的信任模型，不构成对同一系统账户下恶意进程的安全边界。 +审计接口仅由扩展页面经后台服务访问，网页内容脚本不能调用。服务按已连接的浏览器实例隔离列表、详情和删除操作，不采用请求参数中的浏览器身份。本地模式沿用本地连接的信任模型；远程模式使用已配对设备的连接身份。daemon 主机上的文件不构成对同一系统账户下恶意进程的安全边界。 ## 接口概要 diff --git a/packages/dsh-plugin-browserskill/skill/SKILL.md b/packages/dsh-plugin-browserskill/skill/SKILL.md index 3ec84d01..d47e5ef4 100644 --- a/packages/dsh-plugin-browserskill/skill/SKILL.md +++ b/packages/dsh-plugin-browserskill/skill/SKILL.md @@ -5,121 +5,108 @@ description: Browser automation through six injected domain tools. # browser-skill for DeepSeek Harness -Drive the user's logged-in Chromium in an Agent Window. User-window tabs remain protected unless -explicitly borrowed. - -Use the loaded `action` schemas for `browser_session`, `browser_page`, `browser_inspect`, -`browser_interact`, `browser_tabs`, and `browser_assist`; do not guess parameters. -All browser work must use the injected tools directly to preserve ownership, cancellation, -attachments, observation UI, and cleanup. Do not invoke another process to control the browser. +All browser work must use the injected tools directly, in an Agent Window with existing logins. +Do not control the browser through another process. Use the loaded action schemas for parameters. ## Mandatory workflow -Every task owns a bounded plugin session: - -```text -browser_session({ action: "start", ... }) -... use the returned sessionId for browser work ... -browser_session({ action: "stop", session: sessionId }) -``` - -Pass the session when more than one exists; never guess or use another program's id. Stop on -success and failure unless asked to keep it open. Stopping also returns borrowed tabs. - -## Work toward one observable goal - -- Derive a concrete success condition from the user's request. -- Observe, act, and check ambiguous results once. -- Stop acting once success is visible. -- With help enabled, request help for human-only steps or after two attempts make no progress. - With help disabled, use the autonomous handling rules below. - -## Observe, act, observe - -Use `browser_inspect` action `observe` as the primary semantic page view. It returns roles, states, -text, and `@eN` refs. Prefer fresh refs over raw selectors. Refs invalidate after navigation and may -also become stale after large DOM changes, so observe again before the next interaction. - -Use `browser_interact` for click, hover, scroll-to, focus, blur, fill, select, press and wheel. -`scroll-to` reveals a target and returns its visible border-box bounds in top-level viewport CSS -pixels, clipped by ancestors. Partial visibility suffices; hidden/fully clipped targets fail. -This does not test occlusion. Use refs for iframe/shadow-root targets; selectors search the main document. - -`wheel` takes signed `deltaX`/`deltaY` (one nonzero). Optional `target` is scrolled into view first; -otherwise input lands at the viewport centre. Results echo input, not completed scrolling; observe afterwards. - -`focus`/`blur` enter or leave focus-triggered states. Hover-only surfaces appear as -`@e1 button "Products" [hover first: Shoes | Bags]`. Items are labels, not refs: -hover the trigger, observe, then use the revealed item's ref. Click the trigger only if its action is wanted. +1. Define success. Start a session and retain `sessionId`. For a new page: -Escalate reading only as needed: + ```text + browser_session({ action: "start" }) + browser_page({ action: "navigate", session: "", url: "https://example.com" }) + browser_inspect({ action: "observe", session: "" }) + ``` -1. `observe` for normal understanding and interaction refs. -2. `snapshot` when a stricter static accessibility tree is more useful. -3. `html` for exact markup or hidden metadata that semantic views cannot provide. -4. `screenshot` for layout, styling, canvas, images, or requested visual evidence. +2. For an existing user tab, borrow it instead. Replace example IDs/refs with actual + results. Pass `session` when more than one exists; never use foreign IDs. +3. Observe after page changes; check ambiguous results once. Stop acting when success + is visible. On success or failure, call + `browser_session({ action: "stop", session: "" })` unless keeping the session + open is part of the user's request. Stopping returns borrowed tabs, leaving them + open in the user's window. -Do not start with raw HTML or screenshots merely to discover ordinary controls. When interaction is -needed, obtain a fresh observation before acting on screenshot or HTML findings. +## Read and interact -Use `browser_page` for purposeful navigation, history, reload, or a lifecycle wait. Avoid speculative -waits when no navigation is expected. After any page change, discard old refs and observe again. +Prefer `observe` for text/refs; use `snapshot` for static accessibility, `html` for +exact markup, and `screenshot` for visuals. Console/network are bounded read-only +diagnostics; follow sequence cursors. Wait only for expected navigation. -## Respect the Agent Window boundary +To fill an observed field `@e3`: -Use `browser_tabs` to list returned tab ids before selecting, closing, borrowing, or returning tabs. -Borrow a user tab only for the immediate task, and return it as soon as that step is complete. Never -invent a tab id or keep a personal tab borrowed across unrelated work. - -The extension enforces its Automation settings on received requests; old CLIs/daemons may end help locally. -Update all components for full support. -Do not change settings to bypass a prompt. Never repeat denied or expired borrows; inspect unknown outcomes. - -## Ask the human when needed - -With human help enabled, use `browser_assist` action `request-help` for login, captcha, OTP, -payment confirmation, consent, or other human steps. Give a precise prompt and fresh targets; -use completion criteria only for a stable success signal. Resume on continuation or completion; -cancellation and timeout block the step. Observe again before using refs. - -With help disabled, task/host rules and borrow confirmation still apply; no new permission is granted. -`disabled` confirms no human action: re-observe, use existing login state and authorized -inputs, and continue. Where task/host rules allow, models with vision may attempt graphical -verification using screenshots and supported interactions. Phone-only QR scans, face verification, -unavailable SMS codes, and image-only CAPTCHAs for text-only models may remain blocked. Try viable -alternatives and verify results; block only for missing inputs/capability or exhausted options. -Do not repeat unknown effects, request help again, or switch backends. Continue independent work. - -`browser_assist` also resizes the Agent Window or emulates a device for one tab. - -## Debug and recover without wandering +```text +browser_interact({ action: "fill", session: "", target: "@e3", value: "text" }) +``` -Use `browser_inspect` console or network actions only for relevant, bounded, read-only diagnostics. -Continue from returned sequence cursors instead of rereading the same buffer. +Refs invalidate after navigation; large DOM changes may stale them too. Observe again. +Prefer refs for frames/shadow roots; selectors search the main document. Use observe +for ordinary controls, including before acting on HTML or screenshot findings. +Select options by value, not visible label. + +- Hover markers like `[hover first: Shoes | Bags]` list labels, not refs. Hover the + trigger, observe, then use the item's ref. Click the trigger only if its action is wanted. +- `scroll-to` returns ancestor-clipped bounds in top-level viewport CSS pixels. + Partial visibility suffices; hidden/fully clipped targets fail. It does not test occlusion. +- `wheel` uses signed `deltaX`/`deltaY`, at least one nonzero. Optional `target` is + scrolled into view first; otherwise it uses the viewport centre. It reports input, + not scrolling success: observe afterwards. Focus/blur change focus states. + +## Borrowing and human help + +Use `browser_tabs` to list IDs before acting. Borrow for the immediate step and +return promptly. Browser Automation settings +govern confirmation and help; never change them to bypass a prompt or repeat +pending/denied/expired borrows. Inspect unknown outcomes; follow version-error hints. +Remote reads/actions require task-created or borrowed tabs; popups gain no control. +An unowned tab inside the Agent Window needs a user move to a user window before borrowing. + +With help enabled, use `browser_assist` action `request-help` for login, CAPTCHA, +OTP, payment confirmation, consent, or after two attempts without progress. Supply +a precise prompt and fresh targets; completion criteria need a stable success signal. +Resume only on `continued` / `completed`, then observe. Cancellation/timeout blocks +the step; do not repeat the request. Navigation alone is not success. +`browser_assist` also resizes windows or emulates a device for one tab. + +With help disabled, do not request help or re-enable it. `disabled` confirms no human +action or new permission. Re-observe; use existing logins, authorized inputs and +viable alternatives within task/host rules. Vision models may try graphical +verification where authorized. Phone-only QR scans, face verification, missing SMS +codes or image-only tasks for text-only models may remain blocked. Report missing +inputs/capabilities or exhausted alternatives; continue independent work. Never repeat +unknown effects or switch backends to bypass limits. Borrow confirmation still applies. + +## Recover + +- Stale ref: observe, then retry the intended action once. +- Unknown tab/session: list owned resources or start a session; never guess IDs. +- Timeout/unknown effect: inspect before retrying; the action may have happened. +- Unconfirmed fill: read the field. Formatting may satisfy the goal; correct only a + remaining difference instead of blindly refilling or requesting help. +- Other errors: follow the hint; on unrecoverable failure, report and stop the owned session. + +Arbitrary page-script evaluation and interaction recording are intentionally unsupported. +Do not invent tools or bypass these limits. -- Stale ref: observe again and retry the intended action once. -- Unknown tab: list tabs instead of guessing. -- Unknown session: list owned sessions or start one; never try foreign ids. -- Timeout: inspect current state before deciding whether one longer purposeful wait is useful. -- Fill result unconfirmed: observe the field first; the page may have formatted the value. Continue - if the visible result satisfies the user's intent. Otherwise correct the remaining difference; - do not blindly repeat fill or immediately request human help. For other fill errors, follow the - returned hint and inspect current state before retrying. -- Unrecoverable failure: report the blocker and stop the owned session. +## Canvas and continuation -Arbitrary page-script evaluation and interaction recording are intentionally unsupported. Do not -invent tools or route around those limits. +`@eN canvas [visual:screenshot]` is text, not an image. Screenshot the ref when needed; never infer +Canvas names/controls from nearby labels. If images cannot be understood, ask for +an image-capable model and continue with available semantics. -## Canvas and continuation +```text +browser_inspect({ action: "screenshot", session: "", ref: "@e3" }) +browser_interact({ action: "click", session: "", target: "@e3", captureId: "", imageX: 100, imageY: 50 }) +``` -[visual:screenshot] means screenshot that ref first; names are optional, not inferred from -nearby controls. If needed images cannot be understood, ask to switch models and use semantics. -Click via browser_interact(action=click,target=ref,captureId=...,imageX=...,imageY=...), using -ORIGINAL PNG pixels. Captures are single-use, last two minutes, and expire on ref replacement -or a new screenshot of that ref. captureUnavailable means view only; observe and screenshot -again. Counts 1/2 and buttons/modifiers work, not Canvas fill/IME/drag/hover. Repainting is allowed: -verify results; use DOM refs for revealed controls. Inspect before retrying effect_state=unknown. - -No default token cap. With maxTokens, follow nextCursor via observe(cursor=...); each page can -contain many Canvas refs and replaces previous refs. This reads the same observation without -recapture or depth changes. New observe/snapshot or DOM identity changes invalidate continuation. +Use the returned captureId and a point actually seen in ORIGINAL PNG pixels, not +resized display/viewport coordinates. Captures are single-use, last 2m, and expire +on ref replacement or a newer screenshot of that ref. `captureUnavailable` means +view-only: observe and screenshot again before clicking. Counts 1/2 and buttons/ +modifiers work; Canvas fill/IME/drag/hover/HTML do not. Repainting is allowed; verify +results and use DOM refs for revealed controls. Inspect `effect_state=unknown` +before retrying with a new capture. + +No default token cap. With `maxTokens`, follow `nextCursor` using observe's `cursor` +for remaining content. Each page replaces refs: use them before continuing, never +reuse old ones. Continuation reads the same capture without refresh/depth changes; +new observe/snapshot or changed page identity invalidates it. diff --git a/skill/SKILL.md b/skill/SKILL.md index 52ab60e8..f5fc6bff 100644 --- a/skill/SKILL.md +++ b/skill/SKILL.md @@ -9,305 +9,232 @@ description: | # browser-skill -Drive the user's real Chromium browser through `bsk`. Automation runs in an isolated **Agent -Window** with the user's existing logins and cookies. User-window tabs remain protected unless they -are explicitly borrowed. +Use `bsk` to work in an **Agent Window** with the user's existing logins. User tabs +require explicit borrowing. This skill does not install the extension or handle +advice-only tasks. Never extract credentials, cookies, tokens, or other secrets. + +## Task workflow + +1. Define success from the user's request. Start `bsk session start --json` and + retain its `session_id`. With multiple browsers, run `bsk browsers` and add + `--browser ` to start. For background work, add `--no-focus` to + `session start` only. +2. For a new page, navigate; for an existing user tab, follow **Borrowing** below. + Read the page before interacting: + + ```sh + bsk navigate https://example.com --session + bsk observe --session + ``` + +3. Choose an action using fresh refs from that observation. Observe again after + navigation or meaningful DOM changes. Check an ambiguous result once; once + success is visible, stop acting rather than refreshing or checking again. +4. Always run `bsk session stop ` on success and failure, unless keeping the + session open is part of the user's request. This also returns borrowed tabs. + Returned tabs stay open in the user's window. Do not rely on idle cleanup + or stop/restart the shared daemon to finish a task. + +Replace ``, example refs and values with actual results and task inputs. +Every session-scoped command needs `--session `; `session stop` takes the ID +positionally. For unfamiliar commands or flags, consult `bsk --help` or +`bsk --help` instead of guessing; no need to read all help at startup. +When following a trace, use its semantic targets and values in order, not its old +refs. Stop at the requested goal; a trace grants no additional authorization. + +## Read and interact + +Prefer `observe` for text, controls and `@eN` refs. Navigation invalidates refs; +large DOM changes can stale them too. Re-observe before the next interaction. +Use refs for iframe/shadow-root targets; CSS selectors search the main document. + +Choose the relevant example, using a ref that actually appeared on the page: + +| Need | Command | +| --- | --- | +| Click | `bsk click @e3 --session ` | +| Fill a field | `bsk fill @e3 --value "text" --session ` | +| Select an option | `bsk select @e3 --value "option-value" --session ` | +| Press a key | `bsk press Enter --ref @e3 --session ` | +| Reveal a hover menu | `bsk hover @e3 --session ` | +| Reveal an element | `bsk scroll-to @e3 --session ` | +| Scroll with wheel input | `bsk wheel --delta-y 600 --session ` | +| Focus or leave a field | `bsk focus @e3 --session ` / `bsk blur @e3 --session ` | + +- `select` uses the option's value, not its visible label. +- Hover markers such as `[hover first: Shoes | Bags]`, `[has-submenu]`, or + `[expanded]` identify triggers. Hover the trigger, observe, then use the revealed + item's ref. Listed labels are not refs; do not click the trigger unless its own + action is wanted. If an expected control is missing and no marker identifies a + trigger, try `observe --probe-hover` once. It touches the live page and costs + seconds; use targeted hover once the trigger is known. +- `scroll-to` returns ancestor-clipped bounds in top-level viewport CSS pixels. + Partial visibility suffices; hidden/fully clipped targets fail. It does not test + occlusion. `wheel` sends signed deltas (at least one nonzero), not a guaranteed + scroll distance. An optional target is scrolled into view first; without one, + input lands at the viewport centre. Observe to check the page's response. + +Use `snapshot` for a static accessibility tree, `get-html` for exact markup or +hidden metadata, and `screenshot` for visual content or requested visual evidence. +Do not start with HTML/images just to find ordinary controls; obtain fresh refs +before interacting with controls found that way. + +### Large observations + +There is no default token cap. With `observe --max-tokens `, follow a returned +`next_cursor`/`@more` when relevant content remains: -Do not use this skill for tasks with no browser, for extension installation, or when the user only -wants instructions. Never extract credentials, cookies, tokens, or other secrets from pages. +```sh +bsk observe --cursor --session +``` -## Required lifecycle +Each page replaces the ref map: use its refs before continuing and never reuse +refs from earlier pages. Continuation reads the same capture, without refreshing +or hovering; do not combine it with depth changes or hover probing. New observe/ +snapshot or changed page identity invalidates continuation; then observe afresh. -Every browser task owns a bounded session: +## Borrowing and browser settings -```text -1. bsk session start # retain the printed 4-letter session id -2. bsk ... --session # pass it to every session-scoped command -3. bsk session stop # always run on success and error paths -``` +List before borrowing, and return the tab as soon as the relevant step ends: -Do not rely on the idle timeout for cleanup. Stop the session as soon as the goal is met unless the -user explicitly asks to keep it open. Stopping also returns borrowed tabs. - -By default, browser commands auto-start the daemon when needed. Keep the shared daemon running; -task cleanup is `bsk session stop`, not `bsk daemon stop` or `restart`. - -If the agent environment kills background children when each shell command ends (as reported for -Linux WorkBuddy), arrange a persistent daemon outside that per-command sandbox first. The user -can run `BSK_HOME=/absolute/shared/bsk bsk daemon start` in a normal host terminal. A host-managed -background task can instead run `bsk daemon start --foreground` with the same `BSK_HOME`, using -the host's approved execution path. Do not disable sandbox protection for browser task commands. - -In that environment, pass `BSK_HOME=/absolute/shared/bsk BSK_AUTO_START=0` to **every** `bsk` -command. Replace the example path with one dedicated directory that both sides can access, -including its IPC socket; an `export` in one shell tool call may not persist to the next. If the -daemon is unavailable, ask for it to be started in the owning host environment; do not loop on -auto-start, guess a home directory, delete runtime files, or restart the shared daemon. A doctor -warning about local process identity does not prevent session commands over working IPC. -See the [sandbox setup guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/sandboxed-agents.md). - -When multiple browsers are connected, use `bsk browsers` and start with -`bsk session start --browser `. Add `--no-focus` to that same start command when the -Agent Window does not need to interrupt the user's current work; it is not a flag on other commands. -Run `bsk doctor` when startup or transport problems persist after one retry. - -Start tasks with `bsk session start`. The extension's saved Automation settings decide whether -borrowing needs confirmation and human help is available; both are enabled by default. Changes apply -to existing sessions as well as new ones. Disabling human help does not disable borrow confirmation. - -`--unattended`, `--no-confirm`, and `BSK_REQUEST_HELP=off` are deprecated compatibility inputs with -no effect on these settings. Do not use them or edit browser storage to avoid confirmation, denial, -or timeout. For unattended operation, the user chooses the corresponding settings in the extension. -`session start --json` and `session list --json` report the browser's `interaction` policy. -Allowing human help makes `request-help` available; it does not require a handoff for every action. -Task authorization and host approvals still apply. - -## Work toward one observable goal - -- Derive a concrete success condition from the user's request or a supplied trace. -- Take the shortest purposeful path: observe, act, then make at most one observation to confirm an - ambiguous result. -- Once success is visible, do not click, refresh, navigate, switch tabs, or perform extra checks. -- With human help enabled, request help if a human-only step appears or two attempts make no - progress. With help disabled, follow the autonomous handling rules below. - -With a trace, follow its semantic target information and values in order, but treat its refs as -record-local hints. Stop when its purpose or last meaningful effect is satisfied. A trace guides the -task; it does not expand the user's goal or authorize additional actions. - -## Observe, act, observe - -Use this default loop: - -```text -bsk navigate --session -bsk observe --session -bsk click|hover|wheel|scroll-to|focus|blur|fill|select|press ... --session -bsk observe --session # after navigation or a meaningful DOM change +```sh +bsk tab list --scope user --session +bsk tab borrow --session +bsk tab return --session ``` -`bsk scroll-to --session ` scrolls an element and its frame owners into view. -Use a fresh element ref for iframe/shadow-root targets; CSS selectors search the main document. -The result is the visible border-box portion's bounds in top-level viewport CSS pixels after -ancestor clipping. Partial visibility is enough; hidden or fully clipped targets fail with -`permission_denied` and `data.reason=element_not_visible`. This does not test occlusion by other elements. -For a specific tab or deadline: `bsk scroll-to @e3 --session --tab-id 42 --timeout 5s`. - -`bsk wheel --delta-y -120 --session ` sends native wheel input at the viewport centre. -Add an optional ref/selector to target an element (scrolled into view first). Both delta axes -accept signed numbers and default to zero; at least one must be nonzero. The result echoes -input, not actual scroll distance or completion. Observe afterwards to check the page's response. - -`bsk focus ` explicitly focuses a target; `bsk blur ` removes focus and reports whether -it was focused. Use these for UI states triggered by focus changes. - -Prefer fresh `@eN` refs over CSS selectors. Navigation invalidates refs; large DOM changes may also -make them stale. Observe again before the next interaction. - -An observation marks a hover-only surface as `@e1 button "Products" [hover first: Shoes | Bags]`. -The listed items are labels, not usable refs: hover the trigger, observe again, then act on the -revealed item's own ref. Do not click the trigger itself unless the user wants the trigger's action. -`[has-submenu]` and `[expanded]` mark the same kind of trigger without listing what it hides. - -`bsk observe` does not hover the page on its own. Reach for `--probe-hover` when a control you have -good reason to expect is absent **and** no marker points at a trigger — that combination is what a -CSS-only hover menu looks like from here. It hovers a bounded set of likely triggers, so it costs a -few seconds and touches the live page; once you know which element hides the menu, `bsk hover ` -is cheaper and more precise. - -Escalate page reading only as needed: - -1. `bsk observe` for normal semantic understanding, text, controls, and refs. -2. `bsk observe --probe-hover` once when an expected control is missing and no marker points at a - trigger. -3. `bsk snapshot` when a stricter static accessibility tree is more useful. -4. `bsk get-html` for exact markup or hidden metadata that semantic views cannot provide. -5. `bsk screenshot` for layout, styling, canvas, images, or requested visual evidence. - Use `--full-page` when the user wants a long screenshot of the whole ordinary webpage. - -Do not start with raw HTML or screenshots merely to discover ordinary controls. When interaction is -needed, obtain a fresh observation before acting on screenshot or HTML findings. - -## Canvas and observation continuation - -`observe` may place `@eN canvas [visual:screenshot]` near related page controls. Names are -optional: do not infer a table title or controls inside Canvas from adjacent labels. Visual refs -support `screenshot --ref`; point clicks additionally require its `capture_id` and image coordinates. -They do not support fill/hover or HTML extraction. First observe returns text, -not an image. Use the surrounding semantics to decide whether a Canvas screenshot is needed. -If you cannot receive and understand images in this session, tell the user the Canvas contents -cannot be interpreted and ask them to switch to an image-capable model; continue with available -semantic information. BrowserSkill does not detect the model's capabilities. - -There is no default token cap. With an explicit `--max-tokens` limit, an observation may return -`next_cursor` and an `@more` instruction. Use current refs before calling -`bsk observe --cursor --session `: each response replaces the ref map, so refs from -previous pages must not be reused. A response can contain many Canvas entries. Follow cursors -when relevant content remains, rather than repeatedly reading the same prefix. -Continuation reads the same captured observation; it does not refresh or hover the page. Do not -combine it with depth changes or hover probing. A new observe/snapshot replaces the continuation; -if the page identity changed, observe again. Screenshot execution checks current target identity -and geometry, but permits Canvas repainting and does not freeze pixels. - -A Canvas screenshot can return `capture_id`. To click a point you identified in that image, use -`bsk click eN --capture --image-x --image-y --session `. -Use original PNG pixels (returned width/height), not resized display or viewport coordinates. -Captures are single-use, expire after two minutes, and are invalidated by a newer screenshot of -that ref or observation/continuation. With `capture_unavailable`, view the image but observe and -screenshot again before clicking. Click counts 1/2, buttons and modifiers are supported. -After clicking, observe or screenshot to verify the result; use DOM refs for revealed controls. -A completed click does not prove business success. Canvas repainting is allowed; changed identity, -geometry or hit target is rejected. If `effect_state=unknown`, inspect before retrying with a new -capture. Do not infer cell-editing, IME, drag or hover support from point-click capability. - -## Respect the Agent Window boundary - -Normal page writes affect only Agent Window tabs. To operate a user tab, first list it with -`bsk tab list --scope user --session `, then `bsk tab borrow `. Return it immediately -after the relevant step with `bsk tab return `; never invent a tab id or keep a personal tab -borrowed across unrelated work. - -`tab borrow --timeout 120s` changes the confirmation wait (default 60s), not whether approval is -required. Custom waits need daemon and extension protocol 1.2+. Compatible older peers can still -start sessions and borrow with their default wait. The current CLI's `request-help` needs daemon -protocol 1.3; an unsupported operation does not make the connection unusable. Update all three -components for full browser-setting enforcement; older programs may still end help locally. -Repeating a completed borrow in the same session returns -its existing result. Do not repeat pending requests, denied requests, or confirmation timeouts, -or switch to another browser tool to bypass them. If `reason` is `borrow_outcome_unknown`, inspect -tab and session state before continuing; the tab may already have moved. - -## Ask the human when needed - -When human help is enabled (the default), use `bsk request-help` for login, captcha, OTP, payment -confirmation, consent, or another step the user must complete. Give a precise prompt and pass -fresh `--target` refs/selectors when concrete controls can be highlighted. Use completion criteria -only when the page has a clear stable success signal. - -The result `outcome` is one of `continued`, `completed`, `cancelled`, `timed_out`, or `disabled` -(`navigated` is deprecated — never treat navigation as a completion signal). After a human handoff, -resume only after `continued` or `completed`. Treat `cancelled` as rejection and `timed_out` as a -blocker; do not repeat that request. Observe again after control returns before using refs. - -When help is disabled in the extension, make every -reasonable effort to complete the task autonomously with BrowserSkill. Do not call `request-help`. -If a call returns `disabled`, no human action was confirmed: re-observe and continue working rather -than marking the step blocked merely because help is unavailable. - -Disabling help adds no permission: keep task authorization and host restrictions in force. -Use the current page, existing login state, and authorized credentials or codes to complete the -current step. Where the task authorization and host rules allow, a model with image understanding -may attempt graphical verification through screenshots and supported interactions. Phone-only QR -scans, face verification, and unavailable SMS codes may remain blocked; a text-only model may also -leave an image-only CAPTCHA unresolved. Attempt other authorized steps within available capabilities -and verify the actual result before concluding they cannot be completed. - -After a failed attempt, re-observe and try a different viable approach when available. Do not loop -on identical failures or repeat an action whose outcome is unknown. Report a specific blocker only -when required information or capability is missing, or viable approaches are exhausted; continue -independent work. Do not re-enable help or switch browser backends to work around those limits. - -## Command inventory - -This list of names is complete. Never invent a command outside it; read -`bsk --help` for flags instead of guessing them. - -```text -session start|stop|list browsers status doctor update logs -navigate navigate-back navigate-forward reload wait-for-navigation wait-ms -observe snapshot get-html screenshot console network -click hover wheel scroll-to focus blur fill select press evaluate -tab list|create|close|select|borrow|return window resize emulate -upload download request-help record start|stop -``` +Never invent tab IDs or keep a user tab across unrelated work. Do not repeat +pending, denied or timed-out borrows. For `borrow_outcome_unknown`, inspect tab/ +session state first: the tab may already have moved. Do not bypass an outcome +through another browser backend. `tab borrow --timeout 120s` changes only the +confirmation wait (default 60s); custom waits require daemon and extension protocol 1.2+. -Flags and argument forms that are easy to get wrong: +The extension's saved Automation settings control borrow confirmation and human +help independently; both default on and apply to existing sessions too. Read +`interaction` in `session start --json` or `session list --json` when needed. +Deprecated `--unattended`, `--no-confirm`, and `BSK_REQUEST_HELP=off` cannot override +these settings. Never change browser storage/settings to bypass them. Human-help +availability does not require permission for every action or grant extra authority. +`request-help` requires daemon protocol 1.3; update CLI, daemon and extension for +full settings support. A feature's version error does not disable other operations. -```text -bsk fill --value bsk select --value -bsk screenshot --out bsk emulate --device -bsk upload --file bsk download --out -``` +Remote content reads/actions require task-created or borrowed tabs. Page-opened +popups gain no control automatically; an unowned tab inside the Agent Window +needs the user to move it to a user window before borrowing. Remote upload/download +are unsupported; screenshots work. See the [remote guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/remote-extension-connection.md). -`select` matches an option's `value` attribute, not its visible label. Device preset ids are -lowercase and hyphenated, such as `iphone-14`. +## Human steps and recovery -- `console` and `network` provide bounded, read-only debugging evidence. -- `emulate` applies viewport, user-agent, and touch overrides to one tab; new tabs do not inherit - them. Use `--off` to restore the real environment. -- `evaluate` is a last resort when observe plus normal interactions cannot complete the task. With - `--json`, inspect `.ok`: a JavaScript exception may still have CLI exit code 0 because the RPC - succeeded. Never evaluate credential surfaces to read storage, cookies, or auth data. -- `record` captures a user's actions for later replay. Read `bsk record start --help` before use, - and never record banking, SSO, password-manager, or other sensitive pages. +With help enabled, request help for login, CAPTCHA, OTP, payment confirmation, +consent, or after two attempts make no progress: -## Screenshots +```sh +bsk request-help --session --prompt "Please complete sign-in" --target @e3 +``` + +Use a precise prompt and fresh targets; omit `--target` when no control fits. +Use completion criteria only for a clear, stable success signal. + +| Result | Next step | +| --- | --- | +| Help `continued` / `completed` | Observe again, then resume with fresh refs. | +| Help `cancelled` / `timed_out` | Respect rejection or the blocker; do not repeat the request. | +| Help `disabled` | No human action was confirmed. Re-observe and follow the disabled-help rules below. | +| Stale ref | Observe and retry the intended action once. | +| Unknown tab/session | List current tabs/sessions; never guess IDs or use another task's session. | +| Timeout or unknown effect | Inspect current state before retrying; the action may already have happened. | +| `fill_value_mismatch` | Read the field: formatting may still satisfy the request. Correct only a remaining difference; no blind refill or immediate handoff. | +| Unsupported operation | Use available capabilities; suggest updating only if the missing feature is needed. | + +Navigation alone (including deprecated help outcome `navigated`) is not completion. +For other errors, follow the returned hint and inspect the current state. + +**Help disabled:** do not request help or re-enable it. Use existing login state, +authorized inputs and viable alternatives; disabling help adds no permission and +does not remove borrow confirmation or host restrictions. Where authorized, a +vision-capable model may attempt graphical verification. Phone-only QR scans, +face verification, missing SMS codes or image-only tasks for a text-only model +may remain blocked. Report a specific blocker only when inputs/capabilities are +missing or viable approaches are exhausted; continue independent work. Do not loop +on identical failures, repeat unknown effects or switch backends to bypass limits. +On an unrecoverable failure, report the blocker and stop the owned session. + +## Screenshots and Canvas ```sh bsk screenshot --session --out viewport.png -bsk screenshot --session --ref @e3 --out element.png +bsk screenshot --session --ref @e3 --out element.png --json bsk screenshot --session --full-page --out page.png -bsk screenshot --session --full-page --timeout 5m --out page.png ``` -Without `--ref` or `--full-page`, capture only the visible viewport. `--full-page` and -`--ref` are mutually exclusive. Full-page mode scrolls the document from top to bottom, -follows content loaded during scrolling, and restores the original position and styles. -It is page input: use a selected, session-controlled tab in the Agent Window (create or -borrow first), keep the viewport stable, and respect user interrupts. `--tab-id` targets -a specific tab without selecting it. Chrome internal pages, the Web Store, nested scroll -containers and virtualized lists are not supported by automatic full-page capture. - -Capture and PNG encoding default to two minutes. `--timeout` only applies with -`--full-page`; allow your shell runner enough time for that deadline plus file transfer. -Increase it for longer pages, but do not blindly retry an endlessly growing -page or a cancelled request. Ctrl-C cancels. Failure produces no partial output. -The CLI streams the PNG to disk and returns its path; `--json` also reports dimensions -and byte size. An existing `--out` file is replaced only after a full-page image is -received completely. Omitting `--out` uses a temporary path. No popup preview opens. -Use the matching CLI and extension builds; an unknown full-page RPC indicates an older -extension, not a reason to silently substitute a viewport screenshot. - -## File transfer - -`upload` and `download` stage files through the daemon; the agent never touches browser-internal -paths. Treat upload as disclosure to the website, download as accepting website-controlled bytes. - -Upload has two independent mechanisms — choose explicitly, never rely on automatic fallback: - -- **Default (input mode):** for upload buttons, file-input labels, or "upload from computer" - actions. The command clicks the target and intercepts the native file chooser. -- **`--mode drop`:** for reliably identified attachment-receiving areas — an explicit drop zone, - chat composer, email editor, or form attachment area. Do not target page whitespace, generic - containers, or areas whose attachment ownership is ambiguous. - -Decision sequence when uploading: - -1. Try input mode (the default). -2. If it returns `reason=file_input_not_activated` with `effect_state=none`, re-observe. When a - reliable attachment target exists, try `--mode drop` once against that target. -3. Otherwise request help if enabled; when disabled, follow the autonomous handling rules above. -4. **Never** switch mechanisms or repeat when `effect_state` is `unknown` or `committed` — the - browser may already have applied the file. - -A successful drop means Chrome dispatched the native file-drop event; it does not prove the site -accepted the attachment. Observe the page once after the command. - -Download default-refuses to overwrite; pass `--overwrite` when replacing an existing file is -intended. Read `bsk upload --help` and `bsk download --help` for all flags and error details. - -## Recover without wandering - -- Stale ref: observe again and retry the intended action once. -- Unknown tab or session: list current tabs/sessions; never guess identifiers. -- Timeout: inspect current page state before deciding whether one longer purposeful wait is useful. -- Fill result unconfirmed (`fill_value_mismatch`): observe the field first; the page may have - formatted the value. Continue if the visible result satisfies the user's intent. Otherwise correct - the remaining difference; do not blindly repeat fill or immediately request human help. For other - fill errors, follow the returned hint and inspect current state before retrying. -- Unsupported command: continue with available capabilities; suggest updating only when the missing - command is necessary. -- Unrecoverable failure: report the blocker and stop the session in a finally-style path. - -The CLI's current help and error hints are authoritative for flags, parameters, and recovery -details. +Screenshots return a local PNG path; view the image to interpret it. `--out` +replaces an existing file; omitting it uses a temporary path. `--json` includes +dimensions and byte size. `--ref` and `--full-page` cannot be combined. + +Full-page mode scrolls an ordinary webpage and restores its position/styles. +Use a selected, session-controlled tab and stable viewport; `--tab-id` targets a +tab without selecting it. Internal browser pages, the Web Store, nested scrolling +panels and virtualized lists are unsupported. Capture/encoding defaults to 2m; +`--timeout 5m` extends it only in full-page mode. Allow the shell enough time for +capture plus transfer. Respect cancellation; do not blindly retry endless pages +or substitute a viewport image when an older extension rejects full-page capture. +Use matching CLI/extension builds. Ctrl-C cancels; failed full-page captures save +no partial image. + +For `@eN canvas [visual:screenshot]`, observe returns text, not pixels. Screenshot +that ref when its contents matter; never infer Canvas controls or names from +nearby labels. If images cannot be received/understood, explain the limitation, +ask for an image-capable model when needed, and continue with available semantics. + +To click a point seen in a Canvas image, retain that screenshot's `capture_id`: + +```sh +bsk click @e3 --capture --image-x --image-y --session +``` + +Use ORIGINAL PNG coordinates and dimensions, not resized display/viewport pixels. +Captures are single-use, expire after 2m, and are invalidated by ref replacement +(observe/snapshot/continuation) or a newer screenshot of that ref. With +`capture_unavailable`, the image is view-only: observe and screenshot again before +clicking. Counts 1/2, buttons and modifiers work; Canvas fill, IME, drag, hover +and HTML extraction do not. Repainting is allowed; changed identity/geometry/hit +targets are rejected. Verify the result, using DOM refs for revealed controls; +inspect `effect_state=unknown` before retrying with a new capture. + +## Files and other tools + +```sh +bsk upload @e3 --file ./report.pdf --session +bsk download @e3 --out ./report.pdf --session +``` + +Upload discloses the file to the site; download accepts site-controlled bytes. +Use agent-local paths, not browser-internal staging paths. + +- Default upload clicks an upload button/label and intercepts its file chooser. +- If `reason=file_input_not_activated` and `effect_state=none`, re-observe. Try + `--mode drop` once only on a clear attachment target such as a drop zone or + composer, never whitespace or an ambiguous container. Otherwise follow the + human-help rules. There is no automatic fallback between mechanisms. +- Never retry or switch upload modes for `effect_state=unknown` or `committed`. + A successful drop proves dispatch, not site acceptance; observe the attachment. +- Download refuses overwrite by default; add `--overwrite` only when replacement + is intended. Consult each command's help for other flags. + +Use `console` / `network` for bounded read-only diagnostics; follow returned +sequence cursors. `emulate --device iphone-14` affects one tab; `--off` restores it. +`evaluate` is a last resort: inspect JSON `.ok`, since a script exception can have +CLI exit code 0. Never evaluate secrets. `record start` captures user actions; +read its help first and never record banking, SSO or password-manager pages. +Use `bsk --help` to find navigation/history, tab, wait and window commands. + +## Startup problems + +Commands normally auto-start the daemon. After one failed retry, use `bsk doctor`. +If command sandboxes reap background processes, arrange a persistent daemon in +its owning host environment using the [sandbox guide](https://github.com/Tencent/BrowserSkill/blob/main/docs/sandboxed-agents.md). +Use the same accessible `BSK_HOME` and `BSK_AUTO_START=0` on EVERY sandboxed command; +exports may not persist between shell calls. Keep browser commands sandboxed. +A missing daemon needs host-side startup, not repeated auto-start, guessed home +paths, deleted runtime files or a shared-daemon restart. A local process identity +warning permits browser commands when IPC works.