GitHub Copilot SDK port. This is a community fork of the Azure-Samples/computer-use sample, ported from its original OpenAI computer-use implementation to the GitHub Copilot SDK. The computer actions are now exposed as custom Copilot SDK tools and the SDK drives the agentic loop. See Acknowledgments.
A sample application that uses the GitHub Copilot SDK to control a computer through natural language instructions. It captures screenshots, analyzes the GUI, and performs mouse and keyboard actions to complete tasks. The Copilot SDK runs the agentic loop, and the computer-control capabilities are exposed to the model as custom SDK tools.
- Natural language computer control powered by the GitHub Copilot SDK
- Screenshot capture and analysis
- Mouse and keyboard control exposed as custom Copilot tools
- Cross-platform compatibility (Windows, macOS, Linux)
- Screen resolution scaling for consistent AI model input
- Selectable model and reasoning effort
The default model is gpt-5.5, chosen for its balance of capability, speed, and cost on computer-use tasks. You can switch models at any time with --model (for example --model claude-sonnet-4.6), or run --list-models to see what your Copilot account exposes.
- Python 3.11 or higher
- Operating System: Windows, macOS, or Linux
- The GitHub Copilot CLI installed and authenticated. The CLI is bundled with the Python SDK and provides the model access — no separate API key is required.
- Clone the repository:
git clone https://github.com/dc995/computer-use
cd computer-use- Install the required packages:
pip install -r requirements.txt- Authenticate the GitHub Copilot CLI (one time):
copilot auth loginIf you authenticate through the GitHub CLI instead, make sure the Copilot scope is granted:
gh auth login
gh auth refresh --scopes copilotVerify the CLI is available:
copilot --versionThe framework is designed to work directly with your local computer. Here's how to use it:
- Run the example application:
python main.py --instructions "Open web browser and go to microsoft.com"-
The agent will:
- Take screenshots of your screen
- Analyze the visual information
- Call computer-control tools to complete the task
- Observe the resulting screenshot and continue until the task is done
-
After the initial task completes, you can type follow-up instructions, or enter
exit(or a blank line) to quit.
--instructions: The initial task to perform (default: "Open web browser and go to microsoft.com.")--model: The Copilot model to use (default:gpt-5.5)--reasoning-effort: Reasoning effort for models that support it (low,medium,high, orxhigh; omit to use the model default)--list-models: List the models available to your Copilot account and exit--timeout: Seconds to allow each task to run before timing out (default: 1800)--monitor: Which display to view and control —allfor the full virtual desktop spanning every monitor (default), or a 1-based monitor index (1= first/primary). Screenshots and clicks are offset to the chosen region, so the agent can see and act on windows that open on a secondary monitor.--no-done: Suppress the completion indicator. By default, when a run finishes the agent opensdone.html(a full-screen "I'm done" banner) in your default browser so you get a clear visual signal that it's finished. Pass--no-doneto disable it.
Known limitation (mixed-DPI multi-monitor): Screen capture (
mss) and input (pyautogui) share one coordinate space, which works reliably when all monitors use the same display-scaling/DPI. On setups where monitors run at different scaling factors (e.g. a 150% laptop panel beside a 100% external monitor),pyautogui's physical-pixel clicks can drift on the lower-DPI display because it only sets system-level (not per-monitor v2) DPI awareness. Workaround: pin the agent to a single display with--monitor 1(or the relevant index). A future improvement would be to enable per-monitor-v2 DPI awareness and translate coordinates per monitor.
This agent controls your real mouse and keyboard. It is configured to run autonomously: tool permissions are auto-approved (PermissionHandler.approve_all) and a custom system prompt replaces the SDK's default guardrails (system_message uses mode: "replace"). Only run it on a machine where you are comfortable letting it act on your behalf, and supervise it while it works.
For scenarios requiring remote computer control or VM automation, we recommend using Playwright. Playwright provides robust browser automation capabilities and is well-suited for VM-based testing and automation scenarios.
For more information on VM automation with Playwright, please refer to:
This project is a port of the Azure-Samples/computer-use sample (originally an OpenAI computer-use sample) to the GitHub Copilot SDK. It is distributed under the MIT License; the original copyright is retained alongside the port author's. Thanks to the upstream maintainers for the foundation this builds on.
Released under the MIT License.