From d3721811c145389c1e452622f3eb64a61602f6e7 Mon Sep 17 00:00:00 2001 From: abrichr Date: Fri, 28 Aug 2026 11:33:48 -0400 Subject: [PATCH 1/3] docs: rewrite README for a human reader The README opened with a fifteen-line callout that spent most of its words describing openadapt-flow, so anyone who arrived for the training code read two paragraphs about a demonstration compiler first. That point is true and worth keeping, but it is one sentence, not a wall. Three documented APIs did not exist. Checked against openadapt-ml 0.16.3 installed from PyPI into a clean venv: - `AgentPolicy.predict(sample)` is not a method. The real one is `predict_action_from_sample(sample)`, and it returns a 4-tuple of `(Action, thought, state, raw_text)` rather than an object with `.action` and `.thought` attributes. - `QwenVLAdapter(model_name=...)` does not load a model. The constructor takes an already-loaded model and processor; the loading path is `QwenVLAdapter.from_pretrained(model_name)`. - The smoke test cannot run on the core install. `DummyAdapter` raises `ImportError: torch is required` unless the `training` extra is present. Every example is now a command or a snippet that was executed against that venv, shown with its real output. The benchmark table gains back the episode success column, which is 0% for all four configurations and was the most important number missing from the page. The architecture tree and the ecosystem table move to docs/repo_layout.md. Nothing was deleted. Co-Authored-By: Claude Opus 5 --- README.md | 370 +++++++++++++++----------------------------- docs/repo_layout.md | 87 +++++++++++ 2 files changed, 211 insertions(+), 246 deletions(-) create mode 100644 docs/repo_layout.md diff --git a/README.md b/README.md index e5cd7ae..1fa00ab 100644 --- a/README.md +++ b/README.md @@ -1,171 +1,91 @@ -# OpenAdapt-ML - -> [!IMPORTANT] -> **Status: experimental research. Not required by the product.** This package -> explores training and running demo-conditioned vision-language model (VLM) -> agents for GUI automation. It is evidence-generating research work with an -> unstable API, and it is not required to record, compile, or replay a -> workflow. -> -> The OpenAdapt product is a **governed demonstration compiler**: -> [`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow), installed -> via the [`OpenAdapt`](https://github.com/OpenAdaptAI/OpenAdapt) launcher -> (`pip install openadapt`). You record a workflow once, it compiles the -> demonstration into a deterministic, locally executable program, and it replays -> that program with **zero model calls on the healthy path**, halting instead of -> guessing when verification fails. Model training and grounding live here as a -> **research and cost-optimization surface (Phase 2)**, not as part of that -> deterministic replay path. Lifecycle labels for every repository are in the -> [repository lifecycle registry](https://github.com/OpenAdaptAI/.github/blob/main/REPOSITORY_LIFECYCLE.md). +# openadapt-ml [![Tests](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/test.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/test.yml) -[![PyPI version](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) -[![Downloads](https://img.shields.io/pypi/dm/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) -[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) - -OpenAdapt-ML is the research ML layer for [OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt). -It provides the GUI-specific machinery for experimenting with vision-language -model (VLM) agents that automate desktop tasks: canonical schemas for GUI -trajectories, VLM adapters, supervised fine-tuning, visual grounding, online RL -(GRPO) experiments, and demo-conditioned inference. - -## How this fits the product - -OpenAdapt is a governed demonstration compiler. All substrates are first-class, -with honest maturity: Browser is in beta (the full record, compile, and replay -loop runs in CI); Windows, macOS, and RDP are early access; Citrix and VDI are -exploratory. That deterministic replay loop lives in -[`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow) and makes no -model calls when a run is healthy. - -This repository sits deliberately upstream of that path. Everything here is -research aimed at the surfaces where a model may help: repairing or generalizing -a compiled step, grounding UI elements when structural cues are missing, and -reducing cost over time. Treat it as a lab, not a supported API. The APIs, -configs, and results below can and do change. - -## Demos - -**Synthetic Login** (Qwen3-VL-2B fine-tuned on synthetic UI scenarios): - -![Login Demo](experiments/qwen_login/login_demo.gif) -![Registration Demo](experiments/qwen_login/registration_demo.gif) - -## What is here - -- **GUI trajectory schemas.** Pydantic models for `Episode`, `Step`, `Action`, - and `Observation` with JSON Schema export and format converters (WAA, - WebArena). -- **VLM adapters.** A unified interface for Qwen3-VL and Qwen2.5-VL (local) plus - Claude, GPT, and Gemini (inference-only, API-backed), with automatic device - selection (CUDA / MPS / CPU). -- **Supervised fine-tuning (SFT).** TRL `SFTTrainer` with optional Unsloth - optimizations, training LoRA adapters. -- **Online RL (GRPO), experimental.** A Group Relative Policy Optimization - training module that collects rollouts against a live environment. See the - training status note below for what actually runs today. -- **Runtime policy API.** `AgentPolicy` predicts the next GUI action (`CLICK`, - `TYPE`, `DONE`, and related types) from a screenshot and goal. -- **Demo-conditioned inference.** Retrieval-augmented prompting that conditions - on recorded demonstrations for trajectory-aware disambiguation. -- **Grounding.** Locate UI elements via a vision API, oracle bounding boxes, or - Set-of-Marks (SoM) overlays. -- **Recording segmentation.** Turn raw recordings into described, deduplicated - segments. -- **Cloud GPU training.** One-command training pipelines for Lambda Labs, Modal, - and Azure, plus local training. -- **Synthetic data generation.** Configurable UI scenarios (login, registration) - with layout jitter for rapid iteration. - -## Training status (read before you train) - -Model training here is research and cost-optimization work, not the product's -healthy replay path. Two facts matter most: - -- **A base VLM cannot operate Windows out of the box.** In practice you need an - SFT checkpoint (or distillation) before online RL produces any signal. - Un-fine-tuned base models yield near-zero reward on real GUI tasks. -- **The GRPO module has two backends at different maturity.** - `GRPOConfig.backend="standalone"` (the default) is a built-in HuggingFace plus - PEFT trainer intended for single-GPU prototyping and debugging. - `backend="verl"` is an integration point for verl-agent / VAGEN - (GiGPO, multi-GPU); it currently prints setup instructions and raises - `NotImplementedError` rather than running a training job. Supervised - fine-tuning uses TRL's `SFTTrainer` and is the most exercised training path. - -Expect rough edges. This is where experiments happen. - -## Installation +[![PyPI](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) +[![Python](https://img.shields.io/pypi/pyversions/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -```bash -# Core package -pip install openadapt-ml - -# With training dependencies (torch, transformers, TRL, PEFT, datasets) -pip install openadapt-ml[training] +Show a vision-language model a screenshot and a goal, and get back the next GUI +action: click here, type this, stop. This package holds the parts that job +needs, which are a trajectory schema, adapters for Qwen3-VL and the hosted API +models, LoRA fine-tuning, UI grounding, and a policy object you can call from +Python. -# With API-backed VLMs (Claude, GPT) -pip install openadapt-ml[api] +It's for people running training experiments. This is research with an unstable +API, and you don't need it to record or replay a workflow. That's +[openadapt-flow](https://github.com/OpenAdaptAI/openadapt-flow), which compiles +a demonstration and replays it with zero model calls. -# From source -git clone https://github.com/OpenAdaptAI/openadapt-ml.git -cd openadapt-ml -uv sync -``` +[Docs](https://docs.openadapt.ai) · +[Design notes](docs/design.md) · +[Benchmark writeup](docs/qwen_login_experiment.md) · +[Repository layout](docs/repo_layout.md) -Unsloth is optional and installed separately; see the -[Unsloth install guide](https://docs.unsloth.ai/get-started/installation). +![Login demo](experiments/qwen_login/login_demo.gif) +![Registration demo](experiments/qwen_login/registration_demo.gif) -## Quick start +Qwen3-VL-2B, LoRA fine-tuned on the two synthetic scenarios that ship in +`openadapt_ml/ingest/synthetic.py`. Both scenarios jitter their layout between +episodes, so a model that memorized pixel coordinates fails them. -### Run a smoke test (no GPU) +## Try it ```bash -uv run python -m openadapt_ml.scripts.demo_policy --backend dummy +pip install 'openadapt-ml[training]' +python -m openadapt_ml.scripts.demo_policy --backend dummy ``` -### Train on synthetic data +The `training` extra isn't optional for this. A plain `pip install openadapt-ml` +gives you the schema and the converters, but no torch, and the dummy adapter +raises `ImportError: torch is required for DummyAdapter` without it. -```bash -uv run python -m openadapt_ml.scripts.train \ - --config configs/qwen3vl_synthetic.yaml -``` +The smoke test generates one synthetic login episode, builds an SFT-style +sample from it, and runs it through the policy: -### Train on real recordings - -```bash -# Record a workflow with openadapt-capture, then train -uv run python -m openadapt_ml.scripts.train \ - --config configs/qwen3vl_capture.yaml \ - --capture ~/captures/my-workflow \ - --open # Opens the training dashboard in a browser ``` +[user] Goal: Log in with username 'user0' and password 'pass0123' +This is step 1 of 6 (no actions completed yet). -### End-to-end benchmark (train, eval, plot) - -```bash -uv run python -m openadapt_ml.scripts.run_qwen_login_benchmark \ - --config configs/qwen3vl_synthetic_dev.yaml \ - --out-dir experiments/qwen_login/2b_dev +Predicted action: type= coordinates=None text=None +Thought: None +Raw output: DONE() ``` -### Use the policy API +Real output from 0.16.3 on macOS, trimmed. `DONE()` is the whole point of the +dummy backend: it returns a fixed action so the run proves the wiring, not the +model. Swap `--backend qwen3` and it downloads Qwen3-VL-8B and predicts for +real. + +## Use the policy from Python ```python +from openadapt_ml.datasets.next_action import build_next_action_sft_samples +from openadapt_ml.ingest.synthetic import generate_synthetic_episodes +from openadapt_ml.models.dummy_adapter import DummyAdapter from openadapt_ml.runtime.policy import AgentPolicy -from openadapt_ml.models.qwen_vl import QwenVLAdapter -adapter = QwenVLAdapter(model_name="Qwen/Qwen3-VL-2B-Instruct") -policy = AgentPolicy(adapter) +episodes = generate_synthetic_episodes(num_episodes=1, seed=99, output_dir="synthetic/demo") +sample = build_next_action_sft_samples(episodes)[0] + +action, thought, state, raw = AgentPolicy(DummyAdapter()).predict_action_from_sample(sample) +print(action.type, action.coordinates) +print(repr(raw)) +``` -# Given an SFT-style sample (screenshot, goal, chat history): -output = policy.predict(sample) -print(output.action) # Action(type=CLICK, coordinates={"x": 0.45, "y": 0.71}) -print(output.thought) # "Click the Login button" ``` +ActionType.DONE None +'DONE()' +``` + +`predict_action_from_sample` returns a 4-tuple, not an object with attributes. +For a real model, build the adapter with `QwenVLAdapter.from_pretrained(...)` +rather than calling the constructor, which wants an already-loaded model and +processor. -### Use the schema +## Record a trajectory + +Everything here reads and writes one schema, so a WAA episode, a WebArena +episode, and a recording off your own laptop end up the same shape: ```python from openadapt_ml.schema import Episode, Step, Action, Observation, ActionType @@ -187,110 +107,72 @@ episode = Episode( ], success=True, ) +print(episode.episode_id, len(episode.steps), episode.schema_version) ``` -## Architecture - ``` -openadapt_ml/ -├── schema/ # Episode, Step, Action, Observation (Pydantic) + converters -├── models/ # VLM adapters (Qwen3-VL, Qwen2.5-VL, API backends, dummy) -│ └── providers/ # Provider-specific client wiring -├── training/ # Fine-tuning + RL -│ ├── trl_trainer.py # TRL SFTTrainer (+ optional Unsloth) -│ ├── trainer.py # Training orchestration -│ ├── grpo/ # GRPO online RL (standalone default; verl = stub) -│ └── viewer.py # Training dashboard (HTML) -├── runtime/ # Inference: AgentPolicy + action safety gate -├── datasets/ # Episodes -> SFT chat samples -├── ingest/ # Synthetic UI, openadapt-capture loader, generic loader -├── grounding/ # UI element localization (oracle, vision API, SoM) -├── perception/ # Perception integration helpers -├── retrieval/ # Demo-conditioned retrieval for RAG-style prompting -├── segmentation/ # Recording -> described, deduplicated segments -├── baselines/ # Baseline agents and prompt/parse utilities -├── benchmarks/ # ML-specific benchmark agents (PolicyAgent, API, unified) -├── evals/ # Evaluation metrics (grounding, trajectory matching) -├── export/ # Dataset export (Parquet, CLI) -├── cloud/ # Cloud GPU training (Lambda Labs, Modal, Azure, vast.ai) -├── config.py # Settings via pydantic-settings -└── scripts/ # CLI entry points (train, eval, compare, demo) +demo_001 2 1.0.0 ``` -## Benchmark results - -These are controlled synthetic results. They show that the training pipeline -runs end to end, not real-world performance. - -### Synthetic Login (Qwen3-VL-2B with Set-of-Marks) - -| Metric | Score | -|----------------------|----------| -| Action Type Accuracy | **100%** | -| Element Accuracy | **100%** | -| Episode Success Rate | **100%** | - -### Multi-model comparison (Synthetic Login, coordinate mode) - -| Model | Action Accuracy | Coord Error | Click Hit Rate | -|-------------------|-----------------|-------------|----------------| -| Qwen3-VL-2B FT | 0.469 | 0.051 | 0.850 | -| Qwen3-VL-8B FT | 0.286 | 0.004 | 1.000 | -| Claude Sonnet 4.5 | 0.121 | 0.757 | 0.000 | -| GPT-5.1 | 0.183 | 0.057 | 0.600 | - -> This is a controlled synthetic benchmark with roughly three UI elements. It -> validates that the training pipeline works, not real-world accuracy. -> Evaluation on standard benchmarks (WAA, WebArena) is ongoing via -> [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals). - -## Cloud GPU training - -### Lambda Labs +## Train ```bash -export LAMBDA_API_KEY=your_key_here - -# Launch, train, download, and terminate in one command -uv run python -m openadapt_ml.cloud.lambda_labs train \ - --capture ~/captures/my-workflow \ - --goal "Turn off Night Shift in System Settings" -``` +# Synthetic data, no recordings needed +python -m openadapt_ml.scripts.train --config configs/qwen3vl_synthetic.yaml -### Local (CUDA / Apple Silicon) - -```bash -uv run python -m openadapt_ml.cloud.local train \ +# Your own recordings, with the training dashboard +python -m openadapt_ml.scripts.train \ + --config configs/qwen3vl_capture.yaml \ --capture ~/captures/my-workflow --open ``` -## Ecosystem - -OpenAdapt-ML is one component in the OpenAdapt stack: - -| Package | Purpose | -|---------|---------| -| **[OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt)** | Desktop automation platform and launcher (`pip install openadapt`) | -| **[openadapt-flow](https://github.com/OpenAdaptAI/openadapt-flow)** | The demonstration compiler: deterministic, zero-model-call replay on the healthy path | -| **[openadapt-ml](https://github.com/OpenAdaptAI/openadapt-ml)** | This repo: research ML (schemas, VLM adapters, training, inference, grounding) | -| **[openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals)** | Evaluation infrastructure: VM management, pool orchestration, benchmark runners, `oa-vm` CLI | -| **[openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture)** | Lightweight GUI recording and demo sharing | - -> Looking for benchmark evaluation, Azure VM management, or the `oa-vm` CLI? -> Those live in [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals). - -## Documentation +Training runs on a GPU box you rent by the hour. Lambda Labs, Modal, Azure, and +vast.ai each get a one-command wrapper under `openadapt_ml.cloud`, and +`openadapt_ml.cloud.local` does the same thing against CUDA or Apple Silicon. +The guide is [docs/cloud_gpu_training.md](docs/cloud_gpu_training.md). Unsloth +is separate, see the +[Unsloth install guide](https://docs.unsloth.ai/get-started/installation). -- [docs.openadapt.ai](https://docs.openadapt.ai) for the product and the overall - stack. -- [`docs/design.md`](docs/design.md) for system design (schemas, adapters, - training, runtime). -- [`docs/cloud_gpu_training.md`](docs/cloud_gpu_training.md) for the Lambda Labs - and Azure training guide. -- [`docs/qwen_login_experiment.md`](docs/qwen_login_experiment.md) for synthetic - benchmark reproduction. -- [`docs/gemini_grounding.md`](docs/gemini_grounding.md) for the grounding - module. +## What the numbers actually say + +Coordinate mode on the synthetic login scenario, from +[docs/qwen_login_experiment.md](docs/qwen_login_experiment.md) (December 2025): + +| Model | Action accuracy | Coord error | Click hit rate | Episode success | +|---|---|---|---|---| +| Qwen3-VL-2B fine-tuned | 46.9% | 0.051 | 85.0% | 0% | +| Qwen3-VL-8B fine-tuned | 28.6% | 0.004 | 100% | 0% | +| Claude Sonnet 4.5 | 12.1% | 0.757 | 0% | 0% | +| GPT-5.1 | 18.3% | 0.057 | 60.0% | 0% | + +Read the last column first. Not one configuration finished a single episode. +Fine-tuning moves individual-step accuracy and it moves click precision, and +neither of those got any model through a six-step login. Switching from +coordinates to Set-of-Marks element ids does finish episodes: 32 episodes, 384 +steps, 100% on action type, element choice, and episode success, retained in +[`experiments/qwen_login/registration_som_eval.json`](experiments/qwen_login/registration_som_eval.json). + +That's a procedurally generated form with roughly three interactive elements. +It shows the pipeline trains and evaluates end to end. It says nothing about a +real desktop. The later hardened re-runs kept under +`experiments/qwen_login/2b_dev/eval/` report different figures again, on n=32 +and n=4, which is about what you'd expect from samples that small. + +## Where this breaks + +- **A base VLM can't operate Windows.** Un-fine-tuned models score near-zero + reward on real GUI tasks, so online RL has nothing to climb. You need an SFT + checkpoint or a distillation pass before GRPO produces signal at all. +- **`backend="verl"` doesn't train anything.** It prints setup instructions and + raises `NotImplementedError`. The default `backend="standalone"` is a + HuggingFace plus PEFT trainer for single-GPU prototyping, and supervised + fine-tuning through TRL's `SFTTrainer` is the path that gets exercised. +- **The API moves.** Configs, module paths, result formats, and the shape of + what a function hands back all change between releases, with no deprecation + window. +- **The benchmarks are synthetic.** Evaluation against WAA and WebArena lives in + [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals), along with + VM management and the `oa-vm` CLI. ## Contributing @@ -298,21 +180,17 @@ OpenAdapt-ML is one component in the OpenAdapt stack: git clone https://github.com/OpenAdaptAI/openadapt-ml.git cd openadapt-ml uv sync --extra dev --extra training - -# Run tests uv run pytest - -# Lint uv run ruff check . ``` -We use [Conventional Commits](https://www.conventionalcommits.org/) (`feat:`, -`fix:`, `docs:`, and so on) with -[Python Semantic Release](https://python-semantic-release.readthedocs.io/) for -automated versioning and PyPI publishing. +Branches and pull requests, never a push straight to `main`. PR titles need +[Conventional Commits](https://www.conventionalcommits.org/) format, because +[Python Semantic Release](https://python-semantic-release.readthedocs.io/) +parses them to pick the next version and publish to PyPI. ## License -[MIT](LICENSE). OpenAdapt is open core: this repository is permissively licensed, -while private hardening corpora, tuned parameters, and deployment-derived recipes -are intentionally kept out of it. +[MIT](LICENSE). OpenAdapt is open core, so this repository is permissively +licensed while the hardening corpora, tuned parameters, and deployment-derived +recipes stay out of it. diff --git a/docs/repo_layout.md b/docs/repo_layout.md new file mode 100644 index 0000000..be5488c --- /dev/null +++ b/docs/repo_layout.md @@ -0,0 +1,87 @@ +# Repository layout + +Where things live in `openadapt_ml/`, and which other packages this one sits +next to. + +## Package tree + +``` +openadapt_ml/ +├── schema/ # Episode, Step, Action, Observation (Pydantic) + converters +├── models/ # VLM adapters (Qwen3-VL, Qwen2.5-VL, API backends, dummy) +│ └── providers/ # Provider-specific client wiring +├── training/ # Fine-tuning + RL +│ ├── trl_trainer.py # TRL SFTTrainer (+ optional Unsloth) +│ ├── trainer.py # Training orchestration +│ ├── grpo/ # GRPO online RL (standalone default; verl = stub) +│ └── viewer.py # Training dashboard (HTML) +├── runtime/ # Inference: AgentPolicy + action safety gate +├── datasets/ # Episodes -> SFT chat samples (next_action) +├── ingest/ # Synthetic UI, openadapt-capture loader, generic loader +├── grounding/ # UI element localization (oracle, vision API, SoM) +├── perception/ # Perception integration helpers +├── retrieval/ # Demo-conditioned retrieval for RAG-style prompting +├── segmentation/ # Recording -> described, deduplicated segments +├── baselines/ # Baseline agents and prompt/parse utilities +├── benchmarks/ # ML-specific benchmark agents (PolicyAgent, API, unified) +├── evals/ # Evaluation metrics (grounding, trajectory matching) +├── export/ # Dataset export (Parquet, CLI) +├── cloud/ # Cloud GPU training (Lambda Labs, Modal, Azure, vast.ai) +├── config.py # Settings via pydantic-settings +└── scripts/ # CLI entry points (train, eval, compare, demo) +``` + +## What each area does + +**Schemas.** Pydantic models for `Episode`, `Step`, `Action`, and `Observation`, +with JSON Schema export and converters for WAA and WebArena formats. Everything +else in the package reads and writes this shape. + +**VLM adapters.** One interface over Qwen3-VL and Qwen2.5-VL running locally, +plus Claude, GPT, and Gemini for inference only. Device selection across CUDA, +MPS, and CPU is automatic. Build a local adapter with +`QwenVLAdapter.from_pretrained(model_name)`. + +**Supervised fine-tuning.** TRL's `SFTTrainer` training LoRA adapters, with +optional Unsloth optimizations. This is the most exercised training path. + +**Online RL.** A Group Relative Policy Optimization module that collects +rollouts against a live environment. `GRPOConfig.backend` defaults to +`"standalone"`, a HuggingFace plus PEFT trainer for single-GPU prototyping. +`backend="verl"` is an integration point for verl-agent and VAGEN (GiGPO, +multi-GPU) that currently prints setup instructions and raises +`NotImplementedError`. + +**Runtime policy.** `AgentPolicy.predict_action_from_sample(sample)` returns a +4-tuple of `(Action, thought, state, raw_text)`. Action types include `CLICK`, +`TYPE`, `WAIT`, and `DONE`. + +**Demo-conditioned inference.** Retrieval-augmented prompting that conditions on +recorded demonstrations, so a step can be disambiguated by what the human did in +the same situation. + +**Grounding.** Locate a UI element by vision API, by oracle bounding box, or +through Set-of-Marks overlays. See [gemini_grounding.md](gemini_grounding.md). + +**Recording segmentation.** Turn a raw recording into described, deduplicated +segments. + +**Cloud GPU training.** One-command pipelines for Lambda Labs, Modal, Azure, and +vast.ai, plus local CUDA and Apple Silicon. See +[cloud_gpu_training.md](cloud_gpu_training.md). + +**Synthetic data.** Configurable login and registration scenarios with layout +jitter, for iterating without recording anything. + +## Neighbouring packages + +| Package | Purpose | +|---|---| +| [OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt) | Desktop automation platform and launcher (`pip install openadapt`) | +| [openadapt-flow](https://github.com/OpenAdaptAI/openadapt-flow) | The demonstration compiler: deterministic, zero-model-call replay on the healthy path | +| [openadapt-ml](https://github.com/OpenAdaptAI/openadapt-ml) | This repository: schemas, VLM adapters, training, inference, grounding | +| [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals) | Evaluation infrastructure: VM management, pool orchestration, benchmark runners, `oa-vm` CLI | +| [openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture) | Lightweight GUI recording and demo sharing | + +Lifecycle labels for every repository are in the +[repository lifecycle registry](https://github.com/OpenAdaptAI/.github/blob/main/REPOSITORY_LIFECYCLE.md). From 7e0c94bc961212732be621a4c52a8e5545bc27a8 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 28 Aug 2026 13:11:32 -0400 Subject: [PATCH 2/3] docs: restore the research banner and fix four claims AGENTS.md requires a public repo outside the seven product targets to carry a lifecycle state, and records an incident where deleting this banner cost four repos of wasted agent effort. The rewrite removed it. Put it back above the badges. The Set-of-Marks paragraph introduced a login result and then cited registration_som_eval.json. That artifact is the registration scenario: 384 steps over 32 episodes is 12 steps each, and synthetic.py documents registration as 12 steps and 6 elements. Name the scenario and correct three elements to six. The n=4 hardened re-runs are under 8b_hero/eval/, not 2b_dev/eval/, whose two files are both n=32. There is no Azure trainer. cloud/ has train subcommands for lambda_labs, modal_cloud, vast_ai, and local; azure_inference.py is an async inference queue. Drop Azure from the training list and say what it is. The Action line cut 16 of 19 fields with no ellipsis, so it read as though Action has three. Mark the cut and give the count. Training reads --config paths relative to the checkout, and tool.uv.sources points openadapt-capture at ../openadapt-capture, so a fresh clone needs UV_NO_SOURCES=1, which is what test.yml sets. Give the clone and the flag. --- README.md | 48 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 1fa00ab..5616d95 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,14 @@ # openadapt-ml +> [!IMPORTANT] +> **Status: Research. Not required by the product.** This package trains and +> runs vision-language model agents for GUI automation. You do not need it to +> record, compile, or replay a workflow. That is +> [openadapt-flow](https://github.com/OpenAdaptAI/openadapt-flow), installed by +> the [OpenAdapt](https://github.com/OpenAdaptAI/OpenAdapt) launcher. Lifecycle +> labels for every repository are in the +> [repository lifecycle registry](https://github.com/OpenAdaptAI/.github/blob/main/REPOSITORY_LIFECYCLE.md). + [![Tests](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/test.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/test.yml) [![PyPI](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) [![Python](https://img.shields.io/pypi/pyversions/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/) @@ -46,12 +55,13 @@ sample from it, and runs it through the policy: [user] Goal: Log in with username 'user0' and password 'pass0123' This is step 1 of 6 (no actions completed yet). -Predicted action: type= coordinates=None text=None +Predicted action: type= coordinates=None text=None ... Thought: None Raw output: DONE() ``` -Real output from 0.16.3 on macOS, trimmed. `DONE()` is the whole point of the +Real output from 0.16.3 on macOS. `Action` carries 19 fields and all but three +are cut from that line. `DONE()` is the whole point of the dummy backend: it returns a fixed action so the run proves the wiring, not the model. Swap `--backend qwen3` and it downloads Qwen3-VL-8B and predicts for real. @@ -116,6 +126,15 @@ demo_001 2 1.0.0 ## Train +The `--config` paths below are repo-relative, so training needs the checkout +rather than the wheel: + +```bash +git clone https://github.com/OpenAdaptAI/openadapt-ml.git +cd openadapt-ml +UV_NO_SOURCES=1 uv sync --extra training +``` + ```bash # Synthetic data, no recordings needed python -m openadapt_ml.scripts.train --config configs/qwen3vl_synthetic.yaml @@ -126,9 +145,10 @@ python -m openadapt_ml.scripts.train \ --capture ~/captures/my-workflow --open ``` -Training runs on a GPU box you rent by the hour. Lambda Labs, Modal, Azure, and -vast.ai each get a one-command wrapper under `openadapt_ml.cloud`, and +Training runs on a GPU box you rent by the hour. Lambda Labs, Modal, and +vast.ai each get a one-command training wrapper under `openadapt_ml.cloud`, and `openadapt_ml.cloud.local` does the same thing against CUDA or Apple Silicon. +The Azure module there is an async inference queue, not a trainer. The guide is [docs/cloud_gpu_training.md](docs/cloud_gpu_training.md). Unsloth is separate, see the [Unsloth install guide](https://docs.unsloth.ai/get-started/installation). @@ -147,16 +167,18 @@ Coordinate mode on the synthetic login scenario, from Read the last column first. Not one configuration finished a single episode. Fine-tuning moves individual-step accuracy and it moves click precision, and -neither of those got any model through a six-step login. Switching from -coordinates to Set-of-Marks element ids does finish episodes: 32 episodes, 384 -steps, 100% on action type, element choice, and episode success, retained in +neither of those got any model through the login. Switching from coordinates to +Set-of-Marks element ids does finish episodes, on the registration scenario: 32 +episodes, 384 steps, 100% on action type, element choice, and episode success, +retained in [`experiments/qwen_login/registration_som_eval.json`](experiments/qwen_login/registration_som_eval.json). -That's a procedurally generated form with roughly three interactive elements. -It shows the pipeline trains and evaluates end to end. It says nothing about a -real desktop. The later hardened re-runs kept under -`experiments/qwen_login/2b_dev/eval/` report different figures again, on n=32 -and n=4, which is about what you'd expect from samples that small. +That's a procedurally generated form with six interactive elements. It shows +the pipeline trains and evaluates end to end. It says nothing about a real +desktop. The hardened login re-runs report different figures again, n=32 under +`experiments/qwen_login/2b_dev/eval/` and n=4 under +`experiments/qwen_login/8b_hero/eval/`, which is about what you'd expect from +samples that small. ## Where this breaks @@ -179,7 +201,7 @@ and n=4, which is about what you'd expect from samples that small. ```bash git clone https://github.com/OpenAdaptAI/openadapt-ml.git cd openadapt-ml -uv sync --extra dev --extra training +UV_NO_SOURCES=1 uv sync --extra dev --extra training uv run pytest uv run ruff check . ``` From 022aa3990f64df9cee383dca5f33261e73f21370 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 28 Aug 2026 13:11:46 -0400 Subject: [PATCH 3/3] docs: repo_layout listed an Azure trainer that does not exist Same correction as the README. cloud/ has train subcommands for lambda_labs, modal_cloud, vast_ai, and local; azure_inference.py is an inference queue. --- docs/repo_layout.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/repo_layout.md b/docs/repo_layout.md index be5488c..3855a9a 100644 --- a/docs/repo_layout.md +++ b/docs/repo_layout.md @@ -26,7 +26,7 @@ openadapt_ml/ ├── benchmarks/ # ML-specific benchmark agents (PolicyAgent, API, unified) ├── evals/ # Evaluation metrics (grounding, trajectory matching) ├── export/ # Dataset export (Parquet, CLI) -├── cloud/ # Cloud GPU training (Lambda Labs, Modal, Azure, vast.ai) +├── cloud/ # Cloud GPU training (Lambda Labs, Modal, vast.ai) ├── config.py # Settings via pydantic-settings └── scripts/ # CLI entry points (train, eval, compare, demo) ``` @@ -66,8 +66,9 @@ through Set-of-Marks overlays. See [gemini_grounding.md](gemini_grounding.md). **Recording segmentation.** Turn a raw recording into described, deduplicated segments. -**Cloud GPU training.** One-command pipelines for Lambda Labs, Modal, Azure, and -vast.ai, plus local CUDA and Apple Silicon. See +**Cloud GPU training.** One-command pipelines for Lambda Labs, Modal, and +vast.ai, plus local CUDA and Apple Silicon. The Azure module here is an async +inference queue, not a trainer. See [cloud_gpu_training.md](cloud_gpu_training.md). **Synthetic data.** Configurable login and registration scenarios with layout