Add Web Skill Factory: Evolving Reusable, Verified, Code-Native Skills for Web Agents

[Demi Wang (DEM1TASSE)]

What

Adds Web Skill Factory (webwright.skill_factory) — a self-evolving skill library on top of Webwright's code-as-action solves. It turns the final_script.py every solve already produces into reusable, parameterized skills that are plain Python + Playwright: code you can run without a model and compose into the next task instead of re-exploring a site.

solve → gate → group by template → distill → replay-verify → library → next solve reuses

Reuse is resolved out of the agent loop: before a solve starts, route (built on recommend) checks the library and either runs a matching skill directly — no model, no agent — or hands it to the agent as a prior to adapt. The verdict is run / adapt / skip; the agent never spends its own steps querying the library. Both integration points are additive; the agent loop and default config are unchanged.

Why code, not notes:

• Runs standalone, no model. A learned skill re-executes in ~40 s with zero tokens — cron-able, instead of a model re-reading a note and redoing the work every time.
• Real software-engineering surface. Because skills are code, they inherit code's properties for free: tests, versioning, history, encapsulation — executable and verifiable, not prose the model must interpret.
• Verified twice before it lands. An input gate keeps wrong solves out of the material; then the distilled skill must replay its own answers standalone, with no model, before it enters the library.
• Gets stronger with use. New solves widen a skill in place; regression-replay keeps a later change from breaking coverage a skill already had.

Two gates, and a grade that means something

• Input gate — a solve becomes material only if its answer is trustworthy: gold (a known answer), or the default self_verify (shape + non-empty + the agent's own success report).
• Output gate — the distilled skill is replayed standalone, with no model, on its own training taskspecs, and admitted only if it reproduces the recorded answers.

The output gate is what lets a landed skill carry a real grade: executable (replay ran and reproduced — run it directly), reference (replay ran and failed — kept as a labelled prior the agent reads), or unverified (replay skipped).

Modules

Each has a stable interface and a swappable implementation:

module	role
skill_factory/library.py	store — Skill + Library, skills on disk (skill.py + meta.json)
skill_factory/retrieve.py	retrieve — task → most relevant candidates
skill_factory/decide.py	decide — candidates → use / adapt / skip
skill_factory/gate.py	input gate — gold / self_verify / none (keeps wrong solves out)
skill_factory/update.py	evolve — incremental growth; refine distils, parameterizes, decomposes into primitives, and replay-verifies
skill_factory/learn.py	learn <runs_dir> — auto-group runs into templates, gate, evolve; no manifest to write
skill_factory/init.py	init "<need>" — draft a skill.yaml skeleton to fill with ground truth
skill_factory/build.py	build <spec> — solve N instances then learn (parallel, resumable)
skill_factory/llm.py	backend-agnostic LLM via Webwright's Model (no endpoint/key hardcoded)
skill_factory/prompt.py	resolves the library lookup out of the agent loop and injects the chosen skill into the prompt (with_skill_hint)
skill_factory/route.py	out-of-loop entry — recommend, then act: run the skill directly / launch the agent / fall back
skill_factory/execute.py · fill.py · entry_shim.py	run a skill directly via a taskspec; fill its slots from the task; CLI shim so a skill also takes --flags
tools/skill_use.py	recommend(task, library) → run/adapt/skip + chosen skill, how-to-reuse, filled params — the out-of-loop decision route is built on

How it plugs in (no change to the agent loop or default config)

1. Reuse, out of the agent loop — route decides (via recommend) and acts; the decision is resolved before/around the solve, not by the agent mid-loop:

   python -m webwright.skill_factory route --task "<the task>" --library ./library [--start-url ...]

   verdict run (execute the skill directly, no model) / adapt (agent reuses it as a prior) / skip. Without --start-url it only prints the decision (and still runs a directly-runnable skill).
2. Growth after solving — three ways in, by what you already have:

   python -m webwright.skill_factory init  "<one-line need>"          # draft a spec to fill
   python -m webwright.skill_factory build skill.yaml --library ./library   # solve N + learn
   python -m webwright.skill_factory learn outputs/  --library ./library    # distil runs you have

   The manual manifest path (update) remains for benchmarks and logged-in sites.

examples/learned_library/ checks in the skill this produced from 3 real Google Flights solves — five parameters lifted (origin/destination city+code, date), verified on an unseen route three independent ways (from scratch / reuse / standalone, same answer) — with a CI test locking it.

Validation

WebArena: 10 templates across 3 sites — reuse lifts held-out accuracy +15pp and saves steps

10 retrieve-type task templates across shopping-admin / gitlab / map. Per template: 3 train solves build the library (admitted only if gold-verified), 2 held-out instances (unseen parameter values) measure reuse. Every task is solved both with the library and from scratch. Model: gpt-5.4. 100 runs total.

	WITH library	from scratch	Δ
held-out accuracy (20)	70%	55%	+15 pp
held-out avg steps	14.7	17.1	−2.4
train accuracy (30)	86.7%	76.7%	+10 pp
train avg steps	13.7	15.9	−2.2

• Reuse helps most when solving from scratch is expensive. Of 20 held-out tasks, 4 were rescued (failed from scratch, solved with the library) and 6 more solved in fewer steps. Biggest single win: 33 steps → 10.
• Retrieval stayed reliable as the library grew to 10 skills: all 20 held-out solves retrieved their own template's skill, including two near-duplicate gitlab commit skills.
• These numbers are the agent-in-loop path — the agent reading a skill as a prior. The Quick Start's flights example shows the other end: a strict-verified executable skill rerunning an unseen route standalone, with no model.

[Adam (adamlu123)]

update src/webwright/skills/README.md

[Adam (adamlu123)]

I did not find a skill_factory/README.md?

[Adam (adamlu123)]

detailed comments inline.