Fix doctor reporting false failures on valid setups

[dhruv-15-03]

Problem

On a correctly provisioned machine, webwright doctor reports failures for things that
work. Running it on Windows with Firefox installed per
skills/webwright/reference/playwright_patterns.md:

                               Webwright Doctor
┌────────────┬────────┬───────────────────────────────────────────────────────┐
│ Check      │ Status │ Details                                               │
├────────────┼────────┼───────────────────────────────────────────────────────┤
│ Python     │ PASS   │ Python 3.12                                           │
│ Playwright │ PASS   │ playwright installed                                  │
│ Chromium   │ FAIL   │ [WinError 2] The system cannot find the file          │
│            │        │ specified                                             │
│ Screenshot │ FAIL   │ unable to launch Chromium for screenshot validation   │
│            │        │ Fix: playwright install                               │
│ OpenAI Key │ FAIL   │ OPENAI_API_KEY missing                                │
│            │        │ Fix: set the OPENAI_API_KEY environment variable      │
│ Plugins    │ PASS   │ plugin manifests found                                │
└────────────┴────────┴───────────────────────────────────────────────────────┘

3/6 checks passed

Firefox launches and screenshots fine on that machine. All three failures are wrong.

What was wrong

Chromium → Browsers

Two independent bugs in one check:

• It ran the bare playwright console script. That resolves on POSIX but not reliably
  on Windows, so the failure surfaced as a raw [WinError 2] telling the user nothing.
• It only inspected the return code of playwright install --dry-run. That command
  exits 0 whether or not a browser is installed, so on POSIX the check passes
  unconditionally. It could never detect a missing browser — only a missing launcher.

Now invokes sys.executable -m playwright, parses the Install location: lines that
--dry-run prints per engine, and stats them. An engine counts as present only if its
directory exists. Firefox and Chromium are both reported.

Screenshot

• Hard-coded Chromium. The skill reference pins Firefox because Akamai-fronted sites
  reject Chromium on TLS/H2 fingerprinting, so a setup provisioned by following the
  project's own docs fails this check.
• Wrote doctor_test.png into the caller's working directory and left it behind.

Now tries Firefox first, falls back to Chromium, and writes into a TemporaryDirectory.

OpenAI Key → Model Backend

Required OPENAI_API_KEY specifically. An Anthropic or OpenRouter setup reported FAIL,
and so did the Claude Code / Codex plugin path, which needs no key at all because the
host agent drives the loop. Now accepts any backend shipped under webwright/models/,
and when none is set it says so while noting plugin mode does not require one.

Plugins

Resolved the manifest paths against the process working directory, so the check passes
from the repo root and fails from anywhere else. This does not show in the table above
because that run was from the repo root:

# upstream, cwd = src/webwright/
(False, 'missing plugin manifests: Claude, Codex\nFix: configure Claude/Codex plugins')

# this PR, same cwd
(True, 'plugin manifests found (.../Webwright)')

Now walks upward from the working directory to locate the repo root.

After

Same machine, no configuration changed:

                               Webwright Doctor
┌───────────────┬────────┬────────────────────────────────────────────────────┐
│ Check         │ Status │ Details                                            │
├───────────────┼────────┼────────────────────────────────────────────────────┤
│ Python        │ PASS   │ Python 3.12                                        │
│ Playwright    │ PASS   │ playwright installed                               │
│ Browsers      │ PASS   │ firefox available (not installed: chromium)        │
│ Screenshot    │ PASS   │ screenshot capture working (firefox)               │
│ Model Backend │ FAIL   │ no model API key found (checked OPENAI_API_KEY,    │
│               │        │ ANTHROPIC_API_KEY, OPENROUTER_API_KEY)             │
│               │        │ Fix: set one for CLI mode -- not required when     │
│               │        │ running Webwright as a Claude Code / Codex plugin  │
│ Plugins       │ PASS   │ plugin manifests found (.../Webwright)             │
└───────────────┴────────┴────────────────────────────────────────────────────┘

5/6 checks passed

The remaining failure is correct: that machine genuinely has no API key set.

Tests

tests/unit/test_doctor.py asserted isinstance(ok, bool) on each check, which holds
regardless of what the check does. Replaced with tests that pin actual behaviour:

• test_parse_install_locations / _empty — the --dry-run output parser
• test_check_browsers_reports_an_installed_engine
• test_check_model_backend_accepts_anthropic_only / _accepts_openrouter_only
• test_check_model_backend_missing_mentions_plugin_mode
• test_plugin_manifests_found_from_subdirectory
• test_screenshot_does_not_pollute_cwd

9 tests → 22, all passing.

Notes

• Nothing outside run/doctor.py and its tests is touched.
• Two check labels change (Chromium → Browsers, OpenAI Key → Model Backend)
  because both now verify something broader than their old name. Glad to keep the
  original labels if you would rather not change the output.
• An earlier check_browsers draft used sync_playwright() just to read
  executable_path; that leaves a pending driver task and prints
  Task was destroyed but it is pending! to stderr. The subprocess approach avoids
  connecting to the driver at all, which is why it is written this way.

[Copilot]

Pull request overview

This PR fixes webwright doctor reporting false failures on valid setups (notably Windows + Firefox-first configurations) by making checks both more accurate and more aligned with the project’s documented provisioning paths.

Changes:

• Replaces the old Chromium-only install check with a browser-engine check that invokes sys.executable -m playwright install --dry-run, parses reported install locations, and verifies they exist on disk.
• Updates the screenshot validation to try Firefox first (then Chromium) and to write the temporary screenshot into a TemporaryDirectory instead of the caller’s cwd.
• Broadens the “model backend” check to accept OpenAI/Anthropic/OpenRouter env vars and improves plugin manifest resolution by walking upward to find the repo root; expands unit tests to assert real behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/webwright/run/doctor.py	Fixes doctor’s browser detection, screenshot validation, model backend key acceptance, and plugin manifest root detection; updates check labels.
tests/unit/test_doctor.py	Replaces weak type-only assertions with behavior-focused tests covering parsing, browser reporting, model backend env var acceptance, plugin root discovery, and screenshot cwd cleanliness.

Suppressed comments (2)

src/webwright/run/doctor.py:120

• Same as above: this fix hint uses the bare playwright script, which the PR notes is unreliable on Windows. Suggest switching the hint to python -m playwright install firefox so users can actually follow it on Windows.

    return False, (
        "no Playwright browsers installed\nFix: playwright install firefox"
    )

src/webwright/run/doctor.py:171

• The screenshot check’s fix hint also uses the bare playwright script; given the PR motivation (Windows launcher resolution), the hint should be the cross-platform python -m playwright install firefox.

    return False, (
        f"unable to capture a screenshot with any installed browser{detail}\n"
        "Fix: playwright install firefox"
    )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dhruv-15-03]

@microsoft-github-policy-service agree

[dhruv-15-03]

Correction to my previous comment on this PR.

I entered company="Microsoft" in error — I copied the illustrative example
from the CLA message instead of the default option. To be unambiguous:

• I have no employment, contractor, or vendor relationship with Microsoft.
• No employer has intellectual property rights in these submissions.
• I have sole ownership of the intellectual property rights in my
  contributions, and I am not making them in the course of work for any
  employer.

I do not have, and did not intend to claim, authority to bind Microsoft
Corporation or any other entity to this Agreement. Please disregard the
company designation in my earlier comment. Re-agreeing under the correct
option in the comment below.