Add prerelease-test skill for release QA campaigns

.claude/skills/prerelease-test/SKILL.md

@@ -0,0 +1,190 @@
+---
+name: prerelease-test
+description: Run the independent pre-release QA campaign for a Reflex release train. Discovers what shipped by reading every CHANGELOG.md on the pre-release branch, checks each package actually published to PyPI, then exercises the new features end-to-end in real apps driven by a real browser using ONLY published packages, upgrade-tests reflex-examples apps from the previous stable, regression-tests reflex-enterprise demos, audits wheel/sdist packaging, and produces a triaged FINDINGS.md plus a fix-before-release plan. Use this whenever a Reflex release is being prepared or checked — the user mentions a pre-release, an alpha/beta/rc, an `r/pre-*` branch, "about to release", release QA, validating a release candidate, or re-verifying that release-blocking fixes landed in a newer alpha. Also use it for the narrower slices on their own: smoke-testing published packages, upgrade/regression testing example apps against a new version, or auditing that packages ship their `.pyi` stubs in both wheel and sdist.
+---
+
+# Reflex pre-release testing
+
+This skill runs the QA campaign that stands between a pre-release and a release: independent,
+adversarial, end-to-end exercise of what actually got published, from the perspective of a user
+who runs `pip install reflex` and opens a browser.
+
+Its value comes from three habits, not from breadth of assertions:
+
+1. **Only published packages.** Everything installs from PyPI into throwaway venvs. The checkout
+   has unreleased code, different metadata, and a different dependency graph — testing it tells
+   you nothing about what users will get, and a package that failed to publish is itself a
+   release blocker you would otherwise miss.
+2. **Baseline against the previous stable.** A defect that also reproduces on the last release is
+   a bug; one that only reproduces on the new version is a regression and usually a blocker.
+   Without the baseline run you cannot tell them apart, and the triage that follows is guesswork.
+3. **Adversarially verify before reporting.** Have a second agent reproduce each claimed issue
+   from the written repro alone and actively try to refute it. In practice this refutes or
+   reclassifies a meaningful share of findings — and it is what makes the report trustworthy
+   enough for maintainers to act on directly.
+
+## Deliverables
+
+Produce these under `prerelease-testing/<date>-<version>/` in the repo and commit them to a work
+branch as you go (never to `main`):
+
+- `FINDINGS.md` — executive summary, numbered findings with repro + evidence + regression status,
+  refuted claims, and per-cluster summaries. Templates: `references/reporting.md`.
+- `RELEASE_PLAN.md` — triage into fix-before-release vs file-as-issue. Rubric below.
+- One directory per test cluster containing the sample app sources (no `.web/`, `node_modules/`,
+  venvs), the driver scripts, `NOTES.md` with exact rerun commands, logs and screenshots.
+- `README.md` — what each cluster covers and how to reuse it for the next release.
+
+Findings must be reproducible by a stranger from `NOTES.md` alone. That is the bar: a fix agent
+gets handed the finding and nothing else.
+
+## Phases
+
+Work these in order. Phases 2–5 can overlap; 0 and 1 gate everything.
+
+### Phase 0 — Scope discovery
+
+Find the pre-release branch (`git ls-remote --heads origin 'r/pre-*'`, or the user names it), then
+read **every** `CHANGELOG.md` on it: the root one plus `packages/*/CHANGELOG.md`. The top entry of
+each is what this train ships.
+
+Run the discovery script to extract every package's version and confirm each is published:
+
+    uv run --script .claude/skills/prerelease-test/scripts/check_release_versions.py --ref <git-ref>
+
+(Script paths in this skill are repo-root-relative — the repo has its own unrelated `scripts/`
+directory, so the prefix matters.) It exits 1 when a package is confirmed missing and 2 when a
+check could not complete, so a proxy hiccup never reads as a missing package. An unpublished
+version is a finding in its own right — report it immediately rather than working around it.
+
+Read the linked PRs for anything whose intent is not obvious from the changelog line; the GitHub
+MCP tools (`mcp__github__pull_request_read`) do this well. Understanding what a change was *for*
+is what lets you test it as a user rather than as a checklist.
+
+### Phase 1 — De-risk with a smoke test
+
+Before any fan-out: one venv, `reflex init --template blank`, `reflex run`, then drive it in
+Chromium with `.claude/skills/prerelease-test/scripts/drive_app.py`. This shakes out environment
+problems (proxy, bun installs, browser path) once, in a context where you can debug them, instead
+of inside ten parallel agents.
+
+The scripts carry PEP 723 metadata, so `uv run --script <path>` runs each one in its own isolated
+environment (the driver pulls in playwright that way); running them with an existing driver venv's
+interpreter works too.
+
+Also confirm the dependency graph resolves the way the changelog says it should (e.g. optional
+dependencies genuinely absent, sub-packages pinned as intended).
+
+### Phase 2 — Feature exploration fan-out
+
+Decompose the changelog into clusters of related changes and give each to its own agent, with an
+adversarial verification stage behind it. See `references/clusters.md` for how to cut the clusters
+and a standing coverage list; `references/orchestration.md` for the Workflow script, schemas and
+port map.
+
+The instruction that makes this productive: **real-world exploration, not coverage filling.**
+Combine each new feature with the things users actually combine it with — State vars,
+`rx._x.client_state`, `rx.ComponentState`, `@rx.memo` wrapping, `rx.foreach`/`rx.cond`, event
+chains, background tasks, multiple pages and navigation, dev *and* prod mode. Bugs live in the
+interactions, and the original PR's tests already cover the happy path.
+
+Every run inspects all four channels: server log, browser console (errors *and* warnings), network
+tab (failed requests, 4xx/5xx), and the rendered page. Findings frequently show up in a channel
+nobody asserted on.
+
+### Phase 3 — Upgrade regression
+
+Users upgrade in place; that path has its own failure modes (lockfile migration, pruned packages,
+stale `.web/`). Clone `reflex-dev/reflex-examples`, pick apps that span the feature surface —
+including ones using third-party packages (`reflex-local-auth`, `reflex-global-hotkey`) and
+`reflex[db]`/API apps — then for each:
+
+1. Install the **previous stable** (no `--prerelease` flag, so requirements resolve as a user's
+   would), run it, and drive its real user flows in the browser. This is your baseline.
+2. Upgrade the *same* venv and *same* app directory in place to the new version, preserving
+   `.web/` and `reflex.lock/`. Watch the first run's log closely — that is where migration
+   happens. Re-drive the identical flows and compare.
+3. Do one cold run (`rm -rf .web`) to check the fresh-install path converges to the same state.
+
+Diff `.web/package.json` before and after; unexpected dependency changes are findings.
+
+### Phase 4 — Enterprise regression
+
+Downstream breakage is the most common release blocker, because removing a public name is
+invisible until something imports it. Install the **published** `reflex-enterprise` (never the
+checkout) against the new reflex and run its demos — ag-grid, map, dnd, mantine, flow, the MCP
+plugin, OIDC. Baseline against the previous stable whenever something fails, so you can say
+whether the new release broke it.
+
+Before running anything, grep the published enterprise wheel for names the changelog says moved or
+were removed; that finds the breakage in seconds instead of hours.
+
+### Phase 5 — Packaging audit
+
+Audit every package in the train; the discovery script feeds it the whole list. Chain the two with
+`&&`, never a pipe — a pipe reports only the audit's exit status, so a package whose PyPI check
+never completed would be dropped from the list and the audit would still print PASS over what was
+left. The `&&` is what makes that impossible: the audit runs only when discovery exits 0.
+
+    uv run --script .claude/skills/prerelease-test/scripts/check_release_versions.py \
+      --ref <ref> --specs > specs.txt \
+      && xargs uv run --script .claude/skills/prerelease-test/scripts/audit_pyi.py \
+        --manifest-ref <ref> < specs.txt
+
+It verifies each package ships its own generated `.pyi` stubs in **both** wheel and sdist, that
+they are byte-identical between the two, that no package leaks another package's stubs, and that
+counts line up with `pyi_hashes.json` (a package absent from the manifest must ship none).
+
+Stub content hashes legitimately differ from the committed manifest because the build hook
+regenerates them at release time — compare *presence and counts*, and wheel against sdist.
+
+### Phase 6 — Triage and report
+
+Write `FINDINGS.md` first (everything confirmed), then `RELEASE_PLAN.md` splitting findings into:
+
+**Fix before release** — anything meeting one of:
+- a confirmed regression against the previous stable,
+- security-relevant (path traversal, unauthenticated input handling, injection),
+- significant user impact, or trivially small to fix (a one-line guard, a missing default).
+
+**File as issues, fix after** — everything else, including pre-existing defects the campaign
+happened to surface. Downstream (enterprise) issues go to that project's tracker, not this repo's.
+
+Also flag decisions only a maintainer can make — an undocumented behavior change is either a bug
+or a missing changelog entry, and which one it is is their call. Ask rather than assume.
+
+### Phase 7 — Re-verify the next build
+
+When a new alpha ships with the fixes, re-run the *original failing repro* for each finding
+(not just the unit tests) against the new published packages, plus a general smoke. Append the
+pass/fail table to `FINDINGS.md`. When the final ships, do one last stock-install smoke:
+`pip install reflex==<final>` with no prerelease flag, init, run, browser, and confirm the
+resolved graph is all-final.
+
+## Working rules
+
+- **Isolation.** Every agent gets its own venv and its own reserved port range; never install into
+  a shared venv. See the port map in `references/orchestration.md`.
+- **The traps are real.** `references/agent-brief.md` is the brief to hand every agent — it carries
+  the environment gotchas (checkout shadowing, proxy variables breaking installs) and the list of
+  known-benign console noise. Agents that skip it rediscover the same problems and report noise as
+  findings.
+- **Concurrency.** Dev servers are heavy (bun + vite + granian). On a 4-CPU box run about two
+  agents at a time and one dev server per agent; more just makes everything slow and flaky.
+- **Commit as you go.** Each cluster's artifacts land on the work branch when that cluster
+  finishes, so a long campaign is never one unrecoverable batch.
+- **Never fix framework code during the campaign.** Testing and fixing are separate engagements
+  with separate review standards; mixing them costs you the independence that makes the findings
+  credible. Record the repro and move on. (If the user then asks for fixes, that is a new task —
+  one PR per finding, regression test first, `references/reporting.md` has the PR conventions.)
+
+## Scaling the campaign
+
+Full campaign is roughly 30 agents over several hours. Scale down by dropping whole phases rather
+than by testing each phase more shallowly — a shallow pass produces false confidence:
+
+- **Quick check** (~30 min): phases 0, 1, 5, plus grep-based downstream checks from phase 4.
+- **Standard** (~2 h): add phase 2 over the three or four highest-risk clusters.
+- **Full**: everything, with adversarial verification on every claimed issue.
+
+Ask the user which they want if the request is ambiguous and the difference matters.

.claude/skills/prerelease-test/references/agent-brief.md

@@ -0,0 +1,115 @@
+# The agent brief
+
+Copy this into the campaign scratchpad (e.g. `$SB/AGENT_BRIEF.md`), fill the `<...>` placeholders,
+and tell every test agent to read it first. It carries the rules that keep results trustworthy and
+the environment traps that otherwise get rediscovered — expensively — by each agent in turn.
+
+Keep it in a file rather than pasting it into each prompt: agents re-read it when they get
+confused, and updating one file updates the whole fleet.
+
+---
+
+## BRIEF TEMPLATE (copy from here)
+
+You are one of several agents independently exercising the reflex `<VERSION>` pre-release
+(published `<DATE>`) as a real-world user of the framework. Your job: build small sample apps and
+repro scripts for your assigned feature cluster, run them END-TO-END (real server, real Chromium),
+and hunt for anomalies. You REPORT issues; you never fix framework code.
+
+### HARD RULES
+
+1. **NEVER install reflex (or any workspace package) from the local checkout at `<REPO>`.** No
+   `uv sync`, no `uv run`, no `pip install -e`, no `uv pip install .` anywhere under it.
+   Everything installs from PyPI. We are testing what users receive, not what the tree contains.
+2. **Never run python with the checkout as your working directory.** `<REPO>/reflex/` shadows the
+   installed package, so `import reflex` silently picks up unreleased source and your results
+   become fiction. Run scripts from a neutral directory and start each repro with an assertion
+   naming the venv whose python is running it — the shared one below, or your own:
+   ```python
+   import reflex
+
+   assert "<VENV_PATH>" in reflex.__file__, reflex.__file__
+   ```
+   A guard that names some other venv fails on every run and gets deleted, which leaves you with
+   no guard at all.
+3. Reading the checkout is fine and encouraged — release source is on branch `<PRERELEASE_BRANCH>`.
+   For PR context load the GitHub MCP tools via ToolSearch (`select:mcp__github__pull_request_read`).
+4. Do NOT run any `git` write commands (add/commit/checkout/...) in the checkout. The orchestrator
+   commits artifacts.
+5. Do NOT fix bugs you find — record precise repro steps instead.
+6. Kill every server and browser you start before you finish (track PIDs; verify with `ps`). Other
+   agents share this machine — run at most ONE dev server at a time unless your cluster needs two.
+
+### Environment
+
+- Scratchpad root: `SB=<SCRATCHPAD>`
+- **Prebuilt shared venv (READ-ONLY — never install into it):** `$SB/envs/<SHARED_VENV>` has
+  `reflex==<VERSION>` and the alpha sub-packages. Use `$SB/envs/<SHARED_VENV>/bin/reflex` directly.
+- Need other deps or versions? Make your OWN venv:
+  ```
+  uv venv $SB/envs/<yours> --python 3.11
+  uv pip install --python $SB/envs/<yours>/bin/python --prerelease=allow 'reflex==<VERSION>' <extras>
+  ```
+  Always pass `--prerelease=allow` for alphas; PyPI is the default index — never point it at the
+  checkout. Python 3.12/3.13/3.14 are available via `uv venv --python 3.14` etc.
+- **Playwright driver venv:** `$SB/envs/driver/bin/python` (playwright, httpx, websockets).
+  Chromium: `/opt/pw-browsers/chromium` — launch with
+  `p.chromium.launch(executable_path="/opt/pw-browsers/chromium")`.
+  A ready-made driver with console/network capture is at
+  `.claude/skills/prerelease-test/scripts/drive_app.py` in the checkout (read-only use is fine).
+- **Local HTTP needs proxy bypass on the CLIENT side only:** prefix curl/Playwright commands with
+  `NO_PROXY=localhost,127.0.0.1 no_proxy=localhost,127.0.0.1` (curl: `--noproxy '*'`).
+  Do NOT export those variables into the reflex server's environment — it breaks bun's package
+  installs through the proxy, which looks like a framework bug and is not.
+- Set `REFLEX_TELEMETRY_ENABLED=false` for every reflex command.
+- Node and bun are preinstalled. Enterprise apps need `CI=true` to bypass the dev login gate.
+- App working dirs: `$SB/apps/<cluster>/...`. Run servers on YOUR ASSIGNED PORTS:
+  `reflex run --frontend-port <FP> --backend-port <BP>`. The first run does a bun install (1–2 min);
+  poll the frontend URL until it returns 200 for up to ~6 minutes before concluding failure.
+- `--loglevel debug` gives verbose server logs; redirect to a file and actually read it.
+
+### What "testing" means here
+
+- Real-world exploration, not coverage filling. Combine the feature with State vars,
+  `rx._x.client_state`, `@rx.memo` wrapping, `rx.ComponentState`, `rx.foreach`/`rx.cond`, event
+  chains (`yield Other.handler()`), background tasks, multiple pages and navigation — whatever
+  plausibly interacts. The original PR already tested the happy path.
+- Drive the app in Chromium as a user would: click, type, navigate, upload, drag, use the keyboard.
+- On EVERY run capture and inspect: (a) the server log file, (b) browser console messages (errors
+  AND warnings), (c) failed requests / 4xx-5xx responses, (d) screenshots at key moments.
+- Baseline comparisons are what make findings actionable: if behavior looks wrong, check the
+  previous stable (`uv pip install 'reflex==<PREV_STABLE>'` in its own venv). Only-on-new is a
+  regression and high severity; both-versions is context worth noting, not a blocker.
+- Prod matters too: `reflex run --env prod` compiles and serves the built frontend. Test both modes
+  when your feature could differ (hydration, memoization, routing, prerender).
+
+### Known-benign noise — do not report these as findings
+
+- Browser console: the React Router "💿 Hey developer" HydrateFallback log, vite
+  `connecting.../connected` debug lines, the React DevTools download info line.
+- `reflex init` logging a few "Failed to connect to https://registry.npmmirror.com" lines before
+  falling back (this environment blocks that mirror). Do report it if the fallback itself fails.
+- Transient bun "incorrect peer dependency" warnings during a one-time upgrade migration, provided
+  the final lockfile is consistent.
+
+If you see something surprising that is not on this list, investigate it — several real findings
+have surfaced first as an unexplained warning.
+
+### Deliverables (mandatory)
+
+1. Copy reusable artifacts into the repo (plain `cp`/`rsync`, no git):
+   `DEST=<ARTIFACT_ROOT>/<cluster>/`
+   - the app source dirs, EXCLUDING `.web/`, `node_modules/`, `.states/`, `assets/external/`,
+     `*.db`, venvs
+   - your Playwright/repro scripts
+   - `NOTES.md`: what you tested, how to rerun it (exact commands), what you observed, including
+     benign quirks
+2. Return structured findings: every discrete check as a test entry (pass/fail/anomaly/skipped)
+   with enough repro detail that another agent can reproduce a failure from `NOTES.md` and your
+   scripts alone, without your conversation. An "anomaly" is anything surprising (console error,
+   warning, traceback, visual glitch, perf cliff) even when functionality works.
+
+### Timeboxing
+
+Be thorough but keep moving: if one sub-test resists debugging for ~10 minutes, record it as an
+anomaly with logs attached and continue. Finishing the whole cluster beats perfecting one test.