Add prerelease-test skill for release QA campaigns

[masenf]

All Submissions:

• Have you followed the guidelines stated in CONTRIBUTING.md file?
• Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

• New feature (non-breaking change which adds functionality)

Tooling only — no framework code changes, nothing shipped in any package.

New Feature Submission:

• Does your submission pass the tests?
• Have you linted your code locally prior to submission?

Description

Captures the pre-release QA campaign run against the 0.9.9 train as a repeatable skill, so the same analysis can be kicked off before any release instead of being reconstructed from scratch each time. That campaign found 29 verified issues, 11 of which were fixed before 0.9.9 shipped (#6971, #6984, #6985, #6986, #6987, #6989, #6990, #6991, #6992, #6994) with the rest filed as #6972–#6983.

.claude/skills/prerelease-test/SKILL.md lays out the phases: discover what shipped from the pre-release branch changelogs and confirm every package published to PyPI, de-risk with a smoke test, fan out feature clusters that build real apps and drive them in a browser, upgrade-test reflex-examples apps from the previous stable, regression-test reflex-enterprise demos, audit wheel/sdist packaging, then triage into fix-before-release vs file-as-issue.

Three habits are what made the campaign's output actionable, so they are written in explicitly rather than left to chance:

• Install only published packages. The checkout has unreleased code and a different dependency graph, so testing it says nothing about what users get — and a package that failed to publish is itself a blocker you would otherwise miss.
• Baseline against the previous stable. It is the only way to tell a regression from a pre-existing bug, which is the distinction the release decision turns on.
• Verify adversarially before reporting. An independent agent reproduces each claimed issue from the written repro alone and tries to refute it. On the 0.9.9 run this refuted or reclassified a meaningful share of claims, and it doubles as a test that the repro is good enough to hand to a fix agent.

references/ holds the brief given to each test agent (hard rules, environment traps like checkout shadowing of import reflex, and the known-benign console noise list), cluster decomposition guidance, the Workflow explore/verify pipeline with schemas and a port map, and report templates.

scripts/ bundles three helpers the test agents kept rewriting independently:

Script	Purpose	Validated against 0.9.9
check_release_versions.py	changelog versions → PyPI publication check	16 packages, all published
audit_pyi.py	stubs present in wheel and sdist, byte-identical, no foreign stubs, counts vs pyi_hashes.json	121 stubs, clean
drive_app.py	Chromium driver: console/network capture, small JSON action language, benign-noise filtering, exit 0/1	clean and failure paths both exercised

Note on the pyproject change

The one non-skill change is two lines in pyproject.toml. Ruff selects ALL with no .claude exclude, so these scripts are linted by CI. Everything ruff flagged is fixed in the scripts themselves (Args/Returns docstring sections, annotations, EM102, and an action-loop restructure that avoids both PERF401 and PERF203 while still preserving the record of actions that succeeded before a failure). The exception is T201 — printing a report is these scripts' entire purpose — which is whitelisted exactly the way scripts/check_min_deps.py already is.

Test Plan

• uvx ruff@0.16.3 check .claude/skills/prerelease-test/scripts/ and ruff format --check — both clean, using the version locked in uv.lock.
• All three scripts run end-to-end against the released 0.9.9 train, with the results in the table above; drive_app.py was additionally checked for its failure path (a failing action reports the error, keeps the already-successful actions, skips the rest, exits 1) and its noise filter (--all-console shows the lines it normally suppresses).
• No framework code is touched, so the unit suite is unaffected by this diff.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EMjBXPozsNeQNSBZecNH8x

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will not alter performance

✅ 32 untouched benchmarks
⏩ 8 skipped benchmarks¹

---

_{Comparing claude/prerelease-test-skill (3b965b9) with main (dd96aea)}

Footnotes

1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[cubic-dev-ai]

8 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:192">
P2: When the initial page cannot connect or times out, `page.goto` raises before the driver writes `--report` or prints the captured failure details. Catch startup navigation errors, record them in the report, close the browser, and return the documented failure status.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:70">
P2: When a `never-publish-packages` entry also has a `CHANGELOG.md`, `changelog_paths` includes it and `main` reports its missing PyPI version as a release blocker. Read the release configuration and exclude never-published packages before scanning changelogs.

(Based on your team's feedback about excluding never-published packages from release paths.)</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:123">
P2: When PyPI is temporarily unreachable, `pypi_status` returns `published=False`, so `main` counts the package as missing and reports it as unpublished. Track verification errors separately, exclude them from `missing`, and report an indeterminate check without a false publication blocker.</violation>
</file>

<file name=".claude/skills/prerelease-test/references/orchestration.md">

<violation number="1" location=".claude/skills/prerelease-test/references/orchestration.md:87">
P2: When a verify agent dies and its `agent(...)` call resolves to a falsy value, the `.then((v) => ({ issue: iss, verdict: v }))` wrapper turns it into a truthy `{ issue, verdict: null }` entry, so `.filter(Boolean)` keeps it. The explore branch in the same pipeline explicitly guards `if (!res) return { .., agent_died: true }`, but verification has no equivalent guard, so a dead verifier leaves an issue with `verdict: null` in `verified_issues`. Later report parsing (VERDICT_SCHEMA requires `confirmed` and `notes`) will then crash or treat an unverified issue as real in the report. Return `null` from the `.then` when `v` is falsy so the filter drops it.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:156">
P2: When PyPI lacks either artifact, `audit()` omits `expected`, so `main()` raises `KeyError` instead of reporting the missing artifact and aborts the remaining package audits. Include `expected` in this early-return record.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:201">
P2: When a package has no manifest entry, `want` is `None` and the audit accepts any number of own stubs instead of enforcing the documented zero-stub expectation. Track whether a manifest was loaded and treat an absent package entry as an expected count of zero.</violation>
</file>

<file name=".claude/skills/prerelease-test/SKILL.md">

<violation number="1" location=".claude/skills/prerelease-test/SKILL.md:51">
P2: The documented invocation `scripts/check_release_versions.py <git-ref>` is wrong: the script requires the git ref via the `--ref` flag (argparse: `parser.add_argument("--ref", required=True)`). Running the documented form exits with `error: the following arguments are required: --ref` and extracts nothing. Change it to `scripts/check_release_versions.py --ref <git-ref>` so agents following Phase 0 don't hit the argparse error.</violation>

<violation number="2" location=".claude/skills/prerelease-test/SKILL.md:62">
P2: The `scripts/...` paths (check_release_versions.py, drive_app.py, audit_pyi.py) resolve only when run from the skill directory, but the skill's agents operate from the repo root (it says to commit under `prerelease-testing/...` in the repo). These three scripts were added under `.claude/skills/prerelease-test/scripts/`, not repo-root `scripts/` (repo-root `scripts/` has no such files). An agent following the skill from the repo root gets 'command not found'. Use the skill-relative path (e.g. `.claude/skills/prerelease-test/scripts/...`) or state the CWD explicitly.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[greptile-apps]

Greptile Summary

The PR adds a reusable pre-release QA skill covering release discovery, published-package smoke testing, browser-driven feature validation, upgrade and enterprise regression testing, packaging audits, and adversarial finding verification.

• Adds campaign orchestration, agent guidance, cluster decomposition, and reporting templates.
• Adds helpers for checking published versions, auditing wheel and sdist stubs, and driving Chromium-based application tests.
• Configures Ruff to permit report-oriented printing in the new helper scripts.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.claude/skills/prerelease-test/SKILL.md	Defines the complete release-QA workflow and now correctly gates packaging audits on successful release discovery.
.claude/skills/prerelease-test/scripts/check_release_versions.py	Discovers release versions and distinguishes confirmed missing publications from indeterminate PyPI lookup failures.
.claude/skills/prerelease-test/scripts/audit_pyi.py	Audits stub packaging while consistently reporting missing artifacts and separating infrastructure failures from packaging defects.
.claude/skills/prerelease-test/scripts/drive_app.py	Provides browser-driven smoke-test actions with console and network capture plus early action validation.
pyproject.toml	Adds a narrowly scoped Ruff allowance for print-based reporting in the QA helper scripts.

_{Reviews (12): Last reviewed commit: "Make the documented commands runnable an..." | Re-trigger Greptile}

[cubic-dev-ai]

8 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/SKILL.md">

<violation number="1" location=".claude/skills/prerelease-test/SKILL.md:51">
P2: The documented invocation `scripts/check_release_versions.py <git-ref>` fails because the script requires a `--ref <git-ref>` flag (argparse `--ref`, required=True). Document the correct form so a user following Phase 0 doesn't hit an argparse error.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:70">
P2: When a never-published package has a stray `CHANGELOG.md`, `changelog_paths` treats it as part of the release train and can fail QA because that package is not on PyPI. Read the release configuration at `ref` and exclude `never-publish-packages` before checking versions.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:167">
P2: When PyPI returns a non-404 or the lookup fails, `pypi_status()` still sets `published` false, so this list reports a transient outage as an unpublished release blocker. Distinguish confirmed 404s from lookup failures before populating `missing`.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:109">
P2: When auditing packages whose import root differs from their distribution name, this heuristic rejects the package’s own stubs as foreign. Read the package’s actual top-level roots or provide an explicit distribution-to-import mapping instead of assuming hyphens are the only difference.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:156">
P2: When a published version has only one artifact kind, this branch omits `expected`, so `main()` crashes with `KeyError` while printing the result. Include the expected-manifest value in this return record so missing artifacts produce the intended audit failure summary.</violation>

<violation number="3" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:201">
P2: When a package has no entry in a loaded manifest, this check treats it as unchecked instead of expecting zero stubs. Distinguish an unavailable manifest from a loaded manifest, then flag own stubs for packages omitted from the loaded manifest.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:76">
P2: Inline action JSON longer than a filesystem filename makes `candidate.exists()` raise `OSError` before parsing, so documented inline scripts fail instead of running. Catch path-stat errors and parse `raw` as JSON.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/drive_app.py:202">
P2: After the final `click`, `fill`, or `press`, Playwright returns before Reflex's asynchronous event and network work finishes. Reading the report and closing the browser immediately can miss late page errors or failed requests; wait for settlement after the action loop.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Both bot review rounds are addressed — all twelve distinct findings were valid, and each fix was verified by reproducing the reported failure first. Pushed as 0487b40 and e59fdbd.

Round one

Finding	Fix	Verified
pypi_status reports a network failure as unpublished	tri-state published/missing/error; exit 1 for a confirmed gap, 2 for indeterminate	simulated proxy failure → ('error', ...); 404 → ('missing', ...)
never-published packages counted as blockers	read never-publish-packages from the release config and exclude them	resolves {'integrations-docs'} from origin/main
audit_pyi KeyError on a missing artifact	early return carries expected	simulated wheel-only package → clean "missing artifact kinds" report, exit 1 (was KeyError)
manifest-absent package unchecked	with a manifest loaded, absent means zero stubs	reflex-base now shows stubs=0 (manifest: 0); it was unchecked before
dead verifier leaves verdict: null	applied the suggested guard	—
--ref missing from the documented invocation, and bare scripts/... collides with the repo's own scripts/	corrected to full repo-root-relative paths	—

Round two

Finding	Fix	Verified
inline action JSON longer than the filename limit raises OSError before parsing	catch path errors, treat as JSON	Path('x'*1109).exists() raises File name too long; a 1109-byte payload now runs clean
report read before the last action's fallout lands	settle after the action loop	—
import root from a dist-name heuristic	take it from the stub manifest, falling back to the heuristic	13 roots resolved from the manifest
bare python3 shebang bypasses uv	PEP 723 metadata, no shebang, uv run --script — matching scripts/check_min_deps.py and CLAUDE.md	uv run --script runs both stdlib scripts and the audit pipe

On the import-root one specifically: deriving the root from the artifact's own top-level directories would be circular — foreign stubs would define themselves as native and the leak check could never fail — so the manifest is the right source of truth, with the heuristic left as the fallback for a package the manifest doesn't know.

End-to-end re-check after both rounds: discovery finds 16 published packages, the documented Phase 5 pipe audits 121 stubs clean, and the driver passes its clean, long-inline-JSON, action-failure and dead-server paths. ruff check/format clean, and the repo's full local pre-commit suite passes.

---

Generated by Claude Code

[cubic-dev-ai]

4 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:95">
P1: When a tracked `CHANGELOG.md` has no matching version heading, `main()` silently omits that package and can return 0, so the QA campaign misses its publication check. Fail the run or record an error for the changelog instead of continuing.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:141">
P2: When all PyPI files for a version are yanked, this code still reports `published`, so the release check can call a version installable even though normal resolution ignores it. Filter out yanked URLs and treat an all-yanked release as unavailable.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:45">
P2: When a requested release is missing from PyPI or the metadata request fails, `download_artifacts` aborts the whole campaign with a traceback. Catch download failures per spec, record them as problems, and continue to return a controlled nonzero result.</violation>
</file>

<file name=".claude/skills/prerelease-test/references/clusters.md">

<violation number="1" location=".claude/skills/prerelease-test/references/clusters.md:51">
P3: The `scripts/audit_pyi.py` path here is wrong: the repo-root `scripts/` directory (which `SKILL.md` explicitly warns is an unrelated directory) contains no `audit_pyi.py`. The script lives at `.claude/skills/prerelease-test/scripts/audit_pyi.py`, and `SKILL.md` states that skill script paths are repo-root-relative with the `.claude/skills/prerelease-test/` prefix required. An agent following this packaging-cluster brief will get a "No such file or directory" failure. Reference the full path as phase 5 of `SKILL.md` does.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

3 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:98">
P1: When a tracked changelog has no versioned section, `main` silently omits it and can report a clean release with no packages. Treat a missing version heading as an error and return non-zero instead of continuing.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:212">
P2: When `--actions` contains invalid JSON or `null`, `load_actions` raises before `try_action` runs, so the CLI exits with a traceback and never writes the requested report. Parse and validate the action list inside the reporting path, then record the parse error as `action_error`.</violation>
</file>

<file name=".claude/skills/prerelease-test/references/orchestration.md">

<violation number="1" location=".claude/skills/prerelease-test/references/orchestration.md:95">
P3: The pipeline example claims `results.filter(Boolean)` drops dead explorers, but the explorer-died branch returns a truthy marker object (`{ cluster: c.key, agent_died: true }`), so `filter(Boolean)` keeps every one of them. Anyone building this workflow from the doc will expect the final `results` to contain only clusters with genuine (schema-conforming) findings, yet dead clusters survive as `agent_died` objects that do not satisfy FINDINGS_SCHEMA (missing `summary`/`tests`/`issues`/`artifacts_dir`). Either drop died explorers explicitly (e.g. `results.filter(r => !r.agent_died)`) or correct the comment to state dead explorers are kept as markers the orchestrator must handle.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Rounds 3 and 4 addressed — five findings, all valid.

check_release_versions.py

• Indeterminate specs reached the artifact audit (greptile P1). --specs now emits only status == "published" rows. It feeds a downloader, so passing along a package whose PyPI check timed out just moved a controlled indeterminate result downstream and turned it into a traceback. Skipped rows go to stderr, so a short list is never silently mistaken for a complete one; the exit code is unchanged.
• A changelog with no version heading was dropped silently (P1). The package whose publication went unverified was also the one the report never mentioned, which is exactly the failure this script exists to catch. It is now recorded as an error row — printed in the table, excluded from --specs, exit 2.
• Fully yanked releases reported as published (P2). Yanked files are filtered out; an all-yanked version is missing ("published but all files yanked"), and a partial yank shows in the detail column. Also narrowed the try to just the network call so JSON-shape logic can't be swallowed by the broad except.

audit_pyi.py

• download_artifacts aborted the whole audit (P2, and the second half of the greptile P1). Wrapped per package, so an unreachable one fails on its own row and the rest still get audited.

references/clusters.md

• Bare scripts/audit_pyi.py path (P3). Now the repo-root-relative path, matching phase 5 of SKILL.md.

Verification

• Stubbed pypi_status to cover published / missing / indeterminate and confirmed --specs emits only reflex==0.9.9, reports the other two on stderr, and still exits 1.
• Table-tested pypi_status across five PyPI payload shapes: normal, no files, all-yanked, partially-yanked, and yanked key absent.
• Two-changelog run with one missing heading: package appears in the table, exits 2, and --specs skips it.
• audit_pyi.py reflex-base==0.9.9 reflex-does-not-exist-xyz==1.0.0 → reflex-base audits OK, the bogus one reports could not fetch artifacts: HTTPError, exit 1. No traceback.
• Real run against origin/main: all 16 packages OK, exit 0 — no regression.
• ruff format and ruff check clean.

---

Generated by Claude Code

[cubic-dev-ai]

3 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:87">
P2: When `--actions` contains malformed JSON or an unreadable path, `load_actions` raises before the normal failure handling, so the CLI prints a traceback and never writes `--report`. Catch action-script parsing errors in `main`, record them as `action_error`, and continue through the normal report and exit-1 path.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:59">
P2: When a PyPI artifact transfer stalls, `urlretrieve` has no timeout, so the release audit can hang indefinitely instead of recording a fetch failure. Download through `urlopen(..., timeout=60)` or otherwise apply a bounded read timeout.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:229">
P2: When a published wheel or sdist is corrupt, the archive reader raises outside the per-package error handling, aborting the audit and preventing later packages from being checked. Catch archive-read errors here and record them as package problems, as fetch errors already are.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Round 5 — three new findings, all valid, plus one CI note.

SKILL.md phase 5 — "pipeline masks discovery failures" (greptile P1). Correct, and a direct consequence of the round-3 change: once --specs filters out an unchecked package, a shell pipe hands back only the audit's exit status, so the audit prints PASS over a short list and the discovery failure vanishes. Documented as two commands with the spec list in a file, so the discovery exit status is observable on its own.

drive_app.py — malformed --actions (cubic P2). Worse than the report being skipped: load_actions was called inside the browser block, so a typo in the inline JSON cost a Chromium launch, a page load and a settle before the traceback. Now parsed immediately after parse_args, and a JSON value that is not a list says so by name. I've deviated slightly from the suggestion — a malformed CLI argument is a usage error about a page that was never driven, so it goes to stderr with the documented exit 1 rather than into a report describing a run that did not happen.

references/orchestration.md — results.filter(Boolean) (cubic P3). Real mismatch between the code and what a reader would assume. Fixed by the second of the two options offered: keeping the agent_died markers is deliberate — a cluster that produced nothing is a hole in the campaign, and silently dropping it is exactly how a hole gets mistaken for a clean result — so the example now says that instead of implying they are filtered.

The remaining P1 in that round (check_release_versions.py:98, missing version heading) was already fixed in 1f71177; it now records an error row and exits 2.

Verification

• --actions '[{' → INVALID --actions: Expecting property name enclosed in double quotes: line 1 column 3, exit 1, no browser launched. --actions 'null' → expected a JSON list of actions, got NoneType, exit 1.
• Happy path re-checked against a local page: both actions ran, action_error: None, state updated — the pre-parsed list did not change behavior.
• Two-step phase 5 form run for real: discovery writes 16 specs and exits 0 on its own; xargs … --manifest-ref origin/main < specs.txt passes the specs in the right position and audits cleanly.
• ruff format and ruff check clean.

CI: unit-tests (windows-latest, 3.13) failed on 1f71177 in astral-sh/setup-uv — Fetching manifest data from raw.githubusercontent.com … ##[error]fetch failed, with Setup Node and Install Dependencies both skipped. No test body ran, and this PR touches only .claude/skills/ and one ruff per-file-ignore. The pushes since then have re-run it; I'm watching for the result rather than spending a manual re-run.

---

Generated by Claude Code

[cubic-dev-ai]

5 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:100">
P2: When an `internal-packages` entry also has `CHANGELOG.md`, this code treats its ignored changelog heading as the release version and can report a false PyPI blocker. Exclude internal packages from this discovery path as well, matching `reflex-release`'s changelog rules.</violation>
</file>

<file name=".claude/skills/prerelease-test/references/orchestration.md">

<violation number="1" location=".claude/skills/prerelease-test/references/orchestration.md:98">
P3: The comment "results.filter(Boolean) drops dead explorers" is wrong: a dead explorer is already replaced with the truthy placeholder `{ cluster: c.key, agent_died: true }` by `if (!res) return ...`, so `filter(Boolean)` only removes falsy values and the dead-explorer entries survive into the returned results. The placeholder also omits the FINDINGS_SCHEMA-required `tests`, `issues`, and `artifacts_dir`, so downstream report parsing sees an entry shaped like a valid cluster. Either return a falsy value (e.g. `return null`) for dead explorers so the filter drops them, or drop `agent_died` entries explicitly and update the comment.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:53">
P2: When a version has both live and yanked wheel or sdist files, this loop can select the yanked file for auditing. Skip yanked entries before selecting artifacts.</violation>

<violation number="2" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:59">
P2: When a PyPI artifact download stalls, `urlretrieve` calls `urlopen` without a timeout, so this campaign can hang indefinitely despite the 60-second metadata timeout. Download through `urlopen(..., timeout=60)` instead.</violation>

<violation number="3" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:229">
P2: When PyPI serves a corrupt archive, `wheel_stubs` or `sdist_stubs` aborts the audit outside its per-package error handler. Catch archive-read failures around both calls and record the package problem.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Round 6 — two more in audit_pyi.py, both valid, both the same shape as the fetch bug from round 3.

urlretrieve has no timeout (P2). It honours only the global socket timeout, which is unset, so a stalled PyPI transfer would hang the audit indefinitely — worse than the traceback it replaced, because nothing reports it. Downloads now stream through urlopen(..., timeout=60) into the file, which bounds each read and keeps memory flat regardless of artifact size.

A corrupt wheel or sdist aborted the run (P2). wheel_stubs/sdist_stubs raised outside the per-package handling, so one bad archive took the rest of the train with it. Caught per package and recorded as could not read artifacts: <type>.

While fixing these I collapsed the four exits from audit() into a single _record() builder. Three of the six review findings against this file have been "an early return omits a key main() reads" or "an exception escapes the per-package record" — one construction point removes the whole category, and it also settles an inconsistency I'd left behind, where the early returns used expected.get(name) while the success path used expected.get(name, 0).

Verification

• Planted a corrupt wheel in the --keep directory so the download short-circuits: reflex-base reports could not read artifacts: BadZipFile, reflex-components-sonner behind it still audits OK at 1 stub, exit 1. No abort.
• Full 16-package audit with the streaming downloader: 121 stubs, every package matching its manifest count, exit 0 — byte-for-byte the same result as before the change.
• ruff format and ruff check clean.

The three round-5 threads and the greptile SKILL.md P1 are addressed in the previous push; resolving all of them now.

---

Generated by Claude Code

[masenf]

Round 7 — five raised, two new and valid, three already fixed in the previous push.

Internal packages not excluded from discovery (P2, new). Valid, and the same reasoning that made the never-publish-packages exclusion right in round 1. [tool.reflex-release] says internal packages are "patch-released on every push to main that touches them" with "no changelog, no news fragments" — so a changelog heading in one names a version that was never cut, and checking it against PyPI produces a false blocker. Neither internal package has a CHANGELOG.md on main today, so this is a latent trap rather than a live bug, but it fires the moment someone adds one. Both config lists are now read through one EXCLUDED_KEYS loop with the two distinct reasons written down.

Yanked artifacts could be audited (P2, new). Valid: download_artifacts took the last file of each kind, so a version with both a yanked and a live wheel could be audited on the file nobody installs. Yanked entries are skipped. If every file is yanked nothing is selected, which the existing missing artifact kinds branch already reports — and check_release_versions.py flags that version as a blocker before the audit ever runs.

urlretrieve timeout and corrupt-archive handling (P2 ×2) — both fixed in eeb4745/43be3ed, one push ahead of this review.

orchestration.md (P3) — the quoted comment, "results.filter(Boolean) drops dead explorers", isn't in the file; the current text says the opposite, that dead explorers are deliberately kept as agent_died markers because a cluster that produced nothing is a hole in the campaign, and it tells the reader to handle them when writing the report. That is the deliberate choice between the two options the round-5 comment offered, and returning null instead would delete exactly the signal worth keeping.

Verification

• excluded_packages on a synthetic pyproject returns all three names across both keys; with neither key present it returns an empty set (so a config without them excludes nothing rather than everything).
• Against the real origin/main config: skips integrations-docs, reflex-components-internal, reflex-site-shared, and changelog_paths still yields the same 16 changelogs, none under a skipped package.
• download_artifacts with a yanked wheel, a live wheel and a live sdist selects live.whl and never fetches the yanked URL.
• Real discovery against origin/main: 16 packages, all published, exit 0. Round-4 tests still pass.
• ruff format and ruff check clean.

---

Generated by Claude Code

[masenf]

One more, and a good one: "download failures become packaging defects" (greptile P1). Valid — audit_pyi.py was recording an unreachable PyPI as a problem, so infrastructure trouble would fail a healthy release. That is exactly the missing-versus-unknown split check_release_versions.py already makes, and I put it in one script but not the other.

audit() records now carry unchecked separately from problems. Exit 1 means a defect observed in artifacts that were actually read; exit 2 means a package could not be audited at all; 0 is clean. An unreadable archive counts as unchecked rather than a defect, because a truncated download and an artifact PyPI genuinely serves broken are indistinguishable from here — a re-run tells them apart, and calling a healthy release broken is the worse of the two mistakes.

Verification of all three exit paths

• Unreachable package next to a healthy one → ?? row, INDETERMINATE: 1 package(s) could not be audited, exit 2; the healthy package still audits.
• Corrupt wheel planted in the --keep directory → same treatment, exit 2.
• Planted a genuine defect (wheel carrying a reflex_components_radix stub, an own stub present in the sdist but not the wheel, and a count that misses the manifest) → all three problems reported, FAIL, exit 1, with the package behind it still auditing.
• Full 16-package run against the real train → 121 stubs, exit 0.

ruff format and ruff check clean. Resolving the remaining threads: the internal-packages and yanked-artifact findings are fixed in e2854df, the urlretrieve timeout and corrupt-archive ones in eeb4745, and the orchestration.md P3 is answered in my previous comment — the comment it quotes isn't in the file, and keeping the agent_died markers is the deliberate choice between the two options that thread offered.

---

Generated by Claude Code

[cubic-dev-ai]

2 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:235">
P2: When the first or only action fails after triggering page work, the script skips the settle wait and can miss late anomalies. Wait whenever an action was attempted, including failed actions.</violation>
</file>

<file name=".claude/skills/prerelease-test/references/orchestration.md">

<violation number="1" location=".claude/skills/prerelease-test/references/orchestration.md:89">
P2: Dead verifiers are silently filtered out of `verified_issues`, while dead explorers are deliberately kept as `agent_died: true` markers - and this doc's own rationale for the explorer policy applies equally to verifiers. A died verify agent leaves its issue absent from `verified_issues` but still present in the returned full `res.issues`, so downstream it is indistinguishable from an issue excluded by the `slice(0, 4)` fan-out cap or one the verifier never reached. A report writer (whom the doc tells to handle dead explorers explicitly but says nothing about verifiers) can therefore mistake a verification hole for a clean result - the exact silent-drop-looks-clean failure this document warns against for explorers. Record died verifiers as an explicit marker instead of filtering them to `null`.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Round 9 — two findings, both valid, and the second one catches a real inconsistency in my own reasoning.

drive_app.py skipped the settle after a failed action (P2). Valid, and it costs exactly the evidence you want: a click can reach the server and then fail its assertion, so the console error and failed request it caused land after the failure. With nothing recorded as performed, the wait was skipped and that fallout never reached the report. Now settles whenever an action was attempted.

Dead verifiers were silently dropped while dead explorers are kept (P2). This one is a fair hit. In round 1 the guidance was to drop a dead verifier rather than record verdict: null, and I did — then in round 5 I argued the opposite for explorers, that silently dropping a hole is how it gets mistaken for a clean result. Both are right about what's wrong with the other option, and the resolution is the third one neither round proposed: a dead verifier gets its own verifier_died marker. verdict: null would read as a verdict that cleared the issue; dropping the entry left a hole indistinguishable from an issue the slice(0, 4) cap never sent to a verifier. With the marker, all three states are distinguishable, and I noted on the cap what the res.issues/verified_issues difference means so the report writer can tell capped from verified. verdicts no longer needs a filter(Boolean) — nothing in it is falsy now.

Verification — A/B on the settle change, using a page whose only action fails while still doing work (a goto that navigates but times out on networkidle because of an open fetch, with a console.error fired at 1500 ms):

• if performed: → ACTION FAILED: TimeoutError, and no sign of the late error.
• if performed or action_error: → same failure, plus [error] LATE ERROR captured.

ruff format and ruff check clean. CI on 7cb79ee had all 20 build jobs green with the rest still running; the Windows 3.13 job that died in setup-uv last round is running again on the new head.

---

Generated by Claude Code

[masenf]

Round 10 — four raised, two new and valid, one already fixed, one I'm declining.

The agent preamble overpromised ports (P2, new). A genuine inconsistency in my own doc, and the kind that only bites under load: the shared preamble handed every agent <FP>-<FP+19>, but the port map gives a verify agent just 4 ports (3600 + 40i + 4j … +3). A verifier following the bootstrap verbatim would believe it owned 20 and bind into the next verifier's range — precisely the collision the map exists to prevent, and one that would surface as unreproducible cross-talk between agents. The preamble now takes its ends from the table, with a note that explore ranges are 20 wide and verify ranges 4.

The shadowing guard didn't say which venv (P3, new). Fair: assert "<VENV_PATH>" in reflex.__file__ is the single most important instruction in the brief, and it was the one placeholder with three plausible fills (shared venv, the agent's own, the driver). Now says to name the venv whose python is running the repro, plus why it matters — a guard that names the wrong venv fails on every run and gets deleted, which leaves no guard at all.

Dead verifier marker (P2) — fixed in ef6815a, one push ahead of this review. Two threads in this round carry no body at all (just "P2:"); they sit on the verifier and port lines, so I've treated them as duplicates of the bodied findings and resolved them with those.

Declining: root package in never-publish-packages (P2). The scenario is never-publish-packages containing the root package, so the root CHANGELOG.md gets a PyPI check it shouldn't. I don't think this is reachable in a coherent config. never-publish-packages lists directory names under packages/, and the root package is named separately by root-package; putting it in that list would declare that this repo never publishes the framework it exists to publish, and the release pipeline itself would refuse long before my QA script mattered. Guarding it properly would also mean parsing root-package and threading it through, which is real complexity for an impossible input. I'd rather the script stay honest about what it assumes. Happy to add it if you disagree — it's a small change once root-package is parsed.

This looks like the learned rule from round 1 (exclude never-published packages from release paths) being applied one step past where it holds; the round-7 internal-packages finding from the same family was correct and is fixed.

---

Generated by Claude Code

[cubic-dev-ai]

1 existing issue remains and 3 new issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:159">
P2: When PyPI or a proxy returns HTTP 200 JSON without `urls`, `pypi_status()` treats the response as a confirmed missing release and blocks QA. Validate the response shape and return `error` for an indeterminate response instead.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:271">
P2: When a truncated gzip sdist raises `EOFError`, this handler misses it and the audit aborts instead of reporting the package as indeterminate. Catch `EOFError` here so interrupted downloads follow the documented exit-2 path.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:222">
P2: If an action closes the page, this post-action settle call raises outside `try_action` and aborts before the report and `RESULT` line are emitted. Catch the settle failure and guard subsequent page operations so closed-page actions become a recorded anomaly.</violation>
</file>

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic</sub>

[masenf]

Round 11 — three findings, all valid, all fixed in 0e1c17d.

A 200 that isn't PyPI's shape read as a missing release. data.get("urls", []) collapsed "PyPI says this version has no files" and "something else answered with JSON" into the same confirmed-missing verdict. Only urls present and a list means the former; anything else is now unexpected PyPI response shape and indeterminate. Relevant here, where an agent proxy sits in front of every request.

A truncated gzip sdist aborted the audit. Verified before fixing — tarfile.open on a truncated .tar.gz raises EOFError, which subclasses neither OSError nor TarError, so it slipped past the handler written for exactly this case. Added to the tuple.

A crashed or closed page killed the run before the report. The best of the three. The post-action settle and the screenshot both sat outside every guard, so a page an action crashes took down the whole invocation — losing the report, which is the one thing the caller wanted. Both guarded now, with a screenshot_error field so a lost screenshot is recorded rather than silent.

Verification

• pypi_status across six payloads: {"urls":[…]} → published; {"urls":[]} → missing; {}, {"urls":"nope"}, a bare list, and a bare string → all indeterminate. The real "no files" case is still reported as missing.
• Truncated sdist built by cutting a real .tar.gz in half → unchecked: could not read artifacts: EOFError, no problems, so it takes the exit-2 path rather than being called defective.
• Crashed page, A/B with {"goto": "chrome://crash"}:
  • unguarded → playwright._impl._errors.Error: Page.screenshot: Target crashed traceback, no report file written;
  • guarded → ACTION FAILED and SCREENSHOT FAILED both printed, report written, exit 1.

ruff format and ruff check clean.

On the process: that's eleven rounds and 27 findings, 26 of them valid — a good hit rate, and these three were real bugs rather than polish. But each push draws the next review, so the loop is now self-sustaining and the returns are narrowing to error paths I keep adding guards to. I'll stay responsive to anything substantive; if you'd rather freeze it here for human review, say the word and I'll stand down. The one thread I've left open is the round-10 never-publish-packages root-package claim, which I declined and explained above — that one's a judgement call for you.

---

Generated by Claude Code

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:68">
P2: When a download fails after writing part of an artifact and `--keep` is used, this leaves the partial file at the final path, so the next run skips downloading it and remains stuck on the truncated artifact. Download to a temporary `.part` path and atomically rename it only after the copy succeeds, removing the temporary file on failure.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Round 12 — one finding, valid, fixed in 778ac15.

A partial download stuck at the final path. A transfer that died partway wrote its bytes straight to the destination, and --keep skips whatever already exists, so every later run reused the truncated file. What makes it worth fixing rather than shrugging at: the EOFError handling I added in the previous commit reports a bad artifact as unchecked and tells you to re-run to tell a bad download from a genuinely broken artifact — and with a partial file cached, re-running could never clear it. The fix made the underlying bug harder to diagnose, not easier. Downloads now go to .part and only take the real name once the copy completes.

Verification

• Stubbed a stream that yields 4 KB then raises ConnectionResetError: the download fails, and the destination directory is left completely empty — no artifact at the final path, no .part remnant. audit() turns it into one could not fetch artifacts: ConnectionResetError unchecked row with no problems.
• Happy path with --keep: first run downloads and audits reflex-components-sonner==0.9.2 clean at 1 stub with both artifacts kept; second run reuses the cache and gives the identical result, exit 0.

ruff format and ruff check clean.

---

Generated by Claude Code

[cubic-dev-ai]

3 issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:279">
P2: When a wheel has corrupted compressed stub data, `wheel_stubs()` raises `zlib.error`, which this handler does not catch. Catch `zlib.error` along with the other archive-read failures so one unreadable artifact is reported as unchecked and does not abort the remaining package audits.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:166">
P2: When a proxy returns a malformed `urls` list, `pypi_status` crashes instead of reporting an indeterminate check. Validate each entry before calling `.get` or indexing `packagetype`, and return the existing `"error", "unexpected PyPI response shape"` result for malformed file records.</violation>
</file>

<file name=".claude/skills/prerelease-test/SKILL.md">

<violation number="1" location=".claude/skills/prerelease-test/SKILL.md:129">
P2: The two-command form does not actually gate the audit on the discovery check's exit status. If both lines are executed together, the second runs regardless of the first command's non-zero exit, so the truncated-list PASS the prose says it avoids still happens unless the exit code is acted on. Preface with `set -e` (or chain with `&&`) so the audit is skipped when the check returns 1 or 2, and state that the audit must only run after the check exits 0.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 existing issue remains and 2 new issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:166">
P2: When a proxy returns a `urls` list with a malformed entry, `pypi_status` raises instead of returning the documented `error` status, aborting the release check without its exit-2 handling. Validate each URL record and its `packagetype` before filtering it.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:82">
P2: When the screenshot destination cannot be prepared, `mkdir` raises before this guard, so the driver still exits with a traceback instead of reporting `SCREENSHOT FAILED` and writing the QA result. Move screenshot-directory creation inside the same exception handler and record setup failures in `screenshot_error`.</violation>
</file>

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic</sub>

[masenf]

Round 13 — three findings, all valid, fixed in 5bf408a, plus two things I found while verifying them.

The two-command form didn't actually gate anything. The sharpest of the three, because it's a hole in my own round-5 fix: I replaced the pipe to preserve the discovery exit status, then left the two commands on separate lines where the audit runs regardless. The prose described a guarantee the snippet didn't provide. Chained with && now, so a non-zero discovery skips the audit outright.

Malformed urls entries raised out of pypi_status. Also mine: narrowing the try to the network call in round 4 moved the payload parsing outside it, so {"urls": [1, 2]} or an entry without packagetype became a traceback where the old broad except had reported a failed check. Entries are validated now and reuse the existing indeterminate result.

A corrupted deflate stream aborted the audit. Verified before fixing — a wheel with scribbled compressed payload raises zlib.error, which subclasses only Exception, so it missed the handler entirely. Same shape as the EOFError case from round 11.

Two things the verification turned up on its own:

• The report read could not read artifacts: error, because zlib.error.__name__ is literally "error". Exceptions are now named module-qualified unless they're builtins, so it reads zlib.error while ConnectionResetError stays bare.
• A mistyped --ref produced a raw CalledProcessError traceback about subprocess internals — the exact failure mode this PR has spent thirteen rounds eliminating, over what is almost always a typo. It now prints cannot read ref 'x' in <repo>: fatal: Not a valid object name x and exits 2.

Verification

• pypi_status over six payloads: valid entry → published; [1, 2], [{}], [{"packagetype": null}], and a mixed good/bad list → all indeterminate; all-yanked still → missing.
• Corrupted-deflate wheel → unchecked: could not read artifacts: zlib.error, no problems, audit continues.
• The && gate both ways: bad ref → chain exits 1 and the audit never runs; good ref → audit runs, 121 stubs, exit 0.
• Bad --ref → clear message, exit 2. Rounds 4, 7 and 12 regression tests all still pass.

ruff format and ruff check clean.

---

Generated by Claude Code

[masenf]

One more, valid, fixed in c356b8b — and it's a hole one line above the guard I added for it last round.

The screenshot's mkdir sat outside the screenshot's own handler. A destination that can't be created still ended the run in a traceback, which is precisely what that handler existed to prevent. Moved inside. While there I noticed the report write had no guard at all, so I wrapped that too — losing the file shouldn't also lose the findings, and the summary now prints either way.

Verification — with a plain file where the parent directory would have to be:

• --screenshot blocker/shot.png → SCREENSHOT FAILED: FileExistsError: [Errno 17] File exists: 'blocker', exit 1, and the report is still written (492 bytes).
• --report blocker/r.json → REPORT NOT WRITTEN on stderr, full summary and RESULT line still printed, exit 1.

ruff format and ruff check clean.

That's 14 rounds and 34 findings, 33 of them valid — a hit rate worth saying out loud. The last several have all been one shape: a raise escaping a guard, or a guard that didn't quite cover its own setup line. That's a real class of bug and I've been fixing them, but it's also a surface with no natural end, and every push draws the next review. I'm going to stop actively iterating here unless something lands that's materially different in kind — a wrong result, a broken workflow, a security issue — or you tell me to keep going. The PR is in a good state for human review: greptile is at 5/5 with no blocking failure, and the one thread I've deliberately left open is the round-10 root-package claim I declined and explained.

---

Generated by Claude Code

[cubic-dev-ai]

1 existing issue remains and 2 new issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:112">
P2: When the root package is listed in `never-publish-packages` or `internal-packages`, discovery still checks the root `CHANGELOG.md` and can report a false PyPI blocker or emit that package in `--specs`. Apply the exclusion to the root changelog using the configured root package name before appending it.

(Based on your team's feedback about never-publish packages being excluded from all release paths.)</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:253">
P2: When the target page closes or becomes unreadable after a successful load/action, this fallback hides the inspection failure while `clean` remains true. Record the read exception and include it in the failure criteria.</violation>
</file>

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:301">
P2: When `--report` cannot be created or written, this handler only prints a warning and leaves `clean` unchanged. A healthy app therefore returns 0 and prints `RESULT: clean`; mark the run failed after this catch so automation cannot treat a missing requested QA artifact as a pass.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

I said I'd stop unless something materially different turned up. This is it — fixed in dd3dbac.

An unreadable page could report a clean run. Not another missing guard: a wrong result. The title/body read was already wrapped, but the failure only reached the body text, never the clean computation. So a page that died after a successful load and successful actions printed RESULT: clean and exited 0. A QA driver reporting clean when it inspected nothing is the single worst thing this tool can do, and every agent using it would have believed the pass.

Verification — forced the page read to fail on a page with no other anomaly (a static page with an inline favicon, so not even a 404 in the console), and A/B'd the clean criteria:

	RESULT	exit	read_error
baseline, unmodified driver	clean	0	—
read fails, before fix	clean	0	recorded but ignored
read fails, after fix	ANOMALIES FOUND	1	recorded and counted

The baseline row matters as much as the others: a genuinely quiet page still reports clean, so this didn't make the driver paranoid. The failure also prints as PAGE UNREADABLE: … and lands in the report as its own read_error field rather than being folded into action_error, which would misattribute it to an action that actually succeeded.

ruff format and ruff check clean.

The other finding in this round is the root-package never-publish-packages claim again, now widened to internal-packages. My round-10 reasoning is unchanged: those lists hold directory names under packages/, the root package is named separately by root-package, and the scenario needs a config declaring that this repo never publishes reflex. Both that thread and this new one are open for you — I'd rather leave a declined judgement visible than resolve my own disagreement away.

---

Generated by Claude Code

[masenf]

One more, valid, fixed in e860265 — greptile's P1 on audit_pyi.py:18.

When I moved SKILL.md phase 5 from a pipe to &&, I left the pipe in the audit script's own usage docstring. So the script documented exactly the form the skill warns against, and anyone reading the script rather than the skill would copy the version that lets a truncated list print PASS. Both now show the && form, and the docstring says why. grep -rn xargs over the skill confirms no pipe form survives.

I also removed a __pycache__ directory that my test harnesses had created under scripts/ while importing the modules — untracked, never committed, but worth clearing so it can't drift into a future commit.

ruff format and ruff check clean; --help still renders the docstring correctly.

Standing state: the only open thread is the root-package never-publish-packages / internal-packages claim, which I've declined twice with reasoning and am deliberately leaving open for a human call.

---

Generated by Claude Code

[masenf]

Same class as the last one, and this time it's a bug I introduced. Fixed in e2bbca3.

An unwritable --report still reported a pass. The guard I added in c356b8b printed REPORT NOT WRITTEN and carried on without touching clean, so a healthy app with an unwritable report path printed RESULT: clean and exited 0. A caller that asked for a report and got none has no result to trust, and automation would have read the exit status as a pass. That's the second false pass in this file, both introduced by guards I added while fixing tracebacks — worth noting as a pattern: adding a try around a failure is only half the job if the caller's success signal doesn't move with it.

Verification — three cases on a page with no console noise:

	RESULT	exit
healthy page, report writable	clean	0
healthy page, report not writable	ANOMALIES FOUND	1
healthy page, no --report requested	clean	0

The third row is the one that matters for not over-correcting: a run that never asked for a report is still clean.

ruff format and ruff check clean. Greptile's P1 on the usage docstring from the same batch was already fixed in e860265 and its thread resolved; everything else in the batch was re-delivery.

---

Generated by Claude Code

[cubic-dev-ai]

1 existing issue remains and 1 new issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:305">
P2: When `--report` points to an unwritable path, this handler only prints a warning, so a clean page still reports `RESULT: clean` and exits 0 despite the requested QA artifact being missing. Track the report-write failure as a run error while retaining the console summary.</violation>
</file>

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic</sub>

[cubic-dev-ai]

3 issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/drive_app.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/drive_app.py:91">
P2: When `--actions` is a list containing a non-object or multi-key action, `load_actions` accepts it and the driver launches Chromium before reporting an action failure. Validate that every item is a single-entry object before returning the action list so malformed input follows the early `INVALID --actions` path.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/audit_pyi.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/audit_pyi.py:23">
P2: Copying this documented chain fails because neither script is executable and neither has a shebang. Invoke both scripts with `uv run --script`, as the skill's canonical command does.</violation>
</file>

<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:176">
P2: When a response entry has a valid `packagetype` but no boolean `yanked`, this treats it as live and reports `published`; validate `yanked` in the response-shape guard.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".claude/skills/prerelease-test/scripts/check_release_versions.py">

<violation number="1" location=".claude/skills/prerelease-test/scripts/check_release_versions.py:36">
P2: When `root-package` is listed in `internal-packages`, this new exclusion does not apply to the root `CHANGELOG.md`, so the script checks a patch-released package against PyPI and can report a false blocker. Apply the exclusion check to the root changelog before appending it.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[masenf]

Three findings, all valid, fixed in 3b965b9.

The documented invocations didn't run. My own regression from the docstring rewrite one commit earlier: both scripts show bare check_release_versions.py … / audit_pyi.py …, but neither has a shebang or an exec bit — that's deliberate, the PEP 723 metadata is there so uv builds the environment. Anyone copying either documented line got "command not found". Same failure as the pipe: documentation that breaks when used. Both now show uv run --script, matching the skill's canonical form, and say why.

A list of malformed actions slipped past validation. [{"click": "a", "wait": 1}] or [1] passed the "is it a list" check and failed later inside the action loop — after a browser launch and a page load, which is exactly the cost the early exit was added to avoid. Each item must now be a single-key object, since that's what run_action unpacks.

A non-boolean yanked read as live. Taken in a narrower form than suggested: requiring the key would reject a lean-but-legal response, so the guard is isinstance(f.get("yanked", False), bool) — absent still means not yanked, but a value that isn't PyPI's can't have its yank state read and goes indeterminate.

Verification

• load_actions rejects [1], ["click"], [{}], [{"click":"a","wait":1}], [null], and [{"click":"a"}, 7], each with the offending item named; valid lists, [] and None still pass. End-to-end, a malformed list prints INVALID --actions: each action must be a single-key object, got {'click': 'a', 'wait': 1} and exits 1 without launching Chromium.
• pypi_status across five payloads: yanked absent or False → published; True → all-yanked missing; "false" and 0 → indeterminate.
• Both documented commands run verbatim: discovery emits its 16 specs, and the && chain audits 121 stubs with exit 0.
• Rounds 4, 7 and 13 regression tests all still pass.

ruff format and ruff check clean. Greptile is at 5/5 on the previous head with no blocking failure.

---

Generated by Claude Code

[masenf]

Thread housekeeping, so the open list means something when you review.

The root-package never-publish-packages / internal-packages claim has now been raised three times in slightly different words (rounds 10, 12 and 15). I've declined it each time on the same reasoning: those lists hold directory names under packages/, the root package is named separately by root-package, and the scenario requires a config declaring that this repo never publishes reflex — at which point the release pipeline refuses long before my QA script runs. Guarding it properly means parsing and threading root-package for an input that cannot occur.

Three open threads all saying that makes the open list unreadable, so I've resolved the two later duplicates and left the original — r3884855428 — open as the single place the disagreement lives. I'm not resolving my own declined judgement away; it's still there for you to overrule, and it's now the only thread standing between this PR and a clean board.

Everything else is resolved and fixed. Final state: 16 review rounds, 40 findings, 39 of them valid.

---

Generated by Claude Code