Skip to content

chore(verify): rebuild the verify skill around the sim harness as an agent-agnostic tree at tools/verify, with a capture helper and a feature map - #889

Merged
mforce merged 6 commits into
mainfrom
chore/verify-skill
Sep 16, 2026
Merged

mforce merged 6 commits into
mainfrom
chore/verify-skill

Conversation

@mforce

@mforce mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner

Summary

The verify skill predated the Playwright suite in tools/simulation/ui/ that every revamp PR now verifies with, and it had no doctor step, no evidence standard, no cleanup discipline and no feature map. This rebuilds it around the sim harness:

  • Launch and doctor the cluckwork-sim stack (bootstrap, sg docker reset, the Farms: ready marker, the four doctor checks including "was the image built from this checkout's head").
  • Drive through the harness's own fixtures and catalog labels (signIn, the cast personas, tEn, commitNamedPicker, the two viewport projects), with the two-farm rule: captures on readme-farm, write flows on default-farm.
  • Evidence standards and where proofs go; capture.sh takes four 1:1 frames of a route (1280 and 390, light and dark) on the demo farm, prints console errors, and removes its throwaway spec.
  • Cleanup that removes what the run created and never the evidence; the shared-stack rule (one compose project, never reset under another driver).
  • Feature map (features/): Dashboard, Daily entry, Sales, Stock, Flocks, each with sub-features, the user's path, real handles from the existing specs, and the end state that proves it. The unmapped routes are listed for later.
  • The dev-server path stays as the debugging section.

.gitignore's carve-out !.claude/skills/verify/ never applied, because git cannot re-include a file under an excluded directory; the parent rule is now .claude/skills/*, so the helper and the map are tracked as the comment intended.

Proven once, per the skill's own steps

Doctor against the live stack (all healthy, /health/ready 200), then capture.sh verify-skill-proof / readme: four frames in /tmp/verify-skill-proof/ in 8 s, the throwaway spec gone afterwards, git status clean, evidence still on disk after cleanup. The one console line it reports (a single 401 on the sign-in page's session probe) is documented as expected noise.

Both image-pin guards green on the tracked markdown.

Docs-only PR: web and image skip by the #782 gate.

… capture helper and a feature map

The skill predated the Playwright suite in tools/simulation/ui that every
revamp PR now verifies with, and it had no doctor step, no evidence
standard, no cleanup discipline and no feature map. It now launches and
doctors the cluckwork-sim stack, drives it through the harness's own
fixtures and catalog labels, states the proof standards, ships
capture.sh (four 1:1 frames of a route on the demo farm, throwaway spec
removed after), and maps the five features the smoke suite already
covers. The dev-server path stays as the debugging section.

.gitignore's carve-out for .claude/skills/verify/ never applied: git
cannot re-include a file under an excluded directory, so the parent rule
becomes .claude/skills/* and the negation now bites.
@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: f646d7a2-0979-4615-b04f-75c9da3d443e


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

The slug, prefix and route are interpolated into a generated Playwright
spec and a /tmp path; a slug with quotes or braces became code and one
with a separator escaped the directory. Each argument is now validated
against a fixed alphabet before anything is written.
@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

Security review (commit hook) flagged capture.sh: the slug and route were interpolated into the generated spec and the /tmp path unvalidated (code injection, path traversal). Fixed in 2e4f7d6: each argument is held to a fixed alphabet before anything is written; ../x, a quoted slug and a javascript:${x} route now exit 2.

…parses it

The unquoted & in the =~ pattern was a syntax error at parse time, so
every invocation exited 2, the previous commit included. The patterns
live in variables now; rejections and a real capture re-proven.
@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

Correction: 2e4f7d6's route pattern was a bash parse error (an unquoted & inside [[ =~ ]]), so every call exited 2, the rejection "proofs" included. d390f11 moves the patterns into variables; re-proven: ../x, a"b, x${y} and a javascript: route exit 2, and a real capture.sh verify-skill-proof2 / readme produced its four frames in 7 s with the throwaway spec removed.

@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…symlinked into each agent's skill directory, and name it in AGENTS.md

The tree is repo knowledge (stacks, ports, fixtures, proof standards),
not Claude knowledge, and Codex never looked under .claude. It now
lives at tools/verify/; .claude/skills/verify, .codex/skills/verify and
.agents/skills/verify are tracked symlinks to it, with the gitignore
parent rules ending in /* so the carve-outs apply. AGENTS.md carries
the one-paragraph rule pointing at it.
@mforce mforce changed the title chore(verify): rebuild the verify skill around the sim harness, with a capture helper and a feature map chore(verify): rebuild the verify skill around the sim harness as an agent-agnostic tree at tools/verify, with a capture helper and a feature map Sep 16, 2026
@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…issing, and rejects . and .. slugs

Codex CLI review of #889: the trailing || true swallowed every
Playwright failure and the following ls returned 0, so the helper could
report success with no evidence; and the slug alphabet admitted . and
.., so /tmp/.. resolved to /. The exit status now propagates, all four
frames must exist, and a slug must start alphanumeric.
@mforce

mforce commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

Review round (CodeRabbit rate limited; local Codex CLI review of git diff origin/main...HEAD at 428eeca stood in) found two real defects in tools/verify/capture.sh, both fixed in 77e054d:

  1. The trailing || true swallowed every Playwright failure and the following ls returned 0, so the helper could report success with no evidence. The exit status now propagates and all four frames must exist on disk.
  2. The slug alphabet admitted . and .., so /tmp/.. resolved to /. A slug must now start alphanumeric.

Re-proven: ., .. and -x exit 2; a forced Playwright failure (BASE_URL pointed at a dead port) exits non-zero with no frames and the throwaway spec still removed; a real capture still produces its four frames.

@mforce
mforce merged commit cf0998e into main Sep 16, 2026
15 of 16 checks passed
@mforce
mforce deleted the chore/verify-skill branch September 16, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant