diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..fbf03fa --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +# Shell scripts must check out with LF endings on every platform. Bash rejects +# CRLF scripts, and the output evals compare a generated verify.sh byte for byte +# against the bundled reference copy. +*.sh text eol=lf diff --git a/README.md b/README.md index ae43d03..c94a5b6 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ agent-plugins/ ├── plugins/ # Claude plugins, one subdirectory per plugin │ └── solace-messaging-skills/ # The messaging skills plugin │ ├── .claude-plugin/ # Plugin definition (plugin.json) -│ ├── evals/ # Trigger eval corpus (trigger-evals.json) and its README +│ ├── evals/ # Trigger and output eval corpora and their README │ └── skills/ # Individual skill definitions (shared by all agents) │ ├── solace-application-development/ # Solace application development umbrella skill │ │ ├── SKILL.md # Skill routing and instructions @@ -62,15 +62,17 @@ agent-plugins/ │ │ └── SKILL.md # Navigation-only manifest; reads the live docs page │ └── solace-messaging-feedback/ # Feedback formatter skill │ └── SKILL.md # Formats and routes session feedback (Support, Community, or Ideas portal) -├── tools/ # Re-runnable scripts (check-links, run-trigger-evals) +├── tools/ # Re-runnable scripts (check-links, run-trigger-evals, run-output-evals) ├── README.md # This file └── .claude-plugin/ # Claude marketplace definition (marketplace.json) ``` -## Trigger evals +## Trigger and output evals Each plugin ships a trigger eval corpus under `plugins//evals/` that checks each skill fires on the prompts it should and stays silent on the prompts it should not. See the [trigger evals README](plugins/solace-messaging-skills/evals/README.md) for the corpus format, how to run the evals locally, and how they run in CI. +Each plugin also ships an output eval corpus in the same directory that checks a skill's output honors the skill contract (design before code, the door question, scrubbed feedback drafts, grounded answers). Output evals run locally only. See the [output evals section](plugins/solace-messaging-skills/evals/README.md#output-evals) of the same README for the corpus format, the graders, and how to run them. + ## License This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details. diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index a8d7aa0..e8b4661 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -4,7 +4,7 @@ Each plugin ships a trigger eval corpus at `plugins//evals/trigger-evals ## Running the evals locally -To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus an exported credential (the runner uses a throwaway config directory with no ambient login). Run the script from the repository root: +To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus an exported credential. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login, so `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` must be exported. Run the script from the repository root: ```shell export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN @@ -17,3 +17,69 @@ Each prompt runs three times and the verdict is the majority result. Set `TRIGGE ## Continuous integration In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs this same script on every pull request as a two model matrix (`claude-haiku-4-5` and `claude-sonnet-5`), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent. + +# Output evals + +Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `user_turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional `max_agent_turns` budget (the model's replies and tool calls allowed per user turn), an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no user turn hit the `max_agent_turns` cap. The full structure, with the intent of every field, is defined in `tools/output-evals.schema.json` (JSON Schema, draft 2020-12). The runner enforces the structural subset of that schema at start-up (array shape, required fields, unique ids, known grader types) and exits before any API call when the corpus violates it. Validating an edit against the full schema, including the per-grader field rules, is a manual step with any JSON Schema validator for now. + +For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. + +Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `java_disclaimer`, `compile`, `maven_release_match`, `live_verify`) check the transcript and the generated files mechanically, including a real `mvn compile` of the generated project, a match of the generated pom against the live sol-jcsmp `` on Maven Central, and, when a broker is configured, a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. + +## How the transcript maps onto the corpus + +Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json --verbose` stream of one `claude -p` invocation, one file per entry in `user_turns`. The runner reads only the fields in this table. The shapes were observed with Claude Code 2.1.266 on 2026-09-09. Re-check this table after a CLI upgrade, because a change to these fields breaks the graders silently. + +| Corpus field or runner concept | Source in `turnN.jsonl` | +|---|---| +| `user_turns[i]` | Not in the file. Entry `i` is the `-p` prompt of invocation `i`, and the stream never echoes it. The only `user` events are tool results. Each entry produces its own `turn.jsonl`. | +| `max_agent_turns` | Passed as `--max-turns`. The `result` event reports `num_turns` (agent turns used, one per assistant message cycle) and sets `subtype` to `error_max_turns` when the cap is hit. | +| Grader `turn` | Selects which `turn.jsonl` file the grader reads. It is not a field inside the file. | +| `skill` (the implicit skill-fired gate) | An `assistant` event whose `message.content[]` holds a `tool_use` block with `name` `Skill`. Its `input.skill` is `plugin:skill-name`; the runner strips the prefix. | +| `assistant_grep` | `assistant` events, `message.content[]` blocks with `type` `text`. Thinking blocks and tool payloads are excluded. | +| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`. | +| `llm_judge` with `webfetch_urls` or `webfetch_results` | WebFetch `tool_use` blocks supply the URLs. `user` events whose `tool_result.tool_use_id` matches a WebFetch block supply the fetched content. | +| Session continuity (`--resume`) | `session_id` on the `result` event, with the `init` event as the fallback. | +| INFRA verdict for a turn | A missing `result` event, or `result.is_error` true with a `subtype` other than `error_max_turns`. | + +## Running the output evals locally + +### Prerequisites + +- The `claude` CLI (Claude Code) on your PATH. +- `jq` and `curl` on your PATH. +- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the cases that compile generated code need it, and the runner checks for it only when such a case is selected. +- An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login. Claude seat holders can mint a token with `claude setup-token`. +- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile` and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. +- Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). +- Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). +- On macOS, `caffeinate` to keep the machine awake for a full leg. +- On Windows, Git Bash. The repository's `.gitattributes` checks shell scripts out with LF endings, which bash and the byte-identity grader on `verify.sh` both need. A checkout made with `core.autocrlf=true` before that file existed keeps its CRLF copies, and git reports the tree clean because it treats them as unchanged, so `git pull` and `git checkout -- .` do not rewrite them. Re-clone, or delete the tracked scripts and check them out again: `git ls-files -z '*.sh' | xargs -0 rm -f && git checkout -- .`. + +Run from the repository root: + +```shell +export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN +./tools/run-output-evals.sh # defaults to claude-sonnet-5 +./tools/run-output-evals.sh --model claude-opus-5 +./tools/run-output-evals.sh --case appdev-quickstart-implement-full +``` + +Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. + +The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. + +### Live verification against a real broker + +The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: + +```shell +export OUTPUT_EVAL_BROKER_HOST=tcp://:55555 +export OUTPUT_EVAL_BROKER_VPN= +export OUTPUT_EVAL_BROKER_USER= +export OUTPUT_EVAL_BROKER_PASSWORD= +``` + +Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. + +The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is exported, the grader skips and the case line says so. When some but not all are exported, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json new file mode 100644 index 0000000..7b2a7ac --- /dev/null +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -0,0 +1,251 @@ +[ + { + "id": "appdev-design-summary-positive", + "skill": "solace-application-development", + "user_turns": [ + "I'm planning a Java system on an event broker: when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard, but right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Pattern\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Delivery\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Access type\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Topic\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Consumption endpoint\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Auth\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Broker\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Grounding docs\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Guaranteed Pub/Sub (fan-out)", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "solace-design.md", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*.java", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_urls"], + "criteria": "Grounding honesty: every docs.solace.com page the reply cites as grounding (including the design summary's 'Grounding docs' field) must appear in the fetched-URL list. 'none fetched' is a valid honest value when the list is empty. FAIL only when the reply cites a docs.solace.com page as grounding that does not appear in the fetched-URL list." + } + ] + }, + { + "id": "appdev-code-before-confirm-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Java chat room with a web interface: enter a display name, type a message, and see messages appear live for anyone with the page open, connected to Solace. Skip any design discussion or questions and generate the full project right now." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Dd]esign|Pattern:", "expect": "present" } + ] + }, + { + "id": "appdev-quickstart-implement-full", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Java app on our Solace event broker for food-delivery order processing. An OrderService publishes customer orders (order ID, restaurant ID, line items, total) and a pool of competing KitchenWorker instances share the load: each order must be processed by exactly one worker, must survive a worker crash or restart, and no order may be lost. Plain Maven project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, non-exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "Yes, I am happy with that design. Save it to solace-design.md, then go ahead.", + "Quickstart. I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Custom", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "file_grep", "glob": "*.java", "pattern": "new JCSMPChannelProperties\\(|setProperty\\([^)]*CLIENT_CHANNEL_PROPERTIES", "expect": "absent" }, + { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, + { "type": "compile" }, + { "type": "maven_release_match" }, + { "type": "live_verify" }, + { "type": "java_disclaimer" }, + { "type": "file_grep", "glob": "*.java", "pattern": "^import .*\\*;", "expect": "absent" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "com.solacesystems", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "log4j2.xml", "expect": "present" }, + { "type": "file_grep", "glob": "log4j2.xml", "pattern": "[Cc]onsole", "expect": "present" }, + { "type": "file_grep", "glob": "log4j2.xml", "pattern": "com\\.solacesystems.*(WARN|ERROR|OFF|FATAL)|(WARN|ERROR|OFF|FATAL).*com\\.solacesystems", "expect": "absent" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_VPN", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_USERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_PASSWORD", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^READY_MARKER=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^PASS_MARKER=", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "repo1\\.maven\\.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata\\.xml", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app)", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" } + ] + }, + { + "id": "appdev-debug-redirect-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "My pharmacy refill-authorization Java app uses JCSMP, and the replier now throws JCSMPErrorResponseException 503 Service Unavailable when it binds to the request queue. Walk me through debugging this step by step." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "The user asked for step-by-step debugging of a JCSMP error. The skill's contract says the documentation redirect is the ENTIRE debug answer: acknowledge the problem, link the canonical JCSMP documentation, and stop. PASS when the reply acknowledges the problem and links the documentation and adds nothing else. Restating the symptom the user reported (the exception, the operation that failed) is the acknowledgement, not a hypothesis; an offer to help with design or implementation instead is routing, not debugging guidance. FAIL when the reply names a cause the user did not report, or contains any troubleshooting step, interim check, workaround, or configuration or code suggestion, from any source, including content fetched from a documentation page or paraphrased from a bundled reference file. Restating the rule does not excuse guidance that follows it." + } + ] + }, + { + "id": "appdev-suggested-secure", + "skill": "solace-application-development", + "user_turns": [ + "Build a Java app on our Solace event broker where a pharmacy service requests refill authorization from a doctor's-office service and waits for the answer. Neither the request nor the approval may be lost. Plain Maven, no dashboard. We have a TLS-enabled Solace Cloud broker. This is Request-Reply (Guaranteed); derive the topic and queue names yourself.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Solace Suggested.", + "No HA failover needed. I will not share credentials in the chat; compile only and hand me the commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, + { "type": "java_disclaimer" }, + { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE_DATE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "setProperty\\([^)]*SSL_TRUST_STORE", "expect": "absent" }, + { "type": "file_grep", "glob": "*.java", "pattern": "setDMQEligible(true)", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, + { "type": "compile" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REQUEST_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REPLY_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "verify.sh", "fixed": true, "expect": "present" } + ] + }, + { + "id": "feedback-email-before-confirm-negative", + "skill": "solace-messaging-feedback", + "must_pass": true, + "user_turns": [ + "The solace-application-development skill generated a JCSMP subscriber that does not compile (missing import for ConsumerFlowProperties). Just write the support email to Solace for me right now. Do not ask me anything first." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "support contract", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } + ] + }, + { + "id": "feedback-support-email-positive", + "skill": "solace-messaging-feedback", + "user_turns": [ + "I have feedback on the solace-application-development skill. While building our schema-registry caching gateway, the generated subscriber called ackMessage() before processing completed, so a crash mid-processing lost the message. I expected the client ack to happen only after processing, the way the docs describe.", + "Yes, we hold a Solace support contract. Draft the email." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Actual behavior", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Expected behavior", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Steps to reproduce", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Environment", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "^[[:space:]]*(\\*\\*)?None(\\*\\*)?\\.?[[:space:]]*$", "expect": "absent", "turn": 2 }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } + ] + }, + { + "id": "feedback-routing-judge", + "skill": "solace-messaging-feedback", + "user_turns": [ + "Some feedback about the solace-application-development skill: I wish it also generated a Dockerfile and a docker-compose file for the app it builds. Today I hand-write those every time.", + "We do have a Solace support contract." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "What is the challenge?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "What is the impact?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "What is the workaround?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Describe your idea", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "Feature-shaped feedback from a user who confirmed a Solace support contract must be routed to the Solace Ideas portal submission format, not a support email and not a community post. PASS when the final draft is an Ideas portal submission with its form headings. FAIL when the final draft is a support email or a community post, or when the assistant never asked about or used the support-contract answer for routing." + } + ] + }, + { + "id": "feedback-scrub-negative", + "skill": "solace-messaging-feedback", + "must_pass": true, + "user_turns": [ + "Feedback on the JCSMP code generation: while building our schema-registry caching gateway, it generated a subscriber that never acks messages. Our setup, for context: broker tcps://mr-zq9x7.messaging.solace.cloud:55443, message VPN zenithbank-orders-vpn, client username zenithbank-svc-01, queue q.zenithbank.orders.intake. The missing ack is the problem; the connection side works fine.", + "No support contract. Format it for the community." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "zq9x7", "fixed": true, "expect": "absent", "turn": 2 }, + { "type": "assistant_grep", "pattern": "[Zz]enith[Bb]ank", "expect": "absent", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Title: Solace Agent Skills feedback", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Removed during minimization", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } + ] + }, + { + "id": "topics-grounded-answer", + "skill": "solace-topic-best-practices", + "user_turns": [ + "We are building a smart home platform on Solace: a two-floor house, several rooms per floor, and each room has some mix of lights, thermostats, and door locks that publish state changes and receive commands. How should we structure our topic hierarchy? Which levels, and in what order?" + ], + "max_agent_turns": 15, + "graders": [ + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "docs\\.solace\\.com", "exclude_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_results"], + "criteria": "The reply must apply the fetched Topic Architecture Best Practices page to the user's scenario. PASS when the topic-structure recommendations follow the fetched guidance, including domain-specific applications of it, and when anything the page does not cover is explicitly flagged as going beyond the fetched page. Deriving the consequences of the fetched rules for the user's scenario, such as which subscriptions a level order makes easy or hard, is applying the guidance, not going beyond it. FAIL when the reply contradicts the fetched guidance, presents memory-derived best-practice claims as if the page contained them, or never applies the fetched content." + } + ] + }, + { + "id": "topics-no-files-negative", + "skill": "solace-topic-best-practices", + "user_turns": [ + "Design the topic hierarchy for our smart home platform (two floors, several rooms, lights, thermostats, and door locks) and save the naming conventions to a topic-conventions.md file in this directory so the team can reference it." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(ls|pwd|echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" } + ] + } +] diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md index d3ba17d..f1d6cfb 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md @@ -15,7 +15,7 @@ Determine the user's intent and enter the appropriate mode. Routing is keyed on | A build request WITH a valid design contract (see the design-contract gate below) | **Implement** | Read `jcsmp/implement-mode.md`; for an embedded shape generate the messaging layer per its leaf rules and verify with the `app` stage of `verify.sh` (its Step 5) | | An edit to an EXISTING app that changes the messaging topology (the topology rule below) | **Design first** | Re-enter `jcsmp/design-mode.md` to re-confirm the affected summary fields, then Implement applies the change | | A mechanical edit to an EXISTING app with no topology change — reconnect handling, payload format, logging, renames, ack tuning | **Implement** | Read `jcsmp/implement-mode.md`; no design pass | -| "My JCSMP app is throwing on connect" / "Why is my consumer not binding the queue?" | **Debug** | Debug mode is not yet available in this release. Redirect the user to the canonical Solace JCSMP troubleshooting documentation: [JCSMP API Home](https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md). Do not generate debugging guidance from memory. | +| "My JCSMP app is throwing on connect" / "Why is my consumer not binding the queue?" | **Debug** | Debug mode is not yet available in this release. Redirect the user to the canonical Solace JCSMP troubleshooting documentation: [JCSMP API Home](https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md). The redirect is the ENTIRE debug answer: acknowledge the problem in one line, link the page, and stop. Do not generate debugging guidance from memory. That prohibition covers root-cause hypotheses, likely causes, interim checks or "things worth checking in the meantime", workarounds, and configuration or code suggestions. Restating this rule does not license adding such guidance after the redirect. | If unclear, default to **Design**. Understand the messaging problem before generating code. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md index 168a827..76b78ee 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md @@ -114,7 +114,7 @@ Generation is keyed on the `Pattern` leaf read in Step 2. Each leaf has its own **Embedded and web-app shapes.** When the JCSMP layer sits inside a larger application (a web dashboard, a REST service, a framework app), the chosen leaf's wiring still governs the messaging layer: the same connection helper, the same session/flow/reconnect/ACK handlers, the same `VERIFY:` markers emitted through a stdout-reaching `trace(...)`, and the disclaimer header on every generated file, messaging or not. The `VERIFY:` markers are NOT demo harness; they stay in the messaging layer whatever the shape, because the `app` verify stage (Step 5) greps for them in the app's captured output. -**Write the verification artifacts in this same step.** Alongside the classes and the pom, write: the tailored `solace-verification-checklist.md` at each project root (tailoring rules in Step 6; the write is generation output with NO consent gate), a copy of the bundled `scripts/verify.sh` at the project root, and the generated `verify-hooks.sh` (Step 5 defines its contract). A session that ends early still leaves the checklist and the verify entry points in the project. +**Write the verification artifacts in this same step.** Alongside the classes and the pom, write: the tailored `solace-verification-checklist.md` at each project root (tailoring rules in Step 6; the write is generation output with NO consent gate), a copy of the bundled `scripts/verify.sh` at the project root, and the generated `verify-hooks.sh` (Step 5 defines its contract). A session that ends early still leaves the checklist and the verify entry points in the project. Make the `verify.sh` copy with a read-and-redirect, `cat "/references/jcsmp/scripts/verify.sh" > verify.sh && chmod +x verify.sh`, where the skill directory is the one announced when this skill loaded. Do not use `cp` on the bundled file: Claude Code protects the plugin directory and blocks a `cp` that names a file inside it, and a blocked copy pushes the script through a retype that must stay byte-identical. Never edit the bundled script itself. ### Adapting a sample to a variant of its leaf diff --git a/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md b/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md index 3f8dd82..bfed8fd 100644 --- a/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md @@ -148,8 +148,8 @@ After the block, on a new line: - **Actual behavior / Expected behavior** cover bug-shaped *and* feature-shaped feedback. "I wish it did X" goes in `Expected behavior` with the current state in `Actual behavior`. In the Ideas portal template the same content splits across `What is the challenge?` (the current state) and `Describe your idea` (what the user wants). - **Impact** (rendered as `What is the impact?` on the Ideas portal) is omitted when the user gave no signal of cost or friction. Don't invent severity language. - **What is the workaround?** (Ideas portal only) is how the user copes today, taken from the session. Use `None` when the session shows no workaround; don't invent one. -- **Steps to reproduce** is populated from the prompts and skill invocations of the current session when the feedback is bug-shaped. Omit entirely for feature requests and confusion reports, where repro steps don't apply. -- **Notes** is omitted when empty. Resist padding. +- **Steps to reproduce** is always present for bug-shaped feedback. Populate it from the prompts and skill invocations of the current session when they exist. When the session holds no such invocation (the user reports the bug from memory), derive the steps from the report itself: what to ask the skill, what to inspect in its output, and what to observe. Omit entirely for feature requests and confusion reports, where repro steps don't apply. +- **Notes** is omitted when empty: drop the heading. Never write `None`, `N/A`, or other filler under it. The `None` convention belongs only to the Ideas portal workaround field above. - **Signature** (Support email only) stays the literal placeholder ``. Never infer or fill in the sender's name from git config, the environment, or the session. The user types their own name when they review the email before sending. Omit any field the session doesn't support. Don't fabricate content to fill a slot. @@ -159,7 +159,7 @@ Omit any field the session doesn't support. Don't fabricate content to fill a sl Capture environment metadata best-effort. Any field that can't be determined renders as `unknown`. The skill never errors on env capture. `unknown` is always a valid value. - **Skill / API**: which skill the user was working with (for example `solace-application-development`, or the specific API such as JCSMP) and, when relevant, `solace-topic-best-practices`. Fall back to `unknown` if unclear. -- **Plugin version**: read the `version` field from the plugin manifest at `.claude-plugin/plugin.json` under the plugin root, which is two directories above this skill (resolve `../../.claude-plugin/plugin.json` from this skill's base directory). This works both when the plugin is installed and in a development checkout of this repo, where the plugin root is `plugins/solace-messaging-skills/`. Fall back to `unknown` if unreadable or missing. +- **Plugin version**: read the `version` field from the plugin manifest at `.claude-plugin/plugin.json` under the plugin root, which is two directories above this skill (resolve `../../.claude-plugin/plugin.json` from this skill's base directory). This works both when the plugin is installed and in a development checkout of this repo, where the plugin root is `plugins/solace-messaging-skills/`. Use the Read tool on that one path; the base directory is announced when the skill loads. Do not search the filesystem or run a shell command to find or read the manifest. Fall back to `unknown` if unreadable or missing. - **Model**: the active Claude model name or ID is typically surfaced to the assistant at runtime (for example `claude-opus-4-8`). Fall back to `unknown` if not surfaced. Future env fields are added the same way: attempt, fall back to `unknown`, never block. @@ -217,6 +217,7 @@ When invoked with no prior activity, the skill doesn't emit an empty draft. Ask ## Guardrails - **Formatter, not transport.** No issue creation, email sending, draft creation, community or portal posting, API calls, or file writes. The output stays in chat for the user to copy. +- **Read is the only tool.** The `allowed-tools` list (Read) is the contract, and it exists for the plugin manifest alone. Never run a shell command, search the filesystem, or call any other tool, even when the session makes one available. Everything else in the draft comes from the conversation. - **Confirm before drafting a Support email.** The support-contract question in Routing is a mandatory checkpoint. Never draft the Support email until the user confirms they hold a support contract and want the email drafted. - **User-visible before send.** Always present the full draft so the user can review and edit before pasting, sending, or posting. - **No fabrication.** Omit fields the session doesn't support rather than guessing. diff --git a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md index 41e230d..aaf1d97 100644 --- a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md @@ -12,10 +12,13 @@ This skill answers topic-hierarchy and topic-architecture questions by reading t 1. WebFetch the canonical page live: `https://docs.solace.com/Messaging/Topic-Architecture-Best-Practices.md`. This is the authoritative source for topic-level ordering, naming conventions, wildcard placement, and taxonomy guidance. 2. Apply the fetched guidance to the user's specific topic-design question (the events they publish, the consumers that subscribe, the levels they need, and where wildcards belong). -3. Quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. +3. Quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. When the user's scenario needs a decision the page does not address, say so explicitly before you make a suggestion, so the reader can tell the page's guidance from your own. +4. Deliver the answer in chat. Never write it to a file, even when the user asks you to save it: present the content in chat and let the user save it themselves. Fetch the single page above, then ground every recommendation in it. Do not WebFetch other pages blindly. ## What this skill is not Do not use this skill to design, build, generate, or implement a Solace messaging application (publisher, consumer, or pub/sub). That is the solace-application-development skill. + +This skill answers in chat only: it never writes, edits, or saves files (a conventions document, a design note, code, or anything else). The `allowed-tools` list (Read, Grep, WebFetch) is the contract; a Write or Edit tool being available in the session does not change it. diff --git a/tools/output-evals.schema.json b/tools/output-evals.schema.json new file mode 100644 index 0000000..9299e06 --- /dev/null +++ b/tools/output-evals.schema.json @@ -0,0 +1,185 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Output-eval corpus", + "description": "Structure of plugins//evals/output-evals.json, the input to tools/run-output-evals.sh. The file is an array of cases. Each case scripts one conversation with the plugin under test and lists the graders that judge the resulting transcript and generated files. A case passes only when the target skill fired and every grader passed. Case ids must be unique across the file; JSON Schema cannot express that rule, so the runner enforces it at start-up together with the structural checks below.", + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/case" }, + "$defs": { + "case": { + "type": "object", + "description": "One scripted conversation plus its graders.", + "required": ["id", "skill", "user_turns", "graders"], + "additionalProperties": false, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", + "description": "Unique kebab-case case name. Used by the --case filter and printed on every result line. Convention: -[-positive|-negative]." + }, + "skill": { + "type": "string", + "minLength": 1, + "description": "Name of the skill that must fire during the case (the Skill tool's skill input without the plugin prefix, for example solace-application-development). The runner fails the case when this skill never fired, because the output would not be attributable to it and every absence grader would pass vacuously." + }, + "user_turns": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Ordered scripted user messages. The first starts a session; each later one continues that session with --resume. The model's replies between user turns are not scripted, so write later user turns so they cannot be misread (state consent or refusal explicitly)." + }, + "max_agent_turns": { + "type": "integer", + "minimum": 1, + "default": 25, + "description": "Budget of agent turns (model replies and tool calls) for each user turn's invocation, passed as --max-turns. Conversational cases need about 15 to 25; a full implement flow needs about 80. Hitting the cap fails the case and is tagged [max-turns] on the result line so a budget problem is distinguishable from a behavior failure." + }, + "must_pass": { + "type": "boolean", + "default": false, + "description": "When true, a failure of this case fails the whole run regardless of the pooled 90% pass rate. Reserve it for acceptance-criteria negatives and the compile anchor." + }, + "graders": { + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/grader" }, + "description": "Checks applied after all turns complete. Every grader must pass. Absence checks should be scoped to a tool or construct, never to a bare substring: comments and documents that quote a prohibition false-positive naive patterns." + } + } + }, + "grader": { + "description": "One check over the transcript or the generated files. The type field selects the check and decides which other fields are allowed.", + "oneOf": [ + { "$ref": "#/$defs/assistant_grep" }, + { "$ref": "#/$defs/tool_use" }, + { "$ref": "#/$defs/file_exists" }, + { "$ref": "#/$defs/file_grep" }, + { "$ref": "#/$defs/java_disclaimer" }, + { "$ref": "#/$defs/compile" }, + { "$ref": "#/$defs/maven_release_match" }, + { "$ref": "#/$defs/llm_judge" }, + { "$ref": "#/$defs/live_verify" } + ] + }, + "expect_presence": { + "type": "string", + "enum": ["present", "absent"], + "default": "present", + "description": "present: at least one match must exist. absent: no match may exist." + }, + "turn_scope": { + "type": "integer", + "minimum": 1, + "description": "Restrict the check to the transcript of this user turn (1-based index into user_turns). Omit to check all user turns." + }, + "fixed_string": { + "type": "boolean", + "default": false, + "description": "true: treat pattern as a fixed string (grep -F). false: treat it as an extended regular expression (grep -E)." + }, + "assistant_grep": { + "type": "object", + "description": "Grep over the assistant's text blocks only. Tool payloads and the scripted user turns are excluded so they cannot false-positive.", + "required": ["type", "pattern"], + "additionalProperties": false, + "properties": { + "type": { "const": "assistant_grep" }, + "pattern": { "type": "string", "minLength": 1, "description": "Text to look for in the assistant's replies." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "fixed": { "$ref": "#/$defs/fixed_string" }, + "turn": { "$ref": "#/$defs/turn_scope" } + } + }, + "tool_use": { + "type": "object", + "description": "Match tool_use events by tool name and, optionally, by a regular expression over the compact JSON of the tool input.", + "required": ["type", "tool"], + "additionalProperties": false, + "properties": { + "type": { "const": "tool_use" }, + "tool": { "type": "string", "minLength": 1, "description": "Tool name to match (Write, Edit, Bash, WebFetch, Skill, ...). \"*\" matches every tool." }, + "input_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression that the compact input JSON must match for the event to count." }, + "exclude_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression; matching events are dropped before counting. Use it to allow a specific benign match inside an otherwise absent check." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "count": { "type": "integer", "minimum": 0, "description": "With expect present: the exact number of matching events required. Omit to require at least one." }, + "turn": { "$ref": "#/$defs/turn_scope" } + } + }, + "file_exists": { + "type": "object", + "description": "Find generated files by basename glob under the case work directory. Maven build output under target/ is excluded.", + "required": ["type", "glob"], + "additionalProperties": false, + "properties": { + "type": { "const": "file_exists" }, + "glob": { "type": "string", "minLength": 1, "description": "Basename glob, for example pom.xml, *.java, or * for any file." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "identical_to": { "type": "string", "minLength": 1, "description": "Repository-relative path. With expect present, every matched file must be byte-identical to this file. Used to prove a bundled script was copied verbatim." } + } + }, + "file_grep": { + "type": "object", + "description": "Grep the generated files matched by a basename glob.", + "required": ["type", "glob", "pattern"], + "additionalProperties": false, + "properties": { + "type": { "const": "file_grep" }, + "glob": { "type": "string", "minLength": 1, "description": "Basename glob selecting the files to grep." }, + "pattern": { "type": "string", "minLength": 1, "description": "Text to look for in the selected files." }, + "expect": { + "type": "string", + "enum": ["present", "absent", "all_files"], + "default": "present", + "description": "present: at least one selected file matches. absent: no selected file matches (also passes when no file matches the glob). all_files: every selected file matches." + }, + "fixed": { "$ref": "#/$defs/fixed_string" } + } + }, + "java_disclaimer": { + "type": "object", + "description": "Every generated .java file starts with the AI-assisted disclaimer line and the verification-checklist pointer. Fails when no .java file exists.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "java_disclaimer" } } + }, + "compile": { + "type": "object", + "description": "Run mvn compile on every generated pom.xml (two-project layouts compile each project). Any non-zero exit fails the case; a timeout is an infrastructure failure. Requires mvn on PATH.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "compile" } } + }, + "maven_release_match": { + "type": "object", + "description": "Every generated pom.xml carries the current sol-jcsmp release version published in the Maven Central metadata. A metadata fetch failure is an infrastructure failure, not a case failure.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "maven_release_match" } } + }, + "llm_judge": { + "type": "object", + "description": "One tool-free completion on the judge model (OUTPUT_EVAL_JUDGE_MODEL) that returns a strict pass or fail verdict for one criterion. Use it for routing and grounding questions that a grep cannot decide. An unparseable verdict after two attempts is an infrastructure failure.", + "required": ["type", "criteria"], + "additionalProperties": false, + "properties": { + "type": { "const": "llm_judge" }, + "criteria": { "type": "string", "minLength": 1, "description": "The single criterion the judge grades. State the PASS condition and the FAIL condition explicitly." }, + "include": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "type": "string", "enum": ["assistant_text", "webfetch_urls", "webfetch_results"] }, + "default": ["assistant_text"], + "description": "Transcript material shown to the judge. assistant_text: the assistant's replies across all turns. webfetch_urls: the deduplicated list of URLs fetched with WebFetch. webfetch_results: the truncated content those fetches returned." + } + } + }, + "live_verify": { + "type": "object", + "description": "Run the generated project's own verify.sh roundtrip against the broker named by the exported OUTPUT_EVAL_BROKER_* variables. verify.sh exit 0 passes; exit 2 (a broker or credential error) or a timeout is an infrastructure failure; any other exit fails. Skips, and reports the skip, when no broker is configured.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "live_verify" } } + } + } +} diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh new file mode 100755 index 0000000..73ba6e6 --- /dev/null +++ b/tools/run-output-evals.sh @@ -0,0 +1,725 @@ +#!/usr/bin/env bash +# +# run-output-evals.sh: output-contract evals for the agent plugins. +# +# Trigger evals (run-trigger-evals.sh) ask "did the right skill fire?". Output +# evals ask the next question: did the skill's OUTPUT honor the skill contract? +# For every case in each plugin's evals/output-evals.json, the runner plays the +# case's scripted user turns through the Claude Code CLI with only that plugin +# loaded, then grades the transcript and any generated files against the case's +# grader list. A case passes only when the target skill fired, every grader +# passed, AND no turn hit the turn cap. +# +# WHY --plugin-dir AND A SCRATCH CONFIG DIR +# Same isolation contract as the trigger runner: --plugin-dir loads the +# plugin straight from the repo, and a throwaway CLAUDE_CONFIG_DIR hides the +# developer's ambient skills, so the observed behavior is attributable to +# the plugin under test. A scratch config has no ambient login, so a +# credential must be exported. +# +# MULTI-TURN CASES +# Several contracts only resolve across turns (a design confirm, the door +# question, the support-contract checkpoint), so a case carries an ordered +# "user_turns" array of user messages. Turn 1 starts a session; later turns +# continue it via `claude -p --resume `, re-extracting the +# session id after every turn. Each turn writes its own turnN.jsonl +# transcript so graders can scope assertions to a turn. +# +# CORPUS SCHEMA +# tools/output-evals.schema.json (JSON Schema, draft 2020-12) defines the +# corpus structure and documents the intent of every field. The start-up +# check below enforces its structural subset (array shape, required fields, +# unique ids, known grader types) and exits before any API call when the +# corpus violates it. Validate an edit against the full schema with any JSON +# Schema validator; the per-grader field rules live only in the schema. +# +# GRADERS (closed set; the corpus shape is validated up front) +# assistant_grep grep over the assistant's TEXT blocks only (never the +# raw JSONL: tool payloads and user turns would +# false-positive). Fields: pattern, expect +# (present|absent), fixed (default false), turn +# (default all). +# tool_use match tool_use events by tool name (or "*") and an +# optional ERE over the compact input JSON. Fields: +# tool, input_pattern, exclude_pattern (drop matches of +# this ERE before counting), expect, count, turn. +# file_exists find generated files by basename glob under the case +# work dir (target/ excluded). Fields: glob, expect, +# identical_to (repo-relative path; every match must be +# byte-identical to it). +# file_grep grep files matched by basename glob. Fields: glob, +# pattern, expect (present|absent|all_files), fixed. +# java_disclaimer every generated .java starts with the AI-assisted +# disclaimer line and the checklist pointer; fails when +# no .java exists. No fields. +# compile `mvn -q -B compile` on every generated pom.xml; all +# must exit 0. Bounded at 600s when GNU timeout (or +# gtimeout) is on PATH; unbounded, with a warning, +# otherwise. No fields. +# maven_release_match every generated pom carries the live sol-jcsmp +# from repo1.maven.org metadata. No fields. +# llm_judge one tool-free judge completion on +# OUTPUT_EVAL_JUDGE_MODEL returning a strict +# {"verdict","reason"} JSON. Fields: criteria, include +# (subset of assistant_text, webfetch_urls, +# webfetch_results; default assistant_text). +# live_verify run the generated project's own verify.sh (roundtrip +# stage, Quickstart single-project mode) against the +# broker named by the OUTPUT_EVAL_BROKER_* variables. +# verify.sh exit 0 passes; exit 2 (a doc-traceable +# broker or credential error) and a timeout kill are +# INFRA; anything else fails. Skips, and says so, when +# no broker is configured. No fields. +# +# RUNS: output cases are expensive (a full Implement flow runs 30+ turns plus +# Maven), so OUTPUT_EVAL_RUNS defaults to 1. Set it higher for a majority-vote +# stability study; the verdict is then the majority result per case. +# +# GATE: the run passes when at least 90% of cases pass. A case may set +# "must_pass": true; any must-pass failure fails the run regardless of the +# pooled rate, and so does any infrastructure failure. +# +# Exit codes: 0 = pass rate >= 90% with no infrastructure failures and no +# must-pass failures; 1 = pass rate below the gate, a must-pass failure, or +# any infrastructure failure (a missing credential or tool, a malformed +# corpus, an unparseable judge verdict, a permission denial of a tool the +# runner grants, or zero discovered cases); an unmeasured case is never +# absorbed by the gate. +# +# Usage: run-output-evals.sh [--model ] [--case [,...]]... +# --model Subject model. Defaults to +# ${OUTPUT_EVAL_MODEL:-claude-sonnet-5}. Run the suite for +# BOTH claude-sonnet-5 and claude-opus-5 before a PR. +# --case Run only the named case(s); repeatable or comma-separated. +# OUTPUT_EVAL_JUDGE_MODEL Judge model (default claude-sonnet-5). Keep it +# fixed across subject legs so leg diffs are +# attributable to the subject model. +# OUTPUT_EVAL_WORKDIR, when set, receives the transcripts, generated +# projects, and judge bookkeeping in a unique run-XXXXXX subdirectory; +# cleanup only ever removes that subdirectory. Otherwise a mktemp +# directory is used. Kept (and printed) whenever the run fails. +# OUTPUT_EVAL_BROKER_HOST, OUTPUT_EVAL_BROKER_VPN, OUTPUT_EVAL_BROKER_USER, +# OUTPUT_EVAL_BROKER_PASSWORD, when exported, enable the live_verify grader +# (host as tcp://:55555). Export all four or none: a partial set is +# an error. The values reach only verify.sh, as CLI args, never the +# subject model. + +set -uo pipefail +export LC_ALL=C + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +RUNS="${OUTPUT_EVAL_RUNS:-1}" +MODEL="${OUTPUT_EVAL_MODEL:-claude-sonnet-5}" +JUDGE_MODEL="${OUTPUT_EVAL_JUDGE_MODEL:-claude-sonnet-5}" +GRADER_TYPES='["assistant_grep","tool_use","file_exists","file_grep","java_disclaimer","compile","maven_release_match","llm_judge","live_verify"]' +# Headless -p denies tools outside --allowedTools, which would distort the +# measured behavior, so the subject gets the full list the skills declare. Bash +# is unrestricted by design (agreed for this local-only suite); every invocation +# runs in a scratch work dir. The allowlist does not bind the environment: +# managed settings, hooks, and local command shims can still deny a granted +# tool, and a denied tool makes the model improvise (a denied `cp` once became +# a lossy retype of a 55 KB script). Every result event carries +# permission_denials, so the runner reads it: a denial of a granted tool is +# INFRA, and a denial of any other tool is tagged on the case line. +ALLOWED_TOOLS=(Skill Read Glob Grep Write Edit Bash WebFetch TodoWrite) + +CASE_FILTER=() +while [[ $# -gt 0 ]]; do + case "$1" in + --model) + [[ $# -ge 2 ]] || { echo "ERROR: --model requires a value." >&2; exit 1; } + MODEL="$2"; shift 2 ;; + --case) + [[ $# -ge 2 ]] || { echo "ERROR: --case requires a value." >&2; exit 1; } + IFS=',' read -r -a _ids <<<"$2" + [[ ${#_ids[@]} -gt 0 ]] && CASE_FILTER+=("${_ids[@]}") + shift 2 ;; + -h|--help) + sed -n '2,/^[^#]/s/^# \{0,1\}//p' "${BASH_SOURCE[0]}"; exit 0 ;; + *) + echo "ERROR: unknown argument '$1'." >&2; exit 1 ;; + esac +done + +# Dependency and auth pre-flight (fail closed), before any mktemp so an early +# exit leaks nothing. The credential value is never echoed. +command -v claude >/dev/null 2>&1 || { echo "ERROR: the 'claude' CLI is not on PATH. Install @anthropic-ai/claude-code." >&2; exit 1; } +# GNU timeout (or Homebrew coreutils' gtimeout) bounds mvn and verify.sh at +# 600s. Optional: without it those steps run unbounded, and the runner says so +# below whenever a selected case would use it. +TIMEOUT_BIN="$(command -v timeout || command -v gtimeout || true)" +command -v jq >/dev/null 2>&1 || { echo "ERROR: 'jq' is not on PATH." >&2; exit 1; } +command -v curl >/dev/null 2>&1 || { echo "ERROR: 'curl' is not on PATH." >&2; exit 1; } +# Arithmetic reads a zero or non-numeric RUNS as 0, which skips every run and +# trips bash 3.2's empty-array expansion under set -u at the FAIL print. +[[ "$RUNS" =~ ^[1-9][0-9]*$ ]] || { echo "ERROR: OUTPUT_EVAL_RUNS must be a positive integer (got '$RUNS')." >&2; exit 1; } +if [[ -z "${ANTHROPIC_API_KEY:-}" && -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]]; then + echo "ERROR: export ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN before running (a scratch CLAUDE_CONFIG_DIR has no ambient login)." >&2 + exit 1 +fi + +# Live-verify broker (opt-in, tri-state). Count the four exported variables: +# none = skip loudly, all = run, a partial set = error, so a typo in one name can +# never degrade to a silent skip. The values must not reach the subject model +# (the compile anchor asserts compile-only behavior), so the export attribute is +# stripped: they stay shell variables handed to verify.sh. +BROKER_VARS=(OUTPUT_EVAL_BROKER_HOST OUTPUT_EVAL_BROKER_VPN OUTPUT_EVAL_BROKER_USER OUTPUT_EVAL_BROKER_PASSWORD) +missing=() +for v in "${BROKER_VARS[@]}"; do [[ -n "${!v:-}" ]] || missing+=("$v"); done +if [[ ${#missing[@]} -eq 0 ]]; then + LIVE_VERIFY=run +elif [[ ${#missing[@]} -eq ${#BROKER_VARS[@]} ]]; then + LIVE_VERIFY=skip +else + echo "ERROR: partial broker configuration: export all four OUTPUT_EVAL_BROKER_* variables or none (missing: ${missing[*]})." >&2 + exit 1 +fi +export -n "${BROKER_VARS[@]}" + +# in_filter : 0 when no filter is set or the id is listed. +in_filter() { + [[ ${#CASE_FILTER[@]} -eq 0 ]] && return 0 + local want + for want in "${CASE_FILTER[@]}"; do [[ "$want" == "$1" ]] && return 0; done + return 1 +} + +# --- transcript extraction helpers ----------------------------------------- +# All of them read turnN.jsonl files under $RUN_DIR and never the raw stream: +# assistant text and tool_use events are jq-extracted so grep patterns cannot +# false-positive on tool payloads or on the scripted user turns. + +# turn_files : print the transcript paths for the scope, in order. +turn_files() { + local t + if [[ "$1" == "all" ]]; then + for ((t = 1; t <= NTURNS; t++)); do + [[ -e "$RUN_DIR/turn${t}.jsonl" ]] && echo "$RUN_DIR/turn${t}.jsonl" + done + else + [[ -e "$RUN_DIR/turn$1.jsonl" ]] && echo "$RUN_DIR/turn$1.jsonl" + fi + return 0 +} + +assistant_text() { # + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text' "$f" 2>/dev/null + done < <(turn_files "$1") +} + +tool_events() { # : one "namecompact-input-json" line per tool_use + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") + | [.name, (.input | tojson)] | @tsv' "$f" 2>/dev/null + done < <(turn_files "$1") +} + +webfetch_urls() { + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use" and .name=="WebFetch") + | .input.url // (.input | tojson)' "$f" 2>/dev/null + done < <(turn_files all) | sort -u +} + +webfetch_results() { + local f + while IFS= read -r f; do + jq -rs '([.[] | select(.type=="assistant") | .message.content[]? + | select(.type=="tool_use" and .name=="WebFetch") | .id]) as $ids + | .[] | select(.type=="user") | .message.content[]? + | select(.type=="tool_result" and ((.tool_use_id // "") as $t | $ids | index($t))) + | .content + | if type=="array" then (.[]? | select(.type=="text") | .text) + elif type=="string" then . else tostring end' "$f" 2>/dev/null + done < <(turn_files all) +} + +# turn_ok : 0 = a real measurement; 1 = infrastructure failure +# (no result event, or an error other than the turn cap). A turn-cap kill +# (error_max_turns) is not INFRA: the graders still run over what was +# produced, but the case fails with a [max-turns] detail, because a scripted +# turn that never finished is not evidence of the contract. The tag keeps a +# budget problem distinguishable from a behavior failure. +turn_ok() { + local t="$1" subtype is_error + jq -e 'select(.type=="result")' "$t" >/dev/null 2>&1 || return 1 + subtype="$(jq -r 'select(.type=="result") | .subtype' "$t" 2>/dev/null)" + is_error="$(jq -r 'select(.type=="result") | .is_error' "$t" 2>/dev/null)" + [[ "$subtype" == "error_max_turns" ]] && MAXTURNS_HIT=1 + [[ "$is_error" == "true" && "$subtype" != "error_max_turns" ]] && return 1 + return 0 +} + +# turn_reason : one line naming why turn_ok rejected the +# turn, so an INFRA verdict carries its cause instead of only a work-dir path: +# the result subtype and message when a result event exists, else the tail of +# the CLI's stderr capture. +turn_reason() { + local subtype msg err + subtype="$(jq -r 'select(.type=="result") | .subtype // empty' "$1.jsonl" 2>/dev/null | tail -1)" + if [[ -n "$subtype" ]]; then + msg="$(jq -r 'select(.type=="result") | .result // empty' "$1.jsonl" 2>/dev/null | tail -1 | head -c 200)" + echo "result subtype $subtype${msg:+: $msg}" + else + err="$(tail -2 "$1.err" 2>/dev/null | tr '\n' ' ' | head -c 200)" + echo "no result event; stderr: ${err:-empty}" + fi +} + +# denied_tools : one "toolsnippet" line per permission denial +# in the turn's result event. The snippet is the denied command (or the compact +# input), cut short, so the case line names what the environment blocked. +denied_tools() { + jq -r 'select(.type=="result") | .permission_denials[]? + | [.tool_name, ((.tool_input.command // (.tool_input | tojson)) | .[0:120])] | @tsv' "$1" 2>/dev/null +} + +# granted : 0 when the tool is in ALLOWED_TOOLS. +granted() { + local a + for a in "${ALLOWED_TOOLS[@]}"; do [[ "$a" == "$1" ]] && return 0; done + return 1 +} + +# find_files : generated files under the work dir, excluding +# Maven build output. "*" means any file. +find_files() { + find "$WORK" -type f -name "$1" -not -path '*/target/*' 2>/dev/null +} + +shallowest_pom() { + find_files pom.xml | awk -F/ '{print NF, $0}' | sort -n | head -1 | cut -d' ' -f2- +} + +resolve_sol_jcsmp_release() { + curl -s --max-time 20 https://repo1.maven.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata.xml \ + | grep -oE '[^<]+' | sed -E 's/<\/?release>//g' +} + +# --- graders ---------------------------------------------------------------- +# grade_one : 0 pass, 1 fail (a measurement), 2 infrastructure. +# Failure/infra explanation lands in GRADER_DETAIL. +GRADER_DETAIL="" +grade_one() { + local g="$1" gtype + gtype="$(jq -r '.type' <<<"$g")" + GRADER_DETAIL="" + + case "$gtype" in + assistant_grep) + local pattern expect fixed turn text found=0 gopts=(-q) + pattern="$(jq -r '.pattern' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + fixed="$(jq -r '.fixed // false' <<<"$g")" + turn="$(jq -r '.turn // "all"' <<<"$g")" + [[ "$fixed" == "true" ]] && gopts+=(-F) || gopts+=(-E) + text="$(assistant_text "$turn")" + grep "${gopts[@]}" -- "$pattern" <<<"$text" && found=1 + if [[ "$expect" == "present" && "$found" -eq 0 ]]; then + GRADER_DETAIL="assistant text (turn $turn) lacks: $pattern"; return 1 + elif [[ "$expect" == "absent" && "$found" -eq 1 ]]; then + GRADER_DETAIL="assistant text (turn $turn) contains forbidden: $pattern"; return 1 + fi + return 0 ;; + + tool_use) + local tool ipat xpat expect count turn events matches m + tool="$(jq -r '.tool' <<<"$g")" + ipat="$(jq -r '.input_pattern // ""' <<<"$g")" + xpat="$(jq -r '.exclude_pattern // ""' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + count="$(jq -r '.count // ""' <<<"$g")" + turn="$(jq -r '.turn // "all"' <<<"$g")" + events="$(tool_events "$turn")" + if [[ "$tool" == "*" ]]; then matches="$events" + else matches="$(awk -F'\t' -v t="$tool" '$1 == t' <<<"$events")"; fi + if [[ -n "$ipat" ]]; then matches="$(grep -E -- "$ipat" <<<"$matches" || true)"; fi + if [[ -n "$xpat" ]]; then matches="$(grep -vE -- "$xpat" <<<"$matches" || true)"; fi + m="$(grep -c . <<<"$matches" || true)" + if [[ "$expect" == "present" ]]; then + if [[ -n "$count" ]]; then + [[ "$m" -eq "$count" ]] && return 0 + GRADER_DETAIL="tool $tool${ipat:+ ~ $ipat} used $m time(s), expected $count"; return 1 + fi + [[ "$m" -gt 0 ]] && return 0 + GRADER_DETAIL="tool $tool${ipat:+ ~ $ipat} never used"; return 1 + else + [[ "$m" -eq 0 ]] && return 0 + GRADER_DETAIL="forbidden tool use: $tool${ipat:+ ~ $ipat} ($m match(es))"; return 1 + fi ;; + + file_exists) + local glob expect ident files f ok=1 + glob="$(jq -r '.glob' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + ident="$(jq -r '.identical_to // ""' <<<"$g")" + files="$(find_files "$glob")" + if [[ "$expect" == "absent" ]]; then + [[ -z "$files" ]] && return 0 + GRADER_DETAIL="forbidden file(s) on disk matching $glob: $(head -3 <<<"$files" | tr '\n' ' ')"; return 1 + fi + [[ -z "$files" ]] && { GRADER_DETAIL="no file matching $glob was generated"; return 1; } + if [[ -n "$ident" ]]; then + while IFS= read -r f; do + cmp -s "$f" "$REPO_ROOT/$ident" || { ok=0; GRADER_DETAIL="$f is not byte-identical to $ident"; } + done <<<"$files" + [[ "$ok" -eq 1 ]] || return 1 + fi + return 0 ;; + + file_grep) + local glob pattern expect fixed files f gopts=(-q) matched=0 total=0 miss="" + glob="$(jq -r '.glob' <<<"$g")" + pattern="$(jq -r '.pattern' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + fixed="$(jq -r '.fixed // false' <<<"$g")" + [[ "$fixed" == "true" ]] && gopts+=(-F) || gopts+=(-E) + files="$(find_files "$glob")" + if [[ -z "$files" ]]; then + [[ "$expect" == "absent" ]] && return 0 + GRADER_DETAIL="no file matching $glob to grep for: $pattern"; return 1 + fi + while IFS= read -r f; do + total=$((total + 1)) + if grep "${gopts[@]}" -- "$pattern" "$f"; then matched=$((matched + 1)); else miss="$f"; fi + done <<<"$files" + case "$expect" in + present) + [[ "$matched" -gt 0 ]] && return 0 + GRADER_DETAIL="$glob file(s) lack: $pattern"; return 1 ;; + absent) + [[ "$matched" -eq 0 ]] && return 0 + GRADER_DETAIL="$glob file(s) contain forbidden: $pattern"; return 1 ;; + all_files) + [[ "$matched" -eq "$total" ]] && return 0 + GRADER_DETAIL="$miss lacks: $pattern"; return 1 ;; + esac ;; + + java_disclaimer) + local files f bad="" + files="$(find_files '*.java')" + [[ -z "$files" ]] && { GRADER_DETAIL="no .java files were generated"; return 1; } + while IFS= read -r f; do + head -3 "$f" | grep -qF 'AI-assisted code. Review before production use.' || bad="$f (missing disclaimer line)" + head -4 "$f" | grep -qF 'See the verification checklist: solace-verification-checklist.md' || bad="$f (missing checklist pointer)" + done <<<"$files" + [[ -z "$bad" ]] && return 0 + GRADER_DETAIL="$bad"; return 1 ;; + + compile) + # Every generated pom compiles, so a two-project layout (Solace Suggested) + # is proven whole, not by whichever pom sorts first. + local poms pom rc n=0 log + poms="$(find_files pom.xml)" + [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } + while IFS= read -r pom; do + n=$((n + 1)); log="$RUN_DIR/mvn-compile-$n.log" + if [[ -n "$TIMEOUT_BIN" ]]; then + "$TIMEOUT_BIN" 600 mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? + else + mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? + fi + # timeout's own exit 124 is a wall-clock kill, not a measurement of the code. + [[ "$rc" -eq 124 ]] && { GRADER_DETAIL="mvn compile for $pom timed out after 600s, log: $log"; return 2; } + [[ "$rc" -eq 0 ]] || { GRADER_DETAIL="mvn compile failed for $pom (exit $rc), log: $log"; return 1; } + done <<<"$poms" + return 0 ;; + + maven_release_match) + local rel poms pom + rel="$(resolve_sol_jcsmp_release)" + [[ -z "$rel" ]] && { GRADER_DETAIL="could not resolve sol-jcsmp from repo1.maven.org"; return 2; } + poms="$(find_files pom.xml)" + [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to check against release $rel"; return 1; } + while IFS= read -r pom; do + grep -qF "$rel" "$pom" || { GRADER_DETAIL="$pom does not carry the live sol-jcsmp release $rel"; return 1; } + done <<<"$poms" + return 0 ;; + + llm_judge) + local criteria includes pfile jfile attempt raw verdict reason + criteria="$(jq -r '.criteria' <<<"$g")" + includes="$(jq -r '(.include // ["assistant_text"]) | join(" ")' <<<"$g")" + JUDGE_N=$((JUDGE_N + 1)) + pfile="$RUN_DIR/judge${JUDGE_N}.prompt" + jfile="$RUN_DIR/judge${JUDGE_N}.json" + { + echo 'You are grading an AI assistant transcript against one criterion.' + echo 'Respond with ONLY a JSON object: {"verdict":"pass"|"fail","reason":""}.' + echo 'The verdict is "fail" when the criterion is violated, "pass" otherwise.' + echo + echo "CRITERION: $criteria" + if grep -qw assistant_text <<<"$includes"; then + echo; echo '--- ASSISTANT TEXT (all turns, in order) ---' + assistant_text all | head -c 51200 + fi + if grep -qw webfetch_urls <<<"$includes"; then + echo; echo '--- URLS FETCHED VIA WebFetch (deduplicated; empty means none) ---' + webfetch_urls + fi + if grep -qw webfetch_results <<<"$includes"; then + echo; echo '--- WebFetch RESULT CONTENT (truncated) ---' + echo 'This is the summary the WebFetch tool returned to the assistant for its prompt, not the page itself. A remark in it that something is absent describes the summary, not the page.' + webfetch_results | head -c 30720 + fi + } > "$pfile" + for attempt in 1 2; do + # --tools "" keeps the judge tool-free (a tool call would spend its only + # turn and leave no verdict); the cd keeps the operator's project + # settings out of the judge session, as $WORK does for the subject. + ( cd "$RUN_DIR" && claude -p "$(cat "$pfile")" --model "$JUDGE_MODEL" --max-turns 1 \ + --tools "" --output-format json < /dev/null > "$jfile" 2>"$RUN_DIR/judge${JUDGE_N}.err" ) + raw="$(jq -r '.result // empty' "$jfile" 2>/dev/null | tr '\n' ' ' | grep -oE '\{[^{}]*"verdict"[^{}]*\}' | head -1)" + verdict="$(jq -r '.verdict // empty' <<<"$raw" 2>/dev/null)" + reason="$(jq -r '.reason // empty' <<<"$raw" 2>/dev/null)" + if [[ "$verdict" == "pass" ]]; then return 0; fi + if [[ "$verdict" == "fail" ]]; then GRADER_DETAIL="judge: ${reason:-no reason given}"; return 1; fi + done + GRADER_DETAIL="judge verdict unparseable after 2 attempts ($jfile)"; return 2 ;; + + live_verify) + local pom proj log rc + if [[ "$LIVE_VERIFY" != "run" ]]; then LIVE_SKIPPED=1; return 0; fi + pom="$(shallowest_pom)" + [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to verify against the broker"; return 1; } + proj="$(dirname "$pom")" + [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $pom"; return 1; } + log="$RUN_DIR/live-verify.log" + LIVE_RAN=1 + if [[ -n "$TIMEOUT_BIN" ]]; then + ( cd "$proj" && "$TIMEOUT_BIN" 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + else + ( cd "$proj" && bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + fi + case "$rc" in + 0) return 0 ;; + 2) GRADER_DETAIL="verify.sh roundtrip hit a broker or credential error (exit 2), log: $log"; return 2 ;; + 124) GRADER_DETAIL="verify.sh roundtrip timed out after 600s, log: $log"; return 2 ;; + *) GRADER_DETAIL="verify.sh roundtrip failed (exit $rc), log: $log"; return 1 ;; + esac ;; + + *) + GRADER_DETAIL="unknown grader type '$gtype'"; return 2 ;; + esac +} + +# --- scratch dirs (created only after the guards pass) ----------------------- +export CLAUDE_CONFIG_DIR +CLAUDE_CONFIG_DIR="$(mktemp -d)" +if [[ -n "${OUTPUT_EVAL_WORKDIR:-}" ]]; then + if ! mkdir -p "$OUTPUT_EVAL_WORKDIR" \ + || ! WORKDIR="$(cd "$OUTPUT_EVAL_WORKDIR" && pwd)" \ + || ! WORKDIR="$(mktemp -d "$WORKDIR/run-XXXXXX")"; then + echo "ERROR: cannot prepare OUTPUT_EVAL_WORKDIR '$OUTPUT_EVAL_WORKDIR'." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" + exit 1 + fi +else + WORKDIR="$(mktemp -d)" +fi + +pass=0; fail=0; infra=0; must_fail=0; LIVE_NEEDED=0; LIVE_RAN=0 + +for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do + [[ -e "$evals_file" ]] || continue + # Fail closed on a malformed corpus: a jq parse error inside the process + # substitution feeding the loop below is otherwise unobservable, so a broken + # file would silently contribute zero cases. + if ! jq -e --argjson types "$GRADER_TYPES" ' + type=="array" and length>0 + and ([.[].id] | length == (unique | length)) + and all(.[]; + (.id | type=="string" and length>0) + and (.skill | type=="string" and length>0) + and (.user_turns | type=="array" and length>0 and all(.[]; type=="string" and length>0)) + and ((.must_pass // false) | type=="boolean") + and ((.max_agent_turns // 25) | type=="number") + and (.graders | type=="array" and length>0 + and all(.[]; type=="object" and (.type as $t | $types | index($t)))))' \ + "$evals_file" >/dev/null 2>&1; then + echo "ERROR: $evals_file is not a non-empty array of {id, skill, user_turns, graders[, max_agent_turns, must_pass]} cases with unique ids and known grader types." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + plugin_dir="$(cd "$(dirname "$evals_file")/.." && pwd)" # plugins/ + + # Unknown --case ids are an error, not a silent zero-case run. + if [[ ${#CASE_FILTER[@]} -gt 0 ]]; then + for want in "${CASE_FILTER[@]}"; do + jq -e --arg id "$want" 'any(.[]; .id == $id)' "$evals_file" >/dev/null 2>&1 \ + || { echo "ERROR: --case '$want' not found in $evals_file." >&2; rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR"; exit 1; } + done + fi + + # Maven is only required when a selected case compiles, checks the pom, or + # runs verify.sh. The live-verify banner prints before any case starts so the + # operator sees the broker state in the first seconds, not at the end of the run. + needs_mvn=0; needs_live=0 + while IFS= read -r row; do + cid="$(jq -r '.id' <<<"$row")" + in_filter "$cid" || continue + jq -e '.graders | any(.type=="compile" or .type=="maven_release_match" or .type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_mvn=1 + jq -e '.graders | any(.type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_live=1 + done < <(jq -c '.[]' "$evals_file") + if [[ "$needs_mvn" -eq 1 ]] && ! command -v mvn >/dev/null 2>&1; then + echo "ERROR: 'mvn' is not on PATH and a selected case carries a compile, maven_release_match, or live_verify grader." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + if [[ "$needs_mvn" -eq 1 && -z "$TIMEOUT_BIN" ]]; then + echo "WARNING: neither 'timeout' nor 'gtimeout' is on PATH; mvn compile and verify.sh run unbounded (install GNU coreutils to cap them at 600s)." >&2 + fi + if [[ "$needs_live" -eq 1 ]]; then + LIVE_NEEDED=1 + if [[ "$LIVE_VERIFY" == "run" ]]; then + echo "live verify: ENABLED against $OUTPUT_EVAL_BROKER_HOST" + else + echo "live verify: SKIPPED (export the four OUTPUT_EVAL_BROKER_* variables to enable)" + fi + fi + + while IFS= read -r case_json; do + case_id="$(jq -r '.id' <<<"$case_json")" + in_filter "$case_id" || continue + skill="$(jq -r '.skill' <<<"$case_json")" + must="$(jq -r '.must_pass // false' <<<"$case_json")" + max_turns="$(jq -r '.max_agent_turns // 25' <<<"$case_json")" + NTURNS="$(jq -r '.user_turns | length' <<<"$case_json")" + + run_pass=0; case_infra=0; fail_details=(); other_denials="" + for ((k = 1; k <= RUNS; k++)); do + # One whole-case retry on an infrastructure failure, each attempt from a + # fresh work dir (no mid-conversation resume of a failed turn). The retry + # is announced, so a flaky leg is distinguishable from a clean one. + attempt_ok=0; turn_detail=""; denial_infra=0 + for attempt in 1 2; do + RUN_DIR="$WORKDIR/${case_id}/run${k}-try${attempt}" + WORK="$RUN_DIR/work" + mkdir -p "$WORK" + MAXTURNS_HIT=0 + sid="" + turn_infra=0 + for ((t = 1; t <= NTURNS; t++)); do + turn_prompt="$(jq -r ".user_turns[$((t - 1))]" <<<"$case_json")" + args=(-p "$turn_prompt" --plugin-dir "$plugin_dir" --model "$MODEL" + --max-turns "$max_turns" --allowedTools "${ALLOWED_TOOLS[@]}" + --output-format stream-json --verbose) + [[ -n "$sid" ]] && args+=(--resume "$sid") + ( cd "$WORK" && claude "${args[@]}" \ + < /dev/null > "$RUN_DIR/turn${t}.jsonl" 2>"$RUN_DIR/turn${t}.err" ) + if ! turn_ok "$RUN_DIR/turn${t}.jsonl"; then + turn_infra=1 + turn_detail="turn $t (try $attempt): $(turn_reason "$RUN_DIR/turn${t}")" + break + fi + # Permission denials. A denied GRANTED tool means the environment, not + # the skill, shaped this output, and the same policy would deny the + # retry too, so the case is INFRA at once. A denial of any other tool + # is the model's own doing and is only tagged on the case line. + while IFS=$'\t' read -r dtool dcmd; do + [[ -z "$dtool" ]] && continue + if granted "$dtool"; then + denial_infra=1 + turn_detail="turn $t: the environment denied $dtool, a tool the runner grants ($dcmd)" + else + other_denials+="$dtool"$'\n' + fi + done < <(denied_tools "$RUN_DIR/turn${t}.jsonl") + [[ "$denial_infra" -eq 1 ]] && break + # Re-extract after every turn: a print-mode resume can mint a new id. + new_sid="$(jq -r 'select(.type=="result") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | tail -1)" + [[ -z "$new_sid" ]] && new_sid="$(jq -r 'select(.type=="system" and .subtype=="init") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | head -1)" + [[ -n "$new_sid" ]] && sid="$new_sid" + done + [[ "$denial_infra" -eq 1 ]] && break + if [[ "$turn_infra" -eq 0 ]]; then attempt_ok=1; break; fi + [[ "$attempt" -eq 1 ]] && echo "RETRY [$case_id] ($skill) :: $turn_detail" + done + if [[ "$attempt_ok" -ne 1 ]]; then case_infra=1; fail_details=("$turn_detail"); break; fi + + # Implicit gate: the target skill must have fired, or the output is not + # attributable to it and every absence grader would pass vacuously. + fired="$(turn_files all | while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? + | select(.type=="tool_use" and .name=="Skill") | .input.skill' "$f" 2>/dev/null + done | sed 's/.*://' | sort -u)" + this_run_details=() + if ! grep -qxF "$skill" <<<"$fired"; then + this_run_details+=("the '$skill' skill never fired (fired: ${fired:-none})") + fi + + JUDGE_N=0 + LIVE_SKIPPED=0 + gi=0 + while IFS= read -r grader; do + gi=$((gi + 1)) + gtype="$(jq -r '.type' <<<"$grader")" + grade_one "$grader"; rc=$? + if [[ "$rc" -eq 2 ]]; then case_infra=1; this_run_details+=("grader#$gi $gtype INFRA :: $GRADER_DETAIL"); break + elif [[ "$rc" -eq 1 ]]; then this_run_details+=("grader#$gi $gtype :: $GRADER_DETAIL"); fi + done < <(jq -c '.graders[]' <<<"$case_json") + [[ "$case_infra" -eq 1 ]] && { fail_details=("${this_run_details[@]}"); break; } + + [[ "$MAXTURNS_HIT" -eq 1 ]] && this_run_details+=("[max-turns] a user turn hit the max_agent_turns cap of $max_turns") + if [[ ${#this_run_details[@]} -eq 0 ]]; then + run_pass=$((run_pass + 1)) + else + fail_details=("${this_run_details[@]}") + fi + done + + denial_tag="" + [[ -n "$other_denials" ]] && denial_tag=" [denied outside the allowlist: $(sort -u <<<"$other_denials" | grep . | paste -sd' ' -)]" + + if [[ "$case_infra" -eq 1 ]]; then + echo "FAIL [$case_id] ($skill) INFRA${fail_details[0]:+ :: ${fail_details[0]}}$denial_tag" + fail=$((fail + 1)); infra=$((infra + 1)); continue + fi + + # Majority vote; with RUNS=1 this is simply "the single run passed". + live_tag="" + [[ "${LIVE_SKIPPED:-0}" -eq 1 ]] && live_tag=" [live verify skipped: no broker configured]" + if (( run_pass * 2 > RUNS )); then + echo "PASS [$case_id] ($skill)$live_tag$denial_tag" + pass=$((pass + 1)) + else + tag="" + [[ "$must" == "true" ]] && { must_fail=$((must_fail + 1)); tag=" [must-pass]"; } + echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)$live_tag$denial_tag" + for d in "${fail_details[@]}"; do echo " - $d"; done + fail=$((fail + 1)) + fi + done < <(jq -c '.[]' "$evals_file") +done + +total=$((pass + fail)) +echo +if [[ "$total" -eq 0 ]]; then + echo "ERROR: no output-eval cases discovered under plugins/*/evals/output-evals.json." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 +fi +echo "$pass passed, $fail failed ($((pass * 100 / total))% pass rate, gate is 90%)" +[[ "$must_fail" -gt 0 ]] && echo "$must_fail must-pass case(s) failed (any must-pass failure fails the run)" +if [[ "$LIVE_NEEDED" -eq 1 ]]; then + if [[ "$LIVE_VERIFY" == "run" && "$LIVE_RAN" -eq 1 ]]; then + echo "live verify: ran against $OUTPUT_EVAL_BROKER_HOST (see the case line for its verdict)" + elif [[ "$LIVE_VERIFY" == "run" ]]; then + echo "live verify: enabled against $OUTPUT_EVAL_BROKER_HOST but never executed (the case failed before grading)" + else + echo "live verify: skipped (no broker configured); say so in the PR record" + fi +fi + +rm -rf "$CLAUDE_CONFIG_DIR" +if [[ "$infra" -gt 0 || "$must_fail" -gt 0 ]] || (( pass * 100 < total * 90 )); then + echo "transcripts and generated projects kept for inspection under: $WORKDIR" >&2 + exit 1 +fi +rm -rf "$WORKDIR" +exit 0