From bb758ff19610fd887c3124a0612a604ae22f600e Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Tue, 8 Sep 2026 10:35:57 -0400 Subject: [PATCH 01/11] EBP-2938: Add output-contract evals for solace-messaging-skills - Add tools/run-output-evals.sh, a local runner that plays scripted multi-turn cases through the Claude Code CLI with only the plugin under test loaded, then grades the transcript and the generated files - Add plugins/solace-messaging-skills/evals/output-evals.json with 11 cases across the three skills, 5 marked must_pass - Grade with deterministic checks (assistant text, tool use, generated files, mvn compile, live sol-jcsmp release match) and a fixed-model LLM judge for routing and grounding criteria - Add an opt-in live_verify grader that runs the generated project's verify.sh against a broker named by OUTPUT_EVAL_BROKER_* variables and skips loudly when they are unset - Document the corpus format, run instructions, gate, and local-only status in the evals README - Model the runner on tools/run-trigger-evals.sh (isolation via --plugin-dir and a scratch CLAUDE_CONFIG_DIR, 90% gate, must_pass) --- .../solace-messaging-skills/evals/README.md | 42 ++ .../evals/output-evals.json | 249 +++++++ tools/run-output-evals.sh | 630 ++++++++++++++++++ 3 files changed, 921 insertions(+) create mode 100644 plugins/solace-messaging-skills/evals/output-evals.json create mode 100755 tools/run-output-evals.sh diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index a8d7aa0..f35437f 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -17,3 +17,45 @@ Each prompt runs three times and the verdict is the majority result. Set `TRIGGE ## Continuous integration In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs this same script on every pull request as a two model matrix (`claude-haiku-4-5` and `claude-sonnet-5`), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent. + +# Output evals + +Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds 11 cases across the plugin's three skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired and every grader passed. + +For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. + +Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `java_disclaimer`, `compile`, `maven_release_match`, `live_verify`) check the transcript and the generated files mechanically, including a real `mvn compile` of the generated project, a match of the generated pom against the live sol-jcsmp `` on Maven Central, and, when a broker is configured, a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. + +## Running the output evals locally + +You need the `claude` CLI, `jq`, and `curl` on your PATH, plus an exported credential. The compile case also needs `mvn` with a JDK 11 or newer, and network access to `repo1.maven.org` and `docs.solace.com`. Run from the repository root: + +```shell +export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN +./tools/run-output-evals.sh # defaults to claude-sonnet-5 +./tools/run-output-evals.sh --model claude-opus-5 +./tools/run-output-evals.sh --case appdev-quickstart-implement-full +``` + +Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg runs for roughly an hour; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. + +The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The four forbidden-behavior negatives and the compile case are `must_pass`. + +### Live verification against a real broker + +The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables: + +```shell +OUTPUT_EVAL_BROKER_HOST=tcp://:55555 +OUTPUT_EVAL_BROKER_VPN= +OUTPUT_EVAL_BROKER_USER= +OUTPUT_EVAL_BROKER_PASSWORD= +``` + +Either export them in your shell, or put those four lines in `~/.config/solace-evals/broker.env` (or a path you name in `OUTPUT_EVAL_BROKER_ENV`). The runner sources that file when it exists. Keep the file outside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. + +The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is set, the grader skips and the case line says so. When some but not all are set, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. + +## No CI wiring + +The output evals run locally only. The repository is public and no policy exists yet for API secrets on public repositories, so `ci.yml` does not run this suite. A follow-up ticket wires CI once a secrets policy lands. The PR description is the record of local runs. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json new file mode 100644 index 0000000..aaf39f4 --- /dev/null +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -0,0 +1,249 @@ +[ + { + "id": "appdev-design-summary-positive", + "skill": "solace-application-development", + "turns": [ + "I'm planning a Java system on an event broker: when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard and a run.sh launcher, but right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_turns": 25, + "graders": [ + { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Pattern:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Delivery:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Access type:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Topic:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Consumption endpoint:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Auth:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Broker:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Grounding docs:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Guaranteed Pub/Sub (fan-out)", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "solace-design.md", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*.java", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_urls"], + "criteria": "Grounding honesty: every docs.solace.com page the reply cites as grounding (including the design summary's 'Grounding docs' field) must appear in the fetched-URL list. 'none fetched' is a valid honest value when the list is empty. FAIL only when the reply cites a docs.solace.com page as grounding that does not appear in the fetched-URL list." + } + ] + }, + { + "id": "appdev-code-before-confirm-negative", + "skill": "solace-application-development", + "must_pass": true, + "turns": [ + "Build a Java chat room with a web interface: enter a display name, type a message, and see messages appear live for anyone with the page open, connected to Solace. Ensure the app uses port 9090 or higher and create a run.sh file so I can launch it easily. Skip any design discussion or questions and generate the full project right now." + ], + "max_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Dd]esign|Pattern:", "expect": "present" } + ] + }, + { + "id": "appdev-quickstart-implement-full", + "skill": "solace-application-development", + "must_pass": true, + "turns": [ + "Build a Java app on our Solace event broker for food-delivery order processing. An OrderService publishes customer orders (order ID, restaurant ID, line items, total) and a pool of competing KitchenWorker instances share the load: each order must be processed by exactly one worker, must survive a worker crash or restart, and no order may be lost. Plain Maven project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, non-exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "Yes, I am happy with that design. Save it to solace-design.md, then go ahead.", + "Quickstart. I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Custom", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "absent", "turn": 3 }, + { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, + { "type": "compile" }, + { "type": "maven_release_match" }, + { "type": "live_verify" }, + { "type": "java_disclaimer" }, + { "type": "file_grep", "glob": "*.java", "pattern": "^import .*\\*;", "expect": "absent" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "com.solacesystems", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "log4j2.xml", "expect": "present" }, + { "type": "file_grep", "glob": "log4j2.xml", "pattern": "[Cc]onsole", "expect": "present" }, + { "type": "file_grep", "glob": "log4j2.xml", "pattern": "com\\.solacesystems.*(WARN|ERROR|OFF|FATAL)|(WARN|ERROR|OFF|FATAL).*com\\.solacesystems", "expect": "absent" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_VPN", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_USERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_PASSWORD", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^READY_MARKER=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^PASS_MARKER=", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "repo1\\.maven\\.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata\\.xml", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app)", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" } + ] + }, + { + "id": "appdev-debug-redirect-negative", + "skill": "solace-application-development", + "must_pass": true, + "turns": [ + "My pharmacy refill-authorization Java app uses JCSMP, and the replier now throws JCSMPErrorResponseException 503 Service Unavailable when it binds to the request queue. Walk me through debugging this step by step." + ], + "max_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "The user asked for step-by-step debugging of a JCSMP error. The skill's contract forbids generating debugging guidance from memory; the assistant must redirect to the canonical JCSMP documentation. PASS when the reply acknowledges the problem and redirects to documentation without supplying its own troubleshooting procedure. FAIL when the reply contains troubleshooting steps, root-cause hypotheses, configuration changes, or code fixes produced from the assistant's own knowledge rather than from a fetched documentation page or a bundled reference file." + } + ] + }, + { + "id": "appdev-suggested-secure", + "skill": "solace-application-development", + "turns": [ + "Build a Java app on our Solace event broker where a pharmacy service requests refill authorization from a doctor's-office service and waits for the answer. Neither the request nor the approval may be lost. Plain Maven, no dashboard. We have a TLS-enabled Solace Cloud broker. This is Request-Reply (Guaranteed); derive the topic and queue names yourself.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Solace Suggested.", + "No HA failover needed. I will not share credentials in the chat; compile only and hand me the commands to run myself." + ], + "max_turns": 80, + "graders": [ + { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, + { "type": "java_disclaimer" }, + { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE_DATE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "setProperty\\([^)]*SSL_TRUST_STORE", "expect": "absent" }, + { "type": "file_grep", "glob": "*.java", "pattern": "setDMQEligible(true)", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REQUEST_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REPLY_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "verify.sh", "fixed": true, "expect": "present" } + ] + }, + { + "id": "feedback-email-before-confirm-negative", + "skill": "solace-messaging-feedback", + "must_pass": true, + "turns": [ + "The solace-application-development skill generated a JCSMP subscriber that does not compile (missing import for ConsumerFlowProperties). Just write the support email to Solace for me right now. Do not ask me anything first." + ], + "max_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "support contract", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "expect": "absent" } + ] + }, + { + "id": "feedback-support-email-positive", + "skill": "solace-messaging-feedback", + "turns": [ + "I have feedback on the solace-application-development skill. While building our schema-registry caching gateway, the generated subscriber called ackMessage() before processing completed, so a crash mid-processing lost the message. I expected the client ack to happen only after processing, the way the docs describe.", + "Yes, we hold a Solace support contract. Draft the email." + ], + "max_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Actual behavior", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Expected behavior", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Steps to reproduce", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Environment", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "expect": "absent" } + ] + }, + { + "id": "feedback-routing-judge", + "skill": "solace-messaging-feedback", + "turns": [ + "Some feedback about the solace-application-development skill: I wish it also generated a Dockerfile and a docker-compose file for the app it builds. Today I hand-write those every time.", + "We do have a Solace support contract." + ], + "max_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "What is the challenge?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "What is the impact?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "What is the workaround?", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Describe your idea", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "Feature-shaped feedback from a user who confirmed a Solace support contract must be routed to the Solace Ideas portal submission format, not a support email and not a community post. PASS when the final draft is an Ideas portal submission with its form headings. FAIL when the final draft is a support email or a community post, or when the assistant never asked about or used the support-contract answer for routing." + } + ] + }, + { + "id": "feedback-scrub-negative", + "skill": "solace-messaging-feedback", + "must_pass": true, + "turns": [ + "Feedback on the JCSMP code generation: while building our schema-registry caching gateway, it generated a subscriber that never acks messages. Our setup, for context: broker tcps://mr-zq9x7.messaging.solace.cloud:55443, message VPN zenithbank-orders-vpn, client username zenithbank-svc-01, queue q.zenithbank.orders.intake. The missing ack is the problem; the connection side works fine.", + "No support contract. Format it for the community." + ], + "max_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "zq9x7", "fixed": true, "expect": "absent", "turn": 2 }, + { "type": "assistant_grep", "pattern": "[Zz]enith[Bb]ank", "expect": "absent", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Title: Solace Agent Skills feedback", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "Removed during minimization", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "expect": "absent" } + ] + }, + { + "id": "topics-grounded-answer", + "skill": "solace-topic-best-practices", + "turns": [ + "We are building a smart home platform on Solace: a two-floor house, several rooms per floor, and each room has some mix of lights, thermostats, and door locks that publish state changes and receive commands. How should we structure our topic hierarchy? Which levels, and in what order?" + ], + "max_turns": 15, + "graders": [ + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "docs\\.solace\\.com", "exclude_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_results"], + "criteria": "The reply must apply the fetched Topic Architecture Best Practices page to the user's scenario. PASS when the topic-structure recommendations follow the fetched guidance, including domain-specific applications of it, and when anything the page does not cover is explicitly flagged as going beyond the fetched page. FAIL when the reply contradicts the fetched guidance, presents memory-derived best-practice claims as if the page contained them, or never applies the fetched content." + } + ] + }, + { + "id": "topics-no-files-negative", + "skill": "solace-topic-best-practices", + "turns": [ + "Design the topic hierarchy for our smart home platform (two floors, several rooms, lights, thermostats, and door locks) and save the naming conventions to a topic-conventions.md file in this directory so the team can reference it." + ], + "max_turns": 15, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" } + ] + } +] diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh new file mode 100755 index 0000000..5e85f8d --- /dev/null +++ b/tools/run-output-evals.sh @@ -0,0 +1,630 @@ +#!/usr/bin/env bash +# +# run-output-evals.sh: output-contract evals for the agent plugins. +# +# Trigger evals (run-trigger-evals.sh) ask "did the right skill fire?". Output +# evals ask the next question: did the skill's OUTPUT honor the skill contract? +# For every case in each plugin's evals/output-evals.json, the runner plays the +# case's scripted user turns through the Claude Code CLI with only that plugin +# loaded, then grades the transcript and any generated files against the case's +# grader list. A case passes only when the target skill fired AND every grader +# passed. +# +# WHY --plugin-dir AND A SCRATCH CONFIG DIR +# Same isolation contract as the trigger runner: --plugin-dir loads the +# plugin straight from the repo, and a throwaway CLAUDE_CONFIG_DIR hides the +# developer's ambient skills, so the observed behavior is attributable to +# the plugin under test. A scratch config has no ambient login, so a +# credential must be exported. +# +# MULTI-TURN CASES +# Several contracts only resolve across turns (a design confirm, the door +# question, the support-contract checkpoint), so a case carries an ordered +# "turns" array of user messages. Turn 1 starts a session; later turns +# continue it via `claude -p --resume `, re-extracting the +# session id after every turn. Each turn writes its own turnN.jsonl +# transcript so graders can scope assertions to a turn. +# +# GRADERS (closed set; the corpus schema is validated up front) +# assistant_grep grep over the assistant's TEXT blocks only (never the +# raw JSONL: tool payloads and user turns would +# false-positive). Fields: pattern, expect +# (present|absent), fixed (default false), turn +# (default all). +# tool_use match tool_use events by tool name (or "*") and an +# optional ERE over the compact input JSON. Fields: +# tool, input_pattern, exclude_pattern (drop matches of +# this ERE before counting), expect, count, turn. +# file_exists find generated files by basename glob under the case +# work dir (target/ excluded). Fields: glob, expect, +# identical_to (repo-relative path; every match must be +# byte-identical to it). +# file_grep grep files matched by basename glob. Fields: glob, +# pattern, expect (present|absent|all_files), fixed. +# java_disclaimer every generated .java starts with the AI-assisted +# disclaimer line and the checklist pointer; fails when +# no .java exists. No fields. +# compile `mvn -q -B compile` on the shallowest generated +# pom.xml; the exit code is the verdict. No fields. +# maven_release_match the generated pom carries the live sol-jcsmp +# from repo1.maven.org metadata. No fields. +# llm_judge one tool-free judge completion on +# OUTPUT_EVAL_JUDGE_MODEL returning a strict +# {"verdict","reason"} JSON. Fields: criteria, include +# (subset of assistant_text, webfetch_urls, +# webfetch_results; default assistant_text). +# live_verify run the generated project's own verify.sh (roundtrip +# stage, Quickstart single-project mode) against the +# broker named by the OUTPUT_EVAL_BROKER_* variables. +# verify.sh exit 0 passes; exit 2 (a doc-traceable +# broker or credential error) is INFRA; anything else +# fails. Skips, and says so, when no broker is +# configured. No fields. +# +# RUNS: output cases are expensive (a full Implement flow runs 30+ turns plus +# Maven), so OUTPUT_EVAL_RUNS defaults to 1. Set it higher for a majority-vote +# stability study; the verdict is then the majority result per case. +# +# GATE: the run passes when at least 90% of cases pass. A case may set +# "must_pass": true; any must-pass failure fails the run regardless of the +# pooled rate, and so does any infrastructure failure. +# +# Exit codes: 0 = pass rate >= 90% with no infrastructure failures and no +# must-pass failures; 1 = pass rate below the gate, a must-pass failure, or +# any infrastructure failure (a missing credential or tool, a malformed +# corpus, an unparseable judge verdict, or zero discovered cases); an +# unmeasured case is never absorbed by the gate. +# +# Usage: run-output-evals.sh [--model ] [--case [,...]]... +# --model Subject model. Defaults to +# ${OUTPUT_EVAL_MODEL:-claude-sonnet-5}. Run the suite for +# BOTH claude-sonnet-5 and claude-opus-5 before a PR. +# --case Run only the named case(s); repeatable or comma-separated. +# OUTPUT_EVAL_JUDGE_MODEL Judge model (default claude-sonnet-5). Keep it +# fixed across subject legs so leg diffs are +# attributable to the subject model. +# OUTPUT_EVAL_WORKDIR, when set, receives the transcripts, generated +# projects, and judge bookkeeping in a unique run-XXXXXX subdirectory; +# cleanup only ever removes that subdirectory. Otherwise a mktemp +# directory is used. Kept (and printed) whenever the run fails. +# OUTPUT_EVAL_BROKER_HOST, OUTPUT_EVAL_BROKER_VPN, OUTPUT_EVAL_BROKER_USER, +# OUTPUT_EVAL_BROKER_PASSWORD enable the live_verify grader (host as +# tcp://:55555). Set all four or none: a partial set is an error. +# OUTPUT_EVAL_BROKER_ENV names a shell file of KEY=value lines that the +# runner sources first when it exists (default +# ~/.config/solace-evals/broker.env); keep it outside the repository. The +# values reach only verify.sh, as CLI args, never the subject model. + +set -uo pipefail +export LC_ALL=C + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +RUNS="${OUTPUT_EVAL_RUNS:-1}" +MODEL="${OUTPUT_EVAL_MODEL:-claude-sonnet-5}" +JUDGE_MODEL="${OUTPUT_EVAL_JUDGE_MODEL:-claude-sonnet-5}" +GRADER_TYPES='["assistant_grep","tool_use","file_exists","file_grep","java_disclaimer","compile","maven_release_match","llm_judge","live_verify"]' +# Headless -p denies unapproved tools, which would distort the measured +# behavior, so the subject gets the full list the skills declare. Bash is +# unrestricted by design (agreed for this local-only suite); every invocation +# runs in a scratch work dir. +ALLOWED_TOOLS=(Skill Read Glob Grep Write Edit Bash WebFetch TodoWrite) + +CASE_FILTER=() +while [[ $# -gt 0 ]]; do + case "$1" in + --model) + [[ $# -ge 2 ]] || { echo "ERROR: --model requires a value." >&2; exit 1; } + MODEL="$2"; shift 2 ;; + --case) + [[ $# -ge 2 ]] || { echo "ERROR: --case requires a value." >&2; exit 1; } + IFS=',' read -r -a _ids <<<"$2" + [[ ${#_ids[@]} -gt 0 ]] && CASE_FILTER+=("${_ids[@]}") + shift 2 ;; + -h|--help) + sed -n '2,/^[^#]/s/^# \{0,1\}//p' "${BASH_SOURCE[0]}"; exit 0 ;; + *) + echo "ERROR: unknown argument '$1'." >&2; exit 1 ;; + esac +done + +# Dependency and auth pre-flight (fail closed), before any mktemp so an early +# exit leaks nothing. The credential value is never echoed. +command -v claude >/dev/null 2>&1 || { echo "ERROR: the 'claude' CLI is not on PATH. Install @anthropic-ai/claude-code." >&2; exit 1; } +command -v jq >/dev/null 2>&1 || { echo "ERROR: 'jq' is not on PATH." >&2; exit 1; } +command -v curl >/dev/null 2>&1 || { echo "ERROR: 'curl' is not on PATH." >&2; exit 1; } +if [[ -z "${ANTHROPIC_API_KEY:-}" && -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]]; then + echo "ERROR: export ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN before running (a scratch CLAUDE_CONFIG_DIR has no ambient login)." >&2 + exit 1 +fi + +# Live-verify broker (opt-in, tri-state). Source the env file when present, then +# count the four variables: none = skip loudly, all = run, a partial set = error, +# so a typo in one name can never degrade to a silent skip. The values must not +# reach the subject model (the compile anchor asserts compile-only behavior), so +# the export attribute is stripped: they stay shell variables handed to verify.sh. +BROKER_ENV="${OUTPUT_EVAL_BROKER_ENV:-$HOME/.config/solace-evals/broker.env}" +# shellcheck source=/dev/null +[[ -f "$BROKER_ENV" ]] && source "$BROKER_ENV" +BROKER_VARS=(OUTPUT_EVAL_BROKER_HOST OUTPUT_EVAL_BROKER_VPN OUTPUT_EVAL_BROKER_USER OUTPUT_EVAL_BROKER_PASSWORD) +missing=() +for v in "${BROKER_VARS[@]}"; do [[ -n "${!v:-}" ]] || missing+=("$v"); done +if [[ ${#missing[@]} -eq 0 ]]; then + LIVE_VERIFY=run +elif [[ ${#missing[@]} -eq ${#BROKER_VARS[@]} ]]; then + LIVE_VERIFY=skip +else + echo "ERROR: partial broker configuration: set all four OUTPUT_EVAL_BROKER_* variables or none (missing: ${missing[*]})." >&2 + exit 1 +fi +export -n "${BROKER_VARS[@]}" + +# in_filter : 0 when no filter is set or the id is listed. +in_filter() { + [[ ${#CASE_FILTER[@]} -eq 0 ]] && return 0 + local want + for want in "${CASE_FILTER[@]}"; do [[ "$want" == "$1" ]] && return 0; done + return 1 +} + +# --- transcript extraction helpers ----------------------------------------- +# All of them read turnN.jsonl files under $RUN_DIR and never the raw stream: +# assistant text and tool_use events are jq-extracted so grep patterns cannot +# false-positive on tool payloads or on the scripted user turns. + +# turn_files : print the transcript paths for the scope, in order. +turn_files() { + local t + if [[ "$1" == "all" ]]; then + for ((t = 1; t <= NTURNS; t++)); do + [[ -e "$RUN_DIR/turn${t}.jsonl" ]] && echo "$RUN_DIR/turn${t}.jsonl" + done + else + [[ -e "$RUN_DIR/turn$1.jsonl" ]] && echo "$RUN_DIR/turn$1.jsonl" + fi + return 0 +} + +assistant_text() { # + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text' "$f" 2>/dev/null + done < <(turn_files "$1") +} + +tool_events() { # : one "namecompact-input-json" line per tool_use + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") + | [.name, (.input | tojson)] | @tsv' "$f" 2>/dev/null + done < <(turn_files "$1") +} + +webfetch_urls() { + local f + while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use" and .name=="WebFetch") + | .input.url // (.input | tojson)' "$f" 2>/dev/null + done < <(turn_files all) | sort -u +} + +webfetch_results() { + local f + while IFS= read -r f; do + jq -rs '([.[] | select(.type=="assistant") | .message.content[]? + | select(.type=="tool_use" and .name=="WebFetch") | .id]) as $ids + | .[] | select(.type=="user") | .message.content[]? + | select(.type=="tool_result" and ((.tool_use_id // "") as $t | $ids | index($t))) + | .content + | if type=="array" then (.[]? | select(.type=="text") | .text) + elif type=="string" then . else tostring end' "$f" 2>/dev/null + done < <(turn_files all) +} + +# turn_ok : 0 = a real measurement; 1 = infrastructure failure +# (no result event, or an error other than the turn cap). A turn-cap kill +# (error_max_turns) is a valid measurement, tagged so a budget problem is +# distinguishable from a behavior failure. +turn_ok() { + local t="$1" subtype is_error + jq -e 'select(.type=="result")' "$t" >/dev/null 2>&1 || return 1 + subtype="$(jq -r 'select(.type=="result") | .subtype' "$t" 2>/dev/null)" + is_error="$(jq -r 'select(.type=="result") | .is_error' "$t" 2>/dev/null)" + [[ "$subtype" == "error_max_turns" ]] && MAXTURNS_HIT=1 + [[ "$is_error" == "true" && "$subtype" != "error_max_turns" ]] && return 1 + return 0 +} + +# find_files : generated files under the work dir, excluding +# Maven build output. "*" means any file. +find_files() { + find "$WORK" -type f -name "$1" -not -path '*/target/*' 2>/dev/null +} + +shallowest_pom() { + find_files pom.xml | awk -F/ '{print NF, $0}' | sort -n | head -1 | cut -d' ' -f2- +} + +resolve_sol_jcsmp_release() { + curl -s --max-time 20 https://repo1.maven.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata.xml \ + | grep -oE '[^<]+' | sed -E 's/<\/?release>//g' +} + +# --- graders ---------------------------------------------------------------- +# grade_one : 0 pass, 1 fail (a measurement), 2 infrastructure. +# Failure/infra explanation lands in GRADER_DETAIL. +GRADER_DETAIL="" +grade_one() { + local g="$1" gtype + gtype="$(jq -r '.type' <<<"$g")" + GRADER_DETAIL="" + + case "$gtype" in + assistant_grep) + local pattern expect fixed turn text found=0 gopts=(-q) + pattern="$(jq -r '.pattern' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + fixed="$(jq -r '.fixed // false' <<<"$g")" + turn="$(jq -r '.turn // "all"' <<<"$g")" + [[ "$fixed" == "true" ]] && gopts+=(-F) || gopts+=(-E) + text="$(assistant_text "$turn")" + grep "${gopts[@]}" -- "$pattern" <<<"$text" && found=1 + if [[ "$expect" == "present" && "$found" -eq 0 ]]; then + GRADER_DETAIL="assistant text (turn $turn) lacks: $pattern"; return 1 + elif [[ "$expect" == "absent" && "$found" -eq 1 ]]; then + GRADER_DETAIL="assistant text (turn $turn) contains forbidden: $pattern"; return 1 + fi + return 0 ;; + + tool_use) + local tool ipat xpat expect count turn events matches m + tool="$(jq -r '.tool' <<<"$g")" + ipat="$(jq -r '.input_pattern // ""' <<<"$g")" + xpat="$(jq -r '.exclude_pattern // ""' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + count="$(jq -r '.count // ""' <<<"$g")" + turn="$(jq -r '.turn // "all"' <<<"$g")" + events="$(tool_events "$turn")" + if [[ "$tool" == "*" ]]; then matches="$events" + else matches="$(awk -F'\t' -v t="$tool" '$1 == t' <<<"$events")"; fi + if [[ -n "$ipat" ]]; then matches="$(grep -E -- "$ipat" <<<"$matches" || true)"; fi + if [[ -n "$xpat" ]]; then matches="$(grep -vE -- "$xpat" <<<"$matches" || true)"; fi + m="$(grep -c . <<<"$matches" || true)" + if [[ "$expect" == "present" ]]; then + if [[ -n "$count" ]]; then + [[ "$m" -eq "$count" ]] && return 0 + GRADER_DETAIL="tool $tool${ipat:+ ~ $ipat} used $m time(s), expected $count"; return 1 + fi + [[ "$m" -gt 0 ]] && return 0 + GRADER_DETAIL="tool $tool${ipat:+ ~ $ipat} never used"; return 1 + else + [[ "$m" -eq 0 ]] && return 0 + GRADER_DETAIL="forbidden tool use: $tool${ipat:+ ~ $ipat} ($m match(es))"; return 1 + fi ;; + + file_exists) + local glob expect ident files f ok=1 + glob="$(jq -r '.glob' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + ident="$(jq -r '.identical_to // ""' <<<"$g")" + files="$(find_files "$glob")" + if [[ "$expect" == "absent" ]]; then + [[ -z "$files" ]] && return 0 + GRADER_DETAIL="forbidden file(s) on disk matching $glob: $(head -3 <<<"$files" | tr '\n' ' ')"; return 1 + fi + [[ -z "$files" ]] && { GRADER_DETAIL="no file matching $glob was generated"; return 1; } + if [[ -n "$ident" ]]; then + while IFS= read -r f; do + cmp -s "$f" "$REPO_ROOT/$ident" || { ok=0; GRADER_DETAIL="$f is not byte-identical to $ident"; } + done <<<"$files" + [[ "$ok" -eq 1 ]] || return 1 + fi + return 0 ;; + + file_grep) + local glob pattern expect fixed files f gopts=(-q) matched=0 total=0 miss="" + glob="$(jq -r '.glob' <<<"$g")" + pattern="$(jq -r '.pattern' <<<"$g")" + expect="$(jq -r '.expect // "present"' <<<"$g")" + fixed="$(jq -r '.fixed // false' <<<"$g")" + [[ "$fixed" == "true" ]] && gopts+=(-F) || gopts+=(-E) + files="$(find_files "$glob")" + if [[ -z "$files" ]]; then + [[ "$expect" == "absent" ]] && return 0 + GRADER_DETAIL="no file matching $glob to grep for: $pattern"; return 1 + fi + while IFS= read -r f; do + total=$((total + 1)) + if grep "${gopts[@]}" -- "$pattern" "$f"; then matched=$((matched + 1)); else miss="$f"; fi + done <<<"$files" + case "$expect" in + present) + [[ "$matched" -gt 0 ]] && return 0 + GRADER_DETAIL="$glob file(s) lack: $pattern"; return 1 ;; + absent) + [[ "$matched" -eq 0 ]] && return 0 + GRADER_DETAIL="$glob file(s) contain forbidden: $pattern"; return 1 ;; + all_files) + [[ "$matched" -eq "$total" ]] && return 0 + GRADER_DETAIL="$miss lacks: $pattern"; return 1 ;; + esac ;; + + java_disclaimer) + local files f bad="" + files="$(find_files '*.java')" + [[ -z "$files" ]] && { GRADER_DETAIL="no .java files were generated"; return 1; } + while IFS= read -r f; do + head -3 "$f" | grep -qF 'AI-assisted code. Review before production use.' || bad="$f (missing disclaimer line)" + head -4 "$f" | grep -qF 'See the verification checklist: solace-verification-checklist.md' || bad="$f (missing checklist pointer)" + done <<<"$files" + [[ -z "$bad" ]] && return 0 + GRADER_DETAIL="$bad"; return 1 ;; + + compile) + local pom rc + pom="$(shallowest_pom)" + [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } + if command -v timeout >/dev/null 2>&1; then + timeout 600 mvn -q -B -f "$pom" compile > "$RUN_DIR/mvn-compile.log" 2>&1; rc=$? + else + mvn -q -B -f "$pom" compile > "$RUN_DIR/mvn-compile.log" 2>&1; rc=$? + fi + [[ "$rc" -eq 0 ]] && return 0 + GRADER_DETAIL="mvn compile failed (exit $rc), log: $RUN_DIR/mvn-compile.log"; return 1 ;; + + maven_release_match) + local rel pom + rel="$(resolve_sol_jcsmp_release)" + [[ -z "$rel" ]] && { GRADER_DETAIL="could not resolve sol-jcsmp from repo1.maven.org"; return 2; } + pom="$(shallowest_pom)" + [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to check against release $rel"; return 1; } + grep -qF "$rel" "$pom" && return 0 + GRADER_DETAIL="pom does not carry the live sol-jcsmp release $rel"; return 1 ;; + + llm_judge) + local criteria includes pfile jfile attempt raw verdict reason + criteria="$(jq -r '.criteria' <<<"$g")" + includes="$(jq -r '(.include // ["assistant_text"]) | join(" ")' <<<"$g")" + JUDGE_N=$((JUDGE_N + 1)) + pfile="$RUN_DIR/judge${JUDGE_N}.prompt" + jfile="$RUN_DIR/judge${JUDGE_N}.json" + { + echo 'You are grading an AI assistant transcript against one criterion.' + echo 'Respond with ONLY a JSON object: {"verdict":"pass"|"fail","reason":""}.' + echo 'The verdict is "fail" when the criterion is violated, "pass" otherwise.' + echo + echo "CRITERION: $criteria" + if grep -qw assistant_text <<<"$includes"; then + echo; echo '--- ASSISTANT TEXT (all turns, in order) ---' + assistant_text all | head -c 51200 + fi + if grep -qw webfetch_urls <<<"$includes"; then + echo; echo '--- URLS FETCHED VIA WebFetch (deduplicated; empty means none) ---' + webfetch_urls + fi + if grep -qw webfetch_results <<<"$includes"; then + echo; echo '--- WebFetch RESULT CONTENT (truncated) ---' + webfetch_results | head -c 30720 + fi + } > "$pfile" + for attempt in 1 2; do + claude -p "$(cat "$pfile")" --model "$JUDGE_MODEL" --max-turns 1 \ + --output-format json < /dev/null > "$jfile" 2>"$RUN_DIR/judge${JUDGE_N}.err" + raw="$(jq -r '.result // empty' "$jfile" 2>/dev/null | tr '\n' ' ' | grep -oE '\{[^{}]*"verdict"[^{}]*\}' | head -1)" + verdict="$(jq -r '.verdict // empty' <<<"$raw" 2>/dev/null)" + reason="$(jq -r '.reason // empty' <<<"$raw" 2>/dev/null)" + if [[ "$verdict" == "pass" ]]; then return 0; fi + if [[ "$verdict" == "fail" ]]; then GRADER_DETAIL="judge: ${reason:-no reason given}"; return 1; fi + done + GRADER_DETAIL="judge verdict unparseable after 2 attempts ($jfile)"; return 2 ;; + + live_verify) + local pom proj log rc + if [[ "$LIVE_VERIFY" != "run" ]]; then LIVE_SKIPPED=1; return 0; fi + pom="$(shallowest_pom)" + [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to verify against the broker"; return 1; } + proj="$(dirname "$pom")" + [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $pom"; return 1; } + log="$RUN_DIR/live-verify.log" + if command -v timeout >/dev/null 2>&1; then + ( cd "$proj" && timeout 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + else + ( cd "$proj" && bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + fi + case "$rc" in + 0) return 0 ;; + 2) GRADER_DETAIL="verify.sh roundtrip hit a broker or credential error (exit 2), log: $log"; return 2 ;; + *) GRADER_DETAIL="verify.sh roundtrip failed (exit $rc), log: $log"; return 1 ;; + esac ;; + + *) + GRADER_DETAIL="unknown grader type '$gtype'"; return 2 ;; + esac +} + +# --- scratch dirs (created only after the guards pass) ----------------------- +export CLAUDE_CONFIG_DIR +CLAUDE_CONFIG_DIR="$(mktemp -d)" +if [[ -n "${OUTPUT_EVAL_WORKDIR:-}" ]]; then + if ! mkdir -p "$OUTPUT_EVAL_WORKDIR" \ + || ! WORKDIR="$(cd "$OUTPUT_EVAL_WORKDIR" && pwd)" \ + || ! WORKDIR="$(mktemp -d "$WORKDIR/run-XXXXXX")"; then + echo "ERROR: cannot prepare OUTPUT_EVAL_WORKDIR '$OUTPUT_EVAL_WORKDIR'." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" + exit 1 + fi +else + WORKDIR="$(mktemp -d)" +fi + +pass=0; fail=0; infra=0; must_fail=0; LIVE_NEEDED=0 + +for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do + [[ -e "$evals_file" ]] || continue + # Fail closed on a malformed corpus: a jq parse error inside the process + # substitution feeding the loop below is otherwise unobservable, so a broken + # file would silently contribute zero cases. + if ! jq -e --argjson types "$GRADER_TYPES" ' + type=="array" and length>0 + and ([.[].id] | length == (unique | length)) + and all(.[]; + (.id | type=="string" and length>0) + and (.skill | type=="string" and length>0) + and (.turns | type=="array" and length>0 and all(.[]; type=="string" and length>0)) + and ((.must_pass // false) | type=="boolean") + and ((.max_turns // 25) | type=="number") + and (.graders | type=="array" and length>0 + and all(.[]; type=="object" and (.type as $t | $types | index($t)))))' \ + "$evals_file" >/dev/null 2>&1; then + echo "ERROR: $evals_file is not a non-empty array of {id, skill, turns, graders[, max_turns, must_pass]} cases with unique ids and known grader types." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + plugin_dir="$(cd "$(dirname "$evals_file")/.." && pwd)" # plugins/ + + # Unknown --case ids are an error, not a silent zero-case run. + if [[ ${#CASE_FILTER[@]} -gt 0 ]]; then + for want in "${CASE_FILTER[@]}"; do + jq -e --arg id "$want" 'any(.[]; .id == $id)' "$evals_file" >/dev/null 2>&1 \ + || { echo "ERROR: --case '$want' not found in $evals_file." >&2; rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR"; exit 1; } + done + fi + + # Maven is only required when a selected case compiles, checks the pom, or + # runs verify.sh. The live-verify banner prints before any case starts so the + # operator sees the broker state in the first seconds, not after an hour. + needs_mvn=0; needs_live=0 + while IFS= read -r row; do + cid="$(jq -r '.id' <<<"$row")" + in_filter "$cid" || continue + jq -e '.graders | any(.type=="compile" or .type=="maven_release_match" or .type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_mvn=1 + jq -e '.graders | any(.type=="live_verify")' <<<"$row" >/dev/null 2>&1 && needs_live=1 + done < <(jq -c '.[]' "$evals_file") + if [[ "$needs_mvn" -eq 1 ]] && ! command -v mvn >/dev/null 2>&1; then + echo "ERROR: 'mvn' is not on PATH and a selected case carries a compile, maven_release_match, or live_verify grader." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 + fi + if [[ "$needs_live" -eq 1 ]]; then + LIVE_NEEDED=1 + if [[ "$LIVE_VERIFY" == "run" ]]; then + echo "live verify: ENABLED against $OUTPUT_EVAL_BROKER_HOST" + else + echo "live verify: SKIPPED (set the four OUTPUT_EVAL_BROKER_* variables, or fill $BROKER_ENV, to enable)" + fi + fi + + while IFS= read -r case_json; do + case_id="$(jq -r '.id' <<<"$case_json")" + in_filter "$case_id" || continue + skill="$(jq -r '.skill' <<<"$case_json")" + must="$(jq -r '.must_pass // false' <<<"$case_json")" + max_turns="$(jq -r '.max_turns // 25' <<<"$case_json")" + NTURNS="$(jq -r '.turns | length' <<<"$case_json")" + + run_pass=0; case_infra=0; fail_details=() + for ((k = 1; k <= RUNS; k++)); do + # One whole-case retry on an infrastructure failure, each attempt from a + # fresh work dir (no mid-conversation resume of a failed turn). + attempt_ok=0 + for attempt in 1 2; do + RUN_DIR="$WORKDIR/${case_id}/run${k}-try${attempt}" + WORK="$RUN_DIR/work" + mkdir -p "$WORK" + MAXTURNS_HIT=0 + sid="" + turn_infra=0 + for ((t = 1; t <= NTURNS; t++)); do + turn_prompt="$(jq -r ".turns[$((t - 1))]" <<<"$case_json")" + args=(-p "$turn_prompt" --plugin-dir "$plugin_dir" --model "$MODEL" + --max-turns "$max_turns" --allowedTools "${ALLOWED_TOOLS[@]}" + --output-format stream-json --verbose) + [[ -n "$sid" ]] && args+=(--resume "$sid") + ( cd "$WORK" && claude "${args[@]}" \ + < /dev/null > "$RUN_DIR/turn${t}.jsonl" 2>"$RUN_DIR/turn${t}.err" ) + if ! turn_ok "$RUN_DIR/turn${t}.jsonl"; then turn_infra=1; break; fi + # Re-extract after every turn: a print-mode resume can mint a new id. + new_sid="$(jq -r 'select(.type=="result") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | tail -1)" + [[ -z "$new_sid" ]] && new_sid="$(jq -r 'select(.type=="system" and .subtype=="init") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | head -1)" + [[ -n "$new_sid" ]] && sid="$new_sid" + done + if [[ "$turn_infra" -eq 0 ]]; then attempt_ok=1; break; fi + done + if [[ "$attempt_ok" -ne 1 ]]; then case_infra=1; break; fi + + # Implicit gate: the target skill must have fired, or the output is not + # attributable to it and every absence grader would pass vacuously. + fired="$(turn_files all | while IFS= read -r f; do + jq -r 'select(.type=="assistant") | .message.content[]? + | select(.type=="tool_use" and .name=="Skill") | .input.skill' "$f" 2>/dev/null + done | sed 's/.*://' | sort -u)" + this_run_details=() + if ! grep -qxF "$skill" <<<"$fired"; then + this_run_details+=("the '$skill' skill never fired (fired: ${fired:-none})") + fi + + JUDGE_N=0 + LIVE_SKIPPED=0 + gi=0 + while IFS= read -r grader; do + gi=$((gi + 1)) + gtype="$(jq -r '.type' <<<"$grader")" + grade_one "$grader"; rc=$? + if [[ "$rc" -eq 2 ]]; then case_infra=1; this_run_details+=("grader#$gi $gtype INFRA :: $GRADER_DETAIL"); break + elif [[ "$rc" -eq 1 ]]; then this_run_details+=("grader#$gi $gtype :: $GRADER_DETAIL"); fi + done < <(jq -c '.graders[]' <<<"$case_json") + [[ "$case_infra" -eq 1 ]] && { fail_details=("${this_run_details[@]}"); break; } + + [[ "$MAXTURNS_HIT" -eq 1 ]] && this_run_details+=("[max-turns] a turn hit the $max_turns-turn cap") + if [[ ${#this_run_details[@]} -eq 0 ]]; then + run_pass=$((run_pass + 1)) + else + fail_details=("${this_run_details[@]}") + fi + done + + if [[ "$case_infra" -eq 1 ]]; then + echo "FAIL [$case_id] ($skill) INFRA${fail_details[0]:+ :: ${fail_details[0]}}" + fail=$((fail + 1)); infra=$((infra + 1)); continue + fi + + # Majority vote; with RUNS=1 this is simply "the single run passed". + if (( run_pass * 2 > RUNS )); then + live_tag="" + [[ "${LIVE_SKIPPED:-0}" -eq 1 ]] && live_tag=" [live verify skipped: no broker configured]" + echo "PASS [$case_id] ($skill)$live_tag" + pass=$((pass + 1)) + else + tag="" + [[ "$must" == "true" ]] && { must_fail=$((must_fail + 1)); tag=" [must-pass]"; } + echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)" + for d in "${fail_details[@]}"; do echo " - $d"; done + fail=$((fail + 1)) + fi + done < <(jq -c '.[]' "$evals_file") +done + +total=$((pass + fail)) +echo +if [[ "$total" -eq 0 ]]; then + echo "ERROR: no output-eval cases discovered under plugins/*/evals/output-evals.json." >&2 + rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" + exit 1 +fi +echo "$pass passed, $fail failed ($((pass * 100 / total))% pass rate, gate is 90%)" +[[ "$must_fail" -gt 0 ]] && echo "$must_fail must-pass case(s) failed (any must-pass failure fails the run)" +if [[ "$LIVE_NEEDED" -eq 1 ]]; then + if [[ "$LIVE_VERIFY" == "run" ]]; then + echo "live verify: ran against $OUTPUT_EVAL_BROKER_HOST" + else + echo "live verify: skipped (no broker configured); say so in the PR record" + fi +fi + +rm -rf "$CLAUDE_CONFIG_DIR" +if [[ "$infra" -gt 0 || "$must_fail" -gt 0 ]] || (( pass * 100 < total * 90 )); then + echo "transcripts and generated projects kept for inspection under: $WORKDIR" >&2 + exit 1 +fi +rm -rf "$WORKDIR" +exit 0 From 56d029214a401d13d430fdc4f9dcb4dc7f3861ea Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Tue, 8 Sep 2026 10:35:57 -0400 Subject: [PATCH 02/11] EBP-2938: Fix two skill contract gaps found by the output evals - Harden the Debug row in solace-application-development/references/jcsmp.md so the documentation redirect is the entire debug answer; memory-derived hypotheses, interim checks, and fixes are forbidden - State in solace-topic-best-practices/SKILL.md that the skill answers in chat only and never writes files - Both gaps surfaced as failing output-eval cases (appdev-debug-redirect-negative and topics-no-files-negative); both cases pass after these edits --- .../skills/solace-application-development/references/jcsmp.md | 2 +- .../skills/solace-topic-best-practices/SKILL.md | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md index d3ba17d..f1d6cfb 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md @@ -15,7 +15,7 @@ Determine the user's intent and enter the appropriate mode. Routing is keyed on | A build request WITH a valid design contract (see the design-contract gate below) | **Implement** | Read `jcsmp/implement-mode.md`; for an embedded shape generate the messaging layer per its leaf rules and verify with the `app` stage of `verify.sh` (its Step 5) | | An edit to an EXISTING app that changes the messaging topology (the topology rule below) | **Design first** | Re-enter `jcsmp/design-mode.md` to re-confirm the affected summary fields, then Implement applies the change | | A mechanical edit to an EXISTING app with no topology change — reconnect handling, payload format, logging, renames, ack tuning | **Implement** | Read `jcsmp/implement-mode.md`; no design pass | -| "My JCSMP app is throwing on connect" / "Why is my consumer not binding the queue?" | **Debug** | Debug mode is not yet available in this release. Redirect the user to the canonical Solace JCSMP troubleshooting documentation: [JCSMP API Home](https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md). Do not generate debugging guidance from memory. | +| "My JCSMP app is throwing on connect" / "Why is my consumer not binding the queue?" | **Debug** | Debug mode is not yet available in this release. Redirect the user to the canonical Solace JCSMP troubleshooting documentation: [JCSMP API Home](https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md). The redirect is the ENTIRE debug answer: acknowledge the problem in one line, link the page, and stop. Do not generate debugging guidance from memory. That prohibition covers root-cause hypotheses, likely causes, interim checks or "things worth checking in the meantime", workarounds, and configuration or code suggestions. Restating this rule does not license adding such guidance after the redirect. | If unclear, default to **Design**. Understand the messaging problem before generating code. diff --git a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md index 41e230d..5c048b2 100644 --- a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md @@ -13,9 +13,12 @@ This skill answers topic-hierarchy and topic-architecture questions by reading t 1. WebFetch the canonical page live: `https://docs.solace.com/Messaging/Topic-Architecture-Best-Practices.md`. This is the authoritative source for topic-level ordering, naming conventions, wildcard placement, and taxonomy guidance. 2. Apply the fetched guidance to the user's specific topic-design question (the events they publish, the consumers that subscribe, the levels they need, and where wildcards belong). 3. Quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. +4. Deliver the answer in chat. Never write it to a file, even when the user asks you to save it: present the content in chat and let the user save it themselves. Fetch the single page above, then ground every recommendation in it. Do not WebFetch other pages blindly. ## What this skill is not Do not use this skill to design, build, generate, or implement a Solace messaging application (publisher, consumer, or pub/sub). That is the solace-application-development skill. + +This skill answers in chat only: it never writes, edits, or saves files (a conventions document, a design note, code, or anything else). The `allowed-tools` list (Read, Grep, WebFetch) is the contract; a Write or Edit tool being available in the session does not change it. From 4cb25aa77d47823663a198e4c99dbef28d992569 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Tue, 8 Sep 2026 12:16:32 -0400 Subject: [PATCH 03/11] EBP-2938: Simplify output-eval docs, broker input, and case prompts - Remove exact case counts from the evals README so the prose does not drift from the corpus - Remove the "No CI wiring" section from the evals README - Take broker credentials for live_verify from exported OUTPUT_EVAL_BROKER_* variables only; drop the optional env file - Report in the run summary whether live_verify executed, skipped, or never ran because the case failed before grading - Drop the run.sh and port details from two application-development case prompts; no grader depended on them --- .../solace-messaging-skills/evals/README.md | 22 +++++------- .../evals/output-evals.json | 4 +-- tools/run-output-evals.sh | 36 +++++++++---------- 3 files changed, 28 insertions(+), 34 deletions(-) diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index f35437f..6da3716 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -20,7 +20,7 @@ In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs th # Output evals -Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds 11 cases across the plugin's three skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired and every grader passed. +Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired and every grader passed. For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. @@ -39,23 +39,19 @@ export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg runs for roughly an hour; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. -The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The four forbidden-behavior negatives and the compile case are `must_pass`. +The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. ### Live verification against a real broker -The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables: +The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: ```shell -OUTPUT_EVAL_BROKER_HOST=tcp://:55555 -OUTPUT_EVAL_BROKER_VPN= -OUTPUT_EVAL_BROKER_USER= -OUTPUT_EVAL_BROKER_PASSWORD= +export OUTPUT_EVAL_BROKER_HOST=tcp://:55555 +export OUTPUT_EVAL_BROKER_VPN= +export OUTPUT_EVAL_BROKER_USER= +export OUTPUT_EVAL_BROKER_PASSWORD= ``` -Either export them in your shell, or put those four lines in `~/.config/solace-evals/broker.env` (or a path you name in `OUTPUT_EVAL_BROKER_ENV`). The runner sources that file when it exists. Keep the file outside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. +Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. -The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is set, the grader skips and the case line says so. When some but not all are set, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. - -## No CI wiring - -The output evals run locally only. The repository is public and no policy exists yet for API secrets on public repositories, so `ci.yml` does not run this suite. A follow-up ticket wires CI once a secrets policy lands. The PR description is the record of local runs. +The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is exported, the grader skips and the case line says so. When some but not all are exported, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index aaf39f4..f52f63e 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -3,7 +3,7 @@ "id": "appdev-design-summary-positive", "skill": "solace-application-development", "turns": [ - "I'm planning a Java system on an event broker: when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard and a run.sh launcher, but right now I only want help designing the Solace messaging pattern and topology, not the code.", + "I'm planning a Java system on an event broker: when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard, but right now I only want help designing the Solace messaging pattern and topology, not the code.", "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." ], "max_turns": 25, @@ -33,7 +33,7 @@ "skill": "solace-application-development", "must_pass": true, "turns": [ - "Build a Java chat room with a web interface: enter a display name, type a message, and see messages appear live for anyone with the page open, connected to Solace. Ensure the app uses port 9090 or higher and create a run.sh file so I can launch it easily. Skip any design discussion or questions and generate the full project right now." + "Build a Java chat room with a web interface: enter a display name, type a message, and see messages appear live for anyone with the page open, connected to Solace. Skip any design discussion or questions and generate the full project right now." ], "max_turns": 25, "graders": [ diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index 5e85f8d..9f240f1 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -88,12 +88,10 @@ # cleanup only ever removes that subdirectory. Otherwise a mktemp # directory is used. Kept (and printed) whenever the run fails. # OUTPUT_EVAL_BROKER_HOST, OUTPUT_EVAL_BROKER_VPN, OUTPUT_EVAL_BROKER_USER, -# OUTPUT_EVAL_BROKER_PASSWORD enable the live_verify grader (host as -# tcp://:55555). Set all four or none: a partial set is an error. -# OUTPUT_EVAL_BROKER_ENV names a shell file of KEY=value lines that the -# runner sources first when it exists (default -# ~/.config/solace-evals/broker.env); keep it outside the repository. The -# values reach only verify.sh, as CLI args, never the subject model. +# OUTPUT_EVAL_BROKER_PASSWORD, when exported, enable the live_verify grader +# (host as tcp://:55555). Export all four or none: a partial set is +# an error. The values reach only verify.sh, as CLI args, never the +# subject model. set -uo pipefail export LC_ALL=C @@ -138,14 +136,11 @@ if [[ -z "${ANTHROPIC_API_KEY:-}" && -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]]; then exit 1 fi -# Live-verify broker (opt-in, tri-state). Source the env file when present, then -# count the four variables: none = skip loudly, all = run, a partial set = error, -# so a typo in one name can never degrade to a silent skip. The values must not -# reach the subject model (the compile anchor asserts compile-only behavior), so -# the export attribute is stripped: they stay shell variables handed to verify.sh. -BROKER_ENV="${OUTPUT_EVAL_BROKER_ENV:-$HOME/.config/solace-evals/broker.env}" -# shellcheck source=/dev/null -[[ -f "$BROKER_ENV" ]] && source "$BROKER_ENV" +# Live-verify broker (opt-in, tri-state). Count the four exported variables: +# none = skip loudly, all = run, a partial set = error, so a typo in one name can +# never degrade to a silent skip. The values must not reach the subject model +# (the compile anchor asserts compile-only behavior), so the export attribute is +# stripped: they stay shell variables handed to verify.sh. BROKER_VARS=(OUTPUT_EVAL_BROKER_HOST OUTPUT_EVAL_BROKER_VPN OUTPUT_EVAL_BROKER_USER OUTPUT_EVAL_BROKER_PASSWORD) missing=() for v in "${BROKER_VARS[@]}"; do [[ -n "${!v:-}" ]] || missing+=("$v"); done @@ -154,7 +149,7 @@ if [[ ${#missing[@]} -eq 0 ]]; then elif [[ ${#missing[@]} -eq ${#BROKER_VARS[@]} ]]; then LIVE_VERIFY=skip else - echo "ERROR: partial broker configuration: set all four OUTPUT_EVAL_BROKER_* variables or none (missing: ${missing[*]})." >&2 + echo "ERROR: partial broker configuration: export all four OUTPUT_EVAL_BROKER_* variables or none (missing: ${missing[*]})." >&2 exit 1 fi export -n "${BROKER_VARS[@]}" @@ -426,6 +421,7 @@ grade_one() { proj="$(dirname "$pom")" [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $pom"; return 1; } log="$RUN_DIR/live-verify.log" + LIVE_RAN=1 if command -v timeout >/dev/null 2>&1; then ( cd "$proj" && timeout 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? else @@ -457,7 +453,7 @@ else WORKDIR="$(mktemp -d)" fi -pass=0; fail=0; infra=0; must_fail=0; LIVE_NEEDED=0 +pass=0; fail=0; infra=0; must_fail=0; LIVE_NEEDED=0; LIVE_RAN=0 for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do [[ -e "$evals_file" ]] || continue @@ -510,7 +506,7 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do if [[ "$LIVE_VERIFY" == "run" ]]; then echo "live verify: ENABLED against $OUTPUT_EVAL_BROKER_HOST" else - echo "live verify: SKIPPED (set the four OUTPUT_EVAL_BROKER_* variables, or fill $BROKER_ENV, to enable)" + echo "live verify: SKIPPED (export the four OUTPUT_EVAL_BROKER_* variables to enable)" fi fi @@ -614,8 +610,10 @@ fi echo "$pass passed, $fail failed ($((pass * 100 / total))% pass rate, gate is 90%)" [[ "$must_fail" -gt 0 ]] && echo "$must_fail must-pass case(s) failed (any must-pass failure fails the run)" if [[ "$LIVE_NEEDED" -eq 1 ]]; then - if [[ "$LIVE_VERIFY" == "run" ]]; then - echo "live verify: ran against $OUTPUT_EVAL_BROKER_HOST" + if [[ "$LIVE_VERIFY" == "run" && "$LIVE_RAN" -eq 1 ]]; then + echo "live verify: ran against $OUTPUT_EVAL_BROKER_HOST (see the case line for its verdict)" + elif [[ "$LIVE_VERIFY" == "run" ]]; then + echo "live verify: enabled against $OUTPUT_EVAL_BROKER_HOST but never executed (the case failed before grading)" else echo "live verify: skipped (no broker configured); say so in the PR record" fi From 781e6e8273f19ef203185515b5c9e4e876effee6 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Wed, 9 Sep 2026 09:34:01 -0400 Subject: [PATCH 04/11] EBP-2938: Address review findings on the output-eval runner and corpus - Replace the Quickstart failover word check with a code grep for the reconnect block; the Step 6 checklist report legitimately names HA failover - Run the LLM judge tool-free (--tools "") from the run directory so a tool call cannot spend its only turn and project settings stay out of the judge session - Add a compile grader to the Solace Suggested case and make compile and maven_release_match iterate every generated pom for two-project layouts - Align the debug-redirect judge with the Debug row: fail troubleshooting from any source, treat a restated symptom as the acknowledgement - Accept bold field labels in the design summary greps (design mode leaves formatting to the model) - Validate OUTPUT_EVAL_RUNS as a positive integer (a zero value crashed bash 3.2 on an empty array expansion) - Classify a timeout kill (exit 124) as INFRA in compile and live_verify - Carry the cause onto turn-level INFRA lines and print a RETRY line before the second attempt - Show the live-verify skip tag on FAIL lines; document that a max_turns hit fails the case - Drop the run-duration claim from the evals README; mention output evals in the root README Verified locally: claude-sonnet-5 and claude-opus-5 legs over the six affected cases pass, with live verify green against a real broker on both models. --- README.md | 8 +- .../solace-messaging-skills/evals/README.md | 4 +- .../evals/output-evals.json | 21 ++-- tools/run-output-evals.sh | 110 ++++++++++++------ 4 files changed, 93 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index ae43d03..c94a5b6 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ agent-plugins/ ├── plugins/ # Claude plugins, one subdirectory per plugin │ └── solace-messaging-skills/ # The messaging skills plugin │ ├── .claude-plugin/ # Plugin definition (plugin.json) -│ ├── evals/ # Trigger eval corpus (trigger-evals.json) and its README +│ ├── evals/ # Trigger and output eval corpora and their README │ └── skills/ # Individual skill definitions (shared by all agents) │ ├── solace-application-development/ # Solace application development umbrella skill │ │ ├── SKILL.md # Skill routing and instructions @@ -62,15 +62,17 @@ agent-plugins/ │ │ └── SKILL.md # Navigation-only manifest; reads the live docs page │ └── solace-messaging-feedback/ # Feedback formatter skill │ └── SKILL.md # Formats and routes session feedback (Support, Community, or Ideas portal) -├── tools/ # Re-runnable scripts (check-links, run-trigger-evals) +├── tools/ # Re-runnable scripts (check-links, run-trigger-evals, run-output-evals) ├── README.md # This file └── .claude-plugin/ # Claude marketplace definition (marketplace.json) ``` -## Trigger evals +## Trigger and output evals Each plugin ships a trigger eval corpus under `plugins//evals/` that checks each skill fires on the prompts it should and stays silent on the prompts it should not. See the [trigger evals README](plugins/solace-messaging-skills/evals/README.md) for the corpus format, how to run the evals locally, and how they run in CI. +Each plugin also ships an output eval corpus in the same directory that checks a skill's output honors the skill contract (design before code, the door question, scrubbed feedback drafts, grounded answers). Output evals run locally only. See the [output evals section](plugins/solace-messaging-skills/evals/README.md#output-evals) of the same README for the corpus format, the graders, and how to run them. + ## License This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details. diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index 6da3716..80889d4 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -20,7 +20,7 @@ In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs th # Output evals -Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired and every grader passed. +Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no turn hit the `max_turns` cap. For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. @@ -37,7 +37,7 @@ export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN ./tools/run-output-evals.sh --case appdev-quickstart-implement-full ``` -Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg runs for roughly an hour; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. +Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index f52f63e..2af3788 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -9,14 +9,14 @@ "max_turns": 25, "graders": [ { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, - { "type": "assistant_grep", "pattern": "Pattern:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Delivery:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Access type:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Topic:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Consumption endpoint:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Auth:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Broker:", "fixed": true, "expect": "present" }, - { "type": "assistant_grep", "pattern": "Grounding docs:", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Delivery\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Access type\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Topic\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Consumption endpoint\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Auth\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Broker\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Grounding docs\\**:", "expect": "present" }, { "type": "assistant_grep", "pattern": "Guaranteed Pub/Sub (fan-out)", "fixed": true, "expect": "present" }, { "type": "assistant_grep", "pattern": "solace-design.md", "fixed": true, "expect": "present" }, { "type": "tool_use", "tool": "Write", "expect": "absent" }, @@ -57,7 +57,7 @@ { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "Custom", "fixed": true, "expect": "present", "turn": 2 }, - { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "absent", "turn": 3 }, + { "type": "file_grep", "glob": "*.java", "pattern": "new JCSMPChannelProperties\\(|setProperty\\([^)]*CLIENT_CHANNEL_PROPERTIES", "expect": "absent" }, { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, { "type": "compile" }, @@ -107,7 +107,7 @@ { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "llm_judge", - "criteria": "The user asked for step-by-step debugging of a JCSMP error. The skill's contract forbids generating debugging guidance from memory; the assistant must redirect to the canonical JCSMP documentation. PASS when the reply acknowledges the problem and redirects to documentation without supplying its own troubleshooting procedure. FAIL when the reply contains troubleshooting steps, root-cause hypotheses, configuration changes, or code fixes produced from the assistant's own knowledge rather than from a fetched documentation page or a bundled reference file." + "criteria": "The user asked for step-by-step debugging of a JCSMP error. The skill's contract says the documentation redirect is the ENTIRE debug answer: acknowledge the problem, link the canonical JCSMP documentation, and stop. PASS when the reply acknowledges the problem and links the documentation and adds nothing else. Restating the symptom the user reported (the exception, the operation that failed) is the acknowledgement, not a hypothesis; an offer to help with design or implementation instead is routing, not debugging guidance. FAIL when the reply names a cause the user did not report, or contains any troubleshooting step, interim check, workaround, or configuration or code suggestion, from any source, including content fetched from a documentation page or paraphrased from a bundled reference file. Restating the rule does not excuse guidance that follows it." } ] }, @@ -129,6 +129,7 @@ { "type": "file_grep", "glob": "*.java", "pattern": "setProperty\\([^)]*SSL_TRUST_STORE", "expect": "absent" }, { "type": "file_grep", "glob": "*.java", "pattern": "setDMQEligible(true)", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, + { "type": "compile" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REQUEST_RECEIVED", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REPLY_RECEIVED", "fixed": true, "expect": "present" }, { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "expect": "absent" }, diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index 9f240f1..0abd98a 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -7,8 +7,8 @@ # For every case in each plugin's evals/output-evals.json, the runner plays the # case's scripted user turns through the Claude Code CLI with only that plugin # loaded, then grades the transcript and any generated files against the case's -# grader list. A case passes only when the target skill fired AND every grader -# passed. +# grader list. A case passes only when the target skill fired, every grader +# passed, AND no turn hit the turn cap. # # WHY --plugin-dir AND A SCRATCH CONFIG DIR # Same isolation contract as the trigger runner: --plugin-dir loads the @@ -44,9 +44,9 @@ # java_disclaimer every generated .java starts with the AI-assisted # disclaimer line and the checklist pointer; fails when # no .java exists. No fields. -# compile `mvn -q -B compile` on the shallowest generated -# pom.xml; the exit code is the verdict. No fields. -# maven_release_match the generated pom carries the live sol-jcsmp +# compile `mvn -q -B compile` on every generated pom.xml; all +# must exit 0. No fields. +# maven_release_match every generated pom carries the live sol-jcsmp # from repo1.maven.org metadata. No fields. # llm_judge one tool-free judge completion on # OUTPUT_EVAL_JUDGE_MODEL returning a strict @@ -57,9 +57,9 @@ # stage, Quickstart single-project mode) against the # broker named by the OUTPUT_EVAL_BROKER_* variables. # verify.sh exit 0 passes; exit 2 (a doc-traceable -# broker or credential error) is INFRA; anything else -# fails. Skips, and says so, when no broker is -# configured. No fields. +# broker or credential error) and a timeout kill are +# INFRA; anything else fails. Skips, and says so, when +# no broker is configured. No fields. # # RUNS: output cases are expensive (a full Implement flow runs 30+ turns plus # Maven), so OUTPUT_EVAL_RUNS defaults to 1. Set it higher for a majority-vote @@ -131,6 +131,9 @@ done command -v claude >/dev/null 2>&1 || { echo "ERROR: the 'claude' CLI is not on PATH. Install @anthropic-ai/claude-code." >&2; exit 1; } command -v jq >/dev/null 2>&1 || { echo "ERROR: 'jq' is not on PATH." >&2; exit 1; } command -v curl >/dev/null 2>&1 || { echo "ERROR: 'curl' is not on PATH." >&2; exit 1; } +# Arithmetic reads a zero or non-numeric RUNS as 0, which skips every run and +# trips bash 3.2's empty-array expansion under set -u at the FAIL print. +[[ "$RUNS" =~ ^[1-9][0-9]*$ ]] || { echo "ERROR: OUTPUT_EVAL_RUNS must be a positive integer (got '$RUNS')." >&2; exit 1; } if [[ -z "${ANTHROPIC_API_KEY:-}" && -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]]; then echo "ERROR: export ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN before running (a scratch CLAUDE_CONFIG_DIR has no ambient login)." >&2 exit 1 @@ -218,8 +221,10 @@ webfetch_results() { # turn_ok : 0 = a real measurement; 1 = infrastructure failure # (no result event, or an error other than the turn cap). A turn-cap kill -# (error_max_turns) is a valid measurement, tagged so a budget problem is -# distinguishable from a behavior failure. +# (error_max_turns) is not INFRA: the graders still run over what was +# produced, but the case fails with a [max-turns] detail, because a scripted +# turn that never finished is not evidence of the contract. The tag keeps a +# budget problem distinguishable from a behavior failure. turn_ok() { local t="$1" subtype is_error jq -e 'select(.type=="result")' "$t" >/dev/null 2>&1 || return 1 @@ -230,6 +235,22 @@ turn_ok() { return 0 } +# turn_reason : one line naming why turn_ok rejected the +# turn, so an INFRA verdict carries its cause instead of only a work-dir path: +# the result subtype and message when a result event exists, else the tail of +# the CLI's stderr capture. +turn_reason() { + local subtype msg err + subtype="$(jq -r 'select(.type=="result") | .subtype // empty' "$1.jsonl" 2>/dev/null | tail -1)" + if [[ -n "$subtype" ]]; then + msg="$(jq -r 'select(.type=="result") | .result // empty' "$1.jsonl" 2>/dev/null | tail -1 | head -c 200)" + echo "result subtype $subtype${msg:+: $msg}" + else + err="$(tail -2 "$1.err" 2>/dev/null | tr '\n' ' ' | head -c 200)" + echo "no result event; stderr: ${err:-empty}" + fi +} + # find_files : generated files under the work dir, excluding # Maven build output. "*" means any file. find_files() { @@ -356,25 +377,34 @@ grade_one() { GRADER_DETAIL="$bad"; return 1 ;; compile) - local pom rc - pom="$(shallowest_pom)" - [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } - if command -v timeout >/dev/null 2>&1; then - timeout 600 mvn -q -B -f "$pom" compile > "$RUN_DIR/mvn-compile.log" 2>&1; rc=$? - else - mvn -q -B -f "$pom" compile > "$RUN_DIR/mvn-compile.log" 2>&1; rc=$? - fi - [[ "$rc" -eq 0 ]] && return 0 - GRADER_DETAIL="mvn compile failed (exit $rc), log: $RUN_DIR/mvn-compile.log"; return 1 ;; + # Every generated pom compiles, so a two-project layout (Solace Suggested) + # is proven whole, not by whichever pom sorts first. + local poms pom rc n=0 log + poms="$(find_files pom.xml)" + [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } + while IFS= read -r pom; do + n=$((n + 1)); log="$RUN_DIR/mvn-compile-$n.log" + if command -v timeout >/dev/null 2>&1; then + timeout 600 mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? + else + mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? + fi + # timeout's own exit 124 is a wall-clock kill, not a measurement of the code. + [[ "$rc" -eq 124 ]] && { GRADER_DETAIL="mvn compile for $pom timed out after 600s, log: $log"; return 2; } + [[ "$rc" -eq 0 ]] || { GRADER_DETAIL="mvn compile failed for $pom (exit $rc), log: $log"; return 1; } + done <<<"$poms" + return 0 ;; maven_release_match) - local rel pom + local rel poms pom rel="$(resolve_sol_jcsmp_release)" [[ -z "$rel" ]] && { GRADER_DETAIL="could not resolve sol-jcsmp from repo1.maven.org"; return 2; } - pom="$(shallowest_pom)" - [[ -z "$pom" ]] && { GRADER_DETAIL="no generated pom.xml to check against release $rel"; return 1; } - grep -qF "$rel" "$pom" && return 0 - GRADER_DETAIL="pom does not carry the live sol-jcsmp release $rel"; return 1 ;; + poms="$(find_files pom.xml)" + [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to check against release $rel"; return 1; } + while IFS= read -r pom; do + grep -qF "$rel" "$pom" || { GRADER_DETAIL="$pom does not carry the live sol-jcsmp release $rel"; return 1; } + done <<<"$poms" + return 0 ;; llm_judge) local criteria includes pfile jfile attempt raw verdict reason @@ -403,8 +433,11 @@ grade_one() { fi } > "$pfile" for attempt in 1 2; do - claude -p "$(cat "$pfile")" --model "$JUDGE_MODEL" --max-turns 1 \ - --output-format json < /dev/null > "$jfile" 2>"$RUN_DIR/judge${JUDGE_N}.err" + # --tools "" keeps the judge tool-free (a tool call would spend its only + # turn and leave no verdict); the cd keeps the operator's project + # settings out of the judge session, as $WORK does for the subject. + ( cd "$RUN_DIR" && claude -p "$(cat "$pfile")" --model "$JUDGE_MODEL" --max-turns 1 \ + --tools "" --output-format json < /dev/null > "$jfile" 2>"$RUN_DIR/judge${JUDGE_N}.err" ) raw="$(jq -r '.result // empty' "$jfile" 2>/dev/null | tr '\n' ' ' | grep -oE '\{[^{}]*"verdict"[^{}]*\}' | head -1)" verdict="$(jq -r '.verdict // empty' <<<"$raw" 2>/dev/null)" reason="$(jq -r '.reason // empty' <<<"$raw" 2>/dev/null)" @@ -430,6 +463,7 @@ grade_one() { case "$rc" in 0) return 0 ;; 2) GRADER_DETAIL="verify.sh roundtrip hit a broker or credential error (exit 2), log: $log"; return 2 ;; + 124) GRADER_DETAIL="verify.sh roundtrip timed out after 600s, log: $log"; return 2 ;; *) GRADER_DETAIL="verify.sh roundtrip failed (exit $rc), log: $log"; return 1 ;; esac ;; @@ -488,7 +522,7 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do # Maven is only required when a selected case compiles, checks the pom, or # runs verify.sh. The live-verify banner prints before any case starts so the - # operator sees the broker state in the first seconds, not after an hour. + # operator sees the broker state in the first seconds, not at the end of the run. needs_mvn=0; needs_live=0 while IFS= read -r row; do cid="$(jq -r '.id' <<<"$row")" @@ -521,8 +555,9 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do run_pass=0; case_infra=0; fail_details=() for ((k = 1; k <= RUNS; k++)); do # One whole-case retry on an infrastructure failure, each attempt from a - # fresh work dir (no mid-conversation resume of a failed turn). - attempt_ok=0 + # fresh work dir (no mid-conversation resume of a failed turn). The retry + # is announced, so a flaky leg is distinguishable from a clean one. + attempt_ok=0; turn_detail="" for attempt in 1 2; do RUN_DIR="$WORKDIR/${case_id}/run${k}-try${attempt}" WORK="$RUN_DIR/work" @@ -538,15 +573,20 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do [[ -n "$sid" ]] && args+=(--resume "$sid") ( cd "$WORK" && claude "${args[@]}" \ < /dev/null > "$RUN_DIR/turn${t}.jsonl" 2>"$RUN_DIR/turn${t}.err" ) - if ! turn_ok "$RUN_DIR/turn${t}.jsonl"; then turn_infra=1; break; fi + if ! turn_ok "$RUN_DIR/turn${t}.jsonl"; then + turn_infra=1 + turn_detail="turn $t (try $attempt): $(turn_reason "$RUN_DIR/turn${t}")" + break + fi # Re-extract after every turn: a print-mode resume can mint a new id. new_sid="$(jq -r 'select(.type=="result") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | tail -1)" [[ -z "$new_sid" ]] && new_sid="$(jq -r 'select(.type=="system" and .subtype=="init") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | head -1)" [[ -n "$new_sid" ]] && sid="$new_sid" done if [[ "$turn_infra" -eq 0 ]]; then attempt_ok=1; break; fi + [[ "$attempt" -eq 1 ]] && echo "RETRY [$case_id] ($skill) :: $turn_detail" done - if [[ "$attempt_ok" -ne 1 ]]; then case_infra=1; break; fi + if [[ "$attempt_ok" -ne 1 ]]; then case_infra=1; fail_details=("$turn_detail"); break; fi # Implicit gate: the target skill must have fired, or the output is not # attributable to it and every absence grader would pass vacuously. @@ -585,15 +625,15 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do fi # Majority vote; with RUNS=1 this is simply "the single run passed". + live_tag="" + [[ "${LIVE_SKIPPED:-0}" -eq 1 ]] && live_tag=" [live verify skipped: no broker configured]" if (( run_pass * 2 > RUNS )); then - live_tag="" - [[ "${LIVE_SKIPPED:-0}" -eq 1 ]] && live_tag=" [live verify skipped: no broker configured]" echo "PASS [$case_id] ($skill)$live_tag" pass=$((pass + 1)) else tag="" [[ "$must" == "true" ]] && { must_fail=$((must_fail + 1)); tag=" [must-pass]"; } - echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)" + echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)$live_tag" for d in "${fail_details[@]}"; do echo " - $d"; done fail=$((fail + 1)) fi From 12dab5231a16225ac05be8dfe3ef0efcbb200ff5 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Wed, 9 Sep 2026 15:57:37 -0400 Subject: [PATCH 05/11] EBP-2938: Add a corpus JSON Schema and rename the turn fields - Add tools/output-evals.schema.json (JSON Schema draft 2020-12) that defines the output-eval corpus with per-grader field rules and a description for every field, in response to review feedback - Rename the corpus fields turns to user_turns and max_turns to max_agent_turns to separate the scripted user messages from the CLI's agent-turn budget; update the runner and the corpus accordingly - Point the evals README and the runner header at the schema; the runner keeps enforcing the structural subset at start-up - Add a prerequisites list and a transcript-to-corpus mapping table (observed with Claude Code 2.1.266) to the evals README --- .../solace-messaging-skills/evals/README.md | 30 ++- .../evals/output-evals.json | 44 ++--- tools/output-evals.schema.json | 185 ++++++++++++++++++ tools/run-output-evals.sh | 26 ++- 4 files changed, 252 insertions(+), 33 deletions(-) create mode 100644 tools/output-evals.schema.json diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index 80889d4..47923fb 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -20,15 +20,41 @@ In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs th # Output evals -Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional per-invocation `max_turns` budget, an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no turn hit the `max_turns` cap. +Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `user_turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional `max_agent_turns` budget (the model's replies and tool calls allowed per user turn), an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no user turn hit the `max_agent_turns` cap. The full structure, with the intent of every field, is defined in `tools/output-evals.schema.json` (JSON Schema, draft 2020-12). The runner enforces the structural subset of that schema at start-up (array shape, required fields, unique ids, known grader types) and exits before any API call when the corpus violates it. Validating an edit against the full schema, including the per-grader field rules, is a manual step with any JSON Schema validator for now. For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `java_disclaimer`, `compile`, `maven_release_match`, `live_verify`) check the transcript and the generated files mechanically, including a real `mvn compile` of the generated project, a match of the generated pom against the live sol-jcsmp `` on Maven Central, and, when a broker is configured, a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. +## How the transcript maps onto the corpus + +Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json --verbose` stream of one `claude -p` invocation, one file per entry in `user_turns`. The runner reads only the fields in this table. The shapes were observed with Claude Code 2.1.266 on 2026-09-09. Re-check this table after a CLI upgrade, because a change to these fields breaks the graders silently. + +| Corpus field or runner concept | Source in `turnN.jsonl` | +|---|---| +| `user_turns[i]` | Not in the file. Entry `i` is the `-p` prompt of invocation `i`, and the stream never echoes it. The only `user` events are tool results. Each entry produces its own `turn.jsonl`. | +| `max_agent_turns` | Passed as `--max-turns`. The `result` event reports `num_turns` (agent turns used, one per assistant message cycle) and sets `subtype` to `error_max_turns` when the cap is hit. | +| Grader `turn` | Selects which `turn.jsonl` file the grader reads. It is not a field inside the file. | +| `skill` (the implicit skill-fired gate) | An `assistant` event whose `message.content[]` holds a `tool_use` block with `name` `Skill`. Its `input.skill` is `plugin:skill-name`; the runner strips the prefix. | +| `assistant_grep` | `assistant` events, `message.content[]` blocks with `type` `text`. Thinking blocks and tool payloads are excluded. | +| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`. | +| `llm_judge` with `webfetch_urls` or `webfetch_results` | WebFetch `tool_use` blocks supply the URLs. `user` events whose `tool_result.tool_use_id` matches a WebFetch block supply the fetched content. | +| Session continuity (`--resume`) | `session_id` on the `result` event, with the `init` event as the fallback. | +| INFRA verdict for a turn | A missing `result` event, or `result.is_error` true with a `subtype` other than `error_max_turns`. | + ## Running the output evals locally -You need the `claude` CLI, `jq`, and `curl` on your PATH, plus an exported credential. The compile case also needs `mvn` with a JDK 11 or newer, and network access to `repo1.maven.org` and `docs.solace.com`. Run from the repository root: +### Prerequisites + +- The `claude` CLI (Claude Code) on your PATH. +- `jq` and `curl` on your PATH. +- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the cases that compile generated code need it, and the runner checks for it only when such a case is selected. +- An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. The runner uses a throwaway config directory with no ambient login. Claude seat holders can mint a token with `claude setup-token`. +- Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). +- Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). +- On macOS, `caffeinate` to keep the machine awake for a full leg. + +Run from the repository root: ```shell export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index 2af3788..900b3f5 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -2,11 +2,11 @@ { "id": "appdev-design-summary-positive", "skill": "solace-application-development", - "turns": [ + "user_turns": [ "I'm planning a Java system on an event broker: when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard, but right now I only want help designing the Solace messaging pattern and topology, not the code.", "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." ], - "max_turns": 25, + "max_agent_turns": 25, "graders": [ { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, { "type": "assistant_grep", "pattern": "Pattern\\**:", "expect": "present" }, @@ -32,10 +32,10 @@ "id": "appdev-code-before-confirm-negative", "skill": "solace-application-development", "must_pass": true, - "turns": [ + "user_turns": [ "Build a Java chat room with a web interface: enter a display name, type a message, and see messages appear live for anyone with the page open, connected to Solace. Skip any design discussion or questions and generate the full project right now." ], - "max_turns": 25, + "max_agent_turns": 25, "graders": [ { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, @@ -47,12 +47,12 @@ "id": "appdev-quickstart-implement-full", "skill": "solace-application-development", "must_pass": true, - "turns": [ + "user_turns": [ "Build a Java app on our Solace event broker for food-delivery order processing. An OrderService publishes customer orders (order ID, restaurant ID, line items, total) and a pool of competing KitchenWorker instances share the load: each order must be processed by exactly one worker, must survive a worker crash or restart, and no order may be lost. Plain Maven project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, non-exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", "Yes, I am happy with that design. Save it to solace-design.md, then go ahead.", "Quickstart. I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." ], - "max_turns": 80, + "max_agent_turns": 80, "graders": [ { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 2 }, @@ -97,10 +97,10 @@ "id": "appdev-debug-redirect-negative", "skill": "solace-application-development", "must_pass": true, - "turns": [ + "user_turns": [ "My pharmacy refill-authorization Java app uses JCSMP, and the replier now throws JCSMPErrorResponseException 503 Service Unavailable when it binds to the request queue. Walk me through debugging this step by step." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "assistant_grep", "pattern": "https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.md", "fixed": true, "expect": "present" }, { "type": "file_exists", "glob": "*", "expect": "absent" }, @@ -114,13 +114,13 @@ { "id": "appdev-suggested-secure", "skill": "solace-application-development", - "turns": [ + "user_turns": [ "Build a Java app on our Solace event broker where a pharmacy service requests refill authorization from a doctor's-office service and waits for the answer. Neither the request nor the approval may be lost. Plain Maven, no dashboard. We have a TLS-enabled Solace Cloud broker. This is Request-Reply (Guaranteed); derive the topic and queue names yourself.", "Yes, I am happy with that design. No need to save it, go ahead.", "Solace Suggested.", "No HA failover needed. I will not share credentials in the chat; compile only and hand me the commands to run myself." ], - "max_turns": 80, + "max_agent_turns": 80, "graders": [ { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, { "type": "java_disclaimer" }, @@ -140,10 +140,10 @@ "id": "feedback-email-before-confirm-negative", "skill": "solace-messaging-feedback", "must_pass": true, - "turns": [ + "user_turns": [ "The solace-application-development skill generated a JCSMP subscriber that does not compile (missing import for ConsumerFlowProperties). Just write the support email to Solace for me right now. Do not ask me anything first." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, { "type": "assistant_grep", "pattern": "support contract", "fixed": true, "expect": "present" }, @@ -155,11 +155,11 @@ { "id": "feedback-support-email-positive", "skill": "solace-messaging-feedback", - "turns": [ + "user_turns": [ "I have feedback on the solace-application-development skill. While building our schema-registry caching gateway, the generated subscriber called ackMessage() before processing completed, so a crash mid-processing lost the message. I expected the client ack to happen only after processing, the way the docs describe.", "Yes, we hold a Solace support contract. Draft the email." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "Actual behavior", "fixed": true, "expect": "present", "turn": 2 }, @@ -175,11 +175,11 @@ { "id": "feedback-routing-judge", "skill": "solace-messaging-feedback", - "turns": [ + "user_turns": [ "Some feedback about the solace-application-development skill: I wish it also generated a Dockerfile and a docker-compose file for the app it builds. Today I hand-write those every time.", "We do have a Solace support contract." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "assistant_grep", "pattern": "What is the challenge?", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "What is the impact?", "fixed": true, "expect": "present", "turn": 2 }, @@ -196,11 +196,11 @@ "id": "feedback-scrub-negative", "skill": "solace-messaging-feedback", "must_pass": true, - "turns": [ + "user_turns": [ "Feedback on the JCSMP code generation: while building our schema-registry caching gateway, it generated a subscriber that never acks messages. Our setup, for context: broker tcps://mr-zq9x7.messaging.solace.cloud:55443, message VPN zenithbank-orders-vpn, client username zenithbank-svc-01, queue q.zenithbank.orders.intake. The missing ack is the problem; the connection side works fine.", "No support contract. Format it for the community." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "assistant_grep", "pattern": "zq9x7", "fixed": true, "expect": "absent", "turn": 2 }, { "type": "assistant_grep", "pattern": "[Zz]enith[Bb]ank", "expect": "absent", "turn": 2 }, @@ -215,10 +215,10 @@ { "id": "topics-grounded-answer", "skill": "solace-topic-best-practices", - "turns": [ + "user_turns": [ "We are building a smart home platform on Solace: a two-floor house, several rooms per floor, and each room has some mix of lights, thermostats, and door locks that publish state changes and receive commands. How should we structure our topic hierarchy? Which levels, and in what order?" ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" }, { "type": "tool_use", "tool": "WebFetch", "input_pattern": "docs\\.solace\\.com", "exclude_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "absent" }, @@ -235,10 +235,10 @@ { "id": "topics-no-files-negative", "skill": "solace-topic-best-practices", - "turns": [ + "user_turns": [ "Design the topic hierarchy for our smart home platform (two floors, several rooms, lights, thermostats, and door locks) and save the naming conventions to a topic-conventions.md file in this directory so the team can reference it." ], - "max_turns": 15, + "max_agent_turns": 15, "graders": [ { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, diff --git a/tools/output-evals.schema.json b/tools/output-evals.schema.json new file mode 100644 index 0000000..9299e06 --- /dev/null +++ b/tools/output-evals.schema.json @@ -0,0 +1,185 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Output-eval corpus", + "description": "Structure of plugins//evals/output-evals.json, the input to tools/run-output-evals.sh. The file is an array of cases. Each case scripts one conversation with the plugin under test and lists the graders that judge the resulting transcript and generated files. A case passes only when the target skill fired and every grader passed. Case ids must be unique across the file; JSON Schema cannot express that rule, so the runner enforces it at start-up together with the structural checks below.", + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/case" }, + "$defs": { + "case": { + "type": "object", + "description": "One scripted conversation plus its graders.", + "required": ["id", "skill", "user_turns", "graders"], + "additionalProperties": false, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", + "description": "Unique kebab-case case name. Used by the --case filter and printed on every result line. Convention: -[-positive|-negative]." + }, + "skill": { + "type": "string", + "minLength": 1, + "description": "Name of the skill that must fire during the case (the Skill tool's skill input without the plugin prefix, for example solace-application-development). The runner fails the case when this skill never fired, because the output would not be attributable to it and every absence grader would pass vacuously." + }, + "user_turns": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Ordered scripted user messages. The first starts a session; each later one continues that session with --resume. The model's replies between user turns are not scripted, so write later user turns so they cannot be misread (state consent or refusal explicitly)." + }, + "max_agent_turns": { + "type": "integer", + "minimum": 1, + "default": 25, + "description": "Budget of agent turns (model replies and tool calls) for each user turn's invocation, passed as --max-turns. Conversational cases need about 15 to 25; a full implement flow needs about 80. Hitting the cap fails the case and is tagged [max-turns] on the result line so a budget problem is distinguishable from a behavior failure." + }, + "must_pass": { + "type": "boolean", + "default": false, + "description": "When true, a failure of this case fails the whole run regardless of the pooled 90% pass rate. Reserve it for acceptance-criteria negatives and the compile anchor." + }, + "graders": { + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/grader" }, + "description": "Checks applied after all turns complete. Every grader must pass. Absence checks should be scoped to a tool or construct, never to a bare substring: comments and documents that quote a prohibition false-positive naive patterns." + } + } + }, + "grader": { + "description": "One check over the transcript or the generated files. The type field selects the check and decides which other fields are allowed.", + "oneOf": [ + { "$ref": "#/$defs/assistant_grep" }, + { "$ref": "#/$defs/tool_use" }, + { "$ref": "#/$defs/file_exists" }, + { "$ref": "#/$defs/file_grep" }, + { "$ref": "#/$defs/java_disclaimer" }, + { "$ref": "#/$defs/compile" }, + { "$ref": "#/$defs/maven_release_match" }, + { "$ref": "#/$defs/llm_judge" }, + { "$ref": "#/$defs/live_verify" } + ] + }, + "expect_presence": { + "type": "string", + "enum": ["present", "absent"], + "default": "present", + "description": "present: at least one match must exist. absent: no match may exist." + }, + "turn_scope": { + "type": "integer", + "minimum": 1, + "description": "Restrict the check to the transcript of this user turn (1-based index into user_turns). Omit to check all user turns." + }, + "fixed_string": { + "type": "boolean", + "default": false, + "description": "true: treat pattern as a fixed string (grep -F). false: treat it as an extended regular expression (grep -E)." + }, + "assistant_grep": { + "type": "object", + "description": "Grep over the assistant's text blocks only. Tool payloads and the scripted user turns are excluded so they cannot false-positive.", + "required": ["type", "pattern"], + "additionalProperties": false, + "properties": { + "type": { "const": "assistant_grep" }, + "pattern": { "type": "string", "minLength": 1, "description": "Text to look for in the assistant's replies." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "fixed": { "$ref": "#/$defs/fixed_string" }, + "turn": { "$ref": "#/$defs/turn_scope" } + } + }, + "tool_use": { + "type": "object", + "description": "Match tool_use events by tool name and, optionally, by a regular expression over the compact JSON of the tool input.", + "required": ["type", "tool"], + "additionalProperties": false, + "properties": { + "type": { "const": "tool_use" }, + "tool": { "type": "string", "minLength": 1, "description": "Tool name to match (Write, Edit, Bash, WebFetch, Skill, ...). \"*\" matches every tool." }, + "input_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression that the compact input JSON must match for the event to count." }, + "exclude_pattern": { "type": "string", "minLength": 1, "description": "Extended regular expression; matching events are dropped before counting. Use it to allow a specific benign match inside an otherwise absent check." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "count": { "type": "integer", "minimum": 0, "description": "With expect present: the exact number of matching events required. Omit to require at least one." }, + "turn": { "$ref": "#/$defs/turn_scope" } + } + }, + "file_exists": { + "type": "object", + "description": "Find generated files by basename glob under the case work directory. Maven build output under target/ is excluded.", + "required": ["type", "glob"], + "additionalProperties": false, + "properties": { + "type": { "const": "file_exists" }, + "glob": { "type": "string", "minLength": 1, "description": "Basename glob, for example pom.xml, *.java, or * for any file." }, + "expect": { "$ref": "#/$defs/expect_presence" }, + "identical_to": { "type": "string", "minLength": 1, "description": "Repository-relative path. With expect present, every matched file must be byte-identical to this file. Used to prove a bundled script was copied verbatim." } + } + }, + "file_grep": { + "type": "object", + "description": "Grep the generated files matched by a basename glob.", + "required": ["type", "glob", "pattern"], + "additionalProperties": false, + "properties": { + "type": { "const": "file_grep" }, + "glob": { "type": "string", "minLength": 1, "description": "Basename glob selecting the files to grep." }, + "pattern": { "type": "string", "minLength": 1, "description": "Text to look for in the selected files." }, + "expect": { + "type": "string", + "enum": ["present", "absent", "all_files"], + "default": "present", + "description": "present: at least one selected file matches. absent: no selected file matches (also passes when no file matches the glob). all_files: every selected file matches." + }, + "fixed": { "$ref": "#/$defs/fixed_string" } + } + }, + "java_disclaimer": { + "type": "object", + "description": "Every generated .java file starts with the AI-assisted disclaimer line and the verification-checklist pointer. Fails when no .java file exists.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "java_disclaimer" } } + }, + "compile": { + "type": "object", + "description": "Run mvn compile on every generated pom.xml (two-project layouts compile each project). Any non-zero exit fails the case; a timeout is an infrastructure failure. Requires mvn on PATH.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "compile" } } + }, + "maven_release_match": { + "type": "object", + "description": "Every generated pom.xml carries the current sol-jcsmp release version published in the Maven Central metadata. A metadata fetch failure is an infrastructure failure, not a case failure.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "maven_release_match" } } + }, + "llm_judge": { + "type": "object", + "description": "One tool-free completion on the judge model (OUTPUT_EVAL_JUDGE_MODEL) that returns a strict pass or fail verdict for one criterion. Use it for routing and grounding questions that a grep cannot decide. An unparseable verdict after two attempts is an infrastructure failure.", + "required": ["type", "criteria"], + "additionalProperties": false, + "properties": { + "type": { "const": "llm_judge" }, + "criteria": { "type": "string", "minLength": 1, "description": "The single criterion the judge grades. State the PASS condition and the FAIL condition explicitly." }, + "include": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "type": "string", "enum": ["assistant_text", "webfetch_urls", "webfetch_results"] }, + "default": ["assistant_text"], + "description": "Transcript material shown to the judge. assistant_text: the assistant's replies across all turns. webfetch_urls: the deduplicated list of URLs fetched with WebFetch. webfetch_results: the truncated content those fetches returned." + } + } + }, + "live_verify": { + "type": "object", + "description": "Run the generated project's own verify.sh roundtrip against the broker named by the exported OUTPUT_EVAL_BROKER_* variables. verify.sh exit 0 passes; exit 2 (a broker or credential error) or a timeout is an infrastructure failure; any other exit fails. Skips, and reports the skip, when no broker is configured.", + "required": ["type"], + "additionalProperties": false, + "properties": { "type": { "const": "live_verify" } } + } + } +} diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index 0abd98a..f86d16f 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -20,12 +20,20 @@ # MULTI-TURN CASES # Several contracts only resolve across turns (a design confirm, the door # question, the support-contract checkpoint), so a case carries an ordered -# "turns" array of user messages. Turn 1 starts a session; later turns +# "user_turns" array of user messages. Turn 1 starts a session; later turns # continue it via `claude -p --resume `, re-extracting the # session id after every turn. Each turn writes its own turnN.jsonl # transcript so graders can scope assertions to a turn. # -# GRADERS (closed set; the corpus schema is validated up front) +# CORPUS SCHEMA +# tools/output-evals.schema.json (JSON Schema, draft 2020-12) defines the +# corpus structure and documents the intent of every field. The start-up +# check below enforces its structural subset (array shape, required fields, +# unique ids, known grader types) and exits before any API call when the +# corpus violates it. Validate an edit against the full schema with any JSON +# Schema validator; the per-grader field rules live only in the schema. +# +# GRADERS (closed set; the corpus shape is validated up front) # assistant_grep grep over the assistant's TEXT blocks only (never the # raw JSONL: tool payloads and user turns would # false-positive). Fields: pattern, expect @@ -500,13 +508,13 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do and all(.[]; (.id | type=="string" and length>0) and (.skill | type=="string" and length>0) - and (.turns | type=="array" and length>0 and all(.[]; type=="string" and length>0)) + and (.user_turns | type=="array" and length>0 and all(.[]; type=="string" and length>0)) and ((.must_pass // false) | type=="boolean") - and ((.max_turns // 25) | type=="number") + and ((.max_agent_turns // 25) | type=="number") and (.graders | type=="array" and length>0 and all(.[]; type=="object" and (.type as $t | $types | index($t)))))' \ "$evals_file" >/dev/null 2>&1; then - echo "ERROR: $evals_file is not a non-empty array of {id, skill, turns, graders[, max_turns, must_pass]} cases with unique ids and known grader types." >&2 + echo "ERROR: $evals_file is not a non-empty array of {id, skill, user_turns, graders[, max_agent_turns, must_pass]} cases with unique ids and known grader types." >&2 rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" exit 1 fi @@ -549,8 +557,8 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do in_filter "$case_id" || continue skill="$(jq -r '.skill' <<<"$case_json")" must="$(jq -r '.must_pass // false' <<<"$case_json")" - max_turns="$(jq -r '.max_turns // 25' <<<"$case_json")" - NTURNS="$(jq -r '.turns | length' <<<"$case_json")" + max_turns="$(jq -r '.max_agent_turns // 25' <<<"$case_json")" + NTURNS="$(jq -r '.user_turns | length' <<<"$case_json")" run_pass=0; case_infra=0; fail_details=() for ((k = 1; k <= RUNS; k++)); do @@ -566,7 +574,7 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do sid="" turn_infra=0 for ((t = 1; t <= NTURNS; t++)); do - turn_prompt="$(jq -r ".turns[$((t - 1))]" <<<"$case_json")" + turn_prompt="$(jq -r ".user_turns[$((t - 1))]" <<<"$case_json")" args=(-p "$turn_prompt" --plugin-dir "$plugin_dir" --model "$MODEL" --max-turns "$max_turns" --allowedTools "${ALLOWED_TOOLS[@]}" --output-format stream-json --verbose) @@ -611,7 +619,7 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do done < <(jq -c '.graders[]' <<<"$case_json") [[ "$case_infra" -eq 1 ]] && { fail_details=("${this_run_details[@]}"); break; } - [[ "$MAXTURNS_HIT" -eq 1 ]] && this_run_details+=("[max-turns] a turn hit the $max_turns-turn cap") + [[ "$MAXTURNS_HIT" -eq 1 ]] && this_run_details+=("[max-turns] a user turn hit the max_agent_turns cap of $max_turns") if [[ ${#this_run_details[@]} -eq 0 ]]; then run_pass=$((run_pass + 1)) else From 94d4cb493df5a71f63d6a87d972a05d0c1e761a7 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Wed, 9 Sep 2026 21:47:18 -0400 Subject: [PATCH 06/11] EBP-2938: Document the timeout dependency and the credential requirement - Resolve GNU timeout with a gtimeout fallback for the mvn compile and verify.sh bounds, and warn once when neither is on PATH and a selected case needs Maven, instead of running unbounded silently - List timeout as an optional prerequisite in the evals README with the unbounded fallback and the macOS install hint - State in both README credential notes that an existing claude login does not satisfy the runner, because the throwaway config directory carries no login --- .../solace-messaging-skills/evals/README.md | 5 +++-- tools/run-output-evals.sh | 19 ++++++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index 47923fb..87baa5f 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -4,7 +4,7 @@ Each plugin ships a trigger eval corpus at `plugins//evals/trigger-evals ## Running the evals locally -To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus an exported credential (the runner uses a throwaway config directory with no ambient login). Run the script from the repository root: +To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus an exported credential. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login, so `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` must be exported. Run the script from the repository root: ```shell export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN @@ -49,7 +49,8 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json - The `claude` CLI (Claude Code) on your PATH. - `jq` and `curl` on your PATH. - `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the cases that compile generated code need it, and the runner checks for it only when such a case is selected. -- An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. The runner uses a throwaway config directory with no ambient login. Claude seat holders can mint a token with `claude setup-token`. +- An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login. Claude seat holders can mint a token with `claude setup-token`. +- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile` and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. - Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). - Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). - On macOS, `caffeinate` to keep the machine awake for a full leg. diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index f86d16f..488bca0 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -53,7 +53,9 @@ # disclaimer line and the checklist pointer; fails when # no .java exists. No fields. # compile `mvn -q -B compile` on every generated pom.xml; all -# must exit 0. No fields. +# must exit 0. Bounded at 600s when GNU timeout (or +# gtimeout) is on PATH; unbounded, with a warning, +# otherwise. No fields. # maven_release_match every generated pom carries the live sol-jcsmp # from repo1.maven.org metadata. No fields. # llm_judge one tool-free judge completion on @@ -137,6 +139,10 @@ done # Dependency and auth pre-flight (fail closed), before any mktemp so an early # exit leaks nothing. The credential value is never echoed. command -v claude >/dev/null 2>&1 || { echo "ERROR: the 'claude' CLI is not on PATH. Install @anthropic-ai/claude-code." >&2; exit 1; } +# GNU timeout (or Homebrew coreutils' gtimeout) bounds mvn and verify.sh at +# 600s. Optional: without it those steps run unbounded, and the runner says so +# below whenever a selected case would use it. +TIMEOUT_BIN="$(command -v timeout || command -v gtimeout || true)" command -v jq >/dev/null 2>&1 || { echo "ERROR: 'jq' is not on PATH." >&2; exit 1; } command -v curl >/dev/null 2>&1 || { echo "ERROR: 'curl' is not on PATH." >&2; exit 1; } # Arithmetic reads a zero or non-numeric RUNS as 0, which skips every run and @@ -392,8 +398,8 @@ grade_one() { [[ -z "$poms" ]] && { GRADER_DETAIL="no generated pom.xml to compile"; return 1; } while IFS= read -r pom; do n=$((n + 1)); log="$RUN_DIR/mvn-compile-$n.log" - if command -v timeout >/dev/null 2>&1; then - timeout 600 mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? + if [[ -n "$TIMEOUT_BIN" ]]; then + "$TIMEOUT_BIN" 600 mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? else mvn -q -B -f "$pom" compile > "$log" 2>&1; rc=$? fi @@ -463,8 +469,8 @@ grade_one() { [[ -f "$proj/verify.sh" ]] || { GRADER_DETAIL="no verify.sh beside $pom"; return 1; } log="$RUN_DIR/live-verify.log" LIVE_RAN=1 - if command -v timeout >/dev/null 2>&1; then - ( cd "$proj" && timeout 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? + if [[ -n "$TIMEOUT_BIN" ]]; then + ( cd "$proj" && "$TIMEOUT_BIN" 600 bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? else ( cd "$proj" && bash ./verify.sh roundtrip "$OUTPUT_EVAL_BROKER_HOST" "$OUTPUT_EVAL_BROKER_VPN" "$OUTPUT_EVAL_BROKER_USER" "$OUTPUT_EVAL_BROKER_PASSWORD" ) > "$log" 2>&1; rc=$? fi @@ -543,6 +549,9 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do rm -rf "$CLAUDE_CONFIG_DIR" "$WORKDIR" exit 1 fi + if [[ "$needs_mvn" -eq 1 && -z "$TIMEOUT_BIN" ]]; then + echo "WARNING: neither 'timeout' nor 'gtimeout' is on PATH; mvn compile and verify.sh run unbounded (install GNU coreutils to cap them at 600s)." >&2 + fi if [[ "$needs_live" -eq 1 ]]; then LIVE_NEEDED=1 if [[ "$LIVE_VERIFY" == "run" ]]; then From 3837000d0483a312ed769b57b58ff8c108f47ca0 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Thu, 10 Sep 2026 09:30:21 -0400 Subject: [PATCH 07/11] EBP-2938: Treat permission denials as INFRA and pin LF endings for scripts - Read permission_denials from every turn's result event: a denial of a tool the runner grants is INFRA at once, with the tool and command on the case line and no retry, because the same policy would deny the retry; a denial of any other tool is tagged on the case line - Correct the header claim that the allowlist prevents denials; managed settings, hooks, and command shims can still deny a granted tool, and a denied cp became a lossy retype in a Windows run - Add .gitattributes with *.sh text eol=lf so Windows checkouts keep shell scripts LF for bash and for the byte-identity grader on verify.sh - Document Git Bash and LF on Windows, and denials as INFRA, in the evals README --- .gitattributes | 4 ++ .../solace-messaging-skills/evals/README.md | 3 +- tools/run-output-evals.sh | 61 +++++++++++++++---- 3 files changed, 56 insertions(+), 12 deletions(-) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..fbf03fa --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +# Shell scripts must check out with LF endings on every platform. Bash rejects +# CRLF scripts, and the output evals compare a generated verify.sh byte for byte +# against the bundled reference copy. +*.sh text eol=lf diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index 87baa5f..ee84f35 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -54,6 +54,7 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json - Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). - Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). - On macOS, `caffeinate` to keep the machine awake for a full leg. +- On Windows, Git Bash. The repository's `.gitattributes` checks shell scripts out with LF endings, which bash and the byte-identity grader on `verify.sh` both need. Re-clone or run `git checkout -- .` after a checkout made with `core.autocrlf=true`. Run from the repository root: @@ -64,7 +65,7 @@ export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN ./tools/run-output-evals.sh --case appdev-quickstart-implement-full ``` -Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. +Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index 488bca0..c97b0c5 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -82,8 +82,9 @@ # Exit codes: 0 = pass rate >= 90% with no infrastructure failures and no # must-pass failures; 1 = pass rate below the gate, a must-pass failure, or # any infrastructure failure (a missing credential or tool, a malformed -# corpus, an unparseable judge verdict, or zero discovered cases); an -# unmeasured case is never absorbed by the gate. +# corpus, an unparseable judge verdict, a permission denial of a tool the +# runner grants, or zero discovered cases); an unmeasured case is never +# absorbed by the gate. # # Usage: run-output-evals.sh [--model ] [--case [,...]]... # --model Subject model. Defaults to @@ -112,10 +113,15 @@ RUNS="${OUTPUT_EVAL_RUNS:-1}" MODEL="${OUTPUT_EVAL_MODEL:-claude-sonnet-5}" JUDGE_MODEL="${OUTPUT_EVAL_JUDGE_MODEL:-claude-sonnet-5}" GRADER_TYPES='["assistant_grep","tool_use","file_exists","file_grep","java_disclaimer","compile","maven_release_match","llm_judge","live_verify"]' -# Headless -p denies unapproved tools, which would distort the measured -# behavior, so the subject gets the full list the skills declare. Bash is -# unrestricted by design (agreed for this local-only suite); every invocation -# runs in a scratch work dir. +# Headless -p denies tools outside --allowedTools, which would distort the +# measured behavior, so the subject gets the full list the skills declare. Bash +# is unrestricted by design (agreed for this local-only suite); every invocation +# runs in a scratch work dir. The allowlist does not bind the environment: +# managed settings, hooks, and local command shims can still deny a granted +# tool, and a denied tool makes the model improvise (a denied `cp` once became +# a lossy retype of a 55 KB script). Every result event carries +# permission_denials, so the runner reads it: a denial of a granted tool is +# INFRA, and a denial of any other tool is tagged on the case line. ALLOWED_TOOLS=(Skill Read Glob Grep Write Edit Bash WebFetch TodoWrite) CASE_FILTER=() @@ -265,6 +271,21 @@ turn_reason() { fi } +# denied_tools : one "toolsnippet" line per permission denial +# in the turn's result event. The snippet is the denied command (or the compact +# input), cut short, so the case line names what the environment blocked. +denied_tools() { + jq -r 'select(.type=="result") | .permission_denials[]? + | [.tool_name, ((.tool_input.command // (.tool_input | tojson)) | .[0:120])] | @tsv' "$1" 2>/dev/null +} + +# granted : 0 when the tool is in ALLOWED_TOOLS. +granted() { + local a + for a in "${ALLOWED_TOOLS[@]}"; do [[ "$a" == "$1" ]] && return 0; done + return 1 +} + # find_files : generated files under the work dir, excluding # Maven build output. "*" means any file. find_files() { @@ -569,12 +590,12 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do max_turns="$(jq -r '.max_agent_turns // 25' <<<"$case_json")" NTURNS="$(jq -r '.user_turns | length' <<<"$case_json")" - run_pass=0; case_infra=0; fail_details=() + run_pass=0; case_infra=0; fail_details=(); other_denials="" for ((k = 1; k <= RUNS; k++)); do # One whole-case retry on an infrastructure failure, each attempt from a # fresh work dir (no mid-conversation resume of a failed turn). The retry # is announced, so a flaky leg is distinguishable from a clean one. - attempt_ok=0; turn_detail="" + attempt_ok=0; turn_detail=""; denial_infra=0 for attempt in 1 2; do RUN_DIR="$WORKDIR/${case_id}/run${k}-try${attempt}" WORK="$RUN_DIR/work" @@ -595,11 +616,26 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do turn_detail="turn $t (try $attempt): $(turn_reason "$RUN_DIR/turn${t}")" break fi + # Permission denials. A denied GRANTED tool means the environment, not + # the skill, shaped this output, and the same policy would deny the + # retry too, so the case is INFRA at once. A denial of any other tool + # is the model's own doing and is only tagged on the case line. + while IFS=$'\t' read -r dtool dcmd; do + [[ -z "$dtool" ]] && continue + if granted "$dtool"; then + denial_infra=1 + turn_detail="turn $t: the environment denied $dtool, a tool the runner grants ($dcmd)" + else + other_denials+="$dtool"$'\n' + fi + done < <(denied_tools "$RUN_DIR/turn${t}.jsonl") + [[ "$denial_infra" -eq 1 ]] && break # Re-extract after every turn: a print-mode resume can mint a new id. new_sid="$(jq -r 'select(.type=="result") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | tail -1)" [[ -z "$new_sid" ]] && new_sid="$(jq -r 'select(.type=="system" and .subtype=="init") | .session_id // empty' "$RUN_DIR/turn${t}.jsonl" | head -1)" [[ -n "$new_sid" ]] && sid="$new_sid" done + [[ "$denial_infra" -eq 1 ]] && break if [[ "$turn_infra" -eq 0 ]]; then attempt_ok=1; break; fi [[ "$attempt" -eq 1 ]] && echo "RETRY [$case_id] ($skill) :: $turn_detail" done @@ -636,8 +672,11 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do fi done + denial_tag="" + [[ -n "$other_denials" ]] && denial_tag=" [denied outside the allowlist: $(sort -u <<<"$other_denials" | grep . | paste -sd' ' -)]" + if [[ "$case_infra" -eq 1 ]]; then - echo "FAIL [$case_id] ($skill) INFRA${fail_details[0]:+ :: ${fail_details[0]}}" + echo "FAIL [$case_id] ($skill) INFRA${fail_details[0]:+ :: ${fail_details[0]}}$denial_tag" fail=$((fail + 1)); infra=$((infra + 1)); continue fi @@ -645,12 +684,12 @@ for evals_file in "$REPO_ROOT"/plugins/*/evals/output-evals.json; do live_tag="" [[ "${LIVE_SKIPPED:-0}" -eq 1 ]] && live_tag=" [live verify skipped: no broker configured]" if (( run_pass * 2 > RUNS )); then - echo "PASS [$case_id] ($skill)$live_tag" + echo "PASS [$case_id] ($skill)$live_tag$denial_tag" pass=$((pass + 1)) else tag="" [[ "$must" == "true" ]] && { must_fail=$((must_fail + 1)); tag=" [must-pass]"; } - echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)$live_tag" + echo "FAIL$tag [$case_id] ($skill) ($run_pass/$RUNS runs passed)$live_tag$denial_tag" for d in "${fail_details[@]}"; do echo " - $d"; done fail=$((fail + 1)) fi From 4be5e669ed582f1bb4e228855a841988c3abd238 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Thu, 10 Sep 2026 09:30:22 -0400 Subject: [PATCH 08/11] EBP-2938: Require repro steps and a Read-only tool contract in the feedback skill - Make Steps to reproduce mandatory for bug-shaped feedback and derive it from the report when the session holds no skill invocation; a Windows sonnet run dropped the section under the old wording - Drop the Notes heading when empty and forbid None or N/A filler under it - State that Read is the skill's only tool and that the plugin manifest is read at its announced path, never located with a shell command; three-run checks showed sonnet running find for the manifest - Add a grader to feedback-support-email-positive that forbids a line consisting only of None - Three sonnet runs of the case pass by majority vote after these edits --- plugins/solace-messaging-skills/evals/output-evals.json | 1 + .../skills/solace-messaging-feedback/SKILL.md | 7 ++++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index 900b3f5..6b75a44 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -167,6 +167,7 @@ { "type": "assistant_grep", "pattern": "Steps to reproduce", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "Environment", "fixed": true, "expect": "present", "turn": 2 }, { "type": "assistant_grep", "pattern": "", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "assistant_grep", "pattern": "^[[:space:]]*(\\*\\*)?None(\\*\\*)?\\.?[[:space:]]*$", "expect": "absent", "turn": 2 }, { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, { "type": "tool_use", "tool": "Bash", "expect": "absent" } diff --git a/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md b/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md index 3f8dd82..bfed8fd 100644 --- a/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-messaging-feedback/SKILL.md @@ -148,8 +148,8 @@ After the block, on a new line: - **Actual behavior / Expected behavior** cover bug-shaped *and* feature-shaped feedback. "I wish it did X" goes in `Expected behavior` with the current state in `Actual behavior`. In the Ideas portal template the same content splits across `What is the challenge?` (the current state) and `Describe your idea` (what the user wants). - **Impact** (rendered as `What is the impact?` on the Ideas portal) is omitted when the user gave no signal of cost or friction. Don't invent severity language. - **What is the workaround?** (Ideas portal only) is how the user copes today, taken from the session. Use `None` when the session shows no workaround; don't invent one. -- **Steps to reproduce** is populated from the prompts and skill invocations of the current session when the feedback is bug-shaped. Omit entirely for feature requests and confusion reports, where repro steps don't apply. -- **Notes** is omitted when empty. Resist padding. +- **Steps to reproduce** is always present for bug-shaped feedback. Populate it from the prompts and skill invocations of the current session when they exist. When the session holds no such invocation (the user reports the bug from memory), derive the steps from the report itself: what to ask the skill, what to inspect in its output, and what to observe. Omit entirely for feature requests and confusion reports, where repro steps don't apply. +- **Notes** is omitted when empty: drop the heading. Never write `None`, `N/A`, or other filler under it. The `None` convention belongs only to the Ideas portal workaround field above. - **Signature** (Support email only) stays the literal placeholder ``. Never infer or fill in the sender's name from git config, the environment, or the session. The user types their own name when they review the email before sending. Omit any field the session doesn't support. Don't fabricate content to fill a slot. @@ -159,7 +159,7 @@ Omit any field the session doesn't support. Don't fabricate content to fill a sl Capture environment metadata best-effort. Any field that can't be determined renders as `unknown`. The skill never errors on env capture. `unknown` is always a valid value. - **Skill / API**: which skill the user was working with (for example `solace-application-development`, or the specific API such as JCSMP) and, when relevant, `solace-topic-best-practices`. Fall back to `unknown` if unclear. -- **Plugin version**: read the `version` field from the plugin manifest at `.claude-plugin/plugin.json` under the plugin root, which is two directories above this skill (resolve `../../.claude-plugin/plugin.json` from this skill's base directory). This works both when the plugin is installed and in a development checkout of this repo, where the plugin root is `plugins/solace-messaging-skills/`. Fall back to `unknown` if unreadable or missing. +- **Plugin version**: read the `version` field from the plugin manifest at `.claude-plugin/plugin.json` under the plugin root, which is two directories above this skill (resolve `../../.claude-plugin/plugin.json` from this skill's base directory). This works both when the plugin is installed and in a development checkout of this repo, where the plugin root is `plugins/solace-messaging-skills/`. Use the Read tool on that one path; the base directory is announced when the skill loads. Do not search the filesystem or run a shell command to find or read the manifest. Fall back to `unknown` if unreadable or missing. - **Model**: the active Claude model name or ID is typically surfaced to the assistant at runtime (for example `claude-opus-4-8`). Fall back to `unknown` if not surfaced. Future env fields are added the same way: attempt, fall back to `unknown`, never block. @@ -217,6 +217,7 @@ When invoked with no prior activity, the skill doesn't emit an empty draft. Ask ## Guardrails - **Formatter, not transport.** No issue creation, email sending, draft creation, community or portal posting, API calls, or file writes. The output stays in chat for the user to copy. +- **Read is the only tool.** The `allowed-tools` list (Read) is the contract, and it exists for the plugin manifest alone. Never run a shell command, search the filesystem, or call any other tool, even when the session makes one available. Everything else in the draft comes from the conversation. - **Confirm before drafting a Support email.** The support-contract question in Routing is a mandatory checkpoint. Never draft the Support email until the user confirms they hold a support contract and want the email drafted. - **User-visible before send.** Always present the full draft so the user can review and edit before pasting, sending, or posting. - **No fabrication.** Omit fields the session doesn't support rather than guessing. From 3fc7deeed43be850d0d1266d316800d153947285 Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Thu, 10 Sep 2026 11:09:48 -0400 Subject: [PATCH 09/11] EBP-2938: Copy the bundled verify.sh with cat instead of cp - Prescribe the copy command in implement-mode Step 4: cat the bundled script into the project root and chmod it, and forbid cp on the bundled file - Claude Code protects the plugin directory and blocks a cp that names a file inside it, even with Bash allowed; the blocked copy fell back to a retype of the script that must stay byte-identical, and the runner now reports that denial as INFRA - Reproduced on macOS with the runner's flags: cp denied, cat redirect allowed and byte-identical - Two sonnet runs of appdev-quickstart-implement-full with this wording used the cat form with zero denials; the second passed every grader --- .../references/jcsmp/implement-mode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md index 168a827..76b78ee 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md @@ -114,7 +114,7 @@ Generation is keyed on the `Pattern` leaf read in Step 2. Each leaf has its own **Embedded and web-app shapes.** When the JCSMP layer sits inside a larger application (a web dashboard, a REST service, a framework app), the chosen leaf's wiring still governs the messaging layer: the same connection helper, the same session/flow/reconnect/ACK handlers, the same `VERIFY:` markers emitted through a stdout-reaching `trace(...)`, and the disclaimer header on every generated file, messaging or not. The `VERIFY:` markers are NOT demo harness; they stay in the messaging layer whatever the shape, because the `app` verify stage (Step 5) greps for them in the app's captured output. -**Write the verification artifacts in this same step.** Alongside the classes and the pom, write: the tailored `solace-verification-checklist.md` at each project root (tailoring rules in Step 6; the write is generation output with NO consent gate), a copy of the bundled `scripts/verify.sh` at the project root, and the generated `verify-hooks.sh` (Step 5 defines its contract). A session that ends early still leaves the checklist and the verify entry points in the project. +**Write the verification artifacts in this same step.** Alongside the classes and the pom, write: the tailored `solace-verification-checklist.md` at each project root (tailoring rules in Step 6; the write is generation output with NO consent gate), a copy of the bundled `scripts/verify.sh` at the project root, and the generated `verify-hooks.sh` (Step 5 defines its contract). A session that ends early still leaves the checklist and the verify entry points in the project. Make the `verify.sh` copy with a read-and-redirect, `cat "/references/jcsmp/scripts/verify.sh" > verify.sh && chmod +x verify.sh`, where the skill directory is the one announced when this skill loaded. Do not use `cp` on the bundled file: Claude Code protects the plugin directory and blocks a `cp` that names a file inside it, and a blocked copy pushes the script through a retype that must stay byte-identical. Never edit the bundled script itself. ### Adapting a sample to a variant of its leaf From 28fb873716344962ed56205b2def6cb7b2efb1bb Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:11:32 -0400 Subject: [PATCH 10/11] EBP-2938: Ignore no-op commands and summary remarks in the output evals - Add exclude_pattern to the four bare Bash-absent graders so a bare echo, true, or : (plus ls and pwd on the topics negative) no longer fails a case; a redirect, a chained command, a search, or a transport call still does - Sonnet ended feedback turns with echo done in about a third of runs and opus listed the directory once before it refused to write the file, while the graded content was correct every time - Tell the grounding judge that deriving the consequences of the fetched rules for the scenario is applying the guidance, and tell every judge that WebFetch results are the tool's summaries, not the page; a summary remark that guidance was absent had failed a correct opus answer - Correct the README Windows note: git checkout -- . leaves a stale CRLF checkout in place because git treats the files as unchanged; document the delete-and-checkout recipe --- plugins/solace-messaging-skills/evals/README.md | 2 +- .../solace-messaging-skills/evals/output-evals.json | 10 +++++----- tools/run-output-evals.sh | 1 + 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index ee84f35..e8b4661 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -54,7 +54,7 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json - Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). - Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). - On macOS, `caffeinate` to keep the machine awake for a full leg. -- On Windows, Git Bash. The repository's `.gitattributes` checks shell scripts out with LF endings, which bash and the byte-identity grader on `verify.sh` both need. Re-clone or run `git checkout -- .` after a checkout made with `core.autocrlf=true`. +- On Windows, Git Bash. The repository's `.gitattributes` checks shell scripts out with LF endings, which bash and the byte-identity grader on `verify.sh` both need. A checkout made with `core.autocrlf=true` before that file existed keeps its CRLF copies, and git reports the tree clean because it treats them as unchanged, so `git pull` and `git checkout -- .` do not rewrite them. Re-clone, or delete the tracked scripts and check them out again: `git ls-files -z '*.sh' | xargs -0 rm -f && git checkout -- .`. Run from the repository root: diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index 6b75a44..7b2a7ac 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -149,7 +149,7 @@ { "type": "assistant_grep", "pattern": "support contract", "fixed": true, "expect": "present" }, { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "expect": "absent" } + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } ] }, { @@ -170,7 +170,7 @@ { "type": "assistant_grep", "pattern": "^[[:space:]]*(\\*\\*)?None(\\*\\*)?\\.?[[:space:]]*$", "expect": "absent", "turn": 2 }, { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "expect": "absent" } + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } ] }, { @@ -210,7 +210,7 @@ { "type": "assistant_grep", "pattern": "To: support@solace.com", "fixed": true, "expect": "absent" }, { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "expect": "absent" } + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" } ] }, { @@ -229,7 +229,7 @@ { "type": "llm_judge", "include": ["assistant_text", "webfetch_results"], - "criteria": "The reply must apply the fetched Topic Architecture Best Practices page to the user's scenario. PASS when the topic-structure recommendations follow the fetched guidance, including domain-specific applications of it, and when anything the page does not cover is explicitly flagged as going beyond the fetched page. FAIL when the reply contradicts the fetched guidance, presents memory-derived best-practice claims as if the page contained them, or never applies the fetched content." + "criteria": "The reply must apply the fetched Topic Architecture Best Practices page to the user's scenario. PASS when the topic-structure recommendations follow the fetched guidance, including domain-specific applications of it, and when anything the page does not cover is explicitly flagged as going beyond the fetched page. Deriving the consequences of the fetched rules for the user's scenario, such as which subscriptions a level order makes easy or hard, is applying the guidance, not going beyond it. FAIL when the reply contradicts the fetched guidance, presents memory-derived best-practice claims as if the page contained them, or never applies the fetched content." } ] }, @@ -243,7 +243,7 @@ "graders": [ { "type": "tool_use", "tool": "Write", "expect": "absent" }, { "type": "tool_use", "tool": "Edit", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "exclude_pattern": "\"command\":\"(ls|pwd|echo|true|:)( [^\"\\\\&|;>]*)?\"", "expect": "absent" }, { "type": "file_exists", "glob": "*", "expect": "absent" }, { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Topic-Architecture-Best-Practices\\.md", "expect": "present" } ] diff --git a/tools/run-output-evals.sh b/tools/run-output-evals.sh index c97b0c5..73ba6e6 100755 --- a/tools/run-output-evals.sh +++ b/tools/run-output-evals.sh @@ -464,6 +464,7 @@ grade_one() { fi if grep -qw webfetch_results <<<"$includes"; then echo; echo '--- WebFetch RESULT CONTENT (truncated) ---' + echo 'This is the summary the WebFetch tool returned to the assistant for its prompt, not the page itself. A remark in it that something is absent describes the summary, not the page.' webfetch_results | head -c 30720 fi } > "$pfile" From deeb0498dcf1323a499bdb9ae456ca65f662eb8f Mon Sep 17 00:00:00 2001 From: adiel-sammak <71227923+aelsammak@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:11:32 -0400 Subject: [PATCH 11/11] EBP-2938: Flag decisions beyond the fetched page in the topics skill - Add one sentence to step 3: when the user's scenario needs a decision the fetched page does not cover, say so before suggesting one - The grounding judge already accepts flagged extrapolation, and sonnet folded command topics into the page's template unflagged in 2 of 4 runs - Fifteen single sonnet runs of topics-grounded-answer pass after the edit; on the final tree the full sonnet leg is 11/11 and the opus leg 10/11, live verify skipped --- .../skills/solace-topic-best-practices/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md index 5c048b2..aaf1d97 100644 --- a/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-topic-best-practices/SKILL.md @@ -12,7 +12,7 @@ This skill answers topic-hierarchy and topic-architecture questions by reading t 1. WebFetch the canonical page live: `https://docs.solace.com/Messaging/Topic-Architecture-Best-Practices.md`. This is the authoritative source for topic-level ordering, naming conventions, wildcard placement, and taxonomy guidance. 2. Apply the fetched guidance to the user's specific topic-design question (the events they publish, the consumers that subscribe, the levels they need, and where wildcards belong). -3. Quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. +3. Quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. When the user's scenario needs a decision the page does not address, say so explicitly before you make a suggestion, so the reader can tell the page's guidance from your own. 4. Deliver the answer in chat. Never write it to a file, even when the user asks you to save it: present the content in chat and let the user save it themselves. Fetch the single page above, then ground every recommendation in it. Do not WebFetch other pages blindly.