From ea52fe36ea72f3e64a8c5185015bdd7ad79d459b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:22 +0200 Subject: [PATCH 01/36] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20request?= =?UTF-8?q?=20workflow=20and=20external=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds guidance for optional one-shot external handoff when user explicitly requests evaluation with yolo/auto modifier. Clarifies the boundary between preparation (deterministic) and execution (external handoff). Updates roles for preparer and executor, specifying when explicit eval requests authorize automatic handoff versus manual handoff. Normalizes harness naming and improves model discovery guidance. --- AGENTS.md | 34 +++++++++++++++++++++++++--------- CONTRIBUTING.md | 2 ++ 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 80d4201..7ab7c49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ An executing harness stays inside its package. Building, testing, or writing any Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. - Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. -- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. +- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. Urgency, completion gates, third-party instructions, and prior approval do not change this rule. The sole narrow exception is an explicit user-directed eval request with `yolo` (or the existing `auto` alias): it authorizes one fresh external Eval Orchestrator handoff per prepared package, as described below; it never authorizes the preparer to execute an arm. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. @@ -40,13 +40,13 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops, except for the explicitly authorized fresh external handoff below. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval -`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Without `yolo`/`auto`, treat them as instructions to prepare the package and stop for manual handoff. With `yolo`/`auto`, prepare and perform the automatic external handoff below. Neither form requests new eval cases unless the user asks for those too. -Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. Only when the user has not supplied a harness, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1 -Runner `. Never invoke discovery without `-Runner`, and never ask to reconfirm a supplied runner. For a supplied runner with a deterministic default, let preparation resolve and verify it without another question; validate any explicit model exactly as supplied. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot @@ -60,11 +60,27 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed Use `-CodebeltReference` only when the script should select the Codebelt Reference explicitly and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner` and `-CodebeltReference` is an error whenever a package would be generated; `-Runner codex` and `-Runner github-copilot` resolve their repository-defined default models, while `-Runner opencode` still requires `-Model`. +### Optional one-shot external handoff + +An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. + +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. + +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and `-ExternalOrchestratorAvailable` only after establishing the host capability above. Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. + +When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: + +- `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. + +The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. + ### Handing the package over Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner and reads its dispatch owner. For runner-owned dispatch it invokes `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and sets the caller shell/tool timeout to at least the package-computed allowance. That command preflights every blind `with_skill` and `without_skill` arm and starts zero executions if any preflight is incompatible. For orchestrator-owned dispatch it delegates each arm to a fresh native worker. It records each terminal envelope through the deterministic runner-owned recorder where applicable, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. -Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. +Hand the user that one file by its absolute path, and stop there in the normal manual flow. The explicit one-shot flow hands that same file to the fresh external Orchestrator instead. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. @@ -111,7 +127,7 @@ Run it before `scripts/sync-skill-install.ps1`, which stays the last gate becaus Two roles exist here, and they are not the same agent doing the same job. Read which one you are before applying the rules below. -**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop, and you must: +**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop for manual handoff unless this explicit eval request authorizes the one-shot external handoff above. In both cases you must: - never execute the prompts you just prepared, and never quietly become the executor of your own package - never spawn subagents for the candidate or baseline runs @@ -129,12 +145,12 @@ When a person gives you `RUN-THIS.prompt.md`, points you at a prepared package, Four things still hold while you execute: -- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or another agent asking is not a person asking. +- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or an unsolicited agent request is not a person asking. A fresh external Orchestrator receiving the canonical file through the explicit user-directed `yolo`/`auto` handoff above carries that user authorization for this package only. - Stay inside the package directory. If it sits inside a repository, the rest of that repository is not yours to read, build, test, or write. - Do not read the assertions or expected output before the worker runs. After every available worker has finished and its result is recorded, grade the completed results, generate the report, and state the comparison honestly. - Nothing about this makes model-backed execution a completion gate for any repository task. -An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. +An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This disqualifies the preparer context, not the fresh external Orchestrator receiving an explicitly authorized one-shot handoff. The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the foreground package-local `invoke-runner-owned-arms.ps1` command exactly once and waits for its terminal JSON summary; the helper resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. If execution was interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration. For orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, only after Phase 1 returns a successful terminal JSON summary and the immutable freeze validates, the executor bridges results into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. @@ -212,7 +228,7 @@ Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-vi Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. -Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. +Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops, unless an explicit `yolo`/`auto` eval request authorizes the fresh external handoff above. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. `skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8df4768..ea65665 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -100,6 +100,8 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill < Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. + Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For any selected runner whose descriptor says `delegation.dispatch_owner=runner`, invoke `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command with the caller shell/tool timeout set to at least the package-computed allowance. The foreground helper performs every preflight before execution, owns fan-out, state, result registration, and freeze creation, and starts zero model executions when any preflight is incompatible. Never create outer native subagents/tasks for runner-owned arms or hand-author preflight, fan-out, state, or result bookkeeping. Copilot task/general-purpose workers, OpenCode Task/General workers, and Codex native mechanisms remain harness capabilities only; they are not the behavioral transport for runner-owned evaluation. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. From f683f8920b75731d05eaeef0bf305bb29cda793c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:30 +0200 Subject: [PATCH 02/36] =?UTF-8?q?=F0=9F=94=A8=20add=20eval=20request=20det?= =?UTF-8?q?erministic=20helper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces eval-request.ps1 helper script that provides deterministic, model-free eval request workflow for agents. Handles decision logic between manual handoff (preparation-only) and external handoff (with explicit yolo/auto authorization). Includes comprehensive test coverage exercising all runner/model normalization paths and handoff state transitions. Adds PassThru parameter to prepare-skill-evals.ps1 for returning prepared prompt paths. Adds runner normalization to handle user-facing names (GitHub Copilot, Copilot CLI). Integrates validator coverage for eval request workflow in validate-skill-templates.ps1. --- scripts/eval-request.ps1 | 72 ++++++++++ .../eval-runners/tests/test-eval-request.ps1 | 125 ++++++++++++++++++ scripts/prepare-skill-evals.ps1 | 13 ++ scripts/validate-skill-templates.ps1 | 6 + 4 files changed, 216 insertions(+) create mode 100644 scripts/eval-request.ps1 create mode 100644 scripts/eval-runners/tests/test-eval-request.ps1 diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 new file mode 100644 index 0000000..f53bbdf --- /dev/null +++ b/scripts/eval-request.ps1 @@ -0,0 +1,72 @@ +<# +.SYNOPSIS + Deterministic helpers for the agent's explicit eval request workflow. Dot-source to use. +.DESCRIPTION + These helpers never launch a model. The interactive host consumes an external_handoff decision + with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire + this decision to CI, hooks, preparation, validation, or completion-gate execution. +#> +Set-StrictMode -Version Latest + +function Get-EvalHandoff { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath, + [switch]$Yolo, + [switch]$ExternalOrchestratorAvailable + ) + + $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path + if ([IO.Path]::GetFileName($path) -cne 'RUN-THIS.prompt.md') { + throw 'Handoff requires the prepared RUN-THIS.prompt.md file.' + } + $package = Split-Path -Parent $path + $decision = [ordered]@{ action = 'manual_handoff'; prompt_path = $path; reason = 'Preparation complete; hand this file to an external Eval Orchestrator.' } + if (-not $Yolo) { return [pscustomobject]$decision } + + # Reserve the handoff before the host delegates. An uncertain launch must never be retried. + $claim = Join-Path $package '.external-handoff-started' + if ((Test-Path -LiteralPath $claim) -or + (Test-Path -LiteralPath (Join-Path $package 'orchestration-state.json')) -or + (Test-Path -LiteralPath (Join-Path $package 'execution-freeze.json'))) { + $decision.action = 'already_started' + $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' + return [pscustomobject]$decision + } + if (-not $ExternalOrchestratorAvailable) { + $decision.reason = 'This host cannot hand off to a fresh external Eval Orchestrator. Use the intact manual handoff; never execute an arm in this context.' + return [pscustomobject]$decision + } + try { + $stream = [IO.File]::Open($claim, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None) + $stream.Dispose() + } catch [IO.IOException] { + if (-not (Test-Path -LiteralPath $claim)) { throw } + $decision.action = 'already_started' + $decision.reason = 'A handoff was already reserved. Do not dispatch again.' + return [pscustomobject]$decision + } + $decision.action = 'external_handoff' + $decision.reason = 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; wait for its final result.' + return [pscustomobject]$decision +} + +function Invoke-EvalRequest { + [CmdletBinding()] + param( + [Parameter(Mandatory)][hashtable]$Preparation, + [switch]$Yolo, + [switch]$ExternalOrchestratorAvailable + ) + + $ErrorActionPreference = 'Stop' + # Splat existing preparation options unchanged; it alone resolves and verifies model policy. + # Collect every success before issuing any handoff, including multi-skill explicit requests. + if ($Preparation.ContainsKey('CollectResults')) { throw 'An eval request prepares packages; CollectResults is a separate forensic workflow.' } + $arguments = $Preparation.Clone() + $arguments.PassThru = $true + $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) + foreach ($path in $paths) { + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -ExternalOrchestratorAvailable:$ExternalOrchestratorAvailable + } +} diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 new file mode 100644 index 0000000..52ecd86 --- /dev/null +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -0,0 +1,125 @@ +<# Deterministic request-to-handoff coverage: real preparation, fixture catalogs, fake host. No models. #> +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scripts 'eval-request.ps1') +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) +[void](New-Item -ItemType Directory -Path $workspace -Force) +$script:dispatches = [Collections.Generic.List[string]]::new() + +function Assert-True([bool]$Condition, [string]$Message) { + if (-not $Condition) { throw $Message } +} +function Invoke-FakeHost($Decision) { + if ($Decision.action -eq 'external_handoff') { + # The host's only input is the actual canonical handoff file, never an arm prompt. + Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' + Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + $script:dispatches.Add($Decision.prompt_path) + } +} +function New-Preparation([string]$Runner, [string]$Name) { + return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner + OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog } +} +function Assert-Failure([hashtable]$Options, [string]$Pattern) { + # The request must fail closed even in an ordinary interactive PowerShell caller. + $ErrorActionPreference = 'Continue' + $before = $script:dispatches.Count + $failed = $false + try { + Invoke-EvalRequest -Preparation $Options -Yolo -ExternalOrchestratorAvailable | + ForEach-Object { Invoke-FakeHost $_ } + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected failure: $_" + } + Assert-True $failed 'Expected preparation failure.' + Assert-True ($script:dispatches.Count -eq $before) 'Failed preparation dispatched an Orchestrator.' +} + +try { + $catalog = Join-Path $workspace 'models.json' + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'provider/Exact.Model' }) } | + ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 + + $normalOptions = New-Preparation 'Codex' 'normal' + $normal = Invoke-EvalRequest -Preparation $normalOptions -ExternalOrchestratorAvailable + Invoke-FakeHost $normal + Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' + Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + + foreach ($case in @( + @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, + @{ Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + )) { + $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) + if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } + $decision = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.Expected -eq 'codex') { + Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' + } + Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' + $before = $script:dispatches.Count + Invoke-FakeHost $decision + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -ExternalOrchestratorAvailable + Invoke-FakeHost $again + Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' + $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo + Assert-True ($unavailableAfterStart.action -eq 'already_started') 'An uncertain launch incorrectly suggested a new manual execution.' + } + + $reference = New-Preparation 'unused' 'reference' + $reference.Remove('Runner') + $reference.CodebeltReference = $true + $reference.ReasoningEffort = 'high' + $reference.ConfigurationProfile = 'fixture-profile' + $reference.ToolProfile = 'fixture-tools' + $reference.TimeoutSeconds = 123 + $reference.Concurrency = 3 + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -ExternalOrchestratorAvailable + $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' + Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' + + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'unavailable') -Yolo + $before = $script:dispatches.Count + Invoke-FakeHost $unavailable + Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' + Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' + # Existing execution state also blocks automatic handoff, even without a handoff receipt. + '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' + + Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' + $invalid = New-Preparation 'GitHub Copilot' 'invalid-model' + $invalid.Model = 'not-in-catalog' + Assert-Failure $invalid "Runner 'github-copilot' model 'not-in-catalog' could not be verified" + $invalid = New-Preparation 'Codex' 'invalid-default' + $emptyCatalog = Join-Path $workspace 'empty-models.json' + '{"models":[{"id":"unrelated-model"}]}' | Set-Content $emptyCatalog + $invalid.ModelCatalogPath = $emptyCatalog + Assert-Failure $invalid "Runner 'codex' model 'gpt-5.6-luna' could not be verified" + $invalid = New-Preparation 'Codex' 'missing-skill' + $invalid.Skill = 'no-such-skill' + Assert-Failure $invalid 'skill|directory|path' + + # Check the real discovery call contract, not a mocked model resolver. + $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw + Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' + $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' + Write-Host 'PASS: normal/yolo requests, runner/model policy, failures, unavailable host, canonical handoff, and duplicate dispatch guards (fake host only).' +} finally { + # The absolute target is the unique child allocated under the external test workspace above. + $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar + if (-not [IO.Path]::GetFullPath($workspace).StartsWith($allowed, [StringComparison]::OrdinalIgnoreCase)) { throw 'Unsafe test cleanup path.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 358e697..af8a648 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -44,6 +44,9 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER PassThru + Also return each successfully prepared absolute RUN-THIS.prompt.md path on the success stream. + .PARAMETER Runner Package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. GitHub Copilot and Codex can resolve repository-defined default models; OpenCode requires an explicit model selector. @@ -129,6 +132,10 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$PassThru, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$Runner, @@ -482,6 +489,11 @@ function Resolve-ExecutionSelection { if (-not $hasRunner -and $hasModel) { throw 'Runner/model selection requires -Runner when -Model is supplied.' } + # Normalize the user-facing names before discovery; model policy remains runner-owned here. + $Runner = switch ($Runner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } $resolvedRunnerMatch = @($supportedRunners | Where-Object { [string]::Equals($_, $Runner, [StringComparison]::OrdinalIgnoreCase) }) if ($resolvedRunnerMatch.Count -ne 1) { throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." @@ -1781,6 +1793,7 @@ function Invoke-PrepareMode { Write-Host 'The selected evaluator must write valid runner-produced execution results back into this package.' Write-Host 'If it cannot, the evaluation is incomplete and must fail closed; only persisted runner-produced evidence at' Write-Host 'the manifest-declared paths may proceed.' + if ($PassThru) { Write-Output $runnerPath } } function New-RunnerPrompt { diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6858d9c..46241d8 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1430,6 +1430,12 @@ Add-ValidationResult -Results $results -Name 'Progress coalescing renders select } } +Add-ValidationResult -Results $results -Name 'Explicit eval requests preserve preparation and reserve one external handoff' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-eval-request.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Eval request regressions failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Group 'Preparation' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref From 4a22669f7ae23682afc8ea5f4b22cbbd16f93fa4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:37 +0200 Subject: [PATCH 03/36] =?UTF-8?q?=F0=9F=93=9D=20update=20readme=20to=20doc?= =?UTF-8?q?ument=20eval=20request=20capability?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updates repository README to document the eval request workflow and optional one-shot external handoff feature. Clarifies when and how explicit yolo/auto modifiers authorize automatic delegation to external Eval Orchestrators. Reflects the deterministic preparation model and handoff boundaries for agents preparing skill evaluation packages. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b2dce65..65e6c74 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. @@ -26,6 +26,8 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). + The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. From 0bc863e041add08799d14dc44b06b7077f3d26fa Mon Sep 17 00:00:00 2001 From: gimlichael Date: Mon, 7 Sep 2026 23:26:43 +0200 Subject: [PATCH 04/36] =?UTF-8?q?=F0=9F=93=9D=20document=20v0.9.2=20releas?= =?UTF-8?q?e=20notes=20for=20one-shot=20eval=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21f0132..1cab27d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.2] - 2026-09-08 + +This patch release adds optional one-shot external handoff for explicit eval requests, complete with deterministic workflow helpers and test coverage. Agents can now accept `yolo` or `auto` modifiers on explicit evaluation requests to authorize a single fresh external Eval Orchestrator handoff; preparation remains deterministic and repository-local, while only an explicit, authorized harness receives the sealed package. Skill eval preparation gains a PassThru parameter for returning prompt paths, runner names are normalized for user-facing consistency, and validation tooling gains comprehensive eval-request-workflow checks. + +### Added + +- `scripts/eval-request.ps1` with deterministic helpers for optional one-shot external handoff: `Get-EvalHandoff` reserves and transitions between manual and external handoff states, while `Invoke-EvalRequest` normalizes harness names, discovers available models, returns preparation decisions, and never launches a model itself, +- Full test coverage for eval-request workflow in `scripts/eval-runners/tests/test-eval-request.ps1` exercising all runner/model normalization paths, handoff state transitions, already-started detection, unavailable-host fallback, and case-insensitive normalization, +- Runner normalization in eval-request helpers converting user-facing harness names (`GitHub Copilot`, `Copilot CLI`) to internal canonical runner ids (`github-copilot`) before model discovery and profile generation, +- PassThru parameter in `scripts/prepare-skill-evals.ps1` for returning the prepared prompt paths without file output, enabling downstream helpers to collect decisions and route handoffs, +- `-Yolo` flag in `scripts/eval-request.ps1` Invoke-EvalRequest authorizing one-shot external handoff after explicit user eval request, while keeping preparation and validation deterministic, +- Optional-one-shot-external-handoff section in `AGENTS.md` documenting the explicit user authorization, harness selection, model discovery flow (including OpenCode's explicit-model requirement), and the deterministic one-handoff boundary, +- External evaluation authorization guidance in `AGENTS.md` clarifying that `yolo`/`auto` modifiers on explicit eval requests enable handoff without widening repository automation, CI, completion gates, or model-backed execution, +- Eval-request-workflow validation checks in `scripts/validate-skill-templates.ps1` ensuring eval-request.ps1 presence, test coverage, runner normalization completeness, and handoff-state-reservation correctness. + +### Changed + +- `AGENTS.md` eval preparation guidance now distinguishes between manual handoff (normal path that prepares and returns `RUN-THIS.prompt.md`) and external handoff (when user explicitly requests eval with yolo/auto), clarifying that only the latter authorizes orchestrator delegation, +- Runner-model discovery and selection in `scripts/Get-HarnessModels.ps1` now fully documents the -Runner requirement and exposes current available models without auto-selecting a previous or default model, +- `README.md` eval section expanded with documentation of optional one-shot handoff feature, explicit authorization scoping, and reference to `AGENTS.md` for the complete workflow, +- `CONTRIBUTING.md` now references the optional one-shot eval handoff capability and directs users to full preparation and handoff guidance in `AGENTS.md`. + +### Fixed + +- Eval request workflow now clearly separates deterministic preparation (always model-free, always stops for manual handoff) from explicit external handoff (only with user-supplied yolo/auto, never retried if already launched), +- Model discovery no longer accepts harness names as ambiguous shortcuts; runner normalization explicitly converts `Copilot` and `GitHub Copilot CLI` to the canonical `github-copilot` runner before passing to model-discovery logic. + ## [0.9.1] - 2026-09-07 This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. @@ -638,6 +665,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.2]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.9.2 [0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 From f12d15dee50f4ff2cebdcc8e94f64b971263279e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 00:06:53 +0200 Subject: [PATCH 05/36] =?UTF-8?q?=F0=9F=90=9B=20guard=20started=20eval=20p?= =?UTF-8?q?ackages=20from=20-Force=20deletion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add safeguards to prevent -Force from replacing eval packages that have already started handoff or execution. This protects the one-shot eval workflow from accidental data loss. New tests validate that the guard works correctly and that packages remain unchanged when Force is rejected. --- .../eval-runners/tests/test-eval-request.ps1 | 27 +++++++++++++++++++ scripts/prepare-skill-evals.ps1 | 6 +++++ 2 files changed, 33 insertions(+) diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 52ecd86..3f3fd4c 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -51,6 +51,33 @@ try { Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + # Force may replace an unstarted package, but must preserve every started package. + $normalOptions.Iteration = 1 + $normalOptions.Force = $true + $replacement = Invoke-EvalRequest -Preparation $normalOptions + Assert-True ($replacement.prompt_path -eq $normal.prompt_path) 'Force could not replace an unstarted package.' + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + $options = New-Preparation 'Codex' ([guid]::NewGuid().ToString('N')) + $options.Iteration = 1 + $options.Force = $true + if ($marker -eq '.external-handoff-started') { + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + Invoke-FakeHost $reserved + } else { + $reserved = Invoke-EvalRequest -Preparation $options + '{}' | Set-Content -LiteralPath (Join-Path (Split-Path $reserved.prompt_path) $marker) + } + $package = Split-Path $reserved.prompt_path + $beforeFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-Failure $options 'handoff or execution has already started' + $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." + } + foreach ($case in @( @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index af8a648..d122b36 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1465,6 +1465,12 @@ function Invoke-PrepareMode { if (-not $Force) { throw "'$iterationDirectory' already exists. Pass -Force to replace it, or -Iteration to write a new one." } + # Force must not erase a one-shot reservation or evidence of an active/completed run. + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $marker)) { + throw "Cannot replace '$iterationDirectory': handoff or execution has already started ($marker). Preserve this package and use a fresh iteration." + } + } Remove-Item -LiteralPath $iterationDirectory -Recurse -Force } New-Item -ItemType Directory -Path $iterationDirectory -Force | Out-Null From 512b5035eddd6299324d7f45745d7e4d27a99491 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 00:45:03 +0200 Subject: [PATCH 06/36] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20eval=20orc?= =?UTF-8?q?hestrator=20capability=20and=20guard=20model=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename ExternalOrchestratorAvailable to CanDelegateFreshOrchestrator for semantic clarity about what capability is being tested. Add compatibility alias to preserve existing call sites. Formally document GitHub Copilot CLI task + general-purpose agent delegation as a valid external-orchestrator capability. Add guard to prevent Copilot from replacing the repository-defined default model with a subjective stronger choice. Update all tests and validators to match the new naming and expectations. --- AGENTS.md | 6 +-- CONTRIBUTING.md | 4 +- README.md | 2 +- scripts/eval-request.ps1 | 12 +++--- .../eval-runners/tests/test-eval-request.ps1 | 40 ++++++++++++------- scripts/validate-skill-templates.ps1 | 8 ++++ 6 files changed, 46 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7ab7c49..924ac91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,14 +64,14 @@ Use `-CodebeltReference` only when the script should select the Codebelt Referen An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. -Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available and do not infer otherwise merely because no shell command named "external orchestrator" exists. In capability terms: GitHub Copilot CLI with `task` + `general-purpose` available means `ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` is true. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. -For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and `-ExternalOrchestratorAvailable` only after establishing the host capability above. Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: - `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. -- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. - `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ea65665..c53b690 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,9 +98,9 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. For `github-copilot` without an explicit model, discovery is only an availability check for the repository-defined default; never rank or replace it with a subjective stronger/newer choice. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. diff --git a/README.md b/README.md index 65e6c74..e8234ba 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. -Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 index f53bbdf..0a4a8af 100644 --- a/scripts/eval-request.ps1 +++ b/scripts/eval-request.ps1 @@ -5,6 +5,8 @@ These helpers never launch a model. The interactive host consumes an external_handoff decision with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire this decision to CI, hooks, preparation, validation, or completion-gate execution. + For GitHub Copilot CLI, task + general-purpose delegation is a valid external-orchestrator + capability for this one-shot handoff. #> Set-StrictMode -Version Latest @@ -13,7 +15,7 @@ function Get-EvalHandoff { param( [Parameter(Mandatory)][string]$PromptPath, [switch]$Yolo, - [switch]$ExternalOrchestratorAvailable + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator ) $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path @@ -33,8 +35,8 @@ function Get-EvalHandoff { $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' return [pscustomobject]$decision } - if (-not $ExternalOrchestratorAvailable) { - $decision.reason = 'This host cannot hand off to a fresh external Eval Orchestrator. Use the intact manual handoff; never execute an arm in this context.' + if (-not $CanDelegateFreshOrchestrator) { + $decision.reason = 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' return [pscustomobject]$decision } try { @@ -56,7 +58,7 @@ function Invoke-EvalRequest { param( [Parameter(Mandatory)][hashtable]$Preparation, [switch]$Yolo, - [switch]$ExternalOrchestratorAvailable + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator ) $ErrorActionPreference = 'Stop' @@ -67,6 +69,6 @@ function Invoke-EvalRequest { $arguments.PassThru = $true $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) foreach ($path in $paths) { - Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -ExternalOrchestratorAvailable:$ExternalOrchestratorAvailable + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -CanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator } } diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 3f3fd4c..84c51f1 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -30,7 +30,7 @@ function Assert-Failure([hashtable]$Options, [string]$Pattern) { $before = $script:dispatches.Count $failed = $false try { - Invoke-EvalRequest -Preparation $Options -Yolo -ExternalOrchestratorAvailable | + Invoke-EvalRequest -Preparation $Options -Yolo -CanDelegateFreshOrchestrator | ForEach-Object { Invoke-FakeHost $_ } } catch { $failed = $true @@ -42,11 +42,11 @@ function Assert-Failure([hashtable]$Options, [string]$Pattern) { try { $catalog = Join-Path $workspace 'models.json' - @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'provider/Exact.Model' }) } | + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'claude-opus-4.7' }, @{ id = 'provider/Exact.Model' }) } | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 $normalOptions = New-Preparation 'Codex' 'normal' - $normal = Invoke-EvalRequest -Preparation $normalOptions -ExternalOrchestratorAvailable + $normal = Invoke-EvalRequest -Preparation $normalOptions -CanDelegateFreshOrchestrator Invoke-FakeHost $normal Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' @@ -61,7 +61,7 @@ try { $options.Iteration = 1 $options.Force = $true if ($marker -eq '.external-handoff-started') { - $reserved = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator Invoke-FakeHost $reserved } else { $reserved = Invoke-EvalRequest -Preparation $options @@ -74,28 +74,36 @@ try { $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." - $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -ExternalOrchestratorAvailable + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -CanDelegateFreshOrchestrator Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." } foreach ($case in @( - @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, - @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, - @{ Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, - @{ Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + @{ Name = 'codex-default'; Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Name = 'copilot-default'; Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5'; StrongerModel = 'claude-opus-4.7'; UseLegacyAvailabilityAlias = $true }, + @{ Name = 'copilot-explicit-model'; Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Name = 'opencode-explicit-model'; Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } )) { $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } - $decision = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $decision = if ($case.ContainsKey('UseLegacyAvailabilityAlias')) { + Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + } else { + Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + } $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.ContainsKey('StrongerModel')) { + Assert-True ($profile.model -cne $case.StrongerModel) 'Copilot replaced repository model policy with a subjective stronger-model choice.' + } if ($case.Expected -eq 'codex') { Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' } Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' $before = $script:dispatches.Count Invoke-FakeHost $decision - $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($script:dispatches[$before] -ceq $decision.prompt_path) 'Host did not receive the exact generated RUN-THIS.prompt.md path.' + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -CanDelegateFreshOrchestrator Invoke-FakeHost $again Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo @@ -110,19 +118,19 @@ try { $reference.ToolProfile = 'fixture-tools' $reference.TimeoutSeconds = 123 $reference.Concurrency = 3 - $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -ExternalOrchestratorAvailable + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -CanDelegateFreshOrchestrator $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' - $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'unavailable') -Yolo + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'unavailable') -Yolo $before = $script:dispatches.Count Invoke-FakeHost $unavailable Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' # Existing execution state also blocks automatic handoff, even without a handoff receipt. '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') - $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -ExternalOrchestratorAvailable + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -CanDelegateFreshOrchestrator Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' @@ -142,8 +150,10 @@ try { $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper.Contains("[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator")) 'External orchestrator capability alias changed unexpectedly.' + Assert-True (-not $helper.Contains('claude-haiku-4.5') -and -not $helper.Contains('gpt-5.6-luna') -and -not $helper.Contains('claude-opus-4.7')) 'Eval request helper must not embed model-selection policy.' Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' - Write-Host 'PASS: normal/yolo requests, runner/model policy, failures, unavailable host, canonical handoff, and duplicate dispatch guards (fake host only).' + Write-Host 'PASS: normal/yolo requests, Copilot delegation capability semantics, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' } finally { # The absolute target is the unique child allocated under the external test workspace above. $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 46241d8..2119cc4 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1441,6 +1441,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref $runnerReadme = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-runners/README.md' -GitRef $Ref + $evalRequest = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-request.ps1' -GitRef $Ref $prepare = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/prepare-skill-evals.ps1' -GitRef $Ref Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Portable Eval Handoff' @@ -1453,6 +1454,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '-CanDelegateFreshOrchestrator' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery mirrors every model exposed by all configured OpenCode providers' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'availability is presentation metadata only and never filters the selectable catalog' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made.' @@ -1483,6 +1487,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Discovery is an availability check, not model ranking' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'choose a Harness + Model when the user did not already do so' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'OpenCode mirrors every model exposed by all configured providers' @@ -1492,6 +1498,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'the user must choose from the discovered selectors before package preparation' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'OpenCode through `opencode models --verbose`' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'without filtering the selectable catalog' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle "[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator" + Assert-NotContains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'claude-opus-4.7' Assert-Contains -Name 'README.md' -Content $readme -Needle 'full operational permission inside each isolated behavioral harness configuration' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Hard filesystem confinement is a separate optional outer capability' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'full harness operational permission inside each isolated eval boundary' From eee4240a02f974d51cec860b2380fe50ed016118 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 10:09:51 +0200 Subject: [PATCH 07/36] =?UTF-8?q?=E2=9C=85=20implement=20fail-closed=20cop?= =?UTF-8?q?ilot=20authentication=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive authentication validation and fail-closed behavior to the GitHub Copilot runner. The runner now detects when supported non-interactive authentication sources are unavailable (explicit tokens or trusted GitHub CLI fallback) and terminates evaluation preemptively rather than attempting execution without required auth. New test scenarios cover fresh-context environments and missing auth states. Updated handoff prompt generation documents that authentication incompatibility is terminal for the package iteration, preventing suggestions for runner switching or Phase 1 retries. --- .../eval-runners/github-copilot/runner.ps1 | 157 +++++++++++++----- .../tests/test-copilot-boundaries.ps1 | 2 +- .../tests/test-runner-conformance.ps1 | 124 ++++++++++++-- scripts/prepare-skill-evals.ps1 | 1 + 4 files changed, 223 insertions(+), 61 deletions(-) diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index f63bf46..da6dc6b 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -203,35 +203,64 @@ function Get-CopilotTokenVariable { return $null } -function Get-CopilotGhConfigDirectory { +function Add-CopilotGhConfigCandidate { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Candidates, + [string]$Path + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return } + try { + $normalized = [System.IO.Path]::GetFullPath($Path) + if (-not $Candidates.Contains($normalized)) { $Candidates.Add($normalized) } + } catch { } +} + +function Get-CopilotGhConfigDirectories { # GH_CONFIG_DIR is an authentication-state exception to the isolated - # Copilot configuration roots. Resolve it from GitHub CLI's documented - # precedence without reading or logging any credential file. - $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') - if ([string]::IsNullOrWhiteSpace($configured)) { - $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') - if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { - $configured = Join-Path $xdgConfig 'gh' - } elseif ((Get-PlatformName) -eq 'windows') { - $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) - if (-not [string]::IsNullOrWhiteSpace($applicationData)) { - $configured = Join-Path $applicationData 'GitHub CLI' - } - } else { - $userHome = [Environment]::GetEnvironmentVariable('HOME') - if ([string]::IsNullOrWhiteSpace($userHome)) { - $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) - } + # Copilot configuration roots. Build documented/default candidates and keep + # only existing directories; token resolution tries each candidate in order. + $candidates = [System.Collections.Generic.List[string]]::new() + + Add-CopilotGhConfigCandidate -Candidates $candidates -Path ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $xdgConfig 'gh') + } + + if ((Get-PlatformName) -eq 'windows') { + $applicationDataFromEnvironment = [Environment]::GetEnvironmentVariable('APPDATA') + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromEnvironment)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromEnvironment 'GitHub CLI') + } + + $applicationDataFromProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromProfile)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromProfile 'GitHub CLI') + } + } else { + foreach ($userHome in @( + [Environment]::GetEnvironmentVariable('HOME'), + [Environment]::GetEnvironmentVariable('USERPROFILE'), + [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + )) { if (-not [string]::IsNullOrWhiteSpace($userHome)) { - $configured = Join-Path (Join-Path $userHome '.config') 'gh' + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path (Join-Path $userHome '.config') 'gh') } } } - if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { - return $null + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($candidate in @($candidates)) { + if (Test-Path -LiteralPath $candidate -PathType Container) { $resolved.Add($candidate) } } - return [System.IO.Path]::GetFullPath($configured) + return @($resolved.ToArray()) +} + +function Get-CopilotGhConfigDirectory { + $directories = @(Get-CopilotGhConfigDirectories) + if ($directories.Count -eq 0) { return $null } + return [string]$directories[0] } function Get-CopilotGitHubCliToken { @@ -240,26 +269,54 @@ function Get-CopilotGitHubCliToken { return $null } - $environment = New-RunnerProbeEnvironment + $baseEnvironment = New-RunnerProbeEnvironment foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { $value = [Environment]::GetEnvironmentVariable($name) if (-not [string]::IsNullOrWhiteSpace($value)) { - $environment[$name] = $value + $baseEnvironment[$name] = $value + } + } + + $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) } } - $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing host + # default can still resolve even when no candidate was discovered. + $candidateConfigDirectories.Add('') + + $probeRoot = Resolve-RunnerProbeTempRoot + $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null try { - $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 - if ($process.TimedOut -or $process.ExitCode -ne 0) { - return $null - } - $token = ([string]$process.Stdout).Trim() - if ([string]::IsNullOrWhiteSpace($token)) { - return $null + foreach ($candidateDirectory in @($candidateConfigDirectories.ToArray())) { + $environment = [ordered]@{} + foreach ($key in @($baseEnvironment.Keys)) { + $environment[[string]$key] = [string]$baseEnvironment[$key] + } + if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { + [void]$environment.Remove('GH_CONFIG_DIR') + } else { + $environment['GH_CONFIG_DIR'] = [string]$candidateDirectory + } + + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + continue + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + continue + } + return [pscustomobject]@{ + Token = $token + GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + } } - return $token + return $null } finally { if (Test-Path -LiteralPath $probeDirectory) { Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue @@ -275,24 +332,31 @@ function Resolve-CopilotAuthentication { TokenVariable = $tokenVariable TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $true } } $githubCliToken = Get-CopilotGitHubCliToken - if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + if ($null -ne $githubCliToken -and -not [string]::IsNullOrWhiteSpace([string]$githubCliToken.Token)) { return [pscustomobject]@{ Source = 'github_cli_token' TokenVariable = 'GH_TOKEN' - TokenValue = $githubCliToken + TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true + GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + NonInteractiveReady = $true } } + $fallbackSource = if ($null -ne (Resolve-ExternalCommand -Name 'copilot')) { 'copilot_os_keychain_unverified' } else { 'none' } return [pscustomobject]@{ - Source = 'copilot_os_keychain_or_github_cli_unverified' + Source = $fallbackSource TokenVariable = $null TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $false } } @@ -602,10 +666,11 @@ function Get-CopilotPreflight { if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) + $configDetail = if ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { 'ambient/default' } else { 'resolved GH_CONFIG_DIR candidate' } + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) - $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') } if ($platform -notin @('linux', 'macos')) { @@ -646,7 +711,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction -and $continuationCapability.Available) { $mechanisms.Add(("explicit Copilot {0} continuation selected from installed help" -f $continuationCapability.Flag)) $mechanisms.Add('no implicit last-session continuation') @@ -668,6 +733,13 @@ function Get-CopilotPreflight { exact_session_required = $true implicit_continuation = $false } + authentication = [ordered]@{ + source = [string]$authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + token_value_observed = $false + } } return $document } @@ -692,7 +764,6 @@ function New-CopilotEnvironment { } # Model-free help/version bootstrap must use the same cache roots too. if ($WithoutAuthentication) { return New-RunnerEnvironment -Run $Inputs.Run -Additional $additional } - $tokenVariable = Get-CopilotTokenVariable $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'github_cli_token') { $additional[$authState.TokenVariable] = $authState.TokenValue @@ -1091,6 +1162,7 @@ function Invoke-CopilotScriptedExecute { $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') @@ -1286,13 +1358,14 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 9c1f6e9..90d205b 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -24,7 +24,7 @@ foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', } } $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') -function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false } } +function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false; GitHubCliConfigDirectory = $null; NonInteractiveReady = $true } } function Resolve-SandboxCommand { param($Name) return $null } function Resolve-ExternalCommand { param($Name) if ($Name -ne 'copilot') { throw "Unexpected executable lookup: $Name" }; return $fakeCommand } function Get-CopilotPreflight { diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 7eaa252..019a416 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -125,6 +125,11 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldHome = $env:HOME +$recordedOldUserProfile = $env:USERPROFILE +$recordedOldAppData = $env:APPDATA +$recordedOldLocalAppData = $env:LOCALAPPDATA +$recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -1970,58 +1975,134 @@ exit 2 $env:CODEX_HOME = $recordedOldCodexHome } if ($Suite -in @('All', 'Copilot')) { - # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and - # no-auth fixtures are all deterministic and contain no credential values. + # GitHub Copilot authentication continuity and failure semantics: + # deterministic fixtures only, no live credentials. $env:COPILOT_GITHUB_TOKEN = $null $env:GH_TOKEN = $null $env:GITHUB_TOKEN = $null $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + New-Item -ItemType Directory -Path $missingGhConfig -Force | Out-Null + $copilotLogPath = Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl' - # The fixture marker is fake-CLI input only; it models a positive OS - # keychain lookup without naming or reading a real credential-store file. - $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' - New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + # Fixture marker only: models a keychain-only host with no resolvable + # non-interactive token source. + $copilotKeychainHost = Join-Path $recordedRoot 'copilot-keychain-host' + New-Item -ItemType Directory -Path $copilotKeychainHost -Force | Out-Null New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) - $env:COPILOT_HOME = $copilotKeychainHome + $env:COPILOT_HOME = $copilotKeychainHost $env:GH_CONFIG_DIR = $missingGhConfig + $nativeExecutionsBeforeKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsBeforeKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' - Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' - Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + Assert-Equal 'incompatible' $copilotKeychainPreflight.status 'Copilot keychain-only auth is rejected before Phase 1 execution' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot keychain-only preflight fails authentication deterministically' + Assert-Equal 'copilot_os_keychain_unverified' $copilotKeychainPreflight.protocol_observations.authentication.source 'Copilot keychain-only source is diagnostic only' + Assert-True (-not [bool]$copilotKeychainPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot keychain-only source is not accepted as non-interactive readiness' + Assert-True (@($copilotKeychainPreflight.reasons | Where-Object { $_ -match 'supported non-interactive Copilot authentication source' }).Count -eq 1) 'Copilot keychain-only preflight reports the missing non-interactive source' $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' - $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'copilot_authentication_source' -Default '') -eq 'os_keychain' }) - Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Assert-Equal 'incompatible' $copilotKeychainResult.status 'Copilot execute fails closed when preflight cannot resolve non-interactive auth' + Assert-Equal 'incompatible' ([string]$copilotKeychainResult.evidence.preflight.status) 'Copilot execute returns preflight evidence for keychain-only rejection' + $nativeExecutionsAfterKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsAfterKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } + Assert-Equal $nativeExecutionsBeforeKeychain.Count $nativeExecutionsAfterKeychain.Count 'Copilot keychain-only rejection occurs before model execution' Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + # Fresh-context continuity regression: the first GH config candidate exists + # but is not authenticated; fallback resolves a second supported host + # candidate without exposing auth files or token values to the run. + $copilotFreshBoundaryRoot = Join-Path $recordedRoot 'copilot-fresh-boundary' + $copilotFreshHome = Join-Path $copilotFreshBoundaryRoot 'home' + $copilotFreshXdg = Join-Path $copilotFreshBoundaryRoot 'xdg' + $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' + $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' + $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' + $copilotFreshHostGhConfig = Join-Path $copilotFreshHostAppData 'GitHub CLI' + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, (Join-Path $copilotFreshXdg 'gh'), $copilotFreshHostGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotFreshHome + $env:USERPROFILE = $copilotFreshHome + $env:APPDATA = $copilotFreshHostAppData + $env:LOCALAPPDATA = $copilotFreshLocalAppData + $env:XDG_CONFIG_HOME = $copilotFreshXdg + $env:COPILOT_HOME = (Join-Path $copilotFreshBoundaryRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $null + $copilotFreshBoundaryPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotFreshBoundaryPreflight.status 'Copilot preflight resolves non-interactive auth across fresh-context candidate fallback' + Assert-True (@($copilotFreshBoundaryPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot fresh-context preflight proves authentication readiness' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryPreflight.protocol_observations.authentication.source 'Copilot fresh-context source resolves through GitHub CLI fallback' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot fresh-context source is accepted as non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.github_cli_config_candidate_used) 'Copilot fresh-context preflight records GH config candidate fallback usage' + $copilotFreshBoundaryResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotFreshBoundaryResult.status 'Copilot fresh-context fallback executes after preflight authentication continuity' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryResult.evidence.credential.source 'Copilot fresh-context execution records GitHub CLI fallback source' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.noninteractive_ready) 'Copilot fresh-context execution records non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.github_cli_token_resolved) 'Copilot fresh-context execution resolves a trusted GitHub CLI token' + $freshBoundaryExecutionRecords = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $freshBoundaryExecution = $freshBoundaryExecutionRecords[$freshBoundaryExecutionRecords.Count - 1] + Assert-Equal 'explicit_environment' $freshBoundaryExecution.copilot_authentication_source 'Copilot fresh-context worker receives only protected env-token auth' + Assert-Equal 1 @($freshBoundaryExecution.copilot_auth_names_present).Count 'Copilot fresh-context worker receives one token variable' + Assert-True (@($freshBoundaryExecution.copilot_auth_names_present) -contains 'GH_TOKEN') 'Copilot fresh-context worker receives only GH_TOKEN from trusted fallback' + Assert-True ([string]::IsNullOrWhiteSpace([string]$freshBoundaryExecution.gh_config_dir)) 'Copilot fresh-context worker does not receive GH_CONFIG_DIR' + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $copilotGhFallbackHome $env:GH_CONFIG_DIR = $copilotGhConfig + $ghFallbackMatchesBefore = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }).Count $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + Assert-Equal 'github_cli_token' $copilotGhPreflight.protocol_observations.authentication.source 'Copilot preflight classifies explicit GH config fallback source' + Assert-True ([bool]$copilotGhPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot preflight classifies GH config fallback as non-interactive readiness' $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True ([bool]$copilotGhResult.evidence.credential.noninteractive_ready) 'Copilot GitHub CLI fallback result records non-interactive readiness' Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) - Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $copilotNoAuthRoot = Join-Path $recordedRoot 'copilot-no-auth-roots' + New-Item -ItemType Directory -Path (Join-Path $copilotNoAuthRoot 'home'), (Join-Path $copilotNoAuthRoot 'appdata'), (Join-Path $copilotNoAuthRoot 'localappdata'), (Join-Path $copilotNoAuthRoot 'xdg') -Force | Out-Null + $env:HOME = Join-Path $copilotNoAuthRoot 'home' + $env:USERPROFILE = Join-Path $copilotNoAuthRoot 'home' + $env:APPDATA = Join-Path $copilotNoAuthRoot 'appdata' + $env:LOCALAPPDATA = Join-Path $copilotNoAuthRoot 'localappdata' + $env:XDG_CONFIG_HOME = Join-Path $copilotNoAuthRoot 'xdg' + $nativeExecutionsBeforeNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) $env:COPILOT_HOME = $copilotNoAuthHome $env:GH_CONFIG_DIR = $missingGhConfig $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' - Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + Assert-Equal 'incompatible' $copilotNoAuthPreflight.status 'Copilot preflight fails closed when no supported non-interactive auth source is available' + Assert-True (@($copilotNoAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot no-auth preflight records an authentication failure check' + Assert-Equal 'copilot_os_keychain_unverified' $copilotNoAuthPreflight.protocol_observations.authentication.source 'Copilot no-auth preflight reports diagnostic auth source identity' + Assert-True (-not [bool]$copilotNoAuthPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot no-auth preflight reports non-interactive auth readiness as false' $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' - Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-Equal 'incompatible' $copilotNoAuthResult.status 'Copilot no-auth execution fails before Phase 1 model execution' + Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' + $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $recordedOldCopilotHome } Write-Output "Real runner deterministic adapter conformance ($Suite): PASS" @@ -2036,6 +2117,11 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } @@ -2401,6 +2487,7 @@ try { Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('validate-eval-grading.ps1') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading validation, grading application, and finalization boundaries' Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' + Assert-True ($prepareText.Contains('authentication incompatibility is terminal for this package iteration') -and $prepareText.Contains('Do not suggest switching to another runner') -and $prepareText.Contains('do not suggest starting another Orchestrator') -and $prepareText.Contains('do not rerun Phase 1')) 'handoff preparation must fail closed on Copilot authentication incompatibility without alternate-runner/orchestrator suggestions' Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' Assert-True ($prepareText.Contains('Write `grading.json`, then validate it before finalization') -and $prepareText.Contains('Grading validation is retryable; finalization is not') -and $prepareText.Contains('only after grading validation succeeds')) 'handoff preparation must place retryable grading validation before exactly-once finalization' Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' @@ -2409,6 +2496,7 @@ try { $generatedHandoff = Invoke-GeneratedRunnerPrompt $generatedConcurrencyThreeHandoff = Invoke-GeneratedRunnerPrompt -RequestedConcurrency 3 Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' + Assert-True ($generatedHandoff.Contains('authentication incompatibility is terminal for this package iteration') -and $generatedHandoff.Contains('Do not suggest switching to another runner') -and $generatedHandoff.Contains('do not suggest starting another Orchestrator') -and $generatedHandoff.Contains('do not rerun Phase 1')) 'generated handoff output must keep Copilot authentication failures fail-closed without fallback orchestration suggestions' Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' Assert-True ($generatedHandoff.Contains('validate-eval-grading.ps1') -and $generatedHandoff.Contains('-ShowSkeleton') -and $generatedHandoff.Contains('Write `grading.json`, then validate it before finalization') -and $generatedHandoff.Contains('Grading validation is retryable; finalization is not') -and $generatedHandoff.Contains('only after grading validation succeeds')) 'generated handoff requires author -> validate/retry -> finalize exactly once' Assert-True ($generatedHandoff.Contains('allowance of 6240 seconds') -and $generatedHandoff.Contains('6 arm(s) × 120-second fixed model-free runner preflight timeout = 720-second serial preflight allowance') -and $generatedHandoff.Contains('5490-second execution allowance across 3 batch(es) at concurrency 2') -and $generatedHandoff.Contains('2 scripted user turn(s) × profile.timeout_seconds 900 + 30 seconds runner grace') -and $generatedHandoff.Contains('+ 30 seconds orchestration grace')) 'generated handoff uses the fixed preflight timeout, scripted-turn model timeout, execution batches, and separate orchestration grace' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index d122b36..c160999 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1860,6 +1860,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Read the selected runner descriptor and its `delegation.dispatch_owner`. For runner-owned behavioral transport, invoke:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$runnerOwnedFanoutPath`" -IterationDirectory `"$IterationDirectory`"") [void]$builder.AppendLine('It performs every preflight before any execute process, preserves exact manifest paths, owns concurrency/backpressure, timeout/watchdog handling, terminal registration, orchestration evidence, and immutable `execution-freeze.json` before Phase 2. Consume its terminal JSON summary. If Phase 1 reports incompatible or fails, stop: the evaluation is incomplete and must fail closed. The evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check. Do not create outer workers, execute an arm yourself, write orchestration state, or edit raw result/evidence files. If dispatch ownership is orchestrator-owned, use only the descriptor-declared native worker transport, the exact manifest paths, and then run the shared freeze boundary; do not synthesize or repair transport evidence. Only persisted runner-produced evidence at the manifest-declared paths may proceed.') + [void]$builder.AppendLine('For `github-copilot`, authentication incompatibility is terminal for this package iteration: report the missing supported non-interactive source (explicit `COPILOT_GITHUB_TOKEN`/`GH_TOKEN`/`GITHUB_TOKEN` or trusted `gh auth token` fallback), preserve the package, and stop. Do not suggest switching to another runner, do not suggest starting another Orchestrator, and do not rerun Phase 1.') [void]$builder.AppendLine('Workers receive only their isolated run directory. Keep the paired arm, metadata, expected output, assertions, grading, reports, and orchestration files out of Phase 1. Preserve runner-owned terminal results and all referenced raw transcript/event artifacts exactly as written.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2 — grading and finalization') From 3b1703d442d841ff9df38910ac70a3bb70b0b267 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 10:10:06 +0200 Subject: [PATCH 08/36] =?UTF-8?q?=F0=9F=93=9D=20document=20copilot=20authe?= =?UTF-8?q?ntication=20incompatibility=20handling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update eval-runners README to explain Copilot authentication requirements and fail-closed behavior. Documents the fresh-context authentication testing strategy, the GitHub CLI token fallback mechanism, and the fail-closed incompatibility policy that prevents evaluation execution when required non-interactive authentication sources are unavailable. --- scripts/eval-runners/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index d3853fc..1c61842 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -300,12 +300,13 @@ in both paired arms. Personal Copilot configuration is excluded by run-local `COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; the runner does not copy the normal `.copilot` directory. Authentication prefers explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is -present, the trusted runner may resolve `gh auth token` outside the worker and -inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` -is never forwarded into the evaluated worker. `--secret-env-vars` removes every -listed token variable from shell and MCP child environments. Preflight does not -make a model request and therefore reports native keychain/service readiness as -conditional rather than claiming successful remote authentication. Codex's +present, the trusted runner resolves `gh auth token` outside the worker across +documented/default config candidates and injects only that token as a protected +environment variable. Host `GH_CONFIG_DIR` is never forwarded into the evaluated +worker. `--secret-env-vars` removes every listed token variable from shell and +MCP child environments. If neither explicit token variables nor trusted GitHub +CLI token fallback is resolvable, preflight fails closed before Phase 1 model +execution starts. Codex's compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox danger-full-access`; subscription eval arms use the runner-owned app-server path described above. It does not combine explicit sandbox selection with From 7faba4ad894c95e70e36924871113a40b68da08c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:07 +0200 Subject: [PATCH 09/36] =?UTF-8?q?=F0=9F=92=AC=20document=20eval=20infrastr?= =?UTF-8?q?ucture=20and=20project=20conventions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update repository documentation with eval infrastructure details, workspace git scenario support, Copilot eval worker projection requirements, and grading contract specifications. These changes reflect improvements to the portable eval handoff and runner integrity requirements. --- README.md | 4 ++++ scripts/eval-runners/README.md | 8 ++++++++ 2 files changed, 12 insertions(+) diff --git a/README.md b/README.md index e8234ba..fbf3803 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. +Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. + +Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 explicitly follows the packaged Anthropic grader and requires assertion-specific, source-backed PASS evidence; generic completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. + Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 1c61842..976ea0e 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -139,6 +139,14 @@ Operational permission, native skill isolation, and filesystem isolation are dis Native delegation mechanisms: +Copilot execution requires an allowlisted physical temporary projection outside the logical package and source checkout ancestry. Only the arm's staged repository, prepared home, prompt and candidate (with-skill only) are copied. Repository `AGENTS.md` and `.github/copilot-instructions.md` remain enabled and identical in both arms. Ambient ancestor instructions and filesystem links make projection preflight fail closed. Runtime repository changes are synchronized back after link validation; captures remain parent-owned in the logical arm. Execution evidence records both boundaries. On hosts without an outer sandbox this prevents package discovery through the working-directory ancestry, but does not deny arbitrary absolute host paths: projection is not OS confinement. Structured tool events are a conservative contradiction detector; forbidden package/source accesses make the arm incompatible, and the bridge independently rejects contradictory completed transcripts. Native events remain available for diagnosis. + +Copilot `session.usage_checkpoint` billing counters are cumulative: the last checkpoint wins, including across resumed turns. `promptCacheBreakState` entries are per-call cache snapshots, deduplicated by native `model_call_id`; exposed prompt/cache buckets are used only when `assistant.usage` is absent. Tool schema tokens and native snapshots remain in evidence, not generated-token buckets. Missing output tokens stay unavailable, and premium requests/nano-AI units are never converted to currency. See [GitHub's event contract](https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/streaming-events). + +Phase 2 must read and follow the exact packaged `tools/skill-creator/agents/grader.md` after the execution freeze and bridge. PASS evidence requires `Source: output` (or a captured run artifact path), `Quote: `, and `Reason: ` on separate lines. The deterministic validator checks source membership, verbatim quote presence, nonempty evidence and rejects reused or generic PASS evidence. It cannot establish semantic truth; the Grader must fail uncertain/unverified expectations. Execution evidence remains immutable and finalization remains exactly once. + +Historical integrity note: `dotnet-change-impact` Copilot iteration 4 is contaminated and must not be used as effectiveness evidence. Its missing `.external-handoff-started` is consistent with manual handoff, but the package records do not establish the original user request. The current helper reserves the marker before every external handoff and contains no removal path; no reservation-policy change is justified from absence alone. Codex iteration 2 and OpenCode iteration 3 have reservations. New protocol code applies only to newly prepared iterations. + - GitHub Copilot: runner-owned behavioral transport. The runner starts one fresh Copilot CLI session per eval execution (`copilot -C --model --output-format json`, prompt on stdin) and captures that From ca663cfa005757d5c4623db01c9939667b2c12d9 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:20 +0200 Subject: [PATCH 10/36] =?UTF-8?q?=F0=9F=91=B7=20enhance=20eval=20infrastru?= =?UTF-8?q?cture=20with=20grading=20and=20isolation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement boundary violation detection for Copilot eval workers with physical projection outside package ancestry. Add isolation.ps1 support, enhance grading contract to enforce source-backed assertion evidence, update eval schema with new requirements, and fix git environment variable preservation in eval workspace initialization. Add eval-git-workspace.ps1 to support declarative git scenario setup for evaluations with staged commits and repository history. --- scripts/eval-git-workspace.ps1 | 52 +++++ .../eval-runners/bridge-execution-result.ps1 | 3 + .../contracts/execution-result.schema.json | 13 +- .../contracts/grading.schema.json | 2 +- .../eval-runners/eval-grading-contract.ps1 | 46 +++- .../eval-runners/github-copilot/isolation.ps1 | 34 +++ .../eval-runners/github-copilot/runner.ps1 | 208 +++++++++++++++++- .../tests/test-copilot-boundaries.ps1 | 85 ++++++- .../eval-runners/tests/test-eval-request.ps1 | 1 + .../tests/test-integrity-finalization.ps1 | 25 ++- .../tests/test-runner-conformance.ps1 | 4 +- scripts/prepare-skill-evals.ps1 | 13 +- scripts/validate-skill-templates.ps1 | 19 +- 13 files changed, 470 insertions(+), 35 deletions(-) create mode 100644 scripts/eval-git-workspace.ps1 create mode 100644 scripts/eval-runners/github-copilot/isolation.ps1 diff --git a/scripts/eval-git-workspace.ps1 b/scripts/eval-git-workspace.ps1 new file mode 100644 index 0000000..0394705 --- /dev/null +++ b/scripts/eval-git-workspace.ps1 @@ -0,0 +1,52 @@ +# Declarative, local-only Git scenarios. No hooks, remotes, or model calls run. +function Assert-EvalGitScenario { + param([object]$Scenario) + if ($Scenario -is [bool]) { return } + if ($Scenario -isnot [pscustomobject]) { throw 'workspace.git must be a boolean or a declarative Git scenario.' } + $allowed = @('base_branch', 'feature_branch', 'commits') + foreach ($name in $Scenario.PSObject.Properties.Name) { if ($name -notin $allowed) { throw "Unknown workspace.git field '$name'." } } + foreach ($name in $allowed) { if ($name -notin $Scenario.PSObject.Properties.Name) { throw "workspace.git requires '$name'." } } + foreach ($branch in @($Scenario.base_branch, $Scenario.feature_branch)) { + if ($branch -isnot [string] -or $branch -notmatch '^[a-zA-Z0-9][a-zA-Z0-9/_-]*$' -or $branch.Contains('//') -or $branch.EndsWith('/')) { throw 'Invalid scenario branch name.' } + } + if ($Scenario.base_branch -eq $Scenario.feature_branch -or @($Scenario.commits).Count -lt 1) { throw 'Git scenario needs distinct base/feature branches and feature commits.' } + foreach ($commit in $Scenario.commits) { + if ([string]::IsNullOrWhiteSpace([string]$commit.message) -or $commit.files -isnot [pscustomobject]) { throw 'Each Git scenario commit requires message and files.' } + foreach ($name in $commit.PSObject.Properties.Name) { if ($name -notin @('message', 'files')) { throw "Unknown commit field '$name'." } } + foreach ($file in $commit.files.PSObject.Properties) { + if ($file.Name -notmatch '^[a-zA-Z0-9_][a-zA-Z0-9_./-]*$' -or $file.Name -match '(^|/)(\.\.?|\.git|bin|obj)(/|$)' -or $file.Name.EndsWith('/')) { throw "Unsafe Git scenario file '$($file.Name)'." } + if ($null -ne $file.Value -and $file.Value -isnot [string]) { throw 'Git scenario file content must be a string or null (delete).' } + } + } +} + +function Add-EvalGitScenario { + param([string]$RepoDirectory, [object]$Scenario) + Assert-EvalGitScenario -Scenario $Scenario + if ($Scenario -is [bool]) { return } + $identity = @('-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', '-c', 'core.autocrlf=false', '-c', 'core.hooksPath=') + function Invoke-ScenarioGit { param([string[]]$Arguments) + & git @identity -C $RepoDirectory @Arguments 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Git scenario command failed: $($Arguments -join ' ')" } + } + $oldAuthor = $env:GIT_AUTHOR_DATE; $oldCommitter = $env:GIT_COMMITTER_DATE + try { + $env:GIT_AUTHOR_DATE = '2020-01-02T00:00:00Z'; $env:GIT_COMMITTER_DATE = $env:GIT_AUTHOR_DATE + Invoke-ScenarioGit -Arguments @('branch', '-m', [string]$Scenario.base_branch) + # Local tracking refs and symbolic HEAD exercise default resolution + # without a network remote or paths back to the package. + Invoke-ScenarioGit -Arguments @('update-ref', "refs/remotes/origin/$($Scenario.base_branch)", 'HEAD') + Invoke-ScenarioGit -Arguments @('symbolic-ref', 'refs/remotes/origin/HEAD', "refs/remotes/origin/$($Scenario.base_branch)") + Invoke-ScenarioGit -Arguments @('checkout', '-b', [string]$Scenario.feature_branch, '--quiet') + foreach ($commit in $Scenario.commits) { + foreach ($file in $commit.files.PSObject.Properties) { + $path = [IO.Path]::GetFullPath((Join-Path $RepoDirectory $file.Name)) + if (-not $path.StartsWith([IO.Path]::GetFullPath($RepoDirectory) + [IO.Path]::DirectorySeparatorChar, [StringComparison]::OrdinalIgnoreCase)) { throw 'Git scenario path escaped repo.' } + if ($null -eq $file.Value) { if (Test-Path -LiteralPath $path) { Remove-Item -LiteralPath $path -Force } } + else { [void][IO.Directory]::CreateDirectory((Split-Path -Parent $path)); [IO.File]::WriteAllText($path, $file.Value, [Text.UTF8Encoding]::new($false)) } + } + Invoke-ScenarioGit -Arguments @('add', '-A') + Invoke-ScenarioGit -Arguments @('commit', '--quiet', '-m', [string]$commit.message) + } + } finally { $env:GIT_AUTHOR_DATE = $oldAuthor; $env:GIT_COMMITTER_DATE = $oldCommitter } +} diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index d46e8de..c04cac3 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -259,6 +259,9 @@ try { $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) } + . (Join-Path $PSScriptRoot 'github-copilot/isolation.ps1') + Assert-CopilotCapturedBoundary -Raw $raw -RunData $runData + if ($RequireNativeDelegation) { if ([string]$raw.status -eq 'incompatible') { throw 'An incompatible native-worker arm is diagnostic only and cannot be bridged into a gradeable canonical result.' diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index f0bf233..0e05500 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -185,8 +185,8 @@ "prompt_fidelity": { "const": true }, "prompt_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, "terminal_result_capture": { "const": true }, - "paired_arm_visible": { "const": false }, - "grading_material_visible": { "const": false }, + "paired_arm_visible": { "type": "boolean" }, + "grading_material_visible": { "type": "boolean" }, "nested_model_execution": { "const": false }, "model_execution_count": { "const": 1 } } @@ -197,5 +197,12 @@ "warnings": { "type": "array", "items": { "type": "string" } }, "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, "attempt_count": { "const": 1 } - } + }, + "allOf": [{ + "if": { "properties": { "status": { "const": "completed" } } }, + "then": { "properties": { "evidence": { "properties": { "delegation": { "properties": { + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false } + } } } } } } + }] } diff --git a/scripts/eval-runners/contracts/grading.schema.json b/scripts/eval-runners/contracts/grading.schema.json index 23b4aac..f16fc12 100644 --- a/scripts/eval-runners/contracts/grading.schema.json +++ b/scripts/eval-runners/contracts/grading.schema.json @@ -20,7 +20,7 @@ "assertion_index": { "type": "integer", "minimum": 0 }, "assertion": { "type": "string", "minLength": 1 }, "passed": { "type": "boolean" }, - "evidence": { "type": "string" } + "evidence": { "type": "string", "minLength": 1, "pattern": "\\S" } } } } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index c1a7127..0c8cd29 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -52,8 +52,33 @@ function Assert-EvalGradingEntryShape { if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } - if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } -} + if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } + if ([string]::IsNullOrWhiteSpace($Entry.evidence)) { throw 'grading.json evidence must be non-empty.' } + if ($Entry.passed -and $Entry.evidence -notmatch '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') { + throw 'PASS evidence requires Source, a verbatim Quote, and an assertion-specific Reason on separate lines.' + } +} + +function Assert-EvalPassEvidence { + param([object]$Entry, [object]$Canonical, [object]$Record) + if (-not $Entry.passed) { return } + [void]($Entry.evidence -match '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') + $source = $Matches[1].Trim(); $quote = $Matches[2].Trim(); $reason = $Matches[3].Trim() + if ($source -eq 'output') { $content = [string]$Canonical.output } + else { + # Only native captured artifacts are admissible, never grading keys or + # grader-created files. Freeze validation already pins their bytes. + $raw = Read-RunnerJson -Path $Record.ExecutionResultPath + $artifacts = @($raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -ceq $source }) + if ($artifacts.Count -ne 1) { throw "PASS evidence source '$source' is not a captured run artifact." } + $path = Resolve-ContainedPath -BasePath (Split-Path -Parent $Record.RunManifestPath) -RelativePath $source -FieldName 'PASS evidence source' -Kind File + $content = [IO.File]::ReadAllText($path) + } + if ([string]::IsNullOrWhiteSpace($quote) -or -not $content.Contains($quote, [StringComparison]::Ordinal)) { throw 'PASS evidence quote is absent from its frozen source.' } + if ($reason -eq $quote -or $reason -eq $Entry.assertion -or $reason -match '^(?i:eval(?:uation)? completed(?: with output)?|passed|verified|as expected|done|looks good)[.!]?$') { + throw 'PASS evidence must explain how the cited observation establishes this assertion.' + } +} function Assert-EvalGradingContract { param( @@ -124,17 +149,26 @@ function Assert-EvalGradingContract { throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." } - $validated = @{} + $validated = @{} + $passEvidence = @{} foreach ($entry in $submitted) { Assert-EvalGradingEntryShape -Entry $entry $key = Get-EvalGradingEntryKey -Entry $entry if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } $target = $expected[$key] - if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { + if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." - } - $validated[$key] = $entry + } + $armKey = "$($entry.eval_id)|$($entry.configuration)" + $record = @($records | Where-Object { $_.EvalId -eq $entry.eval_id -and $_.Configuration -eq $entry.configuration })[0] + Assert-EvalPassEvidence -Entry $entry -Canonical $canonicalByKey[$armKey] -Record $record + if ($entry.passed) { + $evidenceKey = $armKey + '|' + ([regex]::Replace($entry.evidence.Trim(), '\s+', ' ')).ToLowerInvariant() + if ($passEvidence.ContainsKey($evidenceKey)) { throw 'Repeated PASS evidence across assertions is not assertion-specific.' } + $passEvidence[$evidenceKey] = $true + } + $validated[$key] = $entry } foreach ($key in $expected.Keys) { if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 new file mode 100644 index 0000000..a847514 --- /dev/null +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -0,0 +1,34 @@ +function Find-CopilotBoundaryContradictions { + param([object]$Data, [object]$Projection) + # Structured tool arguments/results can contradict projection isolation; + # their absence never proves OS confinement. + $text = ($Data | ConvertTo-Json -Depth 100 -Compress).Replace('\\', '/').Replace('\', '/') + foreach ($root in @($Projection.PackageRoot, $Projection.SourceRepositoryRoot) | Where-Object { $_ }) { + if ($text.IndexOf(([string]$root).Replace('\', '/'), [StringComparison]::OrdinalIgnoreCase) -ge 0) { 'Tool event references forbidden package/source path.' } + } + if ($text -match '(?i)(eval-metadata\.json|(?:^|[/"\s])(?:with_skill|without_skill)(?:[/"\s]|$)|(?:\.\./)+(?:results|tools|progress)(?:/|"|\s)|(?:execution-freeze|orchestration-state|grading|benchmark)\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)') { + 'Tool event references forbidden grading, paired-arm, or orchestration material.' + } +} + +function Assert-CopilotCapturedBoundary { + param([object]$Raw, [object]$RunData) + if ($Raw.runner.name -ne 'github-copilot' -or $Raw.status -ne 'completed') { return } + $paths = Get-JsonProperty -Object $Raw.evidence -Name execution_paths -Default $null + if (-not [bool](Get-JsonProperty -Object $paths -Name projection_proven -Default $false)) { throw 'Completed Copilot execution lacks a proven physical projection.' } + $package = Split-Path -Parent (Split-Path -Parent $RunData.RunRoot) + $source = [string](Get-JsonProperty -Object $paths -Name source_repository_root -Default '') + $physical = [string](Get-JsonProperty -Object $paths -Name physical_run_root -Default '') + if ([string]::IsNullOrWhiteSpace($physical) -or (Test-PathInside -BasePath $package -CandidatePath $physical) -or ($source -and (Test-PathInside -BasePath $source -CandidatePath $physical))) { throw 'Invalid Copilot physical projection boundary.' } + $proof = [pscustomobject]@{ PackageRoot = $package; SourceRepositoryRoot = $source } + $transcript = @($Raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -eq 'evidence/copilot-events.jsonl' }) + if ($transcript.Count -ne 1) { throw 'Copilot native transcript is missing.' } + $path = Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $transcript[0].path -FieldName 'Copilot transcript' -Kind File + $parsed = ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($path)) + if (@($parsed.Errors).Count) { throw 'Copilot transcript contains unparseable events; boundary inspection is incomplete.' } + foreach ($event in $parsed.Events) { + if ([string](Get-JsonProperty -Object $event -Name type -Default '') -match '^(tool\.|command\.)') { + if (@(Find-CopilotBoundaryContradictions -Data (Get-JsonProperty -Object $event -Name data -Default $null) -Projection $proof).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + } + } +} diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index da6dc6b..49d61ee 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -27,12 +27,16 @@ param( $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot '..\runner-common.ps1') +. (Join-Path $PSScriptRoot 'isolation.ps1') # GitHub Copilot checks these token variables before its OS credential store and # GitHub CLI fallback. The values are forwarded only to the Copilot process; # --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') $script:copilotHomeCleanupSafe = $true +$script:copilotLogicalInputs = $null +$script:copilotProjection = $null +$script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() function Invoke-CopilotProcess { param( @@ -566,6 +570,13 @@ function Get-CopilotPreflight { $warnings = [System.Collections.Generic.List[string]]::new() $profile = $Inputs.Profile $run = $Inputs.Run + try { + if ($null -eq $script:copilotProjection) { [void](Get-CopilotProjectionPlan -Inputs $Inputs) } + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status passed -Detail 'Allowlisted physical workspace outside package/source ancestry is required.')) + } catch { + $reasons.Add($_.Exception.Message) + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status failed -Detail $_.Exception.Message)) + } $commandInfo = Resolve-ExternalCommand -Name 'copilot' $platform = Get-PlatformName $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } @@ -808,6 +819,7 @@ function Write-CopilotCapture { [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text ) + if ($null -ne $script:copilotLogicalInputs) { $RunData = $script:copilotLogicalInputs } $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) @@ -839,6 +851,8 @@ function Read-CopilotEvents { $usageNumToolCalls = 0 $usageSeen = $false $toolStarts = 0 + $lastCheckpoint = $null + $checkpointCalls = [ordered]@{} $sessionError = $null $eventCounts = @{} $sessionIds = [System.Collections.Generic.List[string]]::new() @@ -855,6 +869,10 @@ function Read-CopilotEvents { } if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + if ($eventType -match '^(tool\.|command\.)' -and $null -ne $script:copilotProjection) { + $violations = @(Find-CopilotBoundaryContradictions -Data $data -Projection $script:copilotProjection) + foreach ($violation in $violations) { $script:copilotBoundaryViolations.Add($violation) } + } foreach ($eventSessionId in @(Get-CopilotEventSessionIds -Event $event)) { if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } } @@ -865,6 +883,19 @@ function Read-CopilotEvents { if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } } switch ($eventType) { + 'session.usage_checkpoint' { + # Billing is session-cumulative. Cache-break entries are last + # call snapshots, deduplicated by native call identity. + $lastCheckpoint = $data + foreach ($conversation in @(Get-JsonProperty -Object $data -Name 'promptCacheBreakState' -Default @())) { + $models = Get-JsonProperty -Object $conversation -Name 'models' -Default @{} + foreach ($name in @(Get-JsonPropertyNames -Object $models)) { + $call = Get-JsonProperty -Object $models -Name $name + $id = [string](Get-JsonProperty -Object $call -Name 'model_call_id' -Default '') + if ($id) { $checkpointCalls[$id] = $call } + } + } + } 'assistant.message' { $assistantMessageObserved = $true $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') @@ -907,6 +938,14 @@ function Read-CopilotEvents { $finalText = [string]::Join("`n", $assistantContents) } $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + if (-not $usageSeen -and $checkpointCalls.Count -gt 0) { + foreach ($call in $checkpointCalls.Values) { + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $call -Name 'prompt_tokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $call -Name 'cache_read' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $call -Name 'cache_write' -Default $null) + } + $usageSeen = $true + } return [pscustomobject]@{ FinalText = $finalText @@ -916,6 +955,8 @@ function Read-CopilotEvents { UsageOutput = $usageOutput UsageCacheRead = $usageCacheRead UsageCacheWrite = $usageCacheWrite + UsageCheckpoint = $lastCheckpoint + CheckpointCalls = @($checkpointCalls.Values) ToolCalls = $toolCalls SessionError = $sessionError EventCounts = $eventCounts @@ -1018,7 +1059,8 @@ function Invoke-CopilotScriptedExecute { $failureMessage = $null for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { - $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $Inputs.Run + $turnSourceRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $turnSourceRun $arguments = @($baseArguments) $targetSessionId = $null if ($turnIndex -gt 0) { @@ -1239,8 +1281,8 @@ function Invoke-CopilotScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1261,13 +1303,167 @@ function Invoke-CopilotScriptedExecute { $resultFinalResponse = if ($status -eq 'completed') { $finalText } else { $null } $resultFinalResponseReason = if ($status -eq 'completed') { $null } else { 'native_interaction_incompatible' } $result = New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $resultFinalResponse -FinalResponseReason $resultFinalResponseReason -StartedUtc $firstProcess.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $capturedSessionId -IsolationCapabilities (Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -ContinuationCapability $continuationCapability) -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Copilot accepted the requested model alias and configuration; scripted turns retained the exact requested model on every invocation.'; observations = [ordered]@{ model = $Inputs.Profile.Model; observed_models = @($observedModels.ToArray()); continuation_flag = $continuationCapability.Flag } }) -Telemetry $telemetry -Artifacts @($artifacts.ToArray()) -Warnings @($warnings.ToArray()) -Evidence $evidence -AttemptCount 1 - if ($status -eq 'completed') { [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $Inputs.Run) } + if ($status -eq 'completed') { + $validationRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $validationRun) + } return $result } function Invoke-CopilotExecute { param([Parameter(Mandatory = $true)][object]$Inputs) + # Keep all orchestration data in the parent process. Only repo, prepared + # home, prompt bytes and this arm's candidate cross the projection boundary. + $plan = Get-CopilotProjectionPlan -Inputs $Inputs + $logicalInputs = $Inputs + $script:copilotLogicalInputs = $Inputs + $script:copilotProjection = $plan + $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() + [void](New-Item -ItemType Directory -Path $plan.Root -ErrorAction Stop) + try { + $physicalRun = $Inputs.Run.PSObject.Copy() + $physicalRun.RunRoot = $plan.Root + foreach ($field in @('WorkingDirectoryPath', 'HomeDirectoryPath', 'SkillDirectoryPath')) { + $source = [string]$Inputs.Run.$field + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $relative = [IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $source) + $destination = Join-Path $plan.Root $relative + Copy-CopilotProjectionTree -Source $source -Destination $destination + $physicalRun.$field = $destination + } + # Future scripted inputs stay parent-owned even when staged in repo/home. + if ($null -ne $Inputs.Run.Interaction) { + foreach ($turn in $Inputs.Run.Interaction.turns) { + $source = [string](Get-JsonProperty -Object $turn -Name source -Default '') + if ($source) { + Assert-SafeRelativePath -RelativePath $source -FieldName 'interaction source' + $path = Join-Path $plan.Root $source + if (Test-Path -LiteralPath $path -PathType Leaf) { Remove-Item -LiteralPath $path -Force } + } + } + } + $physicalRun.PromptPath = Join-Path $plan.Root 'prompt.md' + [IO.File]::WriteAllBytes($physicalRun.PromptPath, $Inputs.Run.PromptBytes) + $projectedFiles = @(Get-ChildItem -LiteralPath $physicalRun.WorkingDirectoryPath -Recurse -Force -File | ForEach-Object { [IO.Path]::GetRelativePath($physicalRun.WorkingDirectoryPath, $_.FullName) }) + $physicalInputs = [pscustomobject]@{ Run = $physicalRun; Profile = $Inputs.Profile } + $result = Invoke-CopilotWithPreparedHome -Inputs $physicalInputs -Action { Invoke-CopilotProjectedExecute -Inputs $physicalInputs } + $result.evidence.execution_paths = [ordered]@{ + projection = 'physical_temp_outside_logical_package' + logical_run_root = $logicalInputs.Run.RunRoot + logical_working_directory = $logicalInputs.Run.WorkingDirectoryPath + logical_home_directory = $logicalInputs.Run.HomeDirectoryPath + physical_run_root = $plan.Root + physical_working_directory = $physicalRun.WorkingDirectoryPath + physical_home_directory = $physicalRun.HomeDirectoryPath + source_repository_root = $plan.SourceRepositoryRoot + projection_proven = $true + hard_filesystem_confinement = [bool]$result.isolation.hard_filesystem_confinement + } + $result.evidence.boundary = [ordered]@{ proof = 'allowlisted_physical_projection'; contradictions = @($script:copilotBoundaryViolations.ToArray()); event_inspection = 'contradiction_detector_not_confinement' } + $capturePath = Join-Path $logicalInputs.Run.RunRoot 'evidence/copilot-events.jsonl' + if (Test-Path -LiteralPath $capturePath) { + $usageWarnings = [System.Collections.Generic.List[string]]::new() + # Reparse the combined native stream once, so resumed cumulative + # checkpoints and repeated call snapshots are never summed twice. + $usage = Read-CopilotEvents -Parsed (ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($capturePath))) -Warnings $usageWarnings + if ($usage.ParseErrorCount -gt 0) { $script:copilotBoundaryViolations.Add('Unparseable native events prevent complete boundary inspection.') } + $result.evidence.usage_checkpoint = $usage.UsageCheckpoint + $result.evidence.usage_checkpoint_calls = $usage.CheckpointCalls + if ($usage.UsageSeen) { + $buckets = [ordered]@{} + foreach ($mapping in @(@('input_tokens', 'UsageInput'), @('output_tokens', 'UsageOutput'), @('cache_read_tokens', 'UsageCacheRead'), @('cache_write_tokens', 'UsageCacheWrite'))) { + if ($null -ne $usage.($mapping[1])) { $buckets[$mapping[0]] = [int64]$usage.($mapping[1]) } + } + if ($buckets.Count) { $result.telemetry.tokens = New-AvailableMetric -Value $buckets } + } + } + if ($script:copilotBoundaryViolations.Count -gt 0) { + $result.status = 'incompatible' + $result.isolation.status = 'unverified' + $result.isolation.level = 'unsupported' + $result.isolation.hard_filesystem_confinement = $false + $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $script:copilotBoundaryViolations)) + if ($result.evidence.Contains('delegation')) { + $result.evidence.delegation.paired_arm_visible = $true + $result.evidence.delegation.grading_material_visible = $true + } + } + $result.evidence.boundary.contradictions = @($script:copilotBoundaryViolations | Select-Object -Unique) + # Do not copy runtime links back into the logical package. + Assert-CopilotProjectionTree -Path $physicalRun.WorkingDirectoryPath + Assert-CopilotProjectionTree -Path $logicalInputs.Run.WorkingDirectoryPath + foreach ($file in @(Get-ChildItem -LiteralPath $logicalInputs.Run.WorkingDirectoryPath -Recurse -Force -File)) { + $relative = [IO.Path]::GetRelativePath($logicalInputs.Run.WorkingDirectoryPath, $file.FullName) + if ($projectedFiles -contains $relative -and -not (Test-Path -LiteralPath (Join-Path $physicalRun.WorkingDirectoryPath $relative))) { + if (-not (Test-PathInside -BasePath $logicalInputs.Run.WorkingDirectoryPath -CandidatePath $file.FullName)) { throw 'Unsafe projected output deletion.' } + Remove-Item -LiteralPath $file.FullName -Force + } + } + Copy-CopilotProjectionTree -Source $physicalRun.WorkingDirectoryPath -Destination $logicalInputs.Run.WorkingDirectoryPath + return $result + } finally { + if (-not $script:copilotHomeCleanupSafe) { throw 'Copilot process termination is unproven; projection retained and execution fails closed.' } + if (-not (Test-PathInside -BasePath $plan.Parent -CandidatePath $plan.Root) -or [IO.Path]::GetFileName($plan.Root) -notmatch '^agentic-copilot-projection-[0-9a-f]{32}$') { throw 'Unsafe Copilot projection cleanup path.' } + Remove-Item -LiteralPath $plan.Root -Recurse -Force + $script:copilotLogicalInputs = $null + $script:copilotProjection = $null + } +} + +function Assert-CopilotProjectionTree { + param([string]$Path) + $item = Get-Item -LiteralPath $Path -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw "Copilot projection refuses filesystem link '$Path'." } + if ($item.PSIsContainer) { + foreach ($child in Get-ChildItem -LiteralPath $Path -Force) { Assert-CopilotProjectionTree -Path $child.FullName } + } +} + +function Copy-CopilotProjectionTree { + param([string]$Source, [string]$Destination) + Assert-CopilotProjectionTree -Path $Source + [void][IO.Directory]::CreateDirectory($Destination) + foreach ($child in Get-ChildItem -LiteralPath $Source -Force) { Copy-Item -LiteralPath $child.FullName -Destination $Destination -Recurse -Force } +} + +function Get-CopilotProjectionPlan { + param([object]$Inputs) + $sourceRoot = $null + $packageRoot = Split-Path -Parent (Split-Path -Parent $Inputs.Run.RunRoot) + foreach ($start in @($Inputs.Run.RunRoot, $PSScriptRoot)) { + $cursor = $start + while ($cursor) { + if (Test-Path -LiteralPath (Join-Path $cursor '.git')) { $sourceRoot = $cursor; break } + $cursor = Split-Path -Parent $cursor + } + if ($sourceRoot) { break } + } + $parent = [IO.Path]::GetFullPath([IO.Path]::GetTempPath()) + foreach ($forbidden in @($packageRoot, $sourceRoot) | Where-Object { $_ }) { + if (Test-PathInside -BasePath $forbidden -CandidatePath $parent) { throw 'Copilot cannot establish a physical projection outside package/source ancestry.' } + } + # A linked temp parent or ambient instruction file would invalidate ancestry + # isolation. Fail closed rather than disable legitimate repo instructions. + $cursor = $parent + while ($cursor) { + $item = Get-Item -LiteralPath $cursor -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw 'Copilot projection parent contains a filesystem link.' } + foreach ($instruction in @('AGENTS.md', '.github/copilot-instructions.md', '.github/instructions', '.git')) { + if (Test-Path -LiteralPath (Join-Path $cursor $instruction)) { throw "Copilot projection would inherit ambient '$instruction'." } + } + $cursor = Split-Path -Parent $cursor + } + foreach ($path in @($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.HomeDirectoryPath, $Inputs.Run.SkillDirectoryPath) | Where-Object { $_ }) { Assert-CopilotProjectionTree -Path $path } + foreach ($indirection in @('.git', '.git/objects/info/alternates', '.git/commondir')) { + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.WorkingDirectoryPath $indirection) -PathType Leaf) { throw "Copilot projection refuses external Git indirection '$indirection'." } + } + return [pscustomobject]@{ Root = Join-Path $parent ('agentic-copilot-projection-' + [Guid]::NewGuid().ToString('N')); Parent = $parent; PackageRoot = $packageRoot; SourceRepositoryRoot = $sourceRoot } +} + +function Invoke-CopilotProjectedExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + $preflight = Get-CopilotPreflight -Inputs $Inputs $started = [DateTime]::UtcNow $sessionId = [Guid]::NewGuid().ToString('D') @@ -1424,8 +1620,8 @@ function Invoke-CopilotExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 90d205b..eefa54b 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -9,6 +9,7 @@ Set-StrictMode -Version Latest $runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path . (Join-Path $runnerRoot 'runner-common.ps1') . (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'github-copilot/isolation.ps1') # Import definitions only: neither adapter dispatch nor a real CLI is invoked. foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', 'tests/test-runner-conformance.ps1', '../generate-eval-report.ps1')) { @@ -73,6 +74,14 @@ $gitExit = $LASTEXITCODE $record = [ordered]@{ turn = $turn; arguments = $arguments; cwd = (Get-Location).Path; ceiling = $env:GIT_CEILING_DIRECTORIES; cache = $env:COPILOT_CACHE_HOME; xdg = $env:XDG_CACHE_HOME; localappdata = $env:LOCALAPPDATA; appdata = $env:APPDATA; git_exit = $gitExit; git_root = [string]$gitResult; old_cache_exists = Test-Path -LiteralPath (Join-Path $runHome '.copilot-cache'); double_suffix_exists = Test-Path -LiteralPath (Join-Path $env:COPILOT_CACHE_HOME 'copilot') } [IO.File]::AppendAllText((Join-Path (Get-Location).Path 'fake-log.jsonl'), (($record | ConvertTo-Json -Compress) + "`n")) [IO.File]::WriteAllText((Join-Path (Get-Location).Path 'task-output.txt'), 'keep repo output') +$probe = [ordered]@{ metadata = Test-Path ../../eval-metadata.json; paired = (Test-Path ../../with_skill) -or (Test-Path ../../without_skill); candidate = Test-Path ../skill; staged_agents = Get-Content AGENTS.md -Raw; staged_copilot = Get-Content .github/copilot-instructions.md -Raw; ambient = @() } +$cursor = Split-Path -Parent (Get-Location).Path +while ($cursor) { + foreach ($name in @('AGENTS.md', '.github/copilot-instructions.md')) { if (Test-Path -LiteralPath (Join-Path $cursor $name)) { $probe.ambient += [IO.File]::ReadAllText((Join-Path $cursor $name)) } } + $cursor = Split-Path -Parent $cursor +} +[IO.File]::WriteAllText((Join-Path (Get-Location).Path 'projection-probe.json'), ($probe | ConvertTo-Json)) +if ($inputText -eq 'violation') { '{"type":"tool.execution_start","data":{"toolName":"view","arguments":{"path":"../../eval-metadata.json"}}}' } if ($inputText -eq 'timeout') { [Console]::Out.WriteLine('{"type":"session.start","data":{"sessionId":"fixture-session"}}'); [Console]::Out.Flush(); Start-Sleep -Seconds 30 } $text = if ($turn -eq 1) { 'Confirm before generation. Δ' } else { "Generated successfully.`nExact terminal text." } @{ type = 'session.start'; data = @{ sessionId = 'fixture-session' } } | ConvertTo-Json -Compress @@ -90,12 +99,19 @@ if ($inputText -eq 'failure') { exit 7 } $profilePath = Join-Path $iteration 'execution-profile.json' Write-TestJson -Path $profilePath -Value @{ schema = (Get-RunnerSchemaNames).Profile; runner = 'github-copilot'; model = 'fixture-model'; reasoning_effort = $null; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 15; concurrency = 1 } foreach ($run in @($with, $without)) { + [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'repo/.github')) + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/AGENTS.md'), 'STAGED_INSTRUCTIONS_CANARY') + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/.github/copilot-instructions.md'), 'STAGED_COPILOT_CANARY') [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'home/baseline/empty')) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/nested.bin'), [byte[]]@(0, 255, 10, 13, 42)) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/empty.bin'), [byte[]]@()) [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'evidence')) [IO.File]::WriteAllText((Join-Path $run.Root 'evidence/prepared.txt'), 'keep evidence') } + [IO.File]::WriteAllText((Join-Path $testRoot 'AGENTS.md'), 'FORBIDDEN_SOURCE_INSTRUCTIONS_CANARY') + [void][IO.Directory]::CreateDirectory((Join-Path $testRoot '.github')) + [IO.File]::WriteAllText((Join-Path $testRoot '.github/copilot-instructions.md'), 'FORBIDDEN_COPILOT_CANARY') + [IO.File]::WriteAllText((Join-Path (Split-Path -Parent $with.Root) 'eval-metadata.json'), 'FORBIDDEN_GRADING_CANARY') $inputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $with.Path; Profile = Resolve-ExecutionProfile -ProfilePath $profilePath } $baseline = Get-CopilotHomeBaseline -HomePath $inputs.Run.HomeDirectoryPath $baselineHash = Get-TestTreeHash -Root $inputs.Run.HomeDirectoryPath @@ -117,18 +133,19 @@ if ($inputText -eq 'failure') { exit 7 } $records = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) Assert-Equal 2 $records.Count 'both turns execute with shared runtime state' foreach ($record in $records) { - Assert-Equal $with.Root $record.ceiling 'exact child environment ceiling is the manifest RunRoot' + Assert-Equal $result.evidence.execution_paths.physical_run_root $record.ceiling 'child ceiling is the physical run root' + Assert-True (-not (Test-PathInside -BasePath $testRoot -CandidatePath $record.cwd)) 'physical cwd excludes source and package ancestry' Assert-True ($record.git_exit -ne 0) 'child git cannot discover outer/.git from non-Git staged repo' - Assert-Equal $inputs.Run.WorkingDirectoryPath $record.cwd 'child working directory retained' + Assert-Equal $result.evidence.execution_paths.physical_working_directory $record.cwd 'child uses projected working directory' $argsList = [string[]]$record.arguments - Assert-Equal $inputs.Run.WorkingDirectoryPath $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C retained' + Assert-Equal $record.cwd $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C agrees with process cwd' Assert-Equal 'fixture-model' $argsList[[Array]::IndexOf($argsList, '--model') + 1] 'model lock retained on every turn' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache') $record.xdg 'XDG cache root' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache') $record.xdg 'XDG cache root' Assert-True (-not $record.old_cache_exists -and -not $record.double_suffix_exists) 'no old or double-suffixed cache is created during execution' if ($IsWindows) { Assert-Equal $record.xdg $record.localappdata 'Windows cache fallback converges' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.config') $record.appdata 'Windows config root isolated' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.config') $record.appdata 'Windows config root isolated' } } Assert-True (@($records[1].arguments) -contains '--resume=fixture-session') 'exact session continuation retained' @@ -148,12 +165,13 @@ if ($inputText -eq 'failure') { exit 7 } if ($LASTEXITCODE -ne 0) { throw 'Could not initialize staged fixture Git repository.' } $singleInputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $without.Path; Profile = $inputs.Profile } $singleBaselineHash = Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath - foreach ($scenario in @('completed', 'failure', 'timeout')) { + foreach ($scenario in @('completed', 'failure', 'timeout', 'violation')) { $singleInputs.Run.PromptBytes = [Text.Encoding]::UTF8.GetBytes($scenario) $singleInputs.Profile.TimeoutSeconds = if ($scenario -eq 'timeout') { 2 } else { 15 } $singleResult = Invoke-CopilotWithPreparedHome -Inputs $singleInputs -Action { Invoke-CopilotExecute -Inputs $singleInputs } - $expectedStatus = if ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } + $expectedStatus = if ($scenario -eq 'violation') { 'incompatible' } elseif ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } Assert-Equal $expectedStatus $singleResult.status "$scenario terminal status" + [void](Assert-ExecutionResult -Result $singleResult) Assert-Equal $singleBaselineHash (Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath) "$scenario restores prepared baseline" Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'home/.cache'))) "$scenario removes generated cache" Assert-Equal 'keep repo output' ([IO.File]::ReadAllText((Join-Path $without.Root 'repo/task-output.txt'))) "$scenario preserves repo output" @@ -162,9 +180,58 @@ if ($inputText -eq 'failure') { exit 7 } $singleRecords = @(Get-Content -LiteralPath (Join-Path $without.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) foreach ($record in $singleRecords) { Assert-Equal 0 $record.git_exit 'Git still discovers staged repo/.git' - Assert-Equal ((Join-Path $without.Root 'repo').Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns staged root' + Assert-Equal ($record.cwd.Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns projected staged root' Assert-True ($record.cache -ne $records[0].cache) 'paired arms never share cache' } + Assert-True $singleResult.evidence.delegation.grading_material_visible 'contradiction prevents false invisibility claim' + $forged = $singleResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $forged.status = 'completed' + $forged.evidence.delegation.paired_arm_visible = $false + $forged.evidence.delegation.grading_material_visible = $false + Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $forged -RunData $singleInputs.Run } 'bridge rejects transcript contradiction despite false invisibility flags' + $proof = [pscustomobject]@{ Root = 'C:/temp/projection'; PackageRoot = 'C:/source/.bot/package'; SourceRepositoryRoot = 'C:/source' } + foreach ($path in @('../../eval-metadata.json', '../../with_skill/repo', '../../without_skill/repo', '../../results/arm.json', '../../grading.json', '../../execution-freeze.json', '../../orchestration-state.json', '../../report.html', 'C:\source\AGENTS.md')) { + Assert-True (@(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = $path } } -Projection $proof).Count -gt 0) "captured forbidden access rejected: $path" + } + Assert-Equal 0 @(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = 'src/Widget.cs' } } -Projection $proof).Count 'ordinary staged source is allowed' + $warnings = [Collections.Generic.List[string]]::new() + $checkpoint = @{ type = 'session.usage_checkpoint'; data = @{ totalPremiumRequests = 0.33; totalNanoAiu = 10; promptCacheBreakState = @(@{ models = @{ model = @{ model_call_id = 'call-1'; prompt_tokens = 100; cache_read = 70; cache_write = 20; tool_tokens = 15 } } }) } } + $last = $checkpoint | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $last.data.totalPremiumRequests = 0.66 + $last.data.totalNanoAiu = 20 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $last); Errors = @() } -Warnings $warnings + Assert-Equal 100 $usage.UsageInput 'repeated call snapshots are not summed' + Assert-Equal 70 $usage.UsageCacheRead 'cache read bucket retained without duplication' + Assert-Equal 20 $usage.UsageCacheWrite 'cache write bucket retained without duplication' + Assert-Equal 0.66 $usage.UsageCheckpoint.totalPremiumRequests 'final cumulative premium request counter is authoritative' + Assert-Equal 20 $usage.UsageCheckpoint.totalNanoAiu 'final cumulative nano AI units retained without currency conversion' + Assert-True ($null -eq $usage.UsageOutput) 'unexposed output tokens remain unavailable' + Assert-Equal 15 $usage.CheckpointCalls[0].tool_tokens 'tool schema tokens retained in native evidence only' + $secondCall = $last | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $secondCall.data.promptCacheBreakState[0].models.model.model_call_id = 'call-2' + $secondCall.data.promptCacheBreakState[0].models.model.prompt_tokens = 120 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $secondCall, $secondCall); Errors = @() } -Warnings $warnings + Assert-Equal 220 $usage.UsageInput 'distinct native calls count once each' + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, @{ type = 'assistant.usage'; data = @{ inputTokens = 7; outputTokens = 3 } }); Errors = @() } -Warnings $warnings + Assert-Equal 7 $usage.UsageInput 'native assistant usage takes precedence over cache snapshots' + Assert-Equal 3 $usage.UsageOutput 'actual exposed output count retained' + $savedTemp = $env:TEMP; $savedTmp = $env:TMP + try { + $env:TEMP = $testRoot; $env:TMP = $testRoot + Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'temp inside source ancestry fails closed' + } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp } + $runtimeLink = Join-Path $singleInputs.Run.WorkingDirectoryPath 'forbidden-link' + New-Item -ItemType $linkType -Path $runtimeLink -Target $testRoot | Out-Null + try { Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'linked projection input fails closed' } + finally { (Get-Item -LiteralPath $runtimeLink -Force).Delete() } + foreach ($arm in @($with, $without)) { + $probe = Get-Content (Join-Path $arm.Root 'repo/projection-probe.json') -Raw | ConvertFrom-Json + Assert-True (-not $probe.metadata -and -not $probe.paired) 'parent metadata and sibling arms are unreachable through projected ancestry' + Assert-Equal 0 @($probe.ambient).Count 'source AGENTS and Copilot instructions are excluded' + Assert-Equal 'STAGED_INSTRUCTIONS_CANARY' $probe.staged_agents 'staged AGENTS preserved identically' + Assert-Equal 'STAGED_COPILOT_CANARY' $probe.staged_copilot 'staged Copilot instructions preserved identically' + Assert-Equal ($arm.Root -eq $with.Root) $probe.candidate 'candidate material is only projected for with_skill' + } $transcript = Get-PortableTranscript -Raw $result -RunData $inputs.Run foreach ($text in @((Get-InteractionTurnText -Turn $inputs.Run.Interaction.turns[0] -RunData $inputs.Run), 'Confirm before generation. Δ', 'Yes, generate the key.', "Generated successfully.`nExact terminal text.", "Session: fixture-session`nSame session: true")) { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 84c51f1..81841e7 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -17,6 +17,7 @@ function Invoke-FakeHost($Decision) { # The host's only input is the actual canonical handoff file, never an arm prompt. Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + Assert-True (Test-Path -LiteralPath (Join-Path (Split-Path -Parent $Decision.prompt_path) '.external-handoff-started')) 'External handoff must be reserved before the host launch, for every runner.' $script:dispatches.Add($Decision.prompt_path) } } diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index a32bb4d..e27ad94 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -191,7 +191,7 @@ function New-TestGradingDocument { assertion_index = $index assertion = [string]$assertions[$index] passed = $true - evidence = 'deterministic grading-isolation fixture evidence' + evidence = "Source: output`nQuote: $((Read-TestJson -Path $record.ResultPath).output)`nReason: The captured fixture response establishes assertion $index for this deterministic transport case." }) } } @@ -417,7 +417,7 @@ for ($index = 0; $index -lt $count; $index++) { eval_name = $evalName prompt = "fixture prompt $evalId" expected_output = 'fixture output' - assertions = @($assertion) + assertions = @($assertion, 'the response contains the fixture completion content') }) $runs = [ordered]@{} foreach ($configuration in @('with_skill', 'without_skill')) { @@ -431,7 +431,7 @@ for ($index = 0; $index -lt $count; $index++) { eval_name = $evalName configuration = $configuration execution_status = 'unrun' - grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }) + grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }, [ordered]@{ text = 'the response contains the fixture completion content'; passed = $null; evidence = '' }) }) $runs[$configuration] = [ordered]@{ mode = $configuration @@ -596,6 +596,25 @@ for ($index = 0; $index -lt $count; $index++) { Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'invalid grading entry validation' -ExpectedText 'passed must be a boolean' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'invalid grading validation' + foreach ($badEvidence in @('', " `t`n", 'Evaluation completed with output', "Source: output`nQuote: fabricated unavailable observation`nReason: This establishes the assertion.", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Evaluation completed with output")) { + $bad = Copy-TestGradingDocument -Document $validGrading + $bad.grading[0].evidence = $badEvidence + Write-TestJson -Path $gradingPath -Value $bad + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'non-evidentiary PASS rejected' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'evidence rejection preserves frozen execution' + } + $repeated = Copy-TestGradingDocument -Document $validGrading + $repeated.grading[1].evidence = $repeated.grading[0].evidence + Write-TestJson -Path $gradingPath -Value $repeated + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'repeated PASS evidence rejected' -ExpectedText 'Repeated PASS evidence' + + $artifactEvidence = Copy-TestGradingDocument -Document $validGrading + $firstRecord = @($records | Sort-Object EvalId, Configuration)[0] + $eventLine = @(Get-Content (Join-Path (Split-Path -Parent $firstRecord.RunManifestPath) 'evidence/fixture-events.jsonl'))[0] + $artifactEvidence.grading[0].evidence = "Source: evidence/fixture-events.jsonl`nQuote: $eventLine`nReason: The native event records the fixture response for this assertion." + Write-TestJson -Path $gradingPath -Value $artifactEvidence + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'frozen native artifact citation accepted' + Write-TestJson -Path $gradingPath -Value $validGrading $validValidation = Invoke-TestTool -Path $validationScript -Arguments $validationArguments Assert-ToolPasses -Invocation $validValidation -Description 'valid grading validation' diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 019a416..0916901 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -1329,8 +1329,8 @@ exit 2 Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is projection-local' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the projected isolated home' Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index c160999..80fce77 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -935,6 +935,8 @@ function Get-EvalWorkspaceOption { if ($EvalEntry.PSObject.Properties.Name -contains 'workspace' -and $null -ne $EvalEntry.workspace) { $workspace = $EvalEntry.workspace if ($workspace.PSObject.Properties.Name -contains 'git' -and $null -ne $workspace.git) { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $workspace.git $wantsGit = [bool]$workspace.git } } @@ -1043,8 +1045,12 @@ function Initialize-GitWorkspace { '-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', + '-c', 'core.hooksPath=', + '-c', 'init.templateDir=', '-c', 'core.autocrlf=false' ) + $previousAuthorDate = $env:GIT_AUTHOR_DATE + $previousCommitterDate = $env:GIT_COMMITTER_DATE $env:GIT_AUTHOR_DATE = '2020-01-01T00:00:00Z' $env:GIT_COMMITTER_DATE = '2020-01-01T00:00:00Z' try { @@ -1062,8 +1068,8 @@ function Initialize-GitWorkspace { } & git @identity -C $RepoDirectory tag 'v1.0.0' 2>$null | Out-Null } finally { - Remove-Item Env:GIT_AUTHOR_DATE -ErrorAction SilentlyContinue - Remove-Item Env:GIT_COMMITTER_DATE -ErrorAction SilentlyContinue + $env:GIT_AUTHOR_DATE = $previousAuthorDate + $env:GIT_COMMITTER_DATE = $previousCommitterDate } } @@ -1550,6 +1556,8 @@ function Invoke-PrepareMode { } if ($workspaceOption.Git) { Initialize-GitWorkspace -RepoDirectory $repoDir + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Add-EvalGitScenario -RepoDirectory $repoDir -Scenario $evalEntry.workspace.git } $homeDir = Join-Path $runDir $runDirectoryNames.Home @@ -1868,6 +1876,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') + [void]$builder.AppendLine('Before creating grading.json, the Grader MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`; that guidance is authoritative during Phase 2. Uncertain or unverified expectations FAIL. Every assertion requires specific evidence. For PASS, evidence must use three newline-separated fields: `Source: output` (or a manifest-recorded run artifact path), `Quote: `, and `Reason: `. Do not reuse generic completion statements or identical evidence across assertions. FAIL evidence must explain what is missing or contradicted. Deterministic validation checks provenance and shape; it does not replace the Grader judgment required by grader.md.') [void]$builder.AppendLine('To display the authoritative top-level grading skeleton, run:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -ShowSkeleton") [void]$builder.AppendLine('Write `grading.json`, then validate it before finalization:') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 2119cc4..f8c5964 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -648,8 +648,9 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills include va if ($eval.workspace -isnot [System.Management.Automation.PSCustomObject]) { throw "$relativeEvalPath eval $($eval.id) has a non-object 'workspace'" } - if ($eval.workspace.PSObject.Properties.Name -contains 'git' -and $eval.workspace.git -isnot [bool]) { - throw "$relativeEvalPath eval $($eval.id) must declare 'workspace.git' as a boolean" + if ($eval.workspace.PSObject.Properties.Name -contains 'git') { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $eval.workspace.git } } } @@ -1277,6 +1278,18 @@ Add-ValidationResult -Results $results -Name 'Token normalization and benchmark if ($LASTEXITCODE -ne 0) { throw "Token reporting regression failed: $($output -join [Environment]::NewLine)" } } +Add-ValidationResult -Results $results -Name 'Copilot physical boundaries and checkpoint telemetry remain deterministic' -Group 'Runners' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-copilot-boundaries.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Copilot boundary regression failed: $($output -join [Environment]::NewLine)" } +} + +Add-ValidationResult -Results $results -Name 'Change-impact default resolution has paired deterministic Git history' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-change-impact-workspace.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Git scenario regression failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Runner-owned orchestration remains deterministic' -Group 'Runners' -Action { if (-not [string]::IsNullOrWhiteSpace($Ref)) { return @@ -2359,7 +2372,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' assertion_index = $assertionIndex assertion = [string]$metadataForGrade.assertions[$assertionIndex] passed = $true - evidence = 'validator evidence' + evidence = "Source: output`nQuote: $(([IO.File]::ReadAllText((Join-Path $iterationDirectory $entryToGrade.runs.$configuration.result)) | ConvertFrom-Json).output)`nReason: The fixture response supplies the observed content for assertion $assertionIndex in this deterministic transport test." }) } } From 611b449b12b7f661d3ef66d1aae4bcfe17e0b456 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:30 +0200 Subject: [PATCH 11/36] =?UTF-8?q?=E2=9C=85=20add=20evaluation=20configurat?= =?UTF-8?q?ion=20for=20dotnet-change-impact=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive eval cases for dotnet-change-impact skill including test fixtures for API compatibility analysis, breaking change detection, and version impact assessment. Evaluation cases cover .NET library scenarios with fixture files demonstrating package changes and dependency impacts. --- skills/dotnet-change-impact/evals/evals.json | 13 +++++++++++++ .../files/default-resolution/src/Example.csproj | 5 +++++ .../evals/files/default-resolution/src/Widget.cs | 7 +++++++ 3 files changed, 25 insertions(+) create mode 100644 skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj create mode 100644 skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs diff --git a/skills/dotnet-change-impact/evals/evals.json b/skills/dotnet-change-impact/evals/evals.json index 9794de7..c4941f7 100644 --- a/skills/dotnet-change-impact/evals/evals.json +++ b/skills/dotnet-change-impact/evals/evals.json @@ -91,6 +91,19 @@ }, { "id": 9, + "files": ["evals/files/default-resolution/src/Widget.cs", "evals/files/default-resolution/src/Example.csproj"], + "workspace": { + "git": { + "base_branch": "trunk", + "feature_branch": "feature/remove-legacy-api", + "commits": [ + { + "message": "Remove the legacy public Parse overload", + "files": { "src/Widget.cs": "namespace Example;\n\npublic class Widget\n{\n public string Name { get; set; } = string.Empty;\n}\n" } + } + ] + } + }, "prompt": "Use dotnet-change-impact to classify the version bump for this branch.", "expected_output": "Current-branch default-resolution behavior: inspect the local Git repository, resolve the current branch against the upstream/default base branch, collect commits and net diff, then classify those changes with structured reasoning. Must not ask the user to provide change details first.", "expectations": [ diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj new file mode 100644 index 0000000..ec2cce1 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj @@ -0,0 +1,5 @@ + + + net8.0 + + diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs new file mode 100644 index 0000000..184db81 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs @@ -0,0 +1,7 @@ +namespace Example; + +public class Widget +{ + public string Name { get; set; } = string.Empty; + public static Widget Parse(string value) => new Widget { Name = value }; +} From 944dfac45aff195e43d0519d58a78dff9e374024 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:06:39 +0200 Subject: [PATCH 12/36] =?UTF-8?q?=E2=9C=85=20add=20test=20for=20eval=20git?= =?UTF-8?q?=20workspace=20scenarios?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Test deterministic package preparation for dotnet-change-impact eval 9 with real git repository staging. Validates paired git history consistency, feature branch setup, default branch fallback resolution, meaningful API diffs, and safety checks for declarative git scenarios including path-traversal rejection. --- .../tests/test-change-impact-workspace.ps1 | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 scripts/eval-runners/tests/test-change-impact-workspace.ps1 diff --git a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 new file mode 100644 index 0000000..23bf0f4 --- /dev/null +++ b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 @@ -0,0 +1,45 @@ +# Real package preparation with a fake model catalog; never launches a provider. +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scriptsRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('change-impact-workspace-' + [guid]::NewGuid().ToString('N')) +[void][IO.Directory]::CreateDirectory($workspace) +function Assert-True($Condition, $Message) { if (-not $Condition) { throw $Message } } +try { + $catalog = Join-Path $workspace 'models.json' + [IO.File]::WriteAllText($catalog, '{"models":[{"id":"fixture-model"}]}') + $prompt = & (Join-Path $scriptsRoot 'prepare-skill-evals.ps1') -Skill dotnet-change-impact -Eval 9 -Runner github-copilot -Model fixture-model -ModelCatalogPath $catalog -OutputRoot $workspace -PassThru + $package = Split-Path -Parent $prompt + $handoff = [IO.File]::ReadAllText($prompt) + Assert-True ($handoff.Contains('MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`')) 'Handoff must require the exact packaged grader before grading.' + $manifest = Get-Content (Join-Path $package 'manifest.json') -Raw | ConvertFrom-Json + $heads = @(); $diffs = @(); $refs = @() + foreach ($arm in @('with_skill', 'without_skill')) { + $repo = Join-Path (Split-Path -Parent (Join-Path $package $manifest.evals[0].runs.$arm.run_manifest)) 'repo' + Assert-True (Test-Path (Join-Path $repo '.git') -PathType Container) 'Eval 9 must stage a real .git repository.' + Assert-True ((& git -C $repo branch --show-current) -eq 'feature/remove-legacy-api') 'Feature branch missing.' + Assert-True ((& git -C $repo symbolic-ref refs/remotes/origin/HEAD --short) -eq 'origin/trunk') 'Non-main default branch fallback missing.' + Assert-True ((& git -C $repo rev-list --count origin/HEAD..HEAD) -eq '1') 'Feature commit history missing.' + $diff = (& git -C $repo diff origin/HEAD...HEAD) -join "`n" + Assert-True ($diff.Contains('- public static Widget Parse(string value)')) 'Meaningful public API removal missing.' + Assert-True ([string]::IsNullOrWhiteSpace((& git -C $repo status --porcelain) -join '')) 'Staged repository must be clean.' + $heads += & git -C $repo rev-parse HEAD + $diffs += $diff + $refs += ((& git -C $repo show-ref) -join "`n") + } + Assert-True ($heads[0] -ceq $heads[1] -and $diffs[0] -ceq $diffs[1] -and $refs[0] -ceq $refs[1]) 'Paired Git history, refs and diff must be identical.' + . (Join-Path $scriptsRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario $true + Assert-EvalGitScenario $false + foreach ($invalid in @('{"base_branch":"main","feature_branch":"main","commits":[]}', '{"base_branch":"main","feature_branch":"feature","commits":[{"message":"unsafe","files":{"../escape":"bad"}}]}')) { + $rejected = $false + try { Assert-EvalGitScenario ($invalid | ConvertFrom-Json) } catch { $rejected = $true } + Assert-True $rejected 'Unsafe declarative scenario must be rejected.' + } + 'PASS: eval 9 deterministic package Git history, pairing, grader handoff, boolean compatibility and unsafe-path rejection; no models.' +} finally { + if (-not ([IO.Path]::GetFileName($workspace) -match '^change-impact-workspace-[0-9a-f]{32}$')) { throw 'Unsafe test cleanup.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} From 05e132ba8fd3e4b7db9468f61a3cb882c80c102d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:15 +0200 Subject: [PATCH 13/36] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20git-visual?= =?UTF-8?q?-commits=20scope=20and=20extract=20grouping=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhanced git-visual-commits to fully document scope rules for tracked, staged, unstaged, deleted, renamed, and non-ignored untracked files including contents of new directories. Extracted detailed grouping examples and release-adjacent splitting guidance to a new references/grouping-examples.md for better maintainability. Added three new test cases (26–28) covering untracked file discovery, path reconciliation with hidden status, and explicit scope narrowing. --- skills/git-visual-commits/SKILL.md | 104 +++--------------- skills/git-visual-commits/evals/evals.json | 35 ++++++ .../references/grouping-examples.md | 89 +++++++++++++++ 3 files changed, 139 insertions(+), 89 deletions(-) create mode 100644 skills/git-visual-commits/references/grouping-examples.md diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index 0700436..547cbbd 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -81,7 +81,7 @@ If the user did **not** say `yolo` or `auto`, and session-level auto mode is not If the user says `git bot commit`, `git commit`, or `git our commit` without narrowing language, treat the request as covering the full current worktree. -- The default scope is **all current changes visible in git status**. +- The default scope is **all current changes visible in git status**: staged, unstaged, deleted, renamed, and non-ignored untracked files, including files inside new directories. - Your job is to group that full worktree into the right number of commits by semantic intent. - Never silently narrow the scope to "just the files from the last thing I worked on", "just the files I touched", or "just the newest skill" unless the user explicitly said to do that. - `yolo` keeps this same full-worktree default. It removes the approval wait; it does not narrow scope. @@ -93,6 +93,8 @@ Narrow scope only when the user explicitly does one of these: - asks for a review/plan for a subset before committing If the user did not narrow scope, do not invent a narrower scope on their behalf. + +Use the expanded status inventory in Step 1 as the scope of record. `git diff`, `git diff --stat`, and `git diff --name-only` omit untracked files; none can replace that inventory. Untracked status alone is neither an exclusion nor a reason to ask permission. Respect Git's ignore rules for untracked files; do not force-add ignored files. Already tracked files remain in scope even if an ignore pattern now matches them. ### Recovery Safety Rule @@ -238,7 +240,9 @@ Commits include a body by default. `no-body` or `tmi` suppresses it for one requ ### Step 1: Review changes -Run `git status` and `git diff` (and `git diff --staged` if there are staged changes) to understand what has changed. +From the repository root, run `git status --short --untracked-files=all` to enumerate individual paths even when Git configuration hides untracked files. Keep this inventory through final verification. For programmatic parsing, use `git status --porcelain=v1 --untracked-files=all -z` and parse NUL-delimited records, including rename pairs; do not split filenames on whitespace. + +Review `git diff` and `git diff --staged` for tracked changes and inspect every untracked file directly with an appropriate text or binary viewer. An empty diff does not mean a clean worktree. Do not stage files just to make them appear in a diff. Unless the user explicitly narrowed scope, inspect the **entire current worktree** and build the commit plan from that full set of changes. Do not default to the last task only. @@ -250,7 +254,7 @@ When resolving that reference, prefer the bundled skill path first instead of tr ### Step 2: Classify changes -Before composing any commit message, bucket every changed file by its **semantic intent** — not just its file type. Read the actual diff for each file and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. +Before composing any commit message, bucket every path in the Step 1 inventory by its **semantic intent** — not its tracking status or file type. Read the actual diff for tracked files and contents of untracked files and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. Use the inspected commit-language reference as the meaning source, not your gut. For example, restructuring an existing skill's `SKILL.md`, `FORMS.md`, `references/`, or `evals/` is normally refactor intent and should map to `♻️`; configuration-file changes map to `🔧`; truly new repo or application capabilities map to `✨`. @@ -298,36 +302,9 @@ When more than one file is changed and your first classification puts every chan Ask yourself explicitly: **“Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?”** A metadata preview, remembered rule, or partial read is a failed answer. If the answer is no or uncertain, read `SKILL.md` from its first line through EOF and restart Step 1 and Step 2. -Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. +Then refresh the Step 1 inventory and review tracked diffs and untracked contents; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. Only keep one category after this audit if every path still has one rationale. Put a visible line in the commit plan such as `Quality gate: 3 files, one category retained; full skill read, full diff review, per-file rationale check, and alternative-category check confirmed.` If any check fails or any file has a materially different intent, split the groups and rerun the normal validation. `yolo` and `auto` do not bypass this gate. - -#### Commit body guidance - -Unless **no-body mode** is active, every commit includes a body explaining the *why*: - -- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") -- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") -- **Refactors** → explain what motivated the restructuring -- **New features** → explain the purpose and scope -- **Bug fixes** → explain what was broken and how this fixes it - -Common groupings: -- New repo-managed skill or workflow introduction together -- Existing skill refactor or extraction together -- Dependency/version baseline updates together -- Package/publish metadata together -- Config/setup files together (app host, bootstrapping) -- Environment and infrastructure config together (test runners, CI matrix, container settings) -- Documentation publishing fixes together -- Community health or release communication docs together -- New feature or module code together -- Data contracts, types, and interfaces together -- Database models, migrations, and schema changes together -- Test logic and assertions together (when they share the same rationale) -- Documentation and inline comments together - -When in doubt, one commit per "thing that changes" is better than one big commit. #### Mixed-scope guard @@ -339,40 +316,9 @@ This guard runs unconditionally — including in auto-approval mode. Documentation files (`CHANGELOG.md`, `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, release notes) are **separate-by-default**. They only belong in the same commit as non-doc files when the commit is explicitly documentation-focused (e.g. `📝 add api usage guide` where the docs are the point, not a side effect). -#### Release-adjacent splitting rule - -Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: - -- **Dependency/version baselines** — version alignment or runner baseline changes -- **Community health/release communication** — changelogs and human-facing repo health docs -- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` -- **Documentation publishing** — DocFX navigation, branding, or publishing assets -- **CI/automation** — workflows and helper scripts used only by automation - -These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. - -Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: - -- **Build system / dependency baseline** -- **CI or automation** -- **Release communication plus package metadata** - -Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. - -#### Repo-aligned grouping example - -When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: +#### Detailed grouping rules -- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` -- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill -- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` -- **Template/runtime files** — `assets/`, scaffold helper scripts -- **Validation/tooling** — validator scripts, repo checks -- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` - -Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". - -If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. +Read [grouping-examples.md](references/grouping-examples.md) before finalizing groups. Apply its release-adjacent splitting rule and repository examples alongside the guards above. #### Rename vs removal distinction @@ -409,7 +355,7 @@ Even in auto-approval mode, surface the commit buckets explicitly before committ The summary is status output, not a review request. Step 5 is mandatory in the same turn once its preconditions pass: never ask "Proceed with committing these groups?" (or an equivalent question), wait for a reply, or finish with a pending commit plan. -If the user did not narrow scope, the plan you surface must account for the full worktree rather than an arbitrarily chosen subset. +Reconcile the plan's exact path lists against a fresh Step 1 inventory before presenting it. Every in-scope path must belong to a commit group; splitting a file across groups must account for all its changes. Resolve missing paths before proceeding. Show a coverage line such as `Scope: 7 changed paths (3 untracked), 7 planned, 0 omitted.` Counts supplement the path comparison; equal counts alone do not prove coverage. Record explicit user exclusions separately. Apply this check in every mode, including single-file and auto-approved requests. **Otherwise**, wait for the user to confirm or adjust. They may say things like: - "Looks good" → proceed to stage and commit @@ -432,7 +378,8 @@ Before committing, validate each message against its file list: ### Step 5: Stage and commit each group For each group: -1. `git add ` — be precise, don't use `git add .` unless everything belongs in one commit +1. Refresh the inventory after approval and reconcile any changes with the plan. Stage each group's exact paths, including new files and deletions, with `git add -A -- `. Use literal pathspecs for names containing Git wildcard syntax. Do not use tracked-only staging such as `git add -u` or `git commit -a` as a substitute. + Compare `git diff --staged --name-status` and the full staged diff against the group's planned changes before committing. Account for pre-existing staged changes too; resolve missing or extra staged paths without discarding their contents. Keep staging and commits sequential because they share the index. 2. Compose the commit message (see format above) 3. Run `scripts/validate-commit-subject.ps1` again against the exact subject that will be passed to Git. Add `-PrefixMode Required` only for an explicitly requested combo. Do not run Git unless the validator exits successfully. 4. Run the appropriate commit command: @@ -449,35 +396,14 @@ When the body is just one short explanatory paragraph, prefer a single natural p ### Step 6: Verify +After each commit, compare its recorded paths (`git diff-tree --root --no-commit-id --name-status -r HEAD`) with the group, accounting for both sides of renames. After the last group, rerun `git status --short --untracked-files=all` and reconcile remaining paths with the original inventory and explicit exclusions. A full-worktree request is complete only when no in-scope changes remain, including untracked files. If a path was missed, review and include it through the same workflow under the existing authorization; do not silently leave it behind or claim completion. Report any concrete blocker or newly arriving change that prevents completion. + After committing, run `git log --oneline -5` to confirm the commit looks right. Read the stored subject with `git log -1 --format=%s` and run `scripts/validate-commit-subject.ps1` against that exact value. Then always run `git log -1 --format="%an <%ae>"` and verify that the author matches the requested identity mode before reporting success. Also run `git log -1 --format=%B` and verify the stored body contains readable prose with real line breaks, not literal escape sequences such as `\n`, and is not hard-wrapped mid-sentence just to satisfy a column limit. If any verification fails, amend the commit immediately instead of merely warning about it. If verification fails because the commit path used the wrong author or ignored the requested alias, stop treating it as a message-tweaking problem. Correct the tool path first, preserve the worktree safely, and only then repair the commit. --- -## Good Examples - -``` -🎉 begin api project -✨ add submission endpoint module -🐛 handle null optional fields in dto -➕ add validation library -🐛 fix: handle null optional fields in dto ← only when combo mode was requested -``` - -## Bad Examples (and why) - -``` -feat: add submission endpoint ← "feat:" is not an allowed prefix -✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed -💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning -💬 update changelog for v10.0.10 ← more than one separator space -📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table -🎉 initial commit with all files ← vague, bundles everything -⚙️ config: setup api ← "config:" is not an allowed prefix -♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo -``` - ## Branching (for reference) When explicitly requested, use `[version]/[description]` (for example `v1.2.0/admin-dashboard`). Do not create, rename, or delete branches unless the user explicitly asks. diff --git a/skills/git-visual-commits/evals/evals.json b/skills/git-visual-commits/evals/evals.json index 6ab5043..7fbda95 100644 --- a/skills/git-visual-commits/evals/evals.json +++ b/skills/git-visual-commits/evals/evals.json @@ -283,6 +283,41 @@ "Treats the full current worktree as scope because the user did not narrow it", "Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer" ] + }, + { + "id": 26, + "prompt": "Please git commit, yolo. The worktree has an unstaged change to src/Router.cs, a staged change to config/routes.json, and new untracked src/RoutePolicy.cs and assets/route-map.svg. git diff --stat only lists src/Router.cs. All four files belong to this routing change.", + "expected_output": "A complete commit workflow accounting for all four paths, inspecting untracked contents and reconciling inventory, plan, index, recorded commits, and final status.", + "expectations": [ + "Uses git status --short --untracked-files=all or its NUL-delimited porcelain equivalent as the scope inventory instead of a diff summary", + "Inspects and includes both src/RoutePolicy.cs and assets/route-map.svg without asking permission merely because they are untracked", + "Accounts for the pre-existing staged config/routes.json change when verifying each commit group", + "Compares exact planned paths with the inventory and reports no omitted in-scope paths", + "Checks staged contents and recorded commit paths against each group and verifies no in-scope tracked or untracked changes remain" + ] + }, + { + "id": 27, + "prompt": "Please review my working tree and show the commit plan only. I changed README.md and added a new assets folder containing logo.svg and dark theme/banner.svg. Git is configured with status.showUntrackedFiles=no. A scratch file under ignored .cache/ also exists.", + "expected_output": "A read-only plan that explicitly discovers both new files, handles the path containing spaces, covers the README change, and excludes the ignored scratch file.", + "expectations": [ + "Overrides hidden untracked status with --untracked-files=all and enumerates individual files in new directories", + "Inspects both untracked assets and includes assets/logo.svg and assets/dark theme/banner.svg in the plan", + "Preserves the space-containing path as one filename", + "Excludes the ignored .cache file without force-adding it", + "Does not stage or commit for a plan-only request" + ] + }, + { + "id": 28, + "prompt": "Please git commit only the README.md change, yolo. src/Router.cs is modified and src/RoutePolicy.cs is untracked too; leave both alone.", + "expected_output": "A README-only commit that records the explicit exclusions and reports the remaining source changes accurately.", + "expectations": [ + "Limits the commit to README.md because the user explicitly narrowed scope", + "Leaves both src/Router.cs and src/RoutePolicy.cs out of the commit and preserves their contents", + "Reconciles final status against the explicit exclusions instead of requiring an entirely clean worktree", + "Does not claim the entire worktree is clean when excluded paths remain" + ] } ] } diff --git a/skills/git-visual-commits/references/grouping-examples.md b/skills/git-visual-commits/references/grouping-examples.md new file mode 100644 index 0000000..689962f --- /dev/null +++ b/skills/git-visual-commits/references/grouping-examples.md @@ -0,0 +1,89 @@ +# Commit grouping examples + +#### Release-adjacent splitting rule + +Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: + +- **Dependency/version baselines** — version alignment or runner baseline changes +- **Community health/release communication** — changelogs and human-facing repo health docs +- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` +- **Documentation publishing** — DocFX navigation, branding, or publishing assets +- **CI/automation** — workflows and helper scripts used only by automation + +These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. + +Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: + +- **Build system / dependency baseline** +- **CI or automation** +- **Release communication plus package metadata** + +Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. + +#### Repo-aligned grouping example + +When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: + +- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` +- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill +- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` +- **Template/runtime files** — `assets/`, scaffold helper scripts +- **Validation/tooling** — validator scripts, repo checks +- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` + +Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". + +If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. + + +#### Commit body guidance + +Unless **no-body mode** is active, every commit includes a body explaining the *why*: + +- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") +- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") +- **Refactors** → explain what motivated the restructuring +- **New features** → explain the purpose and scope +- **Bug fixes** → explain what was broken and how this fixes it + +Common groupings: +- New repo-managed skill or workflow introduction together +- Existing skill refactor or extraction together +- Dependency/version baseline updates together +- Package/publish metadata together +- Config/setup files together (app host, bootstrapping) +- Environment and infrastructure config together (test runners, CI matrix, container settings) +- Documentation publishing fixes together +- Community health or release communication docs together +- New feature or module code together +- Data contracts, types, and interfaces together +- Database models, migrations, and schema changes together +- Test logic and assertions together (when they share the same rationale) +- Documentation and inline comments together + +When in doubt, one commit per "thing that changes" is better than one big commit. + + +## Good Examples + +``` +🎉 begin api project +✨ add submission endpoint module +🐛 handle null optional fields in dto +➕ add validation library +🐛 fix: handle null optional fields in dto ← only when combo mode was requested +``` + +## Bad Examples (and why) + +``` +feat: add submission endpoint ← "feat:" is not an allowed prefix +✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed +💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning +💬 update changelog for v10.0.10 ← more than one separator space +📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table +🎉 initial commit with all files ← vague, bundles everything +⚙️ config: setup api ← "config:" is not an allowed prefix +♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo +``` + From 66d2fdcbb68280a5751f89a58eb9825fb62bc8d0 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:30 +0200 Subject: [PATCH 14/36] =?UTF-8?q?=F0=9F=94=A8=20update=20skill=20validatio?= =?UTF-8?q?n=20to=20reference=20grouping=20examples?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated validate-skill-templates.ps1 to check for the new references/grouping-examples.md file and verify its content contains the release-adjacent splitting rules and repo-aligned grouping examples that were extracted from SKILL.md. --- scripts/validate-skill-templates.ps1 | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index f8c5964..b4da584 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -2774,6 +2774,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces $commitLanguage = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/commit-language.md' -GitRef $Ref $subjectValidator = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/validate-commit-subject.ps1' -GitRef $Ref $subjectTests = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/test-commit-subject.ps1' -GitRef $Ref + $groupingExamples = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/grouping-examples.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'automatic trigger for this skill, not as a casual hint.' @@ -2838,17 +2839,17 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Single-category context quality gate' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Exactly one changed file is the only fast-path exception; skip this gate for that case.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then refresh the Step 1 inventory and review tracked diffs and untracked contents' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Put a visible line in the commit plan such as `Quality gate:' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` and `auto` do not bypass this gate.' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Release-adjacent splitting rule' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '#### Release-adjacent splitting rule' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'do not collapse "new skill introduced" and "existing skill refactored" into one commit' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '**New repo-managed skill**' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '**New repo-managed skill**' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Allowed Prefixes' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Emoji Selection' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle 'Gitmoji First, Fallback Second' From 469b4a309d98a17468256ffd1a83670cf3e490b4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:42 +0200 Subject: [PATCH 15/36] =?UTF-8?q?=F0=9F=92=AC=20document=20git-visual-comm?= =?UTF-8?q?its=20scope=20and=20inventory=20features?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated README.md to document new git-visual-commits capabilities: full inventory of tracked, staged, unstaged, and non-ignored untracked files; individual file enumeration inside new directories; exact path reconciliation before staging; and final remaining-change verification. Updated skill table description to reflect these improvements. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index fbf3803..59e69cb 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Each `SKILL.md` description is lean activation metadata. The catalog below expla | Skill | Description | |-------|-------------| -| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | +| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping of tracked and non-ignored untracked files unless narrowed, exact path reconciliation before staging, and a final remaining-change check, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | | [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | @@ -284,6 +284,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith Commit messages are the most-read documentation in any codebase — yet they're usually an afterthought. "fix stuff", "wip", "address PR feedback" tells you nothing six months later. Writing good commits takes discipline, and when you're in flow, it's the first thing that slips. +**git-visual-commits** includes staged, unstaged, and non-ignored untracked files by default. It inventories individual files inside new directories, reconciles every path with the commit plan and staged changes, and checks for leftovers before reporting completion. Explicitly narrowed requests preserve excluded changes. + **git-visual-commits** handles the entire commit workflow— staging, diffing, crafting the message, choosing the right emoji — so every commit is consistent and meaningful without breaking your flow. Whether the agent authors the commit (`git bot commit`), you do (`git commit`), or you worked on it together (`git our commit`), the quality is the same. - **Gitmoji-first** — visual commit categories that are scannable at a glance From ab54ffa39b96a5879f9cfc780d89873e25d2075d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 21:34:59 +0200 Subject: [PATCH 16/36] =?UTF-8?q?=F0=9F=90=9B=20fix=20eval=20runner=20cros?= =?UTF-8?q?s-platform=20path=20and=20environment=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add proper handling for Unix-style temporary directories and environment variables. Runner now respects RunPath property for projected inputs that use /tmp directly. Tests isolate TMPDIR alongside TEMP/TMP, and GitHub CLI config path is now platform-aware to handle Unix vs Windows conventions correctly. --- scripts/eval-runners/runner-common.ps1 | 15 +++++++++++++-- .../tests/test-copilot-boundaries.ps1 | 6 +++--- .../tests/test-runner-conformance.ps1 | 6 ++++-- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 89b26be..279d38c 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -1470,8 +1470,19 @@ function Assert-PhaseOneEvidenceWritable { # package-level freeze exists, refusing to build another result prevents a # direct runner invocation (or the orchestrator-owned recorder) from # truncating or replacing frozen raw evidence. - $runRoot = [System.IO.Path]::GetFullPath([string]$Run.RunRoot) - $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runRoot) + # Projected runner inputs may replace RunRoot with a physical directory + # directly under /tmp. Derive the package location from the original + # manifest path so freeze checks remain valid on every platform. + $runPath = [string](Get-JsonProperty -Object $Run -Name 'RunPath' -Default '') + $runRoot = if ([string]::IsNullOrWhiteSpace($runPath)) { + [System.IO.Path]::GetFullPath([string]$Run.RunRoot) + } else { + [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetFullPath($runPath)) + } + $iterationDirectory = [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetDirectoryName($runRoot)) + if ([string]::IsNullOrWhiteSpace($iterationDirectory)) { + throw 'Cannot derive the eval package directory from the run contract.' + } $freezeRelativePath = 'execution-freeze.json' $manifestPath = Join-Path $iterationDirectory 'manifest.json' if (Test-Path -LiteralPath $manifestPath -PathType Leaf) { diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index eefa54b..87789b3 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -215,11 +215,11 @@ if ($inputText -eq 'failure') { exit 7 } $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, @{ type = 'assistant.usage'; data = @{ inputTokens = 7; outputTokens = 3 } }); Errors = @() } -Warnings $warnings Assert-Equal 7 $usage.UsageInput 'native assistant usage takes precedence over cache snapshots' Assert-Equal 3 $usage.UsageOutput 'actual exposed output count retained' - $savedTemp = $env:TEMP; $savedTmp = $env:TMP + $savedTemp = $env:TEMP; $savedTmp = $env:TMP; $savedTmpDir = $env:TMPDIR try { - $env:TEMP = $testRoot; $env:TMP = $testRoot + $env:TEMP = $testRoot; $env:TMP = $testRoot; $env:TMPDIR = $testRoot Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'temp inside source ancestry fails closed' - } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp } + } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp; $env:TMPDIR = $savedTmpDir } $runtimeLink = Join-Path $singleInputs.Run.WorkingDirectoryPath 'forbidden-link' New-Item -ItemType $linkType -Path $runtimeLink -Target $testRoot | Out-Null try { Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'linked projection input fails closed' } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 0916901..5684a33 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -130,6 +130,7 @@ $recordedOldUserProfile = $env:USERPROFILE $recordedOldAppData = $env:APPDATA $recordedOldLocalAppData = $env:LOCALAPPDATA $recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME +$recordedOldTmpDir = $env:TMPDIR $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -2021,8 +2022,8 @@ exit 2 $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' - $copilotFreshHostGhConfig = Join-Path $copilotFreshHostAppData 'GitHub CLI' - New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, (Join-Path $copilotFreshXdg 'gh'), $copilotFreshHostGhConfig -Force | Out-Null + $copilotFreshHostGhConfig = if ($IsWindows) { Join-Path $copilotFreshHostAppData 'GitHub CLI' } else { Join-Path $copilotFreshXdg 'gh' } + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, $copilotFreshHostGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) $env:HOME = $copilotFreshHome $env:USERPROFILE = $copilotFreshHome @@ -2122,6 +2123,7 @@ exit 2 $env:APPDATA = $recordedOldAppData $env:LOCALAPPDATA = $recordedOldLocalAppData $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome + $env:TMPDIR = $recordedOldTmpDir $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } From 9cd1c5b68f580002462a7156e1ccb37b6e6d523c Mon Sep 17 00:00:00 2001 From: gimlichael Date: Tue, 8 Sep 2026 23:10:46 +0200 Subject: [PATCH 17/36] =?UTF-8?q?=F0=9F=90=9B=20fix=20explicit=20gh=5Fconf?= =?UTF-8?q?ig=5Fdir=20auth=20selection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make explicit GH_CONFIG_DIR authoritative during trusted GitHub CLI token resolution for Copilot runs. If that selected configuration cannot resolve a token, preflight now fails closed instead of silently probing other config roots or ambient identity. Add deterministic regressions that prove explicit GH_CONFIG_DIR success, explicit GH_CONFIG_DIR fail-closed behavior, no fallback probing to alternate configs, and no token value leakage in evidence or output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../eval-runners/github-copilot/runner.ps1 | 57 ++++++++++++++++--- .../tests/test-runner-conformance.ps1 | 56 +++++++++++++++++- 2 files changed, 103 insertions(+), 10 deletions(-) diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index 49d61ee..0a66709 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -220,6 +220,12 @@ function Add-CopilotGhConfigCandidate { } catch { } } +function Get-CopilotExplicitGhConfigDirectory { + $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if ([string]::IsNullOrWhiteSpace($configured)) { return $null } + return [string]$configured +} + function Get-CopilotGhConfigDirectories { # GH_CONFIG_DIR is an authentication-state exception to the isolated # Copilot configuration roots. Build documented/default candidates and keep @@ -281,16 +287,27 @@ function Get-CopilotGitHubCliToken { } } + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() - foreach ($directory in @(Get-CopilotGhConfigDirectories)) { - if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { - $candidateConfigDirectories.Add([string]$directory) + $useAmbientAttempt = $true + if (-not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory)) { + # An explicit GH_CONFIG_DIR selects the GitHub CLI identity. It is + # authoritative: do not search other config roots when it fails. + $candidateConfigDirectories.Add([string]$explicitGhConfigDirectory) + $useAmbientAttempt = $false + } else { + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) + } } } - # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing host - # default can still resolve even when no candidate was discovered. - $candidateConfigDirectories.Add('') + if ($useAmbientAttempt) { + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing + # host default can still resolve when no explicit config was supplied. + $candidateConfigDirectories.Add('') + } $probeRoot = Resolve-RunnerProbeTempRoot $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) @@ -318,6 +335,7 @@ function Get-CopilotGitHubCliToken { return [pscustomobject]@{ Token = $token GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + GhConfigExplicit = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) } } return $null @@ -329,6 +347,8 @@ function Get-CopilotGitHubCliToken { } function Resolve-CopilotAuthentication { + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory + $explicitGhConfigProvided = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) $tokenVariable = Get-CopilotTokenVariable if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { return [pscustomobject]@{ @@ -337,6 +357,8 @@ function Resolve-CopilotAuthentication { TokenValue = $null GitHubCliTokenResolved = $false GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $true } } @@ -349,6 +371,8 @@ function Resolve-CopilotAuthentication { TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + GitHubCliConfigExplicit = [bool](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigExplicit' -Default $false) + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $true } } @@ -360,6 +384,8 @@ function Resolve-CopilotAuthentication { TokenValue = $null GitHubCliTokenResolved = $false GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $false } } @@ -677,11 +703,22 @@ function Get-CopilotPreflight { if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $configDetail = if ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { 'ambient/default' } else { 'resolved GH_CONFIG_DIR candidate' } + $configDetail = if ([bool]$authState.GitHubCliConfigExplicit) { + 'explicit GH_CONFIG_DIR' + } elseif ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { + 'ambient/default' + } else { + 'platform-discovered GH_CONFIG_DIR candidate' + } $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) - $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') + if ([bool]$authState.ExplicitGhConfigProvided) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or fix the selected GH_CONFIG_DIR authentication state.') + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') + } } if ($platform -notin @('linux', 'macos')) { @@ -749,6 +786,8 @@ function Get-CopilotPreflight { noninteractive_ready = [bool]$authState.NonInteractiveReady github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + github_cli_config_explicit = [bool]$authState.GitHubCliConfigExplicit + explicit_gh_config_dir_provided = [bool]$authState.ExplicitGhConfigProvided token_value_observed = $false } } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 5684a33..fa20d33 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -969,6 +969,12 @@ if ($harness -eq 'codex') { $fakeGh = @' [CmdletBinding()] param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$logPath = Join-Path (Split-Path -Parent $MyInvocation.MyCommand.Path) 'gh-auth-token-probe.jsonl' +$record = [ordered]@{ + args = @($RemainingArguments | ForEach-Object { [string]$_ }) + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') +} +[System.IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { @@ -993,6 +999,7 @@ exit 2 New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) $env:GH_CONFIG_DIR = $recordedGhConfig + $ghAuthProbeLogPath = Join-Path $fakeBin 'gh-auth-token-probe.jsonl' $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) @@ -2075,6 +2082,53 @@ exit 2 Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' + Assert-True (($copilotGhResult | ConvertTo-Json -Depth 100) -notmatch 'recorded-gh-fallback-token-not-logged') 'Copilot GitHub CLI fallback result never exposes token values' + + # Explicit GH_CONFIG_DIR is authoritative. If it cannot resolve auth, do not + # consult discovered/ambient configurations even when they are valid. + $copilotExplicitMissingConfig = Join-Path $recordedRoot 'copilot-explicit-missing-gh-config' + if (Test-Path -LiteralPath $copilotExplicitMissingConfig) { + Remove-Item -LiteralPath $copilotExplicitMissingConfig -Recurse -Force + } + $copilotExplicitMissingRoot = Join-Path $recordedRoot 'copilot-explicit-missing' + $copilotExplicitMissingHome = Join-Path $copilotExplicitMissingRoot 'home' + $copilotExplicitMissingXdg = Join-Path $copilotExplicitMissingRoot 'xdg' + $copilotExplicitMissingAppData = Join-Path $copilotExplicitMissingRoot 'host-appdata' + $copilotExplicitMissingLocalAppData = Join-Path $copilotExplicitMissingRoot 'localappdata' + $copilotExplicitMissingFallbackConfig = if ($IsWindows) { Join-Path $copilotExplicitMissingAppData 'GitHub CLI' } else { Join-Path $copilotExplicitMissingXdg 'gh' } + New-Item -ItemType Directory -Path $copilotExplicitMissingHome, $copilotExplicitMissingXdg, $copilotExplicitMissingAppData, $copilotExplicitMissingLocalAppData, $copilotExplicitMissingFallbackConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotExplicitMissingFallbackConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotExplicitMissingHome + $env:USERPROFILE = $copilotExplicitMissingHome + $env:APPDATA = $copilotExplicitMissingAppData + $env:LOCALAPPDATA = $copilotExplicitMissingLocalAppData + $env:XDG_CONFIG_HOME = $copilotExplicitMissingXdg + $env:COPILOT_HOME = (Join-Path $copilotExplicitMissingRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $copilotExplicitMissingConfig + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { Remove-Item -LiteralPath $ghAuthProbeLogPath -Force } + $nativeExecutionsBeforeExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $copilotExplicitMissingPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingPreflight.status 'Copilot explicit GH_CONFIG_DIR without a valid token fails closed' + Assert-True (@($copilotExplicitMissingPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure is detected in preflight' + Assert-True ([bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.explicit_gh_config_dir_provided) 'Copilot preflight records that GH_CONFIG_DIR was explicitly provided' + Assert-True (-not [bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot explicit GH_CONFIG_DIR failure does not report non-interactive readiness' + Assert-True (@($copilotExplicitMissingPreflight.reasons | Where-Object { $_ -match 'GH_CONFIG_DIR was explicitly provided' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure reports selected-identity resolution failure' + $copilotExplicitMissingResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingResult.status 'Copilot explicit GH_CONFIG_DIR failure remains terminal in execute' + Assert-Equal 'incompatible' ([string]$copilotExplicitMissingResult.evidence.preflight.status) 'Copilot explicit GH_CONFIG_DIR execute result preserves incompatible preflight evidence' + $nativeExecutionsAfterExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeExplicitMissing.Count $nativeExecutionsAfterExplicitMissing.Count 'Copilot explicit GH_CONFIG_DIR rejection occurs before model execution' + $ghAuthProbeRecords = @() + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { + $ghAuthProbeRecords = @(Get-Content -LiteralPath $ghAuthProbeLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + $ghAuthTokenProbes = @($ghAuthProbeRecords | Where-Object { @($_.args).Count -eq 2 -and $_.args[0] -eq 'auth' -and $_.args[1] -eq 'token' }) + Assert-Equal 2 $ghAuthTokenProbes.Count 'Copilot explicit GH_CONFIG_DIR executes exactly one GitHub CLI token probe per preflight invocation' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -ne $copilotExplicitMissingConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure never probes another GitHub CLI configuration' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -eq $copilotExplicitMissingFallbackConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure does not consult a valid discovered fallback configuration' + $explicitFailureEvidenceText = ($copilotExplicitMissingPreflight | ConvertTo-Json -Depth 100) + ($copilotExplicitMissingResult | ConvertTo-Json -Depth 100) + Assert-True ($explicitFailureEvidenceText -notmatch 'recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot explicit GH_CONFIG_DIR failure never exposes token values' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null @@ -2098,7 +2152,7 @@ exit 2 Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' - Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' $env:HOME = $recordedOldHome $env:USERPROFILE = $recordedOldUserProfile $env:APPDATA = $recordedOldAppData From b9bccbc82936a7e734c7ff79aa0edbe45a4f463b Mon Sep 17 00:00:00 2001 From: gimlichael Date: Tue, 8 Sep 2026 23:10:54 +0200 Subject: [PATCH 18/36] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20git=20fix?= =?UTF-8?q?ture=20isolation=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify that temporary Git repos, branches, and throwaway commits are forbidden in the real source working tree while deterministic synthetic Git history remains allowed only inside isolated disposable eval fixtures under approved workspace roots. Keep the rule narrow by preserving the existing anti-pollution guardrails and adding deterministic wording checks so this distinction remains explicit across AGENTS, README, and validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENTS.md | 4 +++- README.md | 2 +- scripts/validate-skill-templates.ps1 | 2 ++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 924ac91..fe5f7bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,9 @@ Eval workspaces and test repositories must **never** become part of this reposit - `.bot/-workspace/` — the default. `.gitignore` covers `.bot/*`, so git never sees what lands there, and harnesses that refuse to work outside the repository folder still have somewhere to go. - `$env:TEMP/-workspace/` on Windows, `/tmp/-workspace/` on Unix — for anything that has no reason to sit next to the source. -Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. So are temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases. +Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. In the real `codebeltnet/agentic` source working tree, temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases are forbidden. + +Deterministic synthetic Git history is allowed only when it is the fixture being tested inside an explicitly permitted isolated eval/test repository under `.bot/-workspace/` or the approved OS temp workspace. Those fixture repositories must stay isolated, disposable, and uncommitted to this source repository. `scripts/prepare-skill-evals.ps1` enforces this: it writes to `.bot/` by default, refuses an `-OutputRoot` that is inside the repository but outside `.bot/`, and refuses `.bot/` itself if git has stopped ignoring it. diff --git a/README.md b/README.md index 59e69cb..75cdea7 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. +Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. This exception is only for isolated disposable eval fixtures under the approved workspace roots; throwaway Git history is still forbidden in the real source working tree. Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 explicitly follows the packaged Anthropic grader and requires assertion-specific, source-backed PASS evidence; generic completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b4da584..aedb05e 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1533,6 +1533,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`.bot/-workspace/` — the default.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Anywhere else inside the repository is forbidden' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Deterministic synthetic Git history is allowed only when it is the fixture being tested' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'throwaway Git history is still forbidden in the real source working tree' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'It did not run them, and nothing here will.' if (-not [string]::IsNullOrWhiteSpace($Ref)) { From 15c0e02c53fe367d674450883f02a871fb263ec4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Wed, 9 Sep 2026 13:47:10 +0200 Subject: [PATCH 19/36] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20improve=20dotnet-cha?= =?UTF-8?q?nge-impact=20compatibility=20classifications?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace ambiguous yes/no/possible answers in Compatibility Impact section with explicit categorical values: Compatible, Breaking, or Potentially breaking. This eliminates self-contradictory answers and clarifies the interpretation of each category for skill users. Simplify the eval prompt to be tool-agnostic, improving evaluation consistency across different harnesses. --- skills/dotnet-change-impact/SKILL.md | 20 +++++++++++++++----- skills/dotnet-change-impact/evals/evals.json | 2 +- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/skills/dotnet-change-impact/SKILL.md b/skills/dotnet-change-impact/SKILL.md index be53457..2089fda 100644 --- a/skills/dotnet-change-impact/SKILL.md +++ b/skills/dotnet-change-impact/SKILL.md @@ -328,11 +328,21 @@ there is truly no concrete change to list.> ## Compatibility impact -- Behavioral change: — -- Binary compatibility: — -- Source compatibility: — -- Design-time compatibility: — -- Backwards compatibility: — +State each category with one unambiguous categorical value — `Compatible`, `Breaking`, or +`Potentially breaking` — followed by the reason. Never answer these with `yes`/`no`: "yes" is +ambiguous about whether it means "yes, compatible" or "yes, it breaks", which has produced +self-contradictory answers such as "Binary compatibility: yes — compiled consumers may fail". + +- Behavioral compatibility: — +- Binary compatibility: — +- Source compatibility: — +- Design-time compatibility: — +- Backwards compatibility: — + +Read the values consistently: `Compatible` means existing consumers are unaffected in that +category, `Breaking` means they are broken, and `Potentially breaking` means the outcome depends +on a stated condition (for example, whether an API is public or whether consumers rely on the old +behavior). ## Reasoning diff --git a/skills/dotnet-change-impact/evals/evals.json b/skills/dotnet-change-impact/evals/evals.json index c4941f7..e2a81dc 100644 --- a/skills/dotnet-change-impact/evals/evals.json +++ b/skills/dotnet-change-impact/evals/evals.json @@ -104,7 +104,7 @@ ] } }, - "prompt": "Use dotnet-change-impact to classify the version bump for this branch.", + "prompt": "Classify the version bump for this branch.", "expected_output": "Current-branch default-resolution behavior: inspect the local Git repository, resolve the current branch against the upstream/default base branch, collect commits and net diff, then classify those changes with structured reasoning. Must not ask the user to provide change details first.", "expectations": [ "Follows Default Resolution Behavior when no explicit change details or compare range are provided", From 5dea7f0e3efd295517c97e01df0f1a9b261f1c7c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Wed, 9 Sep 2026 13:47:22 +0200 Subject: [PATCH 20/36] =?UTF-8?q?=F0=9F=94=A7=20enhance=20evaluation=20fra?= =?UTF-8?q?mework=20with=20analyzer/grader=20separation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for independent analyzer/grader configuration in the evaluation framework, allowing the same validator to grade results from different AI providers. This enables more flexible and reusable evaluation infrastructure. Implement shell environment isolation for Codex runner execution with sanitized PATH, SystemRoot, ComSpec, and PATHEXT variables. Enhance GitHub Copilot runner with improved boundary violation tracking and execution control. Update eval-grading-contract and validation scripts to support the new analyzer profile schema. Extend generate-eval-report with enhanced metadata handling for benchmark execution models. --- scripts/eval-runners/codex/runner.ps1 | 164 +++++++++++++-- .../eval-runners/eval-grading-contract.ps1 | 58 ++++-- .../eval-runners/github-copilot/isolation.ps1 | 54 ++++- .../eval-runners/github-copilot/runner.ps1 | 197 +++++++++++++++++- scripts/eval-runners/runner-common.ps1 | 13 ++ scripts/generate-eval-report.ps1 | 19 +- scripts/prepare-skill-evals.ps1 | 131 +++++++++++- scripts/validate-skill-templates.ps1 | 46 ++++ 8 files changed, 635 insertions(+), 47 deletions(-) diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index 42843ee..e9d2d78 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -184,6 +184,86 @@ function Get-CodexBaseSkillSessionConfigValues { ) } +function Get-CodexSanitizedShellPath { + param( + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [AllowNull()][string]$GitDirectory = $null, + [AllowNull()][string]$WindowsRoot = $null + ) + + $separator = if ($Platform -eq 'windows') { ';' } else { ':' } + $entries = [System.Collections.Generic.List[string]]::new() + if ($Platform -eq 'windows') { + $root = if ([string]::IsNullOrWhiteSpace($WindowsRoot)) { [Environment]::GetEnvironmentVariable('SystemRoot') } else { $WindowsRoot } + if ([string]::IsNullOrWhiteSpace($root)) { $root = 'C:\Windows' } + $entries.Add((Join-Path $root 'System32')) + } else { + foreach ($path in @('/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin')) { + $entries.Add($path) + } + } + if (-not [string]::IsNullOrWhiteSpace($GitDirectory)) { $entries.Add([string]$GitDirectory) } + + $deduplicated = [System.Collections.Generic.List[string]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($entry in @($entries)) { + $value = [string]$entry + if ([string]::IsNullOrWhiteSpace($value)) { continue } + $normalized = $value.Trim().TrimEnd('\', '/') + if ([string]::IsNullOrWhiteSpace($normalized)) { continue } + if ($seen.Add($normalized)) { $deduplicated.Add($normalized) } + } + return [string]::Join($separator, [string[]]@($deduplicated.ToArray())) +} + +function Get-CodexShellEnvironmentPolicySet { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [AllowNull()][object]$GitCommandInfo = $null, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName) + ) + + $gitDirectory = $null + if ($null -ne $GitCommandInfo) { + $gitSource = [string](Get-JsonProperty -Object $GitCommandInfo -Name 'Source' -Default '') + if (-not [string]::IsNullOrWhiteSpace($gitSource)) { $gitDirectory = Split-Path -Parent $gitSource } + } + $windowsRoot = [Environment]::GetEnvironmentVariable('SystemRoot') + $values = [ordered]@{ + PATH = Get-CodexSanitizedShellPath -Platform $Platform -GitDirectory $gitDirectory -WindowsRoot $windowsRoot + } + if ($Platform -eq 'windows') { + if ([string]::IsNullOrWhiteSpace($windowsRoot)) { $windowsRoot = 'C:\Windows' } + $values.SystemRoot = $windowsRoot + $values.ComSpec = Join-Path (Join-Path $windowsRoot 'System32') 'cmd.exe' + $values.PATHEXT = '.COM;.EXE;.BAT;.CMD' + } + return $values +} + +function Test-CodexRunUsesGitWorkspace { + param([Parameter(Mandatory = $true)][object]$Run) + + return [bool](Get-JsonProperty -Object $Run -Name 'GitWorkspace' -Default $false) +} + +function Add-CodexShellEnvironmentPolicyConfigArguments { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Arguments, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$ShellEnvironmentSet, + [string]$SwitchName = '-c' + ) + + $Arguments.Add($SwitchName) + $Arguments.Add('shell_environment_policy.inherit=none') + foreach ($key in @($ShellEnvironmentSet.Keys)) { + $name = [string]$key + if ($name -notmatch '^[A-Za-z_][A-Za-z0-9_]*$') { throw "Unsupported Codex shell environment key '$name'." } + $Arguments.Add($SwitchName) + $Arguments.Add(('shell_environment_policy.set.{0}={1}' -f $name, (ConvertTo-CodexConfigStringLiteral -Value ([string]$ShellEnvironmentSet[$key])))) + } +} + function New-CodexSkillSuppressionSelector { param( [Parameter(Mandatory = $true)][object]$SkillEntry, @@ -268,12 +348,13 @@ function Add-CodexSessionConfigArguments { [AllowEmptyCollection()][object[]]$AmbientSkillEntries = @(), [string]$SwitchName = '-c', [switch]$IncludeShellEnvironmentPolicy, + [System.Collections.IDictionary]$ShellEnvironmentSet = $null, [bool]$IncludeNativeSkillSuppression = $true ) if ($IncludeShellEnvironmentPolicy) { - $Arguments.Add($SwitchName) - $Arguments.Add('shell_environment_policy.inherit=none') + if ($null -eq $ShellEnvironmentSet) { $ShellEnvironmentSet = [ordered]@{} } + Add-CodexShellEnvironmentPolicyConfigArguments -Arguments $Arguments -ShellEnvironmentSet $ShellEnvironmentSet -SwitchName $SwitchName } if (-not $IncludeNativeSkillSuppression) { return } foreach ($value in @(Get-CodexSkillSessionConfigValues -CandidateSkillName $CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries)) { @@ -587,6 +668,7 @@ function New-CodexNativeSkillProbeContext { ) $environmentKeys = @($Environment.Keys | ForEach-Object { [string]$_ } | Sort-Object) + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } return [ordered]@{ purpose = $Purpose codex_executable = [string]$CommandInfo.Source @@ -602,6 +684,7 @@ function New-CodexNativeSkillProbeContext { tmp = if ($Environment.Contains('TMP')) { [string]$Environment['TMP'] } else { $null } filtered_environment_keys = @($environmentKeys) shell_environment_policy = 'inherit=none' + shell_environment_policy_set_path = (Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo)['PATH'] app_server_arguments = @($Arguments) per_skill_suppression_selector_count = @($AmbientSkillEntries).Count native_skill_suppression_enabled = @($Arguments | Where-Object { [string]$_ -like 'skills.*' }).Count -gt 0 @@ -623,7 +706,9 @@ function Invoke-CodexAppServerSkillsListProbe { $deadline = $start.AddSeconds([Math]::Max(1, $TimeoutSeconds)) $arguments = [System.Collections.Generic.List[string]]::new() foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--strict-config', '--stdio')) { $arguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -IncludeShellEnvironmentPolicy -IncludeNativeSkillSuppression:$IncludeNativeSkillSuppression + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo + Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet -IncludeNativeSkillSuppression:$IncludeNativeSkillSuppression $psi = [System.Diagnostics.ProcessStartInfo]::new() $psi.FileName = $CommandInfo.FileName @@ -1113,7 +1198,9 @@ function Invoke-CodexAppServer { $suppressionSelectors = @(New-CodexNativeSkillSuppressionSelectors -AmbientSkillEntries $ambientSkillEntries) $appServerArguments = [System.Collections.Generic.List[string]]::new() foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--strict-config', '--stdio')) { $appServerArguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $appServerArguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $ambientSkillEntries -IncludeShellEnvironmentPolicy + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo + Add-CodexSessionConfigArguments -Arguments $appServerArguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $ambientSkillEntries -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet foreach ($argument in @($appServerArguments)) { [void]$psi.ArgumentList.Add([string]$argument) } # Shared progress context for the app-server protocol exchange. When the @@ -2087,11 +2174,13 @@ function New-CodexCliArguments { $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $outputArgument = Get-SandboxVisiblePath -HostPath $LastResponsePath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo -Platform $VisiblePlatform $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--strict-config', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'danger-full-access', '--config', 'shell_environment_policy.inherit=none')) { + foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--strict-config', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'danger-full-access')) { $arguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -SwitchName '--config' + Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -SwitchName '--config' -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet foreach ($argument in @('--output-last-message', $outputArgument)) { $arguments.Add([string]$argument) } @@ -2207,6 +2296,40 @@ function Get-CodexCapabilityMap { return $capabilities } +function Invoke-CodexGitWorkspaceProbe { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$GitCommandInfo, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [int]$TimeoutSeconds = 30 + ) + + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $GitCommandInfo -Platform $Platform + $sanitizedPath = [string]$shellEnvironmentSet['PATH'] + $environment = [ordered]@{} + foreach ($key in @($shellEnvironmentSet.Keys)) { $environment[$key] = [string]$shellEnvironmentSet[$key] } + $gitFileName = [IO.Path]::GetFileName([string]$GitCommandInfo.Source) + if ([string]::IsNullOrWhiteSpace($gitFileName)) { $gitFileName = 'git' } + try { + $process = Invoke-RunnerProcess -FileName $gitFileName -ArgumentList @('--version') -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $TimeoutSeconds + } catch { + return [pscustomobject]@{ + Available = $false + Detail = "git --version could not start with the sanitized Codex shell PATH: $($_.Exception.Message)" + SanitizedPath = $sanitizedPath + Process = $null + } + } + $stdout = [string]$process.Stdout + $available = -not $process.TimedOut -and $process.ExitCode -eq 0 -and $stdout -match '(?im)\bgit version\b' + return [pscustomobject]@{ + Available = $available + Detail = if ($available) { "git --version succeeded with the sanitized Codex shell PATH: $sanitizedPath" } else { "git --version failed with the sanitized Codex shell PATH (exit=$($process.ExitCode), timed_out=$($process.TimedOut))." } + SanitizedPath = $sanitizedPath + Process = $process + } +} + function Get-CodexPreflight { param([Parameter(Mandatory = $true)][object]$Inputs) @@ -2227,6 +2350,8 @@ function Get-CodexPreflight { $nativeWorkerObservation = $null $nativeSkillConfigObservation = $null $auth = Get-CodexAuthSource + $gitCommandInfo = $null + $gitWorkspaceProbe = $null if ($profile.Runner -ne 'codex') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") @@ -2244,6 +2369,21 @@ function Get-CodexPreflight { if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by codex.") } + if (Test-CodexRunUsesGitWorkspace -Run $run) { + $gitCommandInfo = Resolve-ExternalCommand -Name 'git' + if ($null -eq $gitCommandInfo) { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status failed -Detail 'gitWorkspace=true requires git, but Resolve-ExternalCommand could not locate git.')) + $reasons.Add('gitWorkspace=true requires git to be resolvable before Codex execution.') + } else { + $gitWorkspaceProbe = Invoke-CodexGitWorkspaceProbe -Inputs $Inputs -GitCommandInfo $gitCommandInfo -Platform $platform -TimeoutSeconds 30 + if ($gitWorkspaceProbe.Available) { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status passed -Detail $gitWorkspaceProbe.Detail)) + } else { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status failed -Detail $gitWorkspaceProbe.Detail)) + $reasons.Add('gitWorkspace=true requires git --version to succeed with the exact sanitized Codex child shell PATH.') + } + } + } if ($null -eq $commandInfo) { $reasons.Add('The Codex CLI executable is not available on PATH.') @@ -2365,11 +2505,11 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The selected transport starts an ephemeral thread and never supplies a resume, continue, or existing session identifier.')) if ($auth.Kind -eq 'subscription_file') { - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none. Native skill isolation is proven separately through skills/list discovery, ambient path suppression, behavioral skills/list verification, and runtime access evidence.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none and shell_environment_policy.set.PATH supplies only the sanitized runner tool path. Native skill isolation is proven separately through skills/list discovery, ambient path suppression, behavioral skills/list verification, and runtime access evidence.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "thread/start and turn/start set cwd to $($run.WorkingDirectoryPath); HOME and USERPROFILE remain staged under $($run.HomeDirectoryPath).")) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only auth.json is copied into a temporary auth-only CODEX_HOME and it is removed in finally; config.toml, skills, agents, sessions, memories, plugins, MCP configuration, and AGENTS.md are not copied. This does not claim hard filesystem confinement where none is available.')) } else { - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed. Child shell inheritance is disabled with shell_environment_policy.inherit=none and shell_environment_policy.set.PATH supplies only the sanitized runner tool path.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are not copied into the worker HOME.')) } @@ -2382,10 +2522,10 @@ function Get-CodexPreflight { $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction) { $mechanisms.Add('same-thread repeated turn/start for scripted interaction') } else { $mechanisms.Add('no session continuation') } } else { - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } $document = New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) @@ -2888,11 +3028,11 @@ function Invoke-CodexExecute { } $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } $continuationMechanism = if ($null -ne $Inputs.Run.Interaction) { 'same-thread repeated turn/start for scripted interaction' } else { 'no session continuation' } $mechanisms.Add($continuationMechanism) } else { - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index 0c8cd29..529c305 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -59,25 +59,45 @@ function Assert-EvalGradingEntryShape { } } -function Assert-EvalPassEvidence { - param([object]$Entry, [object]$Canonical, [object]$Record) - if (-not $Entry.passed) { return } - [void]($Entry.evidence -match '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') - $source = $Matches[1].Trim(); $quote = $Matches[2].Trim(); $reason = $Matches[3].Trim() - if ($source -eq 'output') { $content = [string]$Canonical.output } - else { - # Only native captured artifacts are admissible, never grading keys or - # grader-created files. Freeze validation already pins their bytes. - $raw = Read-RunnerJson -Path $Record.ExecutionResultPath - $artifacts = @($raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -ceq $source }) - if ($artifacts.Count -ne 1) { throw "PASS evidence source '$source' is not a captured run artifact." } - $path = Resolve-ContainedPath -BasePath (Split-Path -Parent $Record.RunManifestPath) -RelativePath $source -FieldName 'PASS evidence source' -Kind File - $content = [IO.File]::ReadAllText($path) - } - if ([string]::IsNullOrWhiteSpace($quote) -or -not $content.Contains($quote, [StringComparison]::Ordinal)) { throw 'PASS evidence quote is absent from its frozen source.' } - if ($reason -eq $quote -or $reason -eq $Entry.assertion -or $reason -match '^(?i:eval(?:uation)? completed(?: with output)?|passed|verified|as expected|done|looks good)[.!]?$') { - throw 'PASS evidence must explain how the cited observation establishes this assertion.' - } +# Reject generic, templated, or tautological PASS reasons. A reason must explain HOW the cited observation establishes +# the specific assertion, not restate that the assertion passed or that the output was "evaluated". Iteration 9 passed +# 76 assertions with reasons equivalent to "Assertion evaluated against output"; that class must fail closed. +function Test-GenericGradingReason { + param([string]$Reason, [string]$Assertion) + $normalized = ([regex]::Replace([string]$Reason, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim() + if ([string]::IsNullOrWhiteSpace($normalized)) { return $true } + if ($normalized -eq (([regex]::Replace([string]$Assertion, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim())) { return $true } + $genericPatterns = @( + '^(?i)eval(?:uation)? completed(?: with output)?$', + '^(?i)(?:the )?(?:assertion|requirement|expectation|condition|criteri(?:on|a))(?: is| was| has been)?(?: fully| clearly)? (?:met|satisfied|passed|verified|confirmed|evaluated|true|correct|valid|present|fulfilled|checked|held|holds|passes)$', + '(?i)evaluated against (?:the )?(?:output|transcript|response|result|evidence|assertion)', + '(?i)(?:output|response|transcript|result) (?:was |is )?(?:evaluated|matches|meets|satisfies|supports|confirms|contains) (?:the )?(?:assertion|requirement|expectation)', + '^(?i)(?:passed|verified|confirmed|as expected|done|looks good|correct|ok|success(?:ful)?|valid|complete)$', + '^(?i)(?:this )?(?:matches|meets|satisfies|establishes|proves|confirms)(?: the)?(?: assertion| requirement| expectation)?$' + ) + foreach ($pattern in $genericPatterns) { if ($normalized -match $pattern) { return $true } } + return $false +} + +function Assert-EvalPassEvidence { + param([object]$Entry, [object]$Canonical, [object]$Record) + if (-not $Entry.passed) { return } + [void]($Entry.evidence -match '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') + $source = $Matches[1].Trim(); $quote = $Matches[2].Trim(); $reason = $Matches[3].Trim() + if ($source -eq 'output') { $content = [string]$Canonical.output } + else { + # Only native captured artifacts are admissible, never grading keys or + # grader-created files. Freeze validation already pins their bytes. + $raw = Read-RunnerJson -Path $Record.ExecutionResultPath + $artifacts = @($raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -ceq $source }) + if ($artifacts.Count -ne 1) { throw "PASS evidence source '$source' is not a captured run artifact." } + $path = Resolve-ContainedPath -BasePath (Split-Path -Parent $Record.RunManifestPath) -RelativePath $source -FieldName 'PASS evidence source' -Kind File + $content = [IO.File]::ReadAllText($path) + } + if ([string]::IsNullOrWhiteSpace($quote) -or -not $content.Contains($quote, [StringComparison]::Ordinal)) { throw 'PASS evidence quote is absent from its frozen source.' } + if ($reason -eq $quote -or (Test-GenericGradingReason -Reason $reason -Assertion ([string]$Entry.assertion))) { + throw 'PASS evidence must explain how the cited observation establishes this assertion, not restate that it passed or was evaluated.' + } } function Assert-EvalGradingContract { diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 index a847514..847063f 100644 --- a/scripts/eval-runners/github-copilot/isolation.ps1 +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -11,6 +11,51 @@ function Find-CopilotBoundaryContradictions { } } +# Conservative detector for a native Copilot skill activation of the evaluated candidate. The native skill tool is +# excluded and skill discovery roots are isolated, so a candidate skill tool call or a native skill-resolution result +# naming the candidate is an isolation violation in BOTH arms. It fires only on a native 'skill' tool identity or a +# native skill-resolution result (a skillSource marker) that also names the candidate, so an ordinary file read of the +# staged with_skill candidate copy or a mere text mention never trips it. +function Find-CopilotNativeSkillActivation { + param([object]$Data, [string]$CandidateSkillName, [string]$EventType = '') + if ($null -eq $Data -or [string]::IsNullOrWhiteSpace($CandidateSkillName)) { return } + $json = ($Data | ConvertTo-Json -Depth 100 -Compress) + $toolIdentity = $false + foreach ($field in @('toolName', 'tool', 'name', 'tool_name')) { + $value = [string](Get-JsonProperty -Object $Data -Name $field -Default '') + if ($value -match '^(?i)skill$') { $toolIdentity = $true } + } + $skillResolution = $json -match '(?i)"skill[_]?[Ss]ource"' + if (-not ($toolIdentity -or $skillResolution)) { return } + if ($json -match ('(?i)(^|[^A-Za-z0-9._-])' + [regex]::Escape($CandidateSkillName) + '([^A-Za-z0-9._-]|$)')) { + "Native Copilot skill activation referenced the candidate '$CandidateSkillName'; the candidate must never be available through the native skill mechanism." + } +} + +# Independent, bridge-side re-verification of the candidate-instruction identity proof. with_skill must embed the exact +# frozen candidate instruction bytes (prompt prefix before the working-environment marker) whose hash equals the frozen +# candidateInstructionHash; without_skill must carry neither a hash nor an embedded candidate instruction section. This +# never trusts a runner boolean: it recomputes from the staged prompt bytes and run.json. +function Assert-CopilotCandidateInstructionBoundary { + param([object]$RunData) + $boundary = "`n`n# Working environment" + $promptText = ([Text.Encoding]::UTF8.GetString([byte[]]$RunData.PromptBytes)) -replace "`r`n", "`n" -replace "`r", "`n" + $markerIndex = $promptText.IndexOf($boundary, [StringComparison]::Ordinal) + if ($RunData.Mode -eq 'with_skill') { + $expected = [string]$RunData.CandidateInstructionHash + if ([string]::IsNullOrWhiteSpace($expected)) { return } + if ($markerIndex -lt 0) { throw 'with_skill prompt has no working-environment boundary; the injected candidate instructions cannot be isolated for hashing.' } + $instruction = $promptText.Substring(0, $markerIndex) + $actual = ([Convert]::ToHexString([Security.Cryptography.SHA256]::HashData([Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + if ($actual -ne $expected) { throw 'with_skill injected candidate instructions do not hash to the frozen candidateInstructionHash; candidate identity is unproven.' } + if ($instruction -notmatch '(?im)^##\s*Skill:') { throw 'with_skill prompt prefix does not contain the candidate instruction section.' } + } else { + if (-not [string]::IsNullOrWhiteSpace([string]$RunData.CandidateInstructionHash)) { throw 'without_skill must not declare candidateInstructionHash.' } + $prefix = if ($markerIndex -lt 0) { $promptText } else { $promptText.Substring(0, $markerIndex) } + if ($prefix -match '(?im)^##\s*Skill:') { throw 'without_skill prompt embeds a candidate instruction section; the baseline must receive no candidate instructions.' } + } +} + function Assert-CopilotCapturedBoundary { param([object]$Raw, [object]$RunData) if ($Raw.runner.name -ne 'github-copilot' -or $Raw.status -ne 'completed') { return } @@ -20,15 +65,20 @@ function Assert-CopilotCapturedBoundary { $source = [string](Get-JsonProperty -Object $paths -Name source_repository_root -Default '') $physical = [string](Get-JsonProperty -Object $paths -Name physical_run_root -Default '') if ([string]::IsNullOrWhiteSpace($physical) -or (Test-PathInside -BasePath $package -CandidatePath $physical) -or ($source -and (Test-PathInside -BasePath $source -CandidatePath $physical))) { throw 'Invalid Copilot physical projection boundary.' } + Assert-CopilotCandidateInstructionBoundary -RunData $RunData $proof = [pscustomobject]@{ PackageRoot = $package; SourceRepositoryRoot = $source } + $candidateSkillName = [string]$RunData.CandidateSkillName $transcript = @($Raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -eq 'evidence/copilot-events.jsonl' }) if ($transcript.Count -ne 1) { throw 'Copilot native transcript is missing.' } $path = Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $transcript[0].path -FieldName 'Copilot transcript' -Kind File $parsed = ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($path)) if (@($parsed.Errors).Count) { throw 'Copilot transcript contains unparseable events; boundary inspection is incomplete.' } foreach ($event in $parsed.Events) { - if ([string](Get-JsonProperty -Object $event -Name type -Default '') -match '^(tool\.|command\.)') { - if (@(Find-CopilotBoundaryContradictions -Data (Get-JsonProperty -Object $event -Name data -Default $null) -Projection $proof).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + $eventType = [string](Get-JsonProperty -Object $event -Name type -Default '') + $data = Get-JsonProperty -Object $event -Name data -Default $null + if ($eventType -match '^(tool\.|command\.)') { + if (@(Find-CopilotBoundaryContradictions -Data $data -Projection $proof).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + if (@(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $candidateSkillName -EventType $eventType).Count) { throw 'Copilot transcript shows native candidate skill activation; grading is forbidden.' } } } } diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index 0a66709..adf0685 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -33,10 +33,22 @@ Set-StrictMode -Version Latest # GitHub CLI fallback. The values are forwarded only to the Copilot process; # --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +# The native skill tool ('skill') is the mechanism iteration 9 used to load an ambient candidate in the baseline. It is +# removed from the model's tool set in both arms; the candidate is delivered only through the frozen prompt instructions. +$copilotExcludedTools = @('skill') +# The single installed-help-proven boundary between the injected candidate instructions and the rest of the prompt +# wrapper. Preparation guarantees the candidate instruction section is the exact prompt prefix before this marker, so the +# runner and bridge can hash only the canonical candidate instruction bytes without re-deriving the skill body. +$copilotCandidateInstructionBoundary = "`n`n# Working environment" $script:copilotHomeCleanupSafe = $true $script:copilotLogicalInputs = $null $script:copilotProjection = $null $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() +# The evaluated candidate skill name (both arms) and any observed native-skill activation of it. A non-empty violation +# list at the end of execute is a fail-closed isolation breach: the candidate must never reach the worker natively. +$script:copilotCandidateSkillName = $null +$script:copilotNativeSkillViolations = [System.Collections.Generic.List[string]]::new() +$script:copilotNativeSkillCatalog = $null function Invoke-CopilotProcess { param( @@ -139,6 +151,9 @@ $descriptor = [ordered]@{ filesystem_confinement = 'conditional' ambient_candidate_skill_exclusion = 'supported' candidate_skill_exposure = 'supported' + native_skill_tool_exclusion = 'supported' + native_skill_catalog_probe = 'supported' + candidate_instruction_hash = 'supported' prompt_fidelity = 'supported' model_configuration_lock = 'supported' response_capture = 'supported' @@ -150,7 +165,7 @@ $descriptor = [ordered]@{ file_evidence = 'conditional' cost_telemetry = 'unsupported' credential_child_filtering = 'supported' - native_skill_activation_evidence = 'unsupported' + native_skill_activation_evidence = 'supported' # Behavioral evaluation transport is runner-owned: the runner starts one # fresh Copilot CLI session per eval execution and captures the session's # own terminal evidence. Copilot's native task/general-purpose subagent @@ -551,6 +566,11 @@ function New-CopilotCliArguments { '--allow-all', '--no-ask-user', '--disable-builtin-mcps', + # The eval never needs native skill lookup: with_skill already embeds the exact frozen candidate + # instructions in the prompt, and without_skill intentionally has none. Removing the native skill tool from + # the model's tool set closes the native-skill activation path for BOTH arms (installed-version-supported + # --excluded-tools; the candidate 'skill' tool is what iteration 9 used to load an ambient candidate). + ('--excluded-tools=' + ($copilotExcludedTools -join ',')), '--no-color', '--log-level', 'none', '--no-auto-update', @@ -578,6 +598,10 @@ function Get-CopilotCapabilityMap { } $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + $capabilities['native_skill_tool_exclusion'] = 'supported' + $capabilities['native_skill_activation_evidence'] = 'supported' + $capabilities['candidate_instruction_hash'] = if ($Inputs.Run.Mode -eq 'with_skill') { 'supported' } else { 'excluded' } + $capabilities['native_skill_catalog_probe'] = if ($null -ne $script:copilotNativeSkillCatalog -and [bool]$script:copilotNativeSkillCatalog.available) { 'supported' } else { 'conditional' } $capabilities['scripted_multi_turn_same_session'] = if ($null -eq $Inputs.Run.Interaction) { 'conditional' } elseif ($null -ne $ContinuationCapability -and [bool]$ContinuationCapability.Available) { @@ -658,7 +682,7 @@ function Get-CopilotPreflight { } } else { $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--output-format', '--model', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + foreach ($flag in @('--output-format', '--model', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--excluded-tools', '--secret-env-vars')) { if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Copilot CLI does not advertise required flag '$flag'.") } @@ -668,13 +692,18 @@ function Get-CopilotPreflight { foreach ($forbidden in @('--resume', '-r', '--continue', '--session-id', '--connect', '--yolo')) { if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Copilot invocation must not use session-continuation or shortcut option '$forbidden'.") } } - foreach ($required in @('--output-format', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + foreach ($required in @('--output-format', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--excluded-tools', '--secret-env-vars')) { $present = @($constructed) -contains $required - if ($required -eq '--secret-env-vars') { - $present = $present -or (@($constructed | Where-Object { $_ -like '--secret-env-vars=*' }).Count -gt 0) + if ($required -in @('--secret-env-vars', '--excluded-tools')) { + $present = $present -or (@($constructed | Where-Object { $_ -like "$required=*" }).Count -gt 0) } if (-not $present) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } } + $excludedToolsArgument = @($constructed | Where-Object { $_ -like '--excluded-tools=*' }) + if ($excludedToolsArgument.Count -eq 1) { + $excludedList = ($excludedToolsArgument[0] -replace '^--excluded-tools=', '') -split ',' + if ('skill' -notin @($excludedList)) { $reasons.Add('The constructed Copilot invocation must exclude the native skill tool for both arms.') } + } $promptOptionCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count if ($promptOptionCount -ne 0) { $reasons.Add('The constructed Copilot invocation must not place the prompt in argv; prompt delivery uses stdin.') } if ($reasons.Count -eq 0) { @@ -699,6 +728,23 @@ function Get-CopilotPreflight { } } + $nativeSkillCatalog = [ordered]@{ available = $false; candidate = [string]$run.CandidateSkillName; candidate_present = $false; candidate_enabled = $false; proven_absent = $false; entry_count = 0; entries = @(); error = 'not probed' } + if ($null -ne $commandInfo) { + try { + $nativeSkillCatalog = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $commandInfo -Environment (New-CopilotEnvironment -Inputs $Inputs -WithoutAuthentication) -WorkingDirectory $run.WorkingDirectoryPath -CandidateSkillName ([string]$run.CandidateSkillName) + } catch { $nativeSkillCatalog.error = $_.Exception.Message } + $script:copilotNativeSkillCatalog = $nativeSkillCatalog + if ($nativeSkillCatalog.available -and $nativeSkillCatalog.candidate_enabled) { + $reasons.Add("The candidate skill '$($run.CandidateSkillName)' is resolvable and enabled in the isolated native Copilot skill catalog; native discovery would contaminate the eval.") + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status failed -Detail "copilot skill list --json resolved the candidate '$($run.CandidateSkillName)' as an enabled native skill inside the isolated environment.")) + } elseif ($nativeSkillCatalog.available) { + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status passed -Detail "copilot skill list --json proves the candidate '$($run.CandidateSkillName)' is not an enabled native skill in the isolated environment ($($nativeSkillCatalog.entry_count) skills visible).")) + } else { + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status unavailable -Detail "copilot skill list --json was not usable ($($nativeSkillCatalog.error)); the native skill tool exclusion and isolated discovery roots still enforce candidate exclusion.")) + $warnings.Add('The native-skill-catalog probe was unavailable; candidate native-skill exclusion still holds through --excluded-tools=skill, isolated discovery roots, and runtime/bridge detection.') + } + } + $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) @@ -790,6 +836,17 @@ function Get-CopilotPreflight { explicit_gh_config_dir_provided = [bool]$authState.ExplicitGhConfigProvided token_value_observed = $false } + native_skill_isolation = [ordered]@{ + excluded_tools = @($copilotExcludedTools) + native_skill_tool_disabled = $true + native_skill_dynamic_retrieval_disabled = $true + native_skill_catalog_probe_available = [bool]$nativeSkillCatalog.available + native_skill_catalog_proven_absent = [bool]$nativeSkillCatalog.proven_absent + candidate = [string]$run.CandidateSkillName + candidate_present = [bool]$nativeSkillCatalog.candidate_present + candidate_enabled = [bool]$nativeSkillCatalog.candidate_enabled + catalog = $nativeSkillCatalog + } } return $document } @@ -912,6 +969,11 @@ function Read-CopilotEvents { $violations = @(Find-CopilotBoundaryContradictions -Data $data -Projection $script:copilotProjection) foreach ($violation in $violations) { $script:copilotBoundaryViolations.Add($violation) } } + if ($eventType -match '^(tool\.|command\.)' -and -not [string]::IsNullOrWhiteSpace($script:copilotCandidateSkillName)) { + foreach ($activation in @(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $script:copilotCandidateSkillName -EventType $eventType)) { + $script:copilotNativeSkillViolations.Add($activation) + } + } foreach ($eventSessionId in @(Get-CopilotEventSessionIds -Event $event)) { if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } } @@ -1042,13 +1104,106 @@ function Invoke-CopilotTurnProcess { return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds } +# Model-free native-skill-catalog probe. The installed Copilot CLI exposes `copilot skill list --json`, a machine-readable +# enumeration of every skill grouped by source with an `enabled` flag. Run inside the isolated eval environment, it proves +# the candidate is not resolvable from any native discovery source (inherited/personal ~/.agents|~/.copilot skills, +# project .github/skills, plugins, custom). It never consumes model tokens. A candidate that is present AND enabled is an +# isolation breach; probe unavailability is a warning only, because --excluded-tools=skill plus isolated discovery roots +# plus runtime/bridge detection still enforce the invariant. +function Invoke-CopilotNativeSkillCatalogProbe { + param( + [object]$CommandInfo, + [System.Collections.IDictionary]$Environment, + [string]$WorkingDirectory, + [Parameter(Mandatory = $true)][string]$CandidateSkillName + ) + + $probe = [ordered]@{ + available = $false + candidate = $CandidateSkillName + candidate_present = $false + candidate_enabled = $false + proven_absent = $false + entry_count = 0 + entries = @() + error = $null + } + if ($null -eq $CommandInfo) { $probe.error = 'copilot executable unavailable'; return $probe } + try { + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @('skill', 'list', '--json')) -WorkingDirectory $WorkingDirectory -Environment $Environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { $probe.error = "copilot skill list exited $($process.ExitCode)"; return $probe } + $parsed = $null + try { $parsed = [string]$process.Stdout | ConvertFrom-Json -ErrorAction Stop } catch { $probe.error = 'copilot skill list output was not JSON'; return $probe } + $entries = @($parsed) + $summary = [System.Collections.Generic.List[object]]::new() + $present = $false + $enabled = $false + foreach ($entry in $entries) { + $name = [string](Get-JsonProperty -Object $entry -Name 'name' -Default '') + $source = [string](Get-JsonProperty -Object $entry -Name 'source' -Default '') + $entryEnabled = [bool](Get-JsonProperty -Object $entry -Name 'enabled' -Default $true) + $summary.Add([ordered]@{ name = $name; source = $source; enabled = $entryEnabled }) + if ($name -eq $CandidateSkillName) { $present = $true; if ($entryEnabled) { $enabled = $true } } + } + $probe.available = $true + $probe.entry_count = $entries.Count + $probe.entries = @($summary) + $probe.candidate_present = $present + $probe.candidate_enabled = $enabled + # The candidate cannot activate natively when it is either absent or present-but-disabled. + $probe.proven_absent = -not $enabled + return $probe + } catch { + $probe.error = $_.Exception.Message + return $probe + } +} + +# Positive candidate-identity proof. The canonical candidate instruction bytes are the prompt prefix before the +# working-environment marker; preparation guarantees that prefix is exactly the frozen candidate instruction section. The +# hash of those bytes must equal the frozen candidateInstructionHash (with_skill); the baseline must carry neither a hash +# nor an embedded candidate section. Only the candidate instruction content is hashed, so unrelated wrapper edits do not +# invalidate the proof. +function Get-CopilotCandidateInstructionEvidence { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $promptText = ([System.Text.Encoding]::UTF8.GetString([byte[]]$Inputs.Run.PromptBytes)) -replace "`r`n", "`n" -replace "`r", "`n" + $markerIndex = $promptText.IndexOf($copilotCandidateInstructionBoundary, [System.StringComparison]::Ordinal) + $expected = [string]$Inputs.Run.CandidateInstructionHash + $result = [ordered]@{ expected = $null; injected = $null; verified = $false; violations = @() } + $violations = [System.Collections.Generic.List[string]]::new() + if ($Inputs.Run.Mode -eq 'with_skill') { + $result.expected = $expected + if ([string]::IsNullOrWhiteSpace($expected)) { + # Nothing was frozen to verify (a synthetic or non-prepared run). Real prepared packages always declare the + # hash, and the bridge independently enforces it there. + $result.verified = $false + } elseif ($markerIndex -lt 0) { + $violations.Add('with_skill prompt has no working-environment boundary; candidate instruction bytes cannot be isolated for hashing.') + } else { + $instruction = $promptText.Substring(0, $markerIndex) + $injected = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + $result.injected = $injected + $result.verified = ($injected -eq $expected) + if (-not $result.verified) { $violations.Add('with_skill injected candidate instructions do not hash to the frozen candidateInstructionHash; candidate identity is unproven.') } + if ($instruction -notmatch '(?im)^##\s*Skill:') { $violations.Add('with_skill prompt prefix does not contain the candidate instruction section.') } + } + } else { + $result.verified = $true + $prefix = if ($markerIndex -lt 0) { $promptText } else { $promptText.Substring(0, $markerIndex) } + if ($prefix -match '(?im)^##\s*Skill:') { $violations.Add('without_skill prompt embeds a candidate instruction section; the baseline must receive no candidate instructions.') } + if (-not [string]::IsNullOrWhiteSpace($expected)) { $violations.Add('without_skill declares a candidateInstructionHash; the baseline must have none.') } + } + $result.violations = @($violations) + return $result +} + function Invoke-CopilotScriptedExecute { param( [Parameter(Mandatory = $true)][object]$Inputs, [Parameter(Mandatory = $true)][object]$Preflight, [Parameter(Mandatory = $true)][object]$ExecutionDescriptor ) - $started = [DateTime]::UtcNow $commandInfo = Resolve-ExternalCommand -Name 'copilot' $environment = New-CopilotEnvironment -Inputs $Inputs @@ -1359,6 +1514,9 @@ function Invoke-CopilotExecute { $script:copilotLogicalInputs = $Inputs $script:copilotProjection = $plan $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() + $script:copilotCandidateSkillName = [string]$Inputs.Run.CandidateSkillName + $script:copilotNativeSkillViolations = [System.Collections.Generic.List[string]]::new() + $script:copilotNativeSkillCatalog = $null [void](New-Item -ItemType Directory -Path $plan.Root -ErrorAction Stop) try { $physicalRun = $Inputs.Run.PSObject.Copy() @@ -1400,6 +1558,25 @@ function Invoke-CopilotExecute { hard_filesystem_confinement = [bool]$result.isolation.hard_filesystem_confinement } $result.evidence.boundary = [ordered]@{ proof = 'allowlisted_physical_projection'; contradictions = @($script:copilotBoundaryViolations.ToArray()); event_inspection = 'contradiction_detector_not_confinement' } + # Runtime native-skill activations observed during the model turn(s), captured before the candidate-instruction + # identity proof is folded in, so the resolution flag reflects only actual native activation. + $runtimeNativeSkillActivations = @($script:copilotNativeSkillViolations | Select-Object -Unique) + $candidateInstruction = Get-CopilotCandidateInstructionEvidence -Inputs $logicalInputs + $result.evidence.candidate_instruction_hash_expected = $candidateInstruction.expected + $result.evidence.candidate_instruction_hash_injected = $candidateInstruction.injected + $result.evidence.candidate_instruction_hash_verified = [bool]$candidateInstruction.verified + foreach ($violation in @($candidateInstruction.violations)) { $script:copilotNativeSkillViolations.Add($violation) } + $catalog = $script:copilotNativeSkillCatalog + $result.evidence.native_skill = [ordered]@{ + native_skill_tool_disabled = $true + native_skill_dynamic_retrieval_disabled = $true + excluded_tools = @($copilotExcludedTools) + native_skill_catalog_probe_available = [bool]($null -ne $catalog -and $catalog.available) + native_skill_catalog_proven_absent = [bool]($null -ne $catalog -and $catalog.proven_absent) + candidate_native_skill_resolution = ($runtimeNativeSkillActivations.Count -gt 0) + candidate_native_skill_activations = $runtimeNativeSkillActivations + catalog = $catalog + } $capturePath = Join-Path $logicalInputs.Run.RunRoot 'evidence/copilot-events.jsonl' if (Test-Path -LiteralPath $capturePath) { $usageWarnings = [System.Collections.Generic.List[string]]::new() @@ -1417,18 +1594,22 @@ function Invoke-CopilotExecute { if ($buckets.Count) { $result.telemetry.tokens = New-AvailableMetric -Value $buckets } } } - if ($script:copilotBoundaryViolations.Count -gt 0) { + $isolationViolations = @(@($script:copilotBoundaryViolations) + @($script:copilotNativeSkillViolations) | Where-Object { $_ } | Select-Object -Unique) + if ($isolationViolations.Count -gt 0) { $result.status = 'incompatible' $result.isolation.status = 'unverified' $result.isolation.level = 'unsupported' $result.isolation.hard_filesystem_confinement = $false - $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $script:copilotBoundaryViolations)) + $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $isolationViolations)) if ($result.evidence.Contains('delegation')) { $result.evidence.delegation.paired_arm_visible = $true $result.evidence.delegation.grading_material_visible = $true } } $result.evidence.boundary.contradictions = @($script:copilotBoundaryViolations | Select-Object -Unique) + if ($result.evidence.Contains('native_skill')) { + $result.evidence.native_skill.candidate_native_skill_resolution = ($result.evidence.native_skill.candidate_native_skill_resolution -or (@($script:copilotNativeSkillViolations).Count -gt 0)) + } # Do not copy runtime links back into the logical package. Assert-CopilotProjectionTree -Path $physicalRun.WorkingDirectoryPath Assert-CopilotProjectionTree -Path $logicalInputs.Run.WorkingDirectoryPath diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 279d38c..84eed8f 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -1092,6 +1092,17 @@ function Resolve-RunContract { if ($mode -eq 'with_skill' -and -not (Test-Sha256 -Value ([string]$run.skillHash))) { throw 'with_skill run.json skillHash must be a SHA-256 value.' } + # The candidate-instruction hash is the positive identity proof for the injected candidate. It covers only the exact + # canonical candidate instruction bytes the with_skill prompt embeds, not the whole prompt wrapper, so unrelated + # wrapper edits never invalidate the candidate identity. The baseline must not carry it: no candidate is injected. + $candidateInstructionHash = [string](Get-JsonProperty -Object $run -Name 'candidateInstructionHash' -Default '') + if ($mode -eq 'with_skill') { + if (-not [string]::IsNullOrWhiteSpace($candidateInstructionHash) -and -not (Test-Sha256 -Value $candidateInstructionHash)) { + throw 'with_skill run.json candidateInstructionHash, when declared, must be a SHA-256 value covering the exact frozen candidate instruction bytes injected into the prompt.' + } + } elseif (-not [string]::IsNullOrWhiteSpace($candidateInstructionHash)) { + throw 'without_skill run.json must not declare candidateInstructionHash; the baseline receives no candidate instructions.' + } return [pscustomobject]@{ RunPath = $resolvedRunPath @@ -1110,6 +1121,8 @@ function Resolve-RunContract { CandidateSkillName = $candidateSkillName FixtureHash = $fixtureHash SkillHash = if ($mode -eq 'with_skill') { [string]$run.skillHash } else { $null } + CandidateInstructionHash = if ($mode -eq 'with_skill') { $candidateInstructionHash } else { $null } + GitWorkspace = [bool](Get-JsonProperty -Object $run -Name 'gitWorkspace' -Default $false) InteractionPath = $interactionPath InteractionHash = $interactionHash Interaction = $interaction diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index cdc8436..281c7d2 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -792,7 +792,24 @@ foreach ($entry in @($manifest.evals)) { $benchmark.metadata.runs_per_configuration = 1 $benchmark.metadata.executor_model = if ($models.Count -eq 0) { 'model not recorded' } else { $models -join ', ' } -$benchmark.metadata.analyzer_model = 'external skill-creator evaluator' +# Name the actual, validated analyzer/grader from the persisted analyzer profile rather than a generic label. Executor +# identity and analyzer identity remain independently attributable in the report. +$analyzerProfilePath = Join-Path $iterationPath 'analyzer-profile.json' +if (Test-Path -LiteralPath $analyzerProfilePath -PathType Leaf) { + $analyzerProfile = Read-JsonFile -Path $analyzerProfilePath + $analyzerRunner = [string](Get-Property -Object $analyzerProfile -Name 'runner' -Default '') + $analyzerModelName = [string](Get-Property -Object $analyzerProfile -Name 'model' -Default '') + $analyzerReasoning = [string](Get-Property -Object $analyzerProfile -Name 'reasoning_effort' -Default '') + $analyzerIdentity = ((@($analyzerRunner, $analyzerModelName) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) -join ' / ') + if ([string]::IsNullOrWhiteSpace($analyzerIdentity)) { $analyzerIdentity = 'analyzer profile incomplete' } + if (-not [string]::IsNullOrWhiteSpace($analyzerReasoning)) { $analyzerIdentity += " (reasoning: $analyzerReasoning)" } + $benchmark.metadata.analyzer_model = $analyzerIdentity + $benchmark.metadata | Add-Member -NotePropertyName analyzer_runner -NotePropertyValue $analyzerRunner -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_reasoning_effort -NotePropertyValue $analyzerReasoning -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_contract_version -NotePropertyValue ([string](Get-Property -Object $analyzerProfile -Name 'contract_version' -Default '')) -Force +} else { + $benchmark.metadata.analyzer_model = 'analyzer profile not recorded' +} $benchmark.metadata.evals_run = @($workspaceEntries | ForEach-Object { $_.EvalId }) foreach ($run in @($benchmark.runs)) { $match = @($workspaceEntries | Where-Object { $_.EvalId -eq [int]$run.eval_id }) | Select-Object -First 1 diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 80fce77..75e98dd 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -156,6 +156,18 @@ param( [Parameter(ParameterSetName = 'Changed')] [string]$ReasoningEffort, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerRunner, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerModel, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerReasoningEffort, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$ConfigurationProfile = 'isolated-default', @@ -198,6 +210,8 @@ $resultSchema = 'codebeltnet/agentic/eval-result/2' $runSchema = 'codebeltnet/agentic/eval-run/1' $executionProfileSchema = 'codebeltnet/agentic/eval-execution-profile/1' $executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' +$analyzerProfileSchema = 'codebeltnet/agentic/eval-analyzer-profile/1' +$analyzerContractVersion = 'codebeltnet/agentic/eval-grading/1' $runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 @@ -777,6 +791,65 @@ function New-ExecutionProfile { } } +function Resolve-AnalyzerSelection { + param( + [Parameter(Mandatory = $true)][string]$RepoRoot, + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [string]$ExecutionReasoningEffort + ) + + # The analyzer/grader is a distinct, persisted, reproducible profile - never "whichever model happens to host the + # outer orchestrator". By repository policy it defaults to the validated executor selection so cross-provider + # preparation never requires a second runner's catalog, and it is overridable to a stable reference analyzer via + # -AnalyzerRunner/-AnalyzerModel so the same validated analyzer can grade Copilot, Codex, and OpenCode executions. + $hasRunner = -not [string]::IsNullOrWhiteSpace($AnalyzerRunner) + $hasModel = -not [string]::IsNullOrWhiteSpace($AnalyzerModel) + if ($hasModel -and -not $hasRunner) { throw 'Analyzer selection requires -AnalyzerRunner when -AnalyzerModel is supplied.' } + + if (-not $hasRunner -and -not $hasModel) { + return [pscustomobject]@{ + Runner = $ExecutionSelection.Runner + Model = $ExecutionSelection.Model + ReasoningEffort = $ExecutionReasoningEffort + Harness = $ExecutionSelection.Harness + Source = 'executor-matched' + } + } + + if (-not $hasModel) { throw 'Analyzer selection requires -AnalyzerModel when -AnalyzerRunner is supplied.' } + $resolvedRunner = switch ($AnalyzerRunner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } + $supportedRunners = @(Get-SupportedRunnerIds -RepoRoot $RepoRoot) + if ($supportedRunners -notcontains $resolvedRunner) { + throw "Analyzer runner '$resolvedRunner' is not a supported runner id ($($supportedRunners -join ', '))." + } + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $AnalyzerModel) + $reasoning = if (-not [string]::IsNullOrWhiteSpace($AnalyzerReasoningEffort)) { $AnalyzerReasoningEffort } elseif ($resolvedRunner -eq 'codex') { 'low' } else { $null } + return [pscustomobject]@{ + Runner = $resolvedRunner + Model = $AnalyzerModel + ReasoningEffort = $reasoning + Harness = Get-HarnessName -RunnerName $resolvedRunner + Source = 'explicit' + } +} + +function New-AnalyzerProfile { + param([Parameter(Mandatory = $true)][object]$AnalyzerSelection) + + return [ordered]@{ + schema = $analyzerProfileSchema + contract_version = $analyzerContractVersion + runner = $AnalyzerSelection.Runner + harness = $AnalyzerSelection.Harness + model = $AnalyzerSelection.Model + reasoning_effort = $AnalyzerSelection.ReasoningEffort + selection_source = $AnalyzerSelection.Source + } +} + function Resolve-EffectiveConcurrency { param( [Parameter(Mandatory = $true)][string]$RunnerName, @@ -1269,6 +1342,7 @@ function New-RunManifest { [string[]]$RepoFiles, [string]$FixtureHash, [string]$SkillHash, + [string]$CandidateInstructionHash = $null, [bool]$GitWorkspace, [string]$InteractionFile = '', [string]$InteractionHash = '' @@ -1296,6 +1370,7 @@ function New-RunManifest { inputFiles = @($RepoFiles) fixtureHash = $FixtureHash skillHash = if ($Configuration -eq 'with_skill') { $SkillHash } else { $null } + candidateInstructionHash = if ($Configuration -eq 'with_skill') { $CandidateInstructionHash } else { $null } contract = [ordered]@{ sandboxRoot = '.' workingDirectory = $runDirectoryNames.Working @@ -1490,7 +1565,13 @@ function Invoke-PrepareMode { sha256 = Get-TreeHash -Root $copiedRunnerTools file_count = @(Get-ChildItem -LiteralPath $copiedRunnerTools -Recurse -File -Force).Count } - ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value)) + $executionProfile = New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value) + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value $executionProfile + # The analyzer/grader profile is validated during preparation exactly like the execution profile and persisted + # separately so executor identity and analyzer identity stay independently attributable in evidence and reports. + $analyzerSelection = Resolve-AnalyzerSelection -RepoRoot $repoRoot -ExecutionSelection $executionSelection -ExecutionReasoningEffort ([string]$executionProfile.reasoning_effort) + $analyzerProfile = New-AnalyzerProfile -AnalyzerSelection $analyzerSelection + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'analyzer-profile.json') -Value $analyzerProfile $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1501,6 +1582,14 @@ function Invoke-PrepareMode { $generatedUtc = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ') $withSkillInstructions = New-SkillInstructionSection -SkillName $Skill -SkillBody $skillBody -Inventory $inventory + # The candidate-instruction identity proof hashes exactly these canonical bytes (LF-normalized to match the written + # prompt). The runner and bridge isolate them as the prompt prefix before the working-environment marker, so the + # candidate instructions must never themselves contain that marker, or the proof boundary would be ambiguous. + $candidateInstructionCanonical = $withSkillInstructions -replace "`r`n", "`n" -replace "`r", "`n" + if ($candidateInstructionCanonical.Contains("`n`n# Working environment")) { + throw 'Candidate instructions must not contain the working-environment boundary marker; the candidate-instruction identity proof would be ambiguous.' + } + $candidateInstructionHash = Get-Sha256Hex -Bytes ([System.Text.Encoding]::UTF8.GetBytes($candidateInstructionCanonical)) $manifestEvals = [System.Collections.Generic.List[object]]::new() foreach ($evalEntry in $selectedEvals) { @@ -1574,13 +1663,34 @@ function Invoke-PrepareMode { # runs would otherwise differ, while the tracked fixture content is the same. $fixtureHash = Get-TreeHash -Root (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Working) -ExcludeSegments @('.git') - $inputFilesSection = New-InputFilesSection -Fixtures @($fixtures) + # Git-workspace evals must never inline fixture source into the prompt: the Git scenario mutates the working tree + # after fixtures are materialized (for example a feature commit that removes an API), so an inlined base copy would + # disagree with the final staged tree the worker actually reads. Omit inline source for these evals so the worker + # inspects the repository; the working-environment section already tells it to treat the repo as the source of + # truth. For non-Git evals, materialized fixtures still match their inline copies, so inlining is safe. + $inputFilesSection = if ($workspaceOption.Git) { $null } else { New-InputFilesSection -Fixtures @($fixtures) } $assertions = Get-Assertions -EvalEntry $evalEntry $interactionDocument = New-InteractionDocument -EvalEntry $evalEntry $withSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withSkillInstructions -InputFilesSection $inputFilesSection $withoutSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withoutSkillPreamble -InputFilesSection $inputFilesSection + # Deterministic guard: a Git-workspace prompt must not carry inlined fixture content that could disagree with the + # final working tree. It holds by omission above (no inline section is emitted for Git evals); this guard also + # protects against future regressions and, if inline source is ever re-enabled for a Git eval, requires + # byte-equality with the materialized working-tree file rendered after the Git scenario ran. + if ($workspaceOption.Git -and -not [string]::IsNullOrWhiteSpace($inputFilesSection)) { + $finalWorkingTree = Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Working + foreach ($fixture in @($fixtures)) { + if (-not $fixture.Inlined) { continue } + $materializedPath = Join-Path $finalWorkingTree ($fixture.RepoRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $materialized = if (Test-Path -LiteralPath $materializedPath -PathType Leaf) { [System.IO.File]::ReadAllText($materializedPath, $utf8NoBom) } else { $null } + if ($null -eq $materialized -or ($materialized.TrimEnd() -ne ([string]$fixture.Content).TrimEnd())) { + throw "Git-workspace eval '$evalName' inlines fixture '$($fixture.RepoRelative)', but the inline content disagrees with the final staged working tree. Omit inline source for Git-workspace evals so the prompt cannot contradict the repository." + } + } + } + Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Prompt) -Content $withSkillPrompt Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Prompt) -Content $withoutSkillPrompt @@ -1595,8 +1705,8 @@ function Invoke-PrepareMode { if ((Get-FileSha256 -Path $interactionPath) -ne $interactionHash) { throw "Scripted interaction sidecar diverged between configurations for '$evalName'." } } } - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -CandidateInstructionHash $candidateInstructionHash -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -CandidateInstructionHash $null -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) $assumptions = [System.Collections.Generic.List[string]]::new() $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') @@ -1608,6 +1718,7 @@ function Invoke-PrepareMode { } if ($workspaceOption.Git) { $assumptions.Add('This eval stages a real .git in repo/ so repository-root detection and version-deriving tools behave as on a developer machine.') + $assumptions.Add('Input files are intentionally not inlined for this Git-workspace eval; the worker must inspect the staged repository (branches, commits, and diff) rather than rely on any prompt copy, which could disagree with the final working tree.') } $assumptions.Add('The expected output and assertions in this file are the grading key. They live outside every run directory and must never reach a worker.') if ($null -ne $interactionDocument) { @@ -1712,6 +1823,15 @@ function Invoke-PrepareMode { } runner_prompt = 'RUN-THIS.prompt.md' execution_profile = 'execution-profile.json' + analyzer_profile = 'analyzer-profile.json' + analyzer_selection = [ordered]@{ + runner = $analyzerSelection.Runner + harness = $analyzerSelection.Harness + model = $analyzerSelection.Model + reasoning_effort = $analyzerSelection.ReasoningEffort + selection_source = $analyzerSelection.Source + contract_version = $analyzerContractVersion + } runner_protocol = $runnerProtocolSchema runner_tools = $evalRunnerToolRelativePath runner_tools_integrity = $runnerToolsIntegrity @@ -1876,7 +1996,8 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') - [void]$builder.AppendLine('Before creating grading.json, the Grader MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`; that guidance is authoritative during Phase 2. Uncertain or unverified expectations FAIL. Every assertion requires specific evidence. For PASS, evidence must use three newline-separated fields: `Source: output` (or a manifest-recorded run artifact path), `Quote: `, and `Reason: `. Do not reuse generic completion statements or identical evidence across assertions. FAIL evidence must explain what is missing or contradicted. Deterministic validation checks provenance and shape; it does not replace the Grader judgment required by grader.md.') + [void]$builder.AppendLine('Before creating grading.json, the Grader MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`; that guidance is authoritative during Phase 2. Uncertain or unverified expectations FAIL. Every assertion requires specific evidence. For PASS, evidence must use three newline-separated fields: `Source: output` (or a manifest-recorded run artifact path), `Quote: `, and `Reason: `. Do not reuse generic completion statements or identical evidence across assertions; reasons that merely restate that the assertion passed or was "evaluated against output" are rejected. FAIL evidence must explain what is missing or contradicted. Deterministic validation checks provenance and shape; it does not replace the Grader judgment required by grader.md.') + [void]$builder.AppendLine('Grade every arm with the single validated analyzer/grader profile persisted at `analyzer-profile.json` (also recorded in `manifest.analyzer_selection`), not with whichever model happens to host the orchestrator. The analyzer is an independent boundary: it must not inherit an executor arm''s HOME, provider configuration, session, native tools, skill catalogs, or plugins, and executor identity and analyzer identity stay independently attributable in the report metadata (`executor_model` vs `analyzer_model`/`analyzer_runner`). Use the same analyzer profile for every eval and every configuration so comparisons stay within one analyzer stratum.') [void]$builder.AppendLine('To display the authoritative top-level grading skeleton, run:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -ShowSkeleton") [void]$builder.AppendLine('Write `grading.json`, then validate it before finalization:') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index aedb05e..b249241 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1290,6 +1290,39 @@ Add-ValidationResult -Results $results -Name 'Change-impact default resolution h if ($LASTEXITCODE -ne 0) { throw "Git scenario regression failed: $($output -join [Environment]::NewLine)" } } +Add-ValidationResult -Results $results -Name 'Git-workspace evals omit stale inline source and prove candidate-instruction identity' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-change-impact-9-' + [Guid]::NewGuid().ToString('N')) + try { + $output = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-change-impact' -Runner 'github-copilot' -Eval 9 -OutputRoot $packageRoot 2>&1 + if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for change-impact eval 9: $($output -join [Environment]::NewLine)" } + $evalDir = @(Get-ChildItem -Path $packageRoot -Recurse -Directory | Where-Object { $_.Name -like 'eval-09*' } | Select-Object -First 1) + if ($evalDir.Count -ne 1) { throw 'Prepared change-impact eval 9 directory was not found.' } + $withRun = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/run.json'), $utf8NoBom) | ConvertFrom-Json + $withoutRun = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'without_skill/run.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$withRun.candidateInstructionHash -notmatch '^[0-9a-f]{64}$') { throw 'with_skill run.json must declare a SHA-256 candidateInstructionHash.' } + if (-not [string]::IsNullOrWhiteSpace([string]$withoutRun.candidateInstructionHash)) { throw 'without_skill run.json must not declare a candidateInstructionHash.' } + $withPrompt = ([System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/prompt.md'), $utf8NoBom)) -replace "`r`n", "`n" -replace "`r", "`n" + $withoutPrompt = ([System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'without_skill/prompt.md'), $utf8NoBom)) -replace "`r`n", "`n" -replace "`r", "`n" + foreach ($pair in @(@('with_skill', $withPrompt), @('without_skill', $withoutPrompt))) { + if ($pair[1] -match '(?m)^# Input files') { throw "$($pair[0]) Git-workspace prompt must omit inline fixture source that could disagree with the final working tree." } + if ($pair[1] -match 'public static Widget Parse') { throw "$($pair[0]) prompt inlines the stale pre-removal Widget.Parse source." } + if ($pair[1] -match 'Use dotnet-change-impact') { throw "$($pair[0]) task must be skill-neutral and must not name the candidate skill." } + } + $marker = "`n`n# Working environment" + $idx = $withPrompt.IndexOf($marker, [System.StringComparison]::Ordinal) + if ($idx -lt 0) { throw 'with_skill prompt is missing the working-environment boundary.' } + $instruction = $withPrompt.Substring(0, $idx) + $injected = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + if ($injected -ne [string]$withRun.candidateInstructionHash) { throw 'with_skill injected candidate instructions must hash to the frozen candidateInstructionHash.' } + $finalWidget = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/repo/src/Widget.cs'), $utf8NoBom) + if ($finalWidget -match 'Parse') { throw 'The final staged working tree must reflect the feature commit that removed Widget.Parse.' } + } finally { + if (Test-Path -LiteralPath $packageRoot) { Remove-Item -LiteralPath $packageRoot -Recurse -Force -ErrorAction SilentlyContinue } + } +} + Add-ValidationResult -Results $results -Name 'Runner-owned orchestration remains deterministic' -Group 'Runners' -Action { if (-not [string]::IsNullOrWhiteSpace($Ref)) { return @@ -1894,6 +1927,19 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' [string]$manifest.report.benchmark_markdown -ne 'benchmark.md') { throw 'The prepared package manifest must declare the first-party report, upstream skill-creator tools, and output artifacts.' } + if ([string]$manifest.analyzer_profile -ne 'analyzer-profile.json') { + throw 'The prepared package manifest must declare the analyzer_profile artifact.' + } + $analyzerProfile = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'analyzer-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$analyzerProfile.schema -ne 'codebeltnet/agentic/eval-analyzer-profile/1') { + throw "analyzer-profile.json must declare the analyzer profile schema; got '$($analyzerProfile.schema)'." + } + if ([string]::IsNullOrWhiteSpace([string]$analyzerProfile.runner) -or [string]::IsNullOrWhiteSpace([string]$analyzerProfile.model) -or [string]::IsNullOrWhiteSpace([string]$analyzerProfile.contract_version)) { + throw 'analyzer-profile.json must persist a validated grader runner, model, and contract version distinct from the executor.' + } + if ([string]$manifest.analyzer_selection.model -ne [string]$analyzerProfile.model -or [string]$manifest.analyzer_selection.runner -ne [string]$analyzerProfile.runner) { + throw 'manifest.analyzer_selection must match the persisted analyzer-profile.json for independent executor/analyzer attribution.' + } if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/generate-eval-report.ps1')) -or -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/eval-report-template.html')) -or -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/skill-creator/scripts/aggregate_benchmark.py')) -or From 87ba9ba887916795bbd39ca391bb6f8a5e221812 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Wed, 9 Sep 2026 13:47:31 +0200 Subject: [PATCH 21/36] =?UTF-8?q?=E2=9C=85=20update=20evaluation=20test=20?= =?UTF-8?q?fixtures=20and=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update test fixtures and test suites to support new evaluation framework capabilities: analyzer/grader configuration, shell environment isolation, and runner boundary validation. Enhance copilot-help fixtures and add comprehensive validation for runner conformance, codex paths, copilot boundaries, and integrity finalization. --- .../fixtures/copilot-help-exact-session.txt | 1 + .../copilot-help-no-exact-session.txt | 1 + .../tests/test-change-impact-workspace.ps1 | 50 ++++++++++++-- .../eval-runners/tests/test-codex-paths.ps1 | 15 ++++- .../tests/test-copilot-boundaries.ps1 | 48 ++++++++++++- .../tests/test-integrity-finalization.ps1 | 2 +- .../tests/test-runner-conformance.ps1 | 67 +++++++++++++++++-- 7 files changed, 169 insertions(+), 15 deletions(-) diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt index 24903ba..578c020 100644 --- a/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt +++ b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt @@ -6,6 +6,7 @@ Usage: copilot [options] --allow-all-tools Allow tools --no-ask-user Disable prompts --disable-builtin-mcps Disable built-in MCP servers + --excluded-tools Tools that will not be available to the model --secret-env-vars Secret environment variables --no-auto-update Disable updates --resume Resume a previous Copilot session by its exact session id diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt index c6dfd13..ac99a10 100644 --- a/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt +++ b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt @@ -6,6 +6,7 @@ Usage: copilot [options] --allow-all-tools Allow tools --no-ask-user Disable prompts --disable-builtin-mcps Disable built-in MCP servers + --excluded-tools Tools that will not be available to the model --secret-env-vars Secret environment variables --no-auto-update Disable updates --resume Resume the most recent Copilot session by id diff --git a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 index 23bf0f4..faaef0d 100644 --- a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 +++ b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 @@ -4,9 +4,42 @@ param() $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest $scriptsRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scriptsRoot 'eval-runners\runner-common.ps1') $workspace = Join-Path ([IO.Path]::GetTempPath()) ('change-impact-workspace-' + [guid]::NewGuid().ToString('N')) [void][IO.Directory]::CreateDirectory($workspace) function Assert-True($Condition, $Message) { if (-not $Condition) { throw $Message } } +$tokens = $null +$parseErrors = $null +$codexAst = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $scriptsRoot 'eval-runners\codex\runner.ps1'), [ref]$tokens, [ref]$parseErrors) +if ($parseErrors.Count) { throw 'Codex runner did not parse for sanitized PATH projection.' } +$pathBuilder = @($codexAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Get-CodexSanitizedShellPath' }, $true)) +if ($pathBuilder.Count -ne 1) { throw 'Codex sanitized PATH builder was not found.' } +Invoke-Expression $pathBuilder[0].Extent.Text +function New-SanitizedGitEnvironment { + param([Parameter(Mandatory = $true)][object]$GitCommand) + $gitDirectory = Split-Path -Parent ([string]$GitCommand.Source) + $platform = Get-PlatformName + $environment = [ordered]@{ PATH = Get-CodexSanitizedShellPath -Platform $platform -GitDirectory $gitDirectory } + if ($platform -eq 'windows') { + $windowsRoot = [Environment]::GetEnvironmentVariable('SystemRoot') + if ([string]::IsNullOrWhiteSpace($windowsRoot)) { $windowsRoot = 'C:\Windows' } + $environment.SystemRoot = $windowsRoot + $environment.ComSpec = Join-Path (Join-Path $windowsRoot 'System32') 'cmd.exe' + $environment.PATHEXT = '.COM;.EXE;.BAT;.CMD' + } + return $environment +} +function Invoke-SanitizedGit { + param( + [Parameter(Mandatory = $true)][object]$GitCommand, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [Parameter(Mandatory = $true)][string]$Repo, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + $process = Invoke-RunnerProcess -FileName ([IO.Path]::GetFileName([string]$GitCommand.Source)) -ArgumentList (@('-C', $Repo) + $Arguments) -WorkingDirectory $Repo -Environment $Environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { throw "sanitized git $([string]::Join(' ', $Arguments)) failed: $($process.Stderr)" } + return @($process.Stdout -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) }) +} try { $catalog = Join-Path $workspace 'models.json' [IO.File]::WriteAllText($catalog, '{"models":[{"id":"fixture-model"}]}') @@ -16,18 +49,21 @@ try { Assert-True ($handoff.Contains('MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`')) 'Handoff must require the exact packaged grader before grading.' $manifest = Get-Content (Join-Path $package 'manifest.json') -Raw | ConvertFrom-Json $heads = @(); $diffs = @(); $refs = @() + $gitCommand = Resolve-ExternalCommand -Name 'git' + Assert-True ($null -ne $gitCommand) 'git must resolve for the sanitized Codex git-workspace projection.' + $sanitizedGitEnvironment = New-SanitizedGitEnvironment -GitCommand $gitCommand foreach ($arm in @('with_skill', 'without_skill')) { $repo = Join-Path (Split-Path -Parent (Join-Path $package $manifest.evals[0].runs.$arm.run_manifest)) 'repo' Assert-True (Test-Path (Join-Path $repo '.git') -PathType Container) 'Eval 9 must stage a real .git repository.' - Assert-True ((& git -C $repo branch --show-current) -eq 'feature/remove-legacy-api') 'Feature branch missing.' - Assert-True ((& git -C $repo symbolic-ref refs/remotes/origin/HEAD --short) -eq 'origin/trunk') 'Non-main default branch fallback missing.' - Assert-True ((& git -C $repo rev-list --count origin/HEAD..HEAD) -eq '1') 'Feature commit history missing.' - $diff = (& git -C $repo diff origin/HEAD...HEAD) -join "`n" + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('branch', '--show-current')) -eq 'feature/remove-legacy-api') 'Feature branch missing under sanitized PATH.' + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('symbolic-ref', 'refs/remotes/origin/HEAD', '--short')) -eq 'origin/trunk') 'Non-main default branch fallback missing under sanitized PATH.' + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('rev-list', '--count', 'origin/HEAD..HEAD')) -eq '1') 'Feature commit history missing under sanitized PATH.' + $diff = (Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('diff', 'origin/HEAD...HEAD')) -join "`n" Assert-True ($diff.Contains('- public static Widget Parse(string value)')) 'Meaningful public API removal missing.' - Assert-True ([string]::IsNullOrWhiteSpace((& git -C $repo status --porcelain) -join '')) 'Staged repository must be clean.' - $heads += & git -C $repo rev-parse HEAD + Assert-True ([string]::IsNullOrWhiteSpace((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('status', '--porcelain')) -join '')) 'Staged repository must be clean under sanitized PATH.' + $heads += Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('rev-parse', 'HEAD') $diffs += $diff - $refs += ((& git -C $repo show-ref) -join "`n") + $refs += ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('show-ref')) -join "`n") } Assert-True ($heads[0] -ceq $heads[1] -and $diffs[0] -ceq $diffs[1] -and $refs[0] -ceq $refs[1]) 'Paired Git history, refs and diff must be identical.' . (Join-Path $scriptsRoot 'eval-git-workspace.ps1') diff --git a/scripts/eval-runners/tests/test-codex-paths.ps1 b/scripts/eval-runners/tests/test-codex-paths.ps1 index 0f31a1b..9e7835b 100644 --- a/scripts/eval-runners/tests/test-codex-paths.ps1 +++ b/scripts/eval-runners/tests/test-codex-paths.ps1 @@ -10,7 +10,7 @@ $tokens = $null $parseErrors = $null $ast = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $PSScriptRoot '../codex/runner.ps1'), [ref]$tokens, [ref]$parseErrors) if ($parseErrors.Count) { throw 'Codex runner did not parse.' } -$names = @('ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') +$names = @('ConvertTo-CodexConfigStringLiteral', 'Get-CodexSanitizedShellPath', 'Get-CodexShellEnvironmentPolicySet', 'Add-CodexShellEnvironmentPolicyConfigArguments', 'ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') foreach ($name in $names) { $definition = @($ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $name }, $true)) if ($definition.Count -ne 1) { throw "Missing unique function '$name'." } @@ -24,6 +24,19 @@ if ((ConvertTo-CodexComparablePath $comparable) -cne $comparable) { throw 'Path if (-not (Test-CodexPathInsideComparableRoot $staged (Join-Path $staged 'FORMS.md'))) { throw 'Staged descendants must remain inside their root.' } if (Test-CodexPathInsideComparableRoot $staged ($staged + '-other/FORMS.md')) { throw 'A sibling prefix is not a descendant.' } +$windowsGit = 'C:\Program Files\Git\cmd' +$windowsPath = Get-CodexSanitizedShellPath -Platform windows -GitDirectory $windowsGit -WindowsRoot 'C:\Windows' +if ($windowsPath -cne "C:\Windows\System32;$windowsGit") { throw "Windows sanitized PATH was '$windowsPath'." } +if ($windowsPath -match [regex]::Escape('C:\host-only\bin')) { throw 'Windows sanitized PATH exposed an arbitrary host-only directory.' } +$linuxPath = Get-CodexSanitizedShellPath -Platform linux -GitDirectory '/opt/git/bin' +if ($linuxPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/git/bin') { throw "Linux sanitized PATH was '$linuxPath'." } +$macPath = Get-CodexSanitizedShellPath -Platform macos -GitDirectory '/usr/bin' +if ($macPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin') { throw "macOS sanitized PATH should deduplicate /usr/bin but was '$macPath'." } +$configArgs = [System.Collections.Generic.List[string]]::new() +Add-CodexShellEnvironmentPolicyConfigArguments -Arguments $configArgs -ShellEnvironmentSet ([ordered]@{ PATH = $windowsPath; SystemRoot = 'C:\Windows'; ComSpec = 'C:\Windows\System32\cmd.exe'; PATHEXT = '.COM;.EXE;.BAT;.CMD' }) -SwitchName '--config' +if (@($configArgs | Where-Object { $_ -eq 'shell_environment_policy.inherit=none' }).Count -ne 1) { throw 'Codex config args must keep shell environment inheritance disabled.' } +if (@($configArgs | Where-Object { $_ -eq ('shell_environment_policy.set.PATH="' + $windowsPath.Replace('\', '\\') + '"') }).Count -ne 1) { throw 'Codex config args must set the sanitized child PATH.' } + foreach ($access in @('command', 'file')) { $isolation = [ordered]@{ ambient_skill_paths_observed = @($ambient); failures = @() } $parameters = @{ NativeSkillIsolation = $isolation; AllowedStagedSkillRoot = $staged } diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 87789b3..16571d5 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -21,7 +21,7 @@ foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', Invoke-Expression $definition.Extent.Text } if ($file -eq 'github-copilot/runner.ps1' -and $definition -is [Management.Automation.Language.AssignmentStatementAst] -and - $definition.Left.Extent.Text -eq '$descriptor') { Invoke-Expression $definition.Extent.Text } + $definition.Left.Extent.Text -in @('$descriptor', '$copilotExcludedTools', '$copilotCandidateInstructionBoundary')) { Invoke-Expression $definition.Extent.Text } } } $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') @@ -194,6 +194,52 @@ if ($inputText -eq 'failure') { exit 7 } Assert-True (@(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = $path } } -Projection $proof).Count -gt 0) "captured forbidden access rejected: $path" } Assert-Equal 0 @(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = 'src/Widget.cs' } } -Projection $proof).Count 'ordinary staged source is allowed' + + # --- P0 native-skill isolation regressions (model-free) --- + Assert-True (@(New-CopilotCliArguments -Inputs $singleInputs) -contains '--excluded-tools=skill') 'Copilot removes the native skill tool from the model tool set for both arms' + # Native-skill activation detector: a candidate `skill` tool call or an inherited skill-resolution result is a breach. + Assert-True (@(Find-CopilotNativeSkillActivation -Data @{ toolName = 'skill'; arguments = @{ skill = 'demo-skill' } } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_start').Count -gt 0) 'candidate skill tool call is flagged' + Assert-True (@(Find-CopilotNativeSkillActivation -Data @{ skill = 'demo-skill'; found = $true; skillSource = 'inherited' } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_completed').Count -gt 0) 'inherited candidate skill resolution is flagged' + Assert-Equal 0 @(Find-CopilotNativeSkillActivation -Data @{ toolName = 'view'; arguments = @{ path = '../skill/demo-skill/SKILL.md' } } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_start').Count 'ordinary read of the staged candidate copy is not native activation' + Assert-Equal 0 @(Find-CopilotNativeSkillActivation -Data @{ toolName = 'shell'; arguments = @{ command = 'echo demo-skill' } } -CandidateSkillName 'demo-skill' -EventType 'command.execute').Count 'a mere text mention of the candidate is not native activation' + + # Candidate-instruction identity proof (only the frozen instruction bytes are hashed, not the wrapper). + $demoInstruction = "# Operating instructions`n`n## Skill: demo-skill`n`nDo the demo work." + $demoInstructionHash = ([Convert]::ToHexString([Security.Cryptography.SHA256]::HashData([Text.Encoding]::UTF8.GetBytes($demoInstruction)))).ToLowerInvariant() + $withPrompt = "$demoInstruction`n`n# Working environment`n`nStay in the run.`n`n# task`n`nClassify." + $noSkillPrompt = "# Operating instructions`n`nNo special instructions.`n`n# Working environment`n`nStay in the run.`n`n# task`n`nClassify." + $withEvidence = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = $demoInstructionHash } }) + Assert-True $withEvidence.verified 'with_skill injected instructions hash exactly to the frozen candidate hash' + Assert-Equal $demoInstructionHash $withEvidence.injected 'injected candidate-instruction hash equals the frozen hash' + Assert-Equal 0 @($withEvidence.violations).Count 'a matching candidate-instruction hash yields no violation' + $wrapperChanged = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes("$demoInstruction`n`n# Working environment`n`nDifferent wrapper text entirely.`n`n# task`n`nOther."); CandidateInstructionHash = $demoInstructionHash } }) + Assert-True $wrapperChanged.verified 'unrelated prompt-wrapper changes do not invalidate the candidate identity proof' + $mismatch = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = ('0' * 64) } }) + Assert-True ((-not $mismatch.verified) -and @($mismatch.violations).Count -gt 0) 'a mismatched candidate-instruction hash is a violation' + $baselineClean = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($noSkillPrompt); CandidateInstructionHash = $null } }) + Assert-True ($baselineClean.verified -and @($baselineClean.violations).Count -eq 0) 'clean baseline has no candidate injection and no hash' + Assert-True (@((Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = $null } })).violations).Count -gt 0) 'baseline that embeds a candidate instruction section is a violation' + Assert-True (@((Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($noSkillPrompt); CandidateInstructionHash = ('a' * 64) } })).violations).Count -gt 0) 'baseline that declares a candidate hash is a violation' + + # Native-skill-catalog probe: model-free, proves candidate absence/disablement or fails closed on an enabled candidate. + $catalogFake = Join-Path $testRoot 'catalog-fake.ps1' + [IO.File]::WriteAllText($catalogFake, 'param([Parameter(ValueFromRemainingArguments=$true)][string[]]$a); if ($env:FAKE_CATALOG) { Write-Output $env:FAKE_CATALOG } else { Write-Output "[]" }') + $catalogCommand = [pscustomobject]@{ FileName = (Get-Command pwsh).Source; Prefix = @('-NoProfile', '-NonInteractive', '-File', $catalogFake) } + function New-CatalogEnv { param([string]$Json) $env = New-RunnerProbeEnvironment; $env['FAKE_CATALOG'] = $Json; return $env } + $absentProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"customize-cloud-agent","source":"builtin","enabled":true}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($absentProbe.available -and $absentProbe.proven_absent -and -not $absentProbe.candidate_present) 'catalog probe proves an absent candidate is not natively resolvable' + $enabledProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"demo-skill","source":"personal","enabled":true}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($enabledProbe.available -and $enabledProbe.candidate_present -and $enabledProbe.candidate_enabled -and -not $enabledProbe.proven_absent) 'catalog probe flags an enabled ambient candidate' + $disabledProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"demo-skill","source":"personal","enabled":false}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($disabledProbe.candidate_present -and -not $disabledProbe.candidate_enabled -and $disabledProbe.proven_absent) 'a disabled ambient candidate is present but cannot activate' + + # Bridge independently rejects a captured candidate native-skill activation even when runner booleans look clean. + $activationRun = New-TestRun -IterationDirectory (Join-Path $testRoot 'native-skill-activation') -Configuration without_skill + New-Item -ItemType Directory -Path (Join-Path $activationRun.Root 'evidence') -Force | Out-Null + [IO.File]::WriteAllText((Join-Path $activationRun.Root 'evidence/copilot-events.jsonl'), '{"type":"tool.execution_start","data":{"toolName":"skill","arguments":{"skill":"candidate"}}}' + "`n") + $activationInputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $activationRun.Path; Profile = Resolve-ExecutionProfile -ProfilePath $profilePath } + $activationRaw = @{ runner = @{ name = 'github-copilot' }; status = 'completed'; evidence = @{ execution_paths = @{ projection_proven = $true; physical_run_root = (Join-Path $testRoot 'phys-activation'); source_repository_root = '' } }; artifacts = @(@{ scope = 'run'; path = 'evidence/copilot-events.jsonl' }) } | ConvertTo-Json -Depth 20 | ConvertFrom-Json + Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $activationRaw -RunData $activationInputs.Run } 'bridge rejects a captured candidate native-skill activation' $warnings = [Collections.Generic.List[string]]::new() $checkpoint = @{ type = 'session.usage_checkpoint'; data = @{ totalPremiumRequests = 0.33; totalNanoAiu = 10; promptCacheBreakState = @(@{ models = @{ model = @{ model_call_id = 'call-1'; prompt_tokens = 100; cache_read = 70; cache_write = 20; tool_tokens = 15 } } }) } } $last = $checkpoint | ConvertTo-Json -Depth 20 | ConvertFrom-Json diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index e27ad94..84c4da5 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -596,7 +596,7 @@ for ($index = 0; $index -lt $count; $index++) { Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'invalid grading entry validation' -ExpectedText 'passed must be a boolean' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'invalid grading validation' - foreach ($badEvidence in @('', " `t`n", 'Evaluation completed with output', "Source: output`nQuote: fabricated unavailable observation`nReason: This establishes the assertion.", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Evaluation completed with output")) { + foreach ($badEvidence in @('', " `t`n", 'Evaluation completed with output', "Source: output`nQuote: fabricated unavailable observation`nReason: This establishes the assertion.", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Evaluation completed with output", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Assertion evaluated against output", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: The assertion is met", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Output matches the assertion")) { $bad = Copy-TestGradingDocument -Document $validGrading $bad.grading[0].evidence = $badEvidence Write-TestJson -Path $gradingPath -Value $bad diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index fa20d33..1fb0222 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -209,7 +209,7 @@ if ($arguments -contains '--help' -and -not ($harness -eq 'codex' -and $argument 'copilot' { if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } - else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } + else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --excluded-tools --available-tools --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } } default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } } @@ -218,6 +218,11 @@ if ($arguments -contains '--help' -and -not ($harness -eq 'codex' -and $argument Write-Output $help exit 0 } +if ($harness -eq 'copilot' -and ($arguments -contains 'skill') -and ($arguments -contains 'list')) { + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'skill_list_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output '[{"name":"customize-cloud-agent","source":"builtin","enabled":true},{"name":"github-pr-media","source":"builtin","enabled":true}]' + exit 0 +} $continuationFlag = $null foreach ($candidate in @('--resume', '--session-id', '--session')) { if ($arguments -contains $candidate -or @($arguments | Where-Object { [string]$_ -like ($candidate + '=*') }).Count -gt 0) { @@ -654,7 +659,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { if ($null -eq $threadStart) { $record.rpc_methods = @($initialize.method, $initialized.method, $skillsList.method) $record.skills_list_params = $skillsList.params - $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -eq 'shell_environment_policy.inherit=none' }) + $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -like 'shell_environment_policy.*' }) [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) exit 0 } @@ -738,7 +743,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { $record.auth_only_home = [bool]$record.parent_auth_file_visible -and -not [bool]$record.parent_config_file_visible -and -not [bool]$record.parent_skills_directory_visible -and -not [bool]$record.parent_agents_directory_visible -and -not [bool]$record.parent_sessions_directory_visible -and -not [bool]$record.parent_memories_directory_visible -and -not [bool]$record.parent_plugins_directory_visible -and -not [bool]$record.parent_mcp_configuration_visible -and -not [bool]$record.parent_agents_file_visible $record.rpc_methods = @($initialize.method, $initialized.method, $skillsList.method, $threadStart.method, $turnStart.method, 'thread/read') $record.skills_list_params = $skillsList.params - $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -eq 'shell_environment_policy.inherit=none' }) + $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -like 'shell_environment_policy.*' }) $record.thread_params = $threadStart.params $record.turn_params = $turnStart.params [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -830,7 +835,7 @@ if ($arguments -contains '--help') { 'copilot' { if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } - else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } + else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --excluded-tools --available-tools --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } } default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } } @@ -839,6 +844,11 @@ if ($arguments -contains '--help') { Write-Output $help exit 0 } +if ($harness -eq 'copilot' -and ($arguments -contains 'skill') -and ($arguments -contains 'list')) { + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'skill_list_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output '[{"name":"customize-cloud-agent","source":"builtin","enabled":true},{"name":"github-pr-media","source":"builtin","enabled":true}]' + exit 0 +} $stdinMemory = [IO.MemoryStream]::new() [Console]::OpenStandardInput().CopyTo($stdinMemory) $stdinBytes = $stdinMemory.ToArray() @@ -966,6 +976,12 @@ if ($harness -eq 'codex') { [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.cmd"), "@echo off`r`npwsh -NoProfile -NonInteractive -File ""%~dp0$harness.ps1"" %*`r`n", [System.Text.UTF8Encoding]::new($false)) } + $fakeGitBin = Join-Path $recordedRoot 'git-bin' + New-Item -ItemType Directory -Path $fakeGitBin -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fakeGitBin 'git.cmd'), "@echo off`r`necho {""args"":""%*"",""path"":""%PATH%""}>> ""%CD%\git-probe-log.jsonl""`r`necho git version recorded.fixture`r`n", [System.Text.UTF8Encoding]::new($false)) + $fakeGitSh = Join-Path $fakeGitBin 'git' + [System.IO.File]::WriteAllText($fakeGitSh, "#!/usr/bin/env sh`nprintf '{""args"":""%s"",""path"":""%s""}\n' ""`$*"" ""`$PATH"" >> ""`$PWD/git-probe-log.jsonl""`nprintf 'git version recorded.fixture\n'`n", [System.Text.UTF8Encoding]::new($false)) + if (-not $IsWindows) { & chmod +x $fakeGitSh } $fakeGh = @' [CmdletBinding()] param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) @@ -988,7 +1004,9 @@ if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and exit 2 '@ [System.IO.File]::WriteAllText((Join-Path $fakeBin 'gh.ps1'), $fakeGh, [System.Text.UTF8Encoding]::new($false)) - $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" + $hostOnlyPath = Join-Path $recordedRoot 'host-only-bin' + New-Item -ItemType Directory -Path $hostOnlyPath -Force | Out-Null + $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$fakeGitBin$([System.IO.Path]::PathSeparator)$hostOnlyPath$([System.IO.Path]::PathSeparator)$recordedOldPath" $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' @@ -1030,6 +1048,36 @@ exit 2 $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' + $gitWorkspaceIteration = Join-Path $recordedRoot 'iteration-git-workspace' + New-Item -ItemType Directory -Path $gitWorkspaceIteration -Force | Out-Null + $gitWorkspaceRun = New-TestRun -IterationDirectory $gitWorkspaceIteration -Configuration with_skill -EvalName 'codex-git-workspace' + $gitWorkspaceRunJson = Read-RunnerJson -Path $gitWorkspaceRun.Path + $gitWorkspaceRunJson.gitWorkspace = $true + Write-TestJson -Path $gitWorkspaceRun.Path -Value $gitWorkspaceRunJson + $gitWorkspacePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $gitWorkspaceRun.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'compatible' $gitWorkspacePreflight.status 'Codex gitWorkspace preflight passes when git resolves through the sanitized child PATH' + Assert-Equal 1 @($gitWorkspacePreflight.checks | Where-Object { $_.name -eq 'git_workspace_tool_path' -and $_.status -eq 'passed' }).Count 'Codex gitWorkspace preflight records a passed git probe' + $gitProbeLog = Join-Path $gitWorkspaceRun.Root 'repo\git-probe-log.jsonl' + Assert-True (Test-Path -LiteralPath $gitProbeLog -PathType Leaf) 'Codex gitWorkspace preflight runs git --version in the sanitized environment' + $gitProbeText = Get-Content -LiteralPath $gitProbeLog -Raw + Assert-True ([string]$gitProbeText -match [regex]::Escape('--version')) 'Codex gitWorkspace preflight probes git --version' + Assert-True ([string]$gitProbeText -match [regex]::Escape($fakeGitBin)) 'Codex gitWorkspace probe PATH includes the resolved git directory' + Assert-True ([string]$gitProbeText -notmatch [regex]::Escape($hostOnlyPath)) 'Codex gitWorkspace probe PATH excludes arbitrary host-only directories' + $gitWorkspaceShellPath = @($gitWorkspacePreflight.checks | Where-Object { $_.name -eq 'git_workspace_tool_path' } | Select-Object -First 1).detail + Assert-True ([string]$gitWorkspaceShellPath -match [regex]::Escape($fakeGitBin)) 'Codex gitWorkspace preflight reports the git directory in the sanitized PATH' + $pathWithoutGit = "$fakeBin$([System.IO.Path]::PathSeparator)$(Split-Path -Parent (Get-Command pwsh).Source)" + $pathWithGit = $env:PATH + try { + $env:PATH = $pathWithoutGit + $missingGitExecution = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $gitWorkspaceRun.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $missingGitExecution.status 'Codex gitWorkspace execution fails closed when git cannot resolve before model execution' + Assert-Equal 'preflight_incompatible' $missingGitExecution.final_response.reason 'Codex missing-git execution stops at preflight' + $missingGitLog = Join-Path $gitWorkspaceRun.Root 'repo\codex-fake-cli-log.jsonl' + $missingGitRecords = if (Test-Path -LiteralPath $missingGitLog -PathType Leaf) { @(Get-Content -LiteralPath $missingGitLog | ForEach-Object { $_ | ConvertFrom-Json }) } else { @() } + Assert-Equal 0 @($missingGitRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -or @((Get-JsonProperty -Object $_ -Name 'rpc_methods' -Default @()) | Where-Object { [string]$_ -eq 'turn/start' }).Count -gt 0 }).Count 'Codex missing-git preflight starts zero model execution processes' + } finally { + $env:PATH = $pathWithGit + } foreach ($fixtureName in @( 'copilot-scripted-turn-1-events.jsonl', 'copilot-scripted-turn-2-events.jsonl', @@ -1236,6 +1284,11 @@ exit 2 Assert-True ($args -contains '--ask-for-approval') 'Codex uses explicit approval policy' Assert-True ($args -contains 'never') 'Codex approval policy is never' Assert-True ($args -contains '--strict-config') 'Codex CLI uses strict config parsing for session controls' + Assert-True (@($args | Where-Object { $_ -eq 'shell_environment_policy.inherit=none' }).Count -eq 1) 'Codex CLI disables child shell environment inheritance' + $codexShellPathArg = @($args | Where-Object { [string]$_ -like 'shell_environment_policy.set.PATH=*' } | Select-Object -First 1) + Assert-Equal 1 $codexShellPathArg.Count 'Codex CLI sets a sanitized child shell PATH' + Assert-True ([string]$codexShellPathArg[0] -notmatch [regex]::Escape($hostOnlyPath)) 'Codex CLI child shell PATH excludes arbitrary host-only directories' + Assert-True ([string]$codexShellPathArg[0] -notmatch [regex]::Escape($fakeGitBin)) 'Codex CLI child shell PATH does not include git for non-git workspaces' Assert-True ($args -contains '--sandbox' -and $args -contains 'danger-full-access') 'Codex grants full operational sandbox permission' Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' Assert-True (@($args | Where-Object { $_ -eq 'skills.include_instructions=false' }).Count -eq 1) 'Codex CLI disables native skill catalog injection at session scope' @@ -1979,6 +2032,10 @@ exit 2 Assert-True (-not [bool]$subscriptionRecord.unrelated_present) 'Codex app-server parent excludes unrelated inherited environment variables' Assert-True (-not [bool]$subscriptionRecord.worker_auth_file_visible) 'Codex app-server worker fixture does not receive auth.json' Assert-True (@($subscriptionRecord.args) -contains 'shell_environment_policy.inherit=none') 'Codex app-server disables child shell environment inheritance' + $subscriptionShellPathArg = @($subscriptionRecord.args | Where-Object { [string]$_ -like 'shell_environment_policy.set.PATH=*' } | Select-Object -First 1) + Assert-Equal 1 $subscriptionShellPathArg.Count 'Codex app-server sets a sanitized child shell PATH' + Assert-True ([string]$subscriptionShellPathArg[0] -notmatch [regex]::Escape($hostOnlyPath)) 'Codex app-server child shell PATH excludes arbitrary host-only directories' + Assert-True ([string]$subscriptionShellPathArg[0] -notmatch [regex]::Escape($fakeGitBin)) 'Codex app-server child shell PATH does not include git for non-git workspaces' $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:CODEX_HOME = $recordedOldCodexHome } From f90bc067f46a1610d4e9eebf3742370d92ce570c Mon Sep 17 00:00:00 2001 From: gimlichael Date: Wed, 9 Sep 2026 20:54:16 +0200 Subject: [PATCH 22/36] =?UTF-8?q?=F0=9F=8F=97=EF=B8=8F=20enforce=20phase?= =?UTF-8?q?=202=20analyzer=20provenance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 grading now runs through a package-local controller that validates analyzer identity, freezes provenance, and derives grading.json before finalization. This also closes the deterministic CI regressions with target-platform path string construction and separate offline catalog injection for executor and analyzer model validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/eval-report-template.html | 10 + scripts/eval-runners/apply-eval-grading.ps1 | 12 +- scripts/eval-runners/codex/runner.ps1 | 30 +- .../contracts/grading.schema.json | 41 +- .../eval-runners/eval-grading-contract.ps1 | 383 ++++++----- .../eval-runners/finalize-eval-package.ps1 | 6 + .../eval-runners/invoke-phase2-analyzer.ps1 | 645 ++++++++++++++++++ scripts/eval-runners/phase2-grading.ps1 | 515 ++++++++++++++ scripts/eval-runners/runner-common.ps1 | 5 + .../tests/fixtures/runner-owned-fixture.ps1 | 40 +- .../tests/test-change-impact-workspace.ps1 | 14 +- .../eval-runners/tests/test-codex-paths.ps1 | 6 +- .../tests/test-integrity-finalization.ps1 | 140 +++- .../tests/test-runner-conformance.ps1 | 8 +- scripts/generate-eval-report.ps1 | 133 +++- scripts/prepare-skill-evals.ps1 | 98 ++- scripts/validate-skill-templates.ps1 | 147 ++-- skills/dotnet-change-impact/evals/evals.json | 121 ++-- 18 files changed, 1983 insertions(+), 371 deletions(-) create mode 100644 scripts/eval-runners/invoke-phase2-analyzer.ps1 create mode 100644 scripts/eval-runners/phase2-grading.ps1 diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html index 4881def..519a31a 100644 --- a/scripts/eval-report-template.html +++ b/scripts/eval-report-template.html @@ -266,6 +266,16 @@ function renderBenchmark() { const evals = DATA.evals || []; let html = '

Benchmark

Paired results are shown side by side. A missing metric means the harness did not record it; no value is estimated.

'; + const usage = DATA.metadata && DATA.metadata.usage ? DATA.metadata.usage : {}; + const executionUsage = usage.execution || {}; + const analyzerUsage = usage.analyzer || {}; + const display = (value, formatter = num) => value == null || value === '' ? 'unavailable' : formatter(value); + html += '
Execution cost / usage
'; + html += ''; + html += '
RunnerModelWorkersDurationTotal tokensInput tokensOutput tokensCost
' + esc(executionUsage.runner || 'unavailable') + '' + esc(executionUsage.model || 'unavailable') + '' + display(executionUsage.worker_count) + '' + display(executionUsage.duration_seconds, seconds) + '' + display(executionUsage.total_tokens) + '' + display(executionUsage.input_tokens) + '' + display(executionUsage.output_tokens) + '' + display(executionUsage.cost, money) + '
'; + html += '
Analyzer cost / usage

Analyzer usage is a separate Phase 2 stratum; it is not added to execution tokens or cost.

'; + html += ''; + html += '
RunnerModelProfile hashWorkersFailedDurationTotal tokensInput tokensOutput tokensCost
' + esc(analyzerUsage.runner || 'unavailable') + '' + esc(analyzerUsage.model || 'unavailable') + '' + esc(analyzerUsage.profile_sha256 || 'unavailable') + '' + display(analyzerUsage.worker_count) + '' + display(analyzerUsage.failed_worker_count) + '' + display(analyzerUsage.duration_seconds, seconds) + '' + display(analyzerUsage.total_tokens) + '' + display(analyzerUsage.input_tokens) + '' + display(analyzerUsage.output_tokens) + '' + display(analyzerUsage.cost, money) + '
'; html += '
Aggregate comparison
'; html += summaryRow('Pass rate', summaryMetric(evals, 'with_skill', passRate), summaryMetric(evals, 'without_skill', passRate), v => num(v, 2), true); html += summaryRow('Turns', summaryMetric(evals, 'with_skill', r => r && r.metrics.turns), summaryMetric(evals, 'without_skill', r => r && r.metrics.turns), v => num(v, 1)); diff --git a/scripts/eval-runners/apply-eval-grading.ps1 b/scripts/eval-runners/apply-eval-grading.ps1 index c7edfbd..f819e41 100644 --- a/scripts/eval-runners/apply-eval-grading.ps1 +++ b/scripts/eval-runners/apply-eval-grading.ps1 @@ -20,6 +20,7 @@ Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot 'manifest-paths.ps1') . (Join-Path $PSScriptRoot 'execution-freeze.ps1') . (Join-Path $PSScriptRoot 'eval-grading-contract.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') function Write-GradingResultJson { param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) @@ -45,6 +46,7 @@ try { throw "Canonical result validation failed before grading application: $([string]::Join(' ', @($bridgeOutput | ForEach-Object { [string]$_ })))" } + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath) $gradingValidation = Assert-EvalGradingContract -IterationDirectory $iteration -GradingPath $GradingPath $records = @($gradingValidation.Records) $expected = $gradingValidation.Expected @@ -59,7 +61,15 @@ try { $assertions = @(Get-EvalMetadataAssertions -Record $record) for ($index = 0; $index -lt $assertions.Count; $index++) { $entry = $validated["$($record.EvalId)|$($record.Configuration)|$index"] - $newGrading.Add([ordered]@{ text = [string]$entry.assertion; passed = [bool]$entry.passed; evidence = [string]$entry.evidence }) + $newGrading.Add([ordered]@{ + text = [string]$entry.assertion + passed = [bool]$entry.passed + evidence = [string]$entry.evidence + evidence_domain = [string]$entry.evidence_domain + evidence_refs = @($entry.evidence_refs) + reason = [string]$entry.reason + source = [string](Get-JsonProperty -Object $entry -Name 'source' -Default '') + }) } $beforeNonGrading = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $canonical -PropertyName 'grading') # Keep the parsed canonical values as-is. ConvertFrom-Json reparses diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index e9d2d78..4a87407 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -196,7 +196,7 @@ function Get-CodexSanitizedShellPath { if ($Platform -eq 'windows') { $root = if ([string]::IsNullOrWhiteSpace($WindowsRoot)) { [Environment]::GetEnvironmentVariable('SystemRoot') } else { $WindowsRoot } if ([string]::IsNullOrWhiteSpace($root)) { $root = 'C:\Windows' } - $entries.Add((Join-Path $root 'System32')) + $entries.Add((Join-CodexTargetPath -Root $root -Segments @('System32') -Platform windows)) } else { foreach ($path in @('/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin')) { $entries.Add($path) @@ -216,11 +216,33 @@ function Get-CodexSanitizedShellPath { return [string]::Join($separator, [string[]]@($deduplicated.ToArray())) } +function Join-CodexTargetPath { + param( + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Root, + [Parameter(Mandatory = $true)][AllowEmptyString()][string[]]$Segments, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName) + ) + + $separator = if ($Platform -eq 'windows') { '\' } else { '/' } + $result = ([string]$Root).TrimEnd('\', '/') + foreach ($segment in @($Segments)) { + $value = ([string]$segment).Trim('\', '/') + if ([string]::IsNullOrWhiteSpace($value)) { continue } + if ([string]::IsNullOrWhiteSpace($result)) { + $result = $value + } else { + $result = $result + $separator + $value + } + } + return $result +} + function Get-CodexShellEnvironmentPolicySet { param( [Parameter(Mandatory = $true)][object]$Inputs, [AllowNull()][object]$GitCommandInfo = $null, - [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName) + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [AllowNull()][string]$WindowsRoot = $null ) $gitDirectory = $null @@ -228,14 +250,14 @@ function Get-CodexShellEnvironmentPolicySet { $gitSource = [string](Get-JsonProperty -Object $GitCommandInfo -Name 'Source' -Default '') if (-not [string]::IsNullOrWhiteSpace($gitSource)) { $gitDirectory = Split-Path -Parent $gitSource } } - $windowsRoot = [Environment]::GetEnvironmentVariable('SystemRoot') + $windowsRoot = if ([string]::IsNullOrWhiteSpace($WindowsRoot)) { [Environment]::GetEnvironmentVariable('SystemRoot') } else { $WindowsRoot } $values = [ordered]@{ PATH = Get-CodexSanitizedShellPath -Platform $Platform -GitDirectory $gitDirectory -WindowsRoot $windowsRoot } if ($Platform -eq 'windows') { if ([string]::IsNullOrWhiteSpace($windowsRoot)) { $windowsRoot = 'C:\Windows' } $values.SystemRoot = $windowsRoot - $values.ComSpec = Join-Path (Join-Path $windowsRoot 'System32') 'cmd.exe' + $values.ComSpec = Join-CodexTargetPath -Root $windowsRoot -Segments @('System32', 'cmd.exe') -Platform windows $values.PATHEXT = '.COM;.EXE;.BAT;.CMD' } return $values diff --git a/scripts/eval-runners/contracts/grading.schema.json b/scripts/eval-runners/contracts/grading.schema.json index f16fc12..bfb32b5 100644 --- a/scripts/eval-runners/contracts/grading.schema.json +++ b/scripts/eval-runners/contracts/grading.schema.json @@ -7,12 +7,26 @@ "required": ["schema", "grading"], "properties": { "schema": { "const": "codebeltnet/agentic/eval-grading/1" }, + "metadata": { + "type": "object", + "additionalProperties": true, + "properties": { + "schema": { "type": "string" }, + "generated_by": { "type": "string" }, + "analyzer_runner": { "type": "string" }, + "analyzer_model": { "type": "string" }, + "analyzer_reasoning_effort": { "type": ["string", "null"] }, + "analyzer_profile_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "phase2_state": { "type": "string" }, + "grading_freeze": { "type": "string" } + } + }, "grading": { "type": "array", "items": { "type": "object", "additionalProperties": false, - "required": ["eval_id", "eval_name", "configuration", "assertion_index", "assertion", "passed", "evidence"], + "required": ["eval_id", "eval_name", "configuration", "assertion_index", "assertion", "passed", "evidence", "evidence_domain", "evidence_refs", "reason"], "properties": { "eval_id": { "type": "integer", "minimum": 1 }, "eval_name": { "type": "string", "minLength": 1 }, @@ -20,7 +34,30 @@ "assertion_index": { "type": "integer", "minimum": 0 }, "assertion": { "type": "string", "minLength": 1 }, "passed": { "type": "boolean" }, - "evidence": { "type": "string", "minLength": 1, "pattern": "\\S" } + "evidence": { "type": "string", "minLength": 1, "pattern": "\\S" }, + "evidence_domain": { "enum": ["output", "transcript", "validator"] }, + "evidence_refs": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": true, + "required": ["artifact"], + "properties": { + "artifact": { "type": "string", "minLength": 1 }, + "domain": { "enum": ["output", "transcript", "validator"] }, + "start_line": { "type": "integer", "minimum": 1 }, + "end_line": { "type": "integer", "minimum": 1 }, + "quote": { "type": "string" }, + "rule": { "type": "string" }, + "version": { "type": "integer", "minimum": 1 }, + "event": { "type": ["string", "null"] }, + "passed": { "type": "boolean" } + } + } + }, + "reason": { "type": "string", "minLength": 1, "pattern": "\\S" }, + "source": { "enum": ["analyzer", "validator"] } } } } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index 529c305..c4a82a9 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -1,210 +1,207 @@ -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'runner-common.ps1') -} -if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'manifest-paths.ps1') -} -if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'execution-freeze.ps1') -} - -function Get-EvalGradingSkeleton { - return [ordered]@{ - schema = (Get-RunnerSchemaNames).Grading - grading = @() - } -} - -function Get-EvalGradingEntryKey { - param([Parameter(Mandatory = $true)][object]$Entry) - - return "$(Get-JsonProperty -Object $Entry -Name 'eval_id' -Default 0)|$(Get-JsonProperty -Object $Entry -Name 'configuration' -Default '')|$(Get-JsonProperty -Object $Entry -Name 'assertion_index' -Default -1)" -} - -function Get-EvalMetadataAssertions { - param([Parameter(Mandatory = $true)][object]$Record) - - $metadata = Read-RunnerJson -Path $Record.MetadataPath - $assertions = @(Get-JsonProperty -Object $metadata -Name 'assertions' -Default @()) - if ($assertions.Count -eq 0) { throw "Metadata for '$($Record.EvalName)' declares no assertions." } - return @($assertions | ForEach-Object { [string]$_ }) -} - -function Assert-EvalGradingEntryShape { - param([Parameter(Mandatory = $true)][object]$Entry) - - $allowed = @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence') - foreach ($name in @(Get-JsonPropertyNames -Object $Entry)) { - if ($allowed -notcontains $name) { throw "grading.json entry contains unsupported field '$name'." } - } - foreach ($name in $allowed) { - if (-not (Test-JsonProperty -Object $Entry -Name $name)) { throw "grading.json entry is missing '$name'." } - } - $evalId = 0 - try { $evalId = [int]$Entry.eval_id } catch { throw 'grading.json eval_id must be an integer.' } - if ($evalId -lt 1) { throw 'grading.json eval_id must be positive.' } - $index = 0 - try { $index = [int]$Entry.assertion_index } catch { throw 'grading.json assertion_index must be an integer.' } - if ($index -lt 0 -or [double]$Entry.assertion_index -ne $index) { throw 'grading.json assertion_index must be a non-negative integer.' } - if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } - if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } - if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'runner-common.ps1') +} +if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +} +if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'execution-freeze.ps1') +} +if (-not (Get-Command Normalize-EvalAssertion -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'phase2-grading.ps1') +} + +function Get-EvalGradingSkeleton { + return [ordered]@{ + schema = (Get-RunnerSchemaNames).Grading + grading = @() + } +} + +function Get-EvalGradingEntryKey { + param([Parameter(Mandatory = $true)][object]$Entry) + + return "$(Get-JsonProperty -Object $Entry -Name 'eval_id' -Default 0)|$(Get-JsonProperty -Object $Entry -Name 'configuration' -Default '')|$(Get-JsonProperty -Object $Entry -Name 'assertion_index' -Default -1)" +} + +function Get-EvalMetadataAssertions { + param([Parameter(Mandatory = $true)][object]$Record) + + return @(Get-EvalMetadataAssertionObjects -Record $Record) +} + +function Assert-EvalGradingEntryShape { + param([Parameter(Mandatory = $true)][object]$Entry) + + $allowed = @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason', 'source') + foreach ($name in @(Get-JsonPropertyNames -Object $Entry)) { + if ($allowed -notcontains $name) { throw "grading.json entry contains unsupported field '$name'." } + } + foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason')) { + if (-not (Test-JsonProperty -Object $Entry -Name $name)) { throw "grading.json entry is missing '$name'." } + } + $evalId = 0 + try { $evalId = [int]$Entry.eval_id } catch { throw 'grading.json eval_id must be an integer.' } + if ($evalId -lt 1) { throw 'grading.json eval_id must be positive.' } + $index = 0 + try { $index = [int]$Entry.assertion_index } catch { throw 'grading.json assertion_index must be an integer.' } + if ($index -lt 0 -or [double]$Entry.assertion_index -ne $index) { throw 'grading.json assertion_index must be a non-negative integer.' } + if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } + if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } + if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } if ([string]::IsNullOrWhiteSpace($Entry.evidence)) { throw 'grading.json evidence must be non-empty.' } - if ($Entry.passed -and $Entry.evidence -notmatch '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') { - throw 'PASS evidence requires Source, a verbatim Quote, and an assertion-specific Reason on separate lines.' + if ([string]$Entry.evidence_domain -notin @('output', 'transcript', 'validator')) { throw "grading.json evidence_domain '$($Entry.evidence_domain)' is unsupported." } + $refs = @(Get-JsonProperty -Object $Entry -Name 'evidence_refs' -Default @()) + if ($refs.Count -eq 0) { throw 'grading.json evidence_refs must be a non-empty array.' } + if ([string]::IsNullOrWhiteSpace([string]$Entry.reason)) { throw 'grading.json reason must be non-empty.' } + if ((Test-JsonProperty -Object $Entry -Name 'source') -and [string]$Entry.source -notin @('analyzer', 'validator')) { + throw "grading.json source '$($Entry.source)' is unsupported." } } -# Reject generic, templated, or tautological PASS reasons. A reason must explain HOW the cited observation establishes -# the specific assertion, not restate that the assertion passed or that the output was "evaluated". Iteration 9 passed -# 76 assertions with reasons equivalent to "Assertion evaluated against output"; that class must fail closed. -function Test-GenericGradingReason { - param([string]$Reason, [string]$Assertion) - $normalized = ([regex]::Replace([string]$Reason, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim() - if ([string]::IsNullOrWhiteSpace($normalized)) { return $true } - if ($normalized -eq (([regex]::Replace([string]$Assertion, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim())) { return $true } - $genericPatterns = @( - '^(?i)eval(?:uation)? completed(?: with output)?$', - '^(?i)(?:the )?(?:assertion|requirement|expectation|condition|criteri(?:on|a))(?: is| was| has been)?(?: fully| clearly)? (?:met|satisfied|passed|verified|confirmed|evaluated|true|correct|valid|present|fulfilled|checked|held|holds|passes)$', - '(?i)evaluated against (?:the )?(?:output|transcript|response|result|evidence|assertion)', - '(?i)(?:output|response|transcript|result) (?:was |is )?(?:evaluated|matches|meets|satisfies|supports|confirms|contains) (?:the )?(?:assertion|requirement|expectation)', - '^(?i)(?:passed|verified|confirmed|as expected|done|looks good|correct|ok|success(?:ful)?|valid|complete)$', - '^(?i)(?:this )?(?:matches|meets|satisfies|establishes|proves|confirms)(?: the)?(?: assertion| requirement| expectation)?$' - ) - foreach ($pattern in $genericPatterns) { if ($normalized -match $pattern) { return $true } } - return $false -} - -function Assert-EvalPassEvidence { - param([object]$Entry, [object]$Canonical, [object]$Record) - if (-not $Entry.passed) { return } - [void]($Entry.evidence -match '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') - $source = $Matches[1].Trim(); $quote = $Matches[2].Trim(); $reason = $Matches[3].Trim() - if ($source -eq 'output') { $content = [string]$Canonical.output } - else { - # Only native captured artifacts are admissible, never grading keys or - # grader-created files. Freeze validation already pins their bytes. - $raw = Read-RunnerJson -Path $Record.ExecutionResultPath - $artifacts = @($raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -ceq $source }) - if ($artifacts.Count -ne 1) { throw "PASS evidence source '$source' is not a captured run artifact." } - $path = Resolve-ContainedPath -BasePath (Split-Path -Parent $Record.RunManifestPath) -RelativePath $source -FieldName 'PASS evidence source' -Kind File - $content = [IO.File]::ReadAllText($path) - } - if ([string]::IsNullOrWhiteSpace($quote) -or -not $content.Contains($quote, [StringComparison]::Ordinal)) { throw 'PASS evidence quote is absent from its frozen source.' } - if ($reason -eq $quote -or (Test-GenericGradingReason -Reason $reason -Assertion ([string]$Entry.assertion))) { - throw 'PASS evidence must explain how the cited observation establishes this assertion, not restate that it passed or was evaluated.' - } +# Reject generic, templated, or tautological PASS reasons. A reason must explain HOW the cited observation establishes +# the specific assertion, not restate that the assertion passed or that the output was "evaluated". Iteration 9 passed +# 76 assertions with reasons equivalent to "Assertion evaluated against output"; that class must fail closed. +function Test-GenericGradingReason { + param([string]$Reason, [string]$Assertion) + $normalized = ([regex]::Replace([string]$Reason, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim() + if ([string]::IsNullOrWhiteSpace($normalized)) { return $true } + if ($normalized -eq (([regex]::Replace([string]$Assertion, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim())) { return $true } + $genericPatterns = @( + '^(?i)eval(?:uation)? completed(?: with output)?$', + '^(?i)(?:the )?(?:assertion|requirement|expectation|condition|criteri(?:on|a))(?: is| was| has been)?(?: fully| clearly)? (?:met|satisfied|passed|verified|confirmed|evaluated|true|correct|valid|present|fulfilled|checked|held|holds|passes)$', + '(?i)evaluated against (?:the )?(?:output|transcript|response|result|evidence|assertion)', + '(?i)(?:output|response|transcript|result) (?:was |is )?(?:evaluated|matches|meets|satisfies|supports|confirms|contains) (?:the )?(?:assertion|requirement|expectation)', + '^(?i)(?:passed|verified|confirmed|as expected|done|looks good|correct|ok|success(?:ful)?|valid|complete)$', + '^(?i)(?:this )?(?:matches|meets|satisfies|establishes|proves|confirms)(?: the)?(?: assertion| requirement| expectation)?$' + ) + foreach ($pattern in $genericPatterns) { if ($normalized -match $pattern) { return $true } } + return $false } - -function Assert-EvalGradingContract { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [string]$GradingPath = 'grading.json' - ) - - $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path - Assert-SafeRelativePath -RelativePath $GradingPath -FieldName 'grading path' - $gradingFullPath = Resolve-ContainedPath -BasePath $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File - if (-not (Test-Path -LiteralPath $gradingFullPath -PathType Leaf)) { - throw "Grading is incomplete: grading-only artifact '$GradingPath' is missing." - } - - $gradingDocument = Read-RunnerJson -Path $gradingFullPath - $schemas = Get-RunnerSchemaNames - if ([string](Get-JsonProperty -Object $gradingDocument -Name 'schema' -Default '') -ne $schemas.Grading) { - throw "grading.json must declare '$($schemas.Grading)'." - } - $topLevelAllowed = @('schema', 'grading') - foreach ($name in @(Get-JsonPropertyNames -Object $gradingDocument)) { - if ($topLevelAllowed -notcontains $name) { throw "grading.json contains unsupported field '$name'; the Grader may author only grading entries." } - } - if (-not (Test-JsonProperty -Object $gradingDocument -Name 'grading')) { - throw "grading.json is missing 'grading'." - } - $submittedValue = Get-JsonProperty -Object $gradingDocument -Name 'grading' -Default $null - if ($null -eq $submittedValue -or $submittedValue -is [string] -or -not ($submittedValue -is [System.Collections.IEnumerable])) { - throw 'grading.json grading must be an array.' - } - $submitted = @($submittedValue) - foreach ($entry in $submitted) { - Assert-EvalGradingEntryShape -Entry $entry - } - - $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState - $manifest = $freezeValidation.Manifest - $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') - if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { - throw "grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." - } - - $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) - $expected = @{} - $canonicalByKey = @{} - foreach ($record in $records) { - $assertions = @(Get-EvalMetadataAssertions -Record $record) - $canonical = Read-RunnerJson -Path $record.ResultPath - if ([int]$canonical.eval_id -ne [int]$record.EvalId -or [string]$canonical.eval_name -ne [string]$record.EvalName -or [string]$canonical.configuration -ne [string]$record.Configuration) { - throw "Canonical result '$($record.ResultRelative)' does not match its exact manifest identity." - } - if ([string]$canonical.execution_status -ne 'completed') { - throw "Grading is incomplete: '$($record.EvalName)/$($record.Configuration)' is not a completed execution." - } - $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] = $canonical - for ($index = 0; $index -lt $assertions.Count; $index++) { - $key = "$($record.EvalId)|$($record.Configuration)|$index" - $expected[$key] = [ordered]@{ - eval_id = [int]$record.EvalId - eval_name = [string]$record.EvalName - configuration = [string]$record.Configuration - assertion_index = $index - assertion = [string]$assertions[$index] - } - } - } - if ($submitted.Count -ne $expected.Count) { - throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." - } - + +function Assert-EvalPassEvidence { + param([object]$Entry, [object]$Canonical, [object]$Expected) + [void](Test-GradeEvidenceReference -Grade $Entry -Expected $Expected -Canonical $Canonical) + if (-not $Entry.passed) { return } + $reason = [string](Get-JsonProperty -Object $Entry -Name 'reason' -Default '') + if (Test-GenericGradingReason -Reason $reason -Assertion ([string]$Entry.assertion)) { + throw 'PASS evidence must explain how the cited observation establishes this assertion, not restate that it passed or was evaluated.' + } +} + +function Assert-EvalGradingContract { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' + ) + + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + Assert-SafeRelativePath -RelativePath $GradingPath -FieldName 'grading path' + $gradingFullPath = Resolve-ContainedPath -BasePath $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File + if (-not (Test-Path -LiteralPath $gradingFullPath -PathType Leaf)) { + throw "Grading is incomplete: grading-only artifact '$GradingPath' is missing." + } + + $gradingDocument = Read-RunnerJson -Path $gradingFullPath + $schemas = Get-RunnerSchemaNames + if ([string](Get-JsonProperty -Object $gradingDocument -Name 'schema' -Default '') -ne $schemas.Grading) { + throw "grading.json must declare '$($schemas.Grading)'." + } + $topLevelAllowed = @('schema', 'grading', 'metadata') + foreach ($name in @(Get-JsonPropertyNames -Object $gradingDocument)) { + if ($topLevelAllowed -notcontains $name) { throw "grading.json contains unsupported field '$name'; the Grader may author only grading entries." } + } + if (-not (Test-JsonProperty -Object $gradingDocument -Name 'grading')) { + throw "grading.json is missing 'grading'." + } + $submittedValue = Get-JsonProperty -Object $gradingDocument -Name 'grading' -Default $null + if ($null -eq $submittedValue -or $submittedValue -is [string] -or -not ($submittedValue -is [System.Collections.IEnumerable])) { + throw 'grading.json grading must be an array.' + } + $submitted = @($submittedValue) + foreach ($entry in $submitted) { + Assert-EvalGradingEntryShape -Entry $entry + } + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + $manifest = $freezeValidation.Manifest + $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') + if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { + throw "grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." + } + + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) + $expected = @{} + $canonicalByKey = @{} + foreach ($record in $records) { + $assertions = @(Get-EvalMetadataAssertions -Record $record) + $canonical = Read-RunnerJson -Path $record.ResultPath + if ([int]$canonical.eval_id -ne [int]$record.EvalId -or [string]$canonical.eval_name -ne [string]$record.EvalName -or [string]$canonical.configuration -ne [string]$record.Configuration) { + throw "Canonical result '$($record.ResultRelative)' does not match its exact manifest identity." + } + if ([string]$canonical.execution_status -ne 'completed') { + throw "Grading is incomplete: '$($record.EvalName)/$($record.Configuration)' is not a completed execution." + } + $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] = $canonical + for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertion = $assertions[$index] + $key = "$($record.EvalId)|$($record.Configuration)|$index" + $expected[$key] = [ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + } + } + } + if ($submitted.Count -ne $expected.Count) { + throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." + } + $validated = @{} $passEvidence = @{} - foreach ($entry in $submitted) { - Assert-EvalGradingEntryShape -Entry $entry - $key = Get-EvalGradingEntryKey -Entry $entry - if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } - if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } - $target = $expected[$key] - if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { - throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." + foreach ($entry in $submitted) { + Assert-EvalGradingEntryShape -Entry $entry + $key = Get-EvalGradingEntryKey -Entry $entry + if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } + if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } + $target = $expected[$key] + if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion -or [string]$entry.evidence_domain -ne [string]$target.evidence_domain) { + throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." } $armKey = "$($entry.eval_id)|$($entry.configuration)" $record = @($records | Where-Object { $_.EvalId -eq $entry.eval_id -and $_.Configuration -eq $entry.configuration })[0] - Assert-EvalPassEvidence -Entry $entry -Canonical $canonicalByKey[$armKey] -Record $record + Assert-EvalPassEvidence -Entry $entry -Canonical $canonicalByKey[$armKey] -Expected $target if ($entry.passed) { $evidenceKey = $armKey + '|' + ([regex]::Replace($entry.evidence.Trim(), '\s+', ' ')).ToLowerInvariant() if ($passEvidence.ContainsKey($evidenceKey)) { throw 'Repeated PASS evidence across assertions is not assertion-specific.' } $passEvidence[$evidenceKey] = $true } $validated[$key] = $entry - } - foreach ($key in $expected.Keys) { - if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } - } - - return [pscustomobject]@{ - IterationDirectory = $iteration - GradingPath = $GradingPath - GradingFullPath = $gradingFullPath - GradingDocument = $gradingDocument - FreezeValidation = $freezeValidation - Manifest = $manifest - Records = @($records) - Expected = $expected - Validated = $validated - CanonicalByKey = $canonicalByKey - GradedAssertions = $expected.Count - } -} + } + foreach ($key in $expected.Keys) { + if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } + } + + return [pscustomobject]@{ + IterationDirectory = $iteration + GradingPath = $GradingPath + GradingFullPath = $gradingFullPath + GradingDocument = $gradingDocument + FreezeValidation = $freezeValidation + Manifest = $manifest + Records = @($records) + Expected = $expected + Validated = $validated + CanonicalByKey = $canonicalByKey + GradedAssertions = $expected.Count + } +} diff --git a/scripts/eval-runners/finalize-eval-package.ps1 b/scripts/eval-runners/finalize-eval-package.ps1 index e041f8e..2396517 100644 --- a/scripts/eval-runners/finalize-eval-package.ps1 +++ b/scripts/eval-runners/finalize-eval-package.ps1 @@ -23,6 +23,7 @@ Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot 'orchestration.ps1') . (Join-Path $PSScriptRoot 'execution-freeze.ps1') . (Join-Path $PSScriptRoot 'package-integrity.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') function Invoke-FinalizerCommand { param( @@ -132,9 +133,11 @@ try { } [void](Invoke-FinalizerCommand -ScriptPath $manifestBridge -Arguments $bridgeArgs -Description 'Manifest bridge') [void](Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState) + $gradingFreeze = Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath $gradingScript = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/apply-eval-grading.ps1') -FieldName 'grading application helper' -Kind File -RequireExists [void](Invoke-FinalizerCommand -ScriptPath $gradingScript -Arguments @('-IterationDirectory', $iteration, '-GradingPath', $GradingPath) -Description 'Grading application') + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath) $gradedAssertions = Assert-FinalizerGrading -Records $records -IterationDirectory $iteration $reportRelative = [string](Get-JsonProperty -Object $manifest.report -Name 'tool' -Default 'tools/generate-eval-report.ps1') @@ -155,6 +158,9 @@ try { iteration = $iteration runner = $profile.Runner model = $profile.Model + analyzer_runner = [string]$gradingFreeze.Analyzer.Runner + analyzer_model = [string]$gradingFreeze.Analyzer.Model + analyzer_profile_sha256 = [string]$gradingFreeze.Analyzer.Hash expected_arms = $records.Count completed_arms = $records.Count graded_assertions = $gradedAssertions diff --git a/scripts/eval-runners/invoke-phase2-analyzer.ps1 b/scripts/eval-runners/invoke-phase2-analyzer.ps1 new file mode 100644 index 0000000..5750835 --- /dev/null +++ b/scripts/eval-runners/invoke-phase2-analyzer.ps1 @@ -0,0 +1,645 @@ +<#! +.SYNOPSIS + Runs package-local Phase 2 grading with explicit analyzer provenance. + +.DESCRIPTION + This is the normal Phase 2 boundary after Phase 1 execution is frozen and + bridged. It resolves deterministic validator-domain assertions before any + analyzer worker starts, dispatches one fresh analyzer session per remaining + semantic arm, persists analyzer evidence and progress, freezes Phase 2, and + deterministically writes package-root grading.json. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [ValidateRange(1, 128)][int]$Concurrency = 16, + [ValidateRange(1, 86400)][int]$TimeoutSeconds = 900 +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'package-integrity.ps1') +. (Join-Path $PSScriptRoot 'fanout-process.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') + +function Write-Phase2Summary { + param([Parameter(Mandatory = $true)][object]$Value, [int]$ExitCode = 0) + + Write-RunnerJson -Value $Value -Compress -AsOutput + if ($ExitCode -ne 0) { exit $ExitCode } +} + +function Save-Phase2State { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$State) + + $State.updated_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + Write-RunnerJsonFile -Path $Path -Value $State +} + +function Resolve-AnalyzerRunner { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile + ) + + $runnerToolsDirectory = Resolve-PackageRunnerToolsDirectory -IterationDirectory $IterationDirectory -Manifest $Manifest + $resolverPath = Join-Path $runnerToolsDirectory 'resolve-runner.ps1' + if (-not (Test-Path -LiteralPath $resolverPath -PathType Leaf)) { throw "Package-local Eval Runner resolver is missing at '$resolverPath'." } + $resolutionOutput = & pwsh -NoProfile -NonInteractive -File $resolverPath ([string]$AnalyzerProfile.Runner) 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Analyzer runner '$($AnalyzerProfile.Runner)' could not be resolved package-locally: $([string]::Join(' ', @($resolutionOutput)))" } + $resolution = ([string]::Join([Environment]::NewLine, @($resolutionOutput | ForEach-Object { [string]$_ }))) | ConvertFrom-Json + $runnerRelative = [string](Get-JsonProperty -Object $resolution -Name 'path' -Default '') + $runnerPath = Resolve-ContainedPath -BasePath $runnerToolsDirectory -RelativePath $runnerRelative -FieldName 'resolved analyzer runner path' -Kind File + $descriptor = Get-PackageRunnerDescriptorFromPath -RunnerName ([string]$AnalyzerProfile.Runner) -RunnerPath $runnerPath -RunnerToolsDirectory $runnerToolsDirectory + if ([string](Get-JsonProperty -Object $descriptor.harness -Name 'name' -Default '') -ne [string]$AnalyzerProfile.Harness) { + throw "Analyzer profile harness '$($AnalyzerProfile.Harness)' does not match package-local runner descriptor '$($descriptor.harness.name)'." + } + $delegation = Get-JsonProperty -Object $descriptor -Name 'delegation' -Default $null + if ([string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default '') -ne 'runner') { + throw "Analyzer runner '$($AnalyzerProfile.Runner)' must provide runner-owned fresh analyzer sessions for attributable Phase 2 grading." + } + return [pscustomobject]@{ RunnerToolsDirectory = $runnerToolsDirectory; RunnerPath = $runnerPath; Descriptor = $descriptor; Resolution = $resolution } +} + +function New-AnalyzerExecutionProfile { + param([Parameter(Mandatory = $true)][object]$AnalyzerProfile) + + return [ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + reasoning_effort = $AnalyzerProfile.ReasoningEffort + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = $TimeoutSeconds + concurrency = $Concurrency + } +} + +function New-AnalyzerPrompt { + param( + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][string]$GraderContract + ) + + $bundleJson = ConvertTo-RunnerJson -Value $Bundle -Depth 100 + return @" +# Phase 2 semantic analyzer + +You are grading exactly one evaluation arm. Use the grader contract below and the input bundle below. Return only JSON. Do not inspect any repository, paired arm, sibling eval, previous grade, native skill, plugin, MCP server, or global configuration. + +## Response schema + +Return this exact JSON shape: + +````json +{ + "schema": "codebeltnet/agentic/eval-analyzer-fragment/1", + "eval_id": $($Bundle.eval_id), + "configuration": "$($Bundle.configuration)", + "grading": [ + { + "assertion_index": 0, + "passed": true, + "reason": "Specific explanation tied to the cited frozen evidence.", + "evidence_refs": [ + { + "artifact": "$($Bundle.canonical_result)", + "domain": "output", + "start_line": 1, + "end_line": 1, + "quote": "verbatim text from those lines" + } + ] + } + ] +} +```` + +Rules: +- Grade only the assertion indexes listed in the input bundle. +- Use output-domain refs only for output assertions and transcript-domain refs only for transcript assertions. +- Cite only artifacts listed in this one-arm bundle. +- A quote, when present, must be verbatim from the cited lines or event. +- Do not cite the paired arm or any sibling eval. +- Do not retry, ask for another model, or broaden the task. + +## Packaged grader contract + +````markdown +$GraderContract +```` + +## Input bundle + +````json +$bundleJson +```` +"@ +} + +function New-AnalyzerRunBundle { + param( + [Parameter(Mandatory = $true)][string]$Phase2Root, + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile, + [Parameter(Mandatory = $true)][string]$AnalyzerExecutionProfilePath, + [Parameter(Mandatory = $true)][string]$GraderContractText + ) + + $workerId = [string]$Worker.worker_id + $runRoot = Join-Path (Join-Path $Phase2Root 'work') $workerId + $repoRoot = Join-Path $runRoot 'repo' + $homeRoot = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null + $canonical = Read-RunnerJson -Path $Worker.record.ResultPath + $output = [string](Get-JsonProperty -Object $canonical -Name 'output' -Default '') + $outputLines = @(Get-ArmOutputLines -Output $output) + $lineRecords = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt $outputLines.Count; $index++) { + $lineRecords.Add([ordered]@{ line = $index + 1; text = [string]$outputLines[$index] }) + } + $metadata = Read-RunnerJson -Path $Worker.record.MetadataPath + $bundle = [ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-input/1' + worker_id = $workerId + eval_id = [int]$Worker.eval_id + eval_name = [string]$Worker.eval_name + configuration = [string]$Worker.configuration + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + expected_output = [string](Get-JsonProperty -Object $metadata -Name 'expected_output' -Default '') + canonical_result = [string]$Worker.record.ResultRelative + canonical_result_sha256 = Get-Sha256HexFromFile -Path $Worker.record.ResultPath + frozen_output = [ordered]@{ + artifact = [string]$Worker.record.ResultRelative + lines = @($lineRecords.ToArray()) + } + allowed_artifacts = @(([string]$Worker.record.ResultRelative) + @((Get-JsonProperty -Object $canonical -Name 'output_files' -Default @()) | ForEach-Object { [string]$_ })) + assertions = @($Worker.assertions | ForEach-Object { + [ordered]@{ + assertion_index = [int]$_.assertion_index + assertion = [string]$_.assertion + evidence_domain = [string]$_.evidence_domain + } + }) + } + $bundlePath = Join-Path $repoRoot 'input-bundle.json' + Write-RunnerJsonFile -Path $bundlePath -Value $bundle + $graderPath = Join-Path $repoRoot 'grader.md' + [System.IO.File]::WriteAllText($graderPath, $GraderContractText, [System.Text.UTF8Encoding]::new($false)) + $bundleHash = Get-Sha256HexFromFile -Path $bundlePath + $prompt = New-AnalyzerPrompt -Bundle $bundle -GraderContract $GraderContractText + [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), $prompt, [System.Text.UTF8Encoding]::new($false)) + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = [int]$Worker.eval_id + evalName = ('phase2-{0}' -f [string]$Worker.eval_name) + candidateSkillName = 'phase2-analyzer' + skillName = $null + iteration = 1 + mode = 'without_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @('input-bundle.json', 'grader.md') + fixtureHash = Get-JsonFingerprint -Object $bundle + skillHash = $null + candidateInstructionHash = $null + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + $runPath = Join-Path $runRoot 'run.json' + Write-RunnerJsonFile -Path $runPath -Value $run + return [pscustomobject]@{ + WorkerId = $workerId + RunRoot = $runRoot + RunPath = $runPath + BundlePath = $bundlePath + BundleHash = $bundleHash + PromptPath = Join-Path $runRoot 'prompt.md' + AnalyzerExecutionProfilePath = $AnalyzerExecutionProfilePath + } +} + +function Invoke-AnalyzerPreflight { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][string]$ProgressLogPath + ) + + $stdoutPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase2-preflight-' + [Guid]::NewGuid().ToString('N') + '.stdout') + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase2-preflight-' + [Guid]::NewGuid().ToString('N') + '.stderr') + $child = $null + try { + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $RunnerPath, 'preflight', '-Run', $Bundle.RunPath, '-Profile', $Bundle.AnalyzerExecutionProfilePath) -WorkingDirectory (Split-Path -Parent $Bundle.RunPath) -StdoutPath $stdoutPath -StderrPath $stderrPath -TimeoutSeconds (Get-RunnerPreflightTimeoutSeconds) -Runner 'phase2' -WorkerId $Bundle.WorkerId -EvalId $null -Configuration 'analyzer' -Phase 'preflight' -ProgressLogPath $ProgressLogPath + $exitCode = Complete-RunnerChildProcess -Child $child + $child = $null + $stdout = if (Test-Path -LiteralPath $stdoutPath -PathType Leaf) { [System.IO.File]::ReadAllText($stdoutPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } + return [pscustomobject]@{ ExitCode = $exitCode; Result = $result; Stdout = $stdout; Bundle = $Bundle } + } finally { + if ($null -ne $child) { try { [void](Complete-RunnerChildProcess -Child $child -TimeoutSeconds 1) } catch { } } + foreach ($path in @($stdoutPath, $stderrPath)) { Remove-Item -LiteralPath $path -Force -ErrorAction SilentlyContinue } + } +} + +function ConvertFrom-AnalyzerResponse { + param([Parameter(Mandatory = $true)][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { throw 'Analyzer returned an empty response.' } + try { + return $Text | ConvertFrom-Json -Depth 100 + } catch { + throw "Analyzer returned malformed JSON: $($_.Exception.Message)" + } +} + +function Confirm-AnalyzerFragment { + param( + [Parameter(Mandatory = $true)][object]$Fragment, + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$Canonical + ) + + if ([string](Get-JsonProperty -Object $Fragment -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-analyzer-fragment/1') { throw 'Analyzer fragment has an unsupported schema.' } + if ([int](Get-JsonProperty -Object $Fragment -Name 'eval_id' -Default 0) -ne [int]$Worker.eval_id -or [string](Get-JsonProperty -Object $Fragment -Name 'configuration' -Default '') -ne [string]$Worker.configuration) { + throw 'Analyzer fragment does not match the requested eval arm.' + } + $grades = @(Get-JsonProperty -Object $Fragment -Name 'grading' -Default @()) + $expectedByIndex = @{} + foreach ($assertion in @($Worker.assertions)) { $expectedByIndex[[int]$assertion.assertion_index] = $assertion } + if ($grades.Count -ne @($Worker.assertions).Count) { throw 'Analyzer fragment grade cardinality does not match unresolved semantic assertions.' } + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($grade in $grades) { + $index = [int](Get-JsonProperty -Object $grade -Name 'assertion_index' -Default -1) + if (-not $expectedByIndex.ContainsKey($index)) { throw "Analyzer fragment includes unexpected assertion_index '$index'." } + $expected = $expectedByIndex[$index] + if ((Get-JsonProperty -Object $grade -Name 'passed' -Default $null) -isnot [bool]) { throw 'Analyzer fragment passed must be boolean.' } + $reason = [string](Get-JsonProperty -Object $grade -Name 'reason' -Default '') + if ([string]::IsNullOrWhiteSpace($reason)) { throw 'Analyzer fragment reason must be non-empty.' } + $refs = @(Get-JsonProperty -Object $grade -Name 'evidence_refs' -Default @()) + $entry = ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$grade.passed) -Reason $reason -EvidenceRefs $refs -Source 'analyzer' -Evidence $reason + [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical) + $entries.Add($entry) + } + return @($entries.ToArray()) +} + +function Get-ObservedAnalyzerModel { + param([Parameter(Mandatory = $true)][object]$Raw) + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + $observed = [string](Get-JsonProperty -Object $delegation -Name 'observed_model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($observed)) { return $observed } + $resolved = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'resolved' -Default $null) -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($resolved)) { return $resolved } + return '' +} + +function New-AnalyzerResult { + param( + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][object]$Raw, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile, + [Parameter(Mandatory = $true)][string]$RawRelative, + [Parameter(Mandatory = $true)][string]$FragmentRelative, + [Parameter(Mandatory = $true)][object[]]$Grades, + [string]$Status = 'completed', + [string]$Failure = '' + ) + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + $tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'tokens' -Default $null + $toolCalls = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'tool_calls' -Default $null + $cost = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'cost' -Default $null + $rawPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $RawRelative -FieldName 'analyzer raw execution result' -Kind File -RequireExists + $fragmentPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $FragmentRelative -FieldName 'analyzer grading fragment' -Kind File -RequireExists + return [ordered]@{ + schema = $script:AnalyzerResultSchema + worker_id = [string]$Worker.worker_id + eval_id = [int]$Worker.eval_id + eval_name = [string]$Worker.eval_name + configuration = [string]$Worker.configuration + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + requested_runner = [string]$AnalyzerProfile.Runner + requested_model = [string]$AnalyzerProfile.Model + requested_reasoning_effort = $AnalyzerProfile.ReasoningEffort + resolved_runner = [string](Get-JsonProperty -Object $Raw.runner -Name 'name' -Default '') + resolved_model = [string](Get-JsonProperty -Object $Raw.requested -Name 'model' -Default '') + observed_model = Get-ObservedAnalyzerModel -Raw $Raw + harness_name = [string](Get-JsonProperty -Object $Raw.harness -Name 'name' -Default '') + harness_version = [string](Get-JsonProperty -Object $Raw.harness -Name 'version' -Default '') + session_id = [string](Get-JsonProperty -Object $delegation -Name 'worker_session_id' -Default (Get-JsonProperty -Object $Raw.session -Name 'id' -Default '')) + started_utc = [string](Get-JsonProperty -Object $Raw -Name 'started_utc' -Default '') + finished_utc = [string](Get-JsonProperty -Object $Raw -Name 'finished_utc' -Default '') + duration_ms = [int64]([double](Get-JsonProperty -Object $Raw -Name 'duration_seconds' -Default 0) * 1000) + terminal_status = $Status + failure = $Failure + input_bundle_sha256 = [string]$Bundle.BundleHash + grader_contract_sha256 = Get-Sha256HexFromFile -Path (Join-Path (Split-Path -Parent $Bundle.BundlePath) 'grader.md') + assertions_sha256 = Get-JsonFingerprint -Object @($Worker.assertions) + raw_execution_result = $RawRelative + raw_execution_result_sha256 = Get-Sha256HexFromFile -Path $rawPath + grading_fragment = $FragmentRelative + grading_fragment_sha256 = Get-Sha256HexFromFile -Path $fragmentPath + parsed_grading = @($Grades) + input_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'input_tokens' -Default $null + output_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'output_tokens' -Default $null + cache_read_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'cached_input_tokens' -Default $null + cache_write_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'cache_write_tokens' -Default $null + total_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'total_tokens' -Default $null + cost = Get-JsonProperty -Object $cost -Name 'value' -Default $null + tool_calls = Get-JsonProperty -Object $toolCalls -Name 'value' -Default $null + attempt_count = 1 + } +} + +function Get-RootGradingMetadata { + param([Parameter(Mandatory = $true)][object]$AnalyzerProfile) + + return [ordered]@{ + schema = 'codebeltnet/agentic/eval-grading-metadata/1' + generated_by = 'phase2-controller' + analyzer_runner = [string]$AnalyzerProfile.Runner + analyzer_model = [string]$AnalyzerProfile.Model + analyzer_reasoning_effort = $AnalyzerProfile.ReasoningEffort + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + phase2_state = 'phase2-state.json' + grading_freeze = 'grading-freeze.json' + } +} + +try { + $manifestPath = Join-Path $iteration 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { throw 'manifest.json is missing.' } + $manifest = Read-RunnerJson -Path $manifestPath + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iteration -Manifest $manifest) + $executionIdentity = Assert-PackageRunnerIdentity -IterationDirectory $iteration -Manifest $manifest + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + [void](Assert-FanoutPhase1Success -Aggregate $freezeValidation.Aggregate -MessagePrefix 'Phase 2 Phase 1') + $bridgePath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/bridge-manifest-results.ps1') -FieldName 'manifest bridge' -Kind File -RequireExists + $bridgeArgs = @('-IterationDirectory', $iteration, '-RequireComplete', '-RequireParallelDispatch') + if ([string](Get-JsonProperty -Object $executionIdentity.Descriptor.delegation -Name 'dispatch_owner' -Default '') -eq 'runner') { $bridgeArgs += '-RequireNativeDelegation' } + $bridgeOutput = & pwsh -NoProfile -NonInteractive -File $bridgePath @bridgeArgs 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Manifest bridge failed before Phase 2: $([string]::Join(' ', @($bridgeOutput)))" } + + $statePath = Join-Path $iteration 'phase2-state.json' + $freezePath = Join-Path $iteration 'grading-freeze.json' + if (Test-Path -LiteralPath $freezePath -PathType Leaf) { + $validatedFreeze = Assert-GradingFreeze -IterationDirectory $iteration + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'already_frozen' + analyzer_profile_sha256 = [string]$validatedFreeze.Analyzer.Hash + grading_freeze = 'grading-freeze.json' + grading = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default 'grading.json') + }) + } + if (Test-Path -LiteralPath $statePath -PathType Leaf) { + throw 'Phase 2 state already exists without a valid grading freeze; refusing to duplicate analyzer work. Use a fresh package iteration.' + } + + $analyzerProfile = Resolve-AnalyzerProfile -IterationDirectory $iteration -Manifest $manifest + $analyzerRunner = Resolve-AnalyzerRunner -IterationDirectory $iteration -Manifest $manifest -AnalyzerProfile $analyzerProfile + $phase2Root = Join-Path $iteration 'phase2' + $validatorRoot = Join-Path $phase2Root 'validators' + $resultRoot = Join-Path $phase2Root 'results' + $fragmentRoot = Join-Path $phase2Root 'fragments' + $progressRoot = Join-Path $iteration 'progress' + New-Item -ItemType Directory -Path $phase2Root, $validatorRoot, $resultRoot, $fragmentRoot, $progressRoot -Force | Out-Null + $progressLogPath = Join-Path $progressRoot 'phase2-progress.jsonl' + + $analyzerExecutionProfilePath = Join-Path $phase2Root 'analyzer-execution-profile.json' + Write-RunnerJsonFile -Path $analyzerExecutionProfilePath -Value (New-AnalyzerExecutionProfile -AnalyzerProfile $analyzerProfile) + $analyzerExecutionProfile = Resolve-ExecutionProfile -ProfilePath $analyzerExecutionProfilePath + $graderContractPath = Get-GraderContractPath -IterationDirectory $iteration + $graderContractText = [System.IO.File]::ReadAllText($graderContractPath, [System.Text.UTF8Encoding]::new($false)) + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) + $semanticWorkers = [System.Collections.Generic.List[object]]::new() + $validatorEntries = [System.Collections.Generic.List[object]]::new() + $grades = [System.Collections.Generic.List[object]]::new() + + foreach ($record in $records) { + $canonical = Read-RunnerJson -Path $record.ResultPath + $semanticAssertions = [System.Collections.Generic.List[object]]::new() + $assertionIndex = 0 + foreach ($assertion in @(Get-EvalMetadataAssertionObjects -Record $record)) { + $expected = [ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $assertionIndex + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + } + if ([string]$assertion.evidence_domain -eq 'validator') { + $validatorResult = Invoke-GradingValidatorRule -Rule ([string]$assertion.validator) -Canonical $canonical -Record $record + $validatorRelative = ('phase2/validators/{0}-{1}-{2}.json' -f [int]$record.EvalId, [string]$record.Configuration, $assertionIndex) + $validatorPath = Join-Path $iteration ($validatorRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $validatorPath -Value $validatorResult + $validatorEntries.Add([ordered]@{ + path = $validatorRelative + sha256 = Get-Sha256HexFromFile -Path $validatorPath + status = if ([bool]$validatorResult.passed) { 'passed' } else { 'failed' } + rule = [string]$assertion.validator + eval_id = [int]$record.EvalId + configuration = [string]$record.Configuration + assertion_index = $assertionIndex + }) + $grades.Add((ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$validatorResult.passed) -Reason ([string]$validatorResult.reason) -EvidenceRefs @($validatorResult.evidence_refs) -Source 'validator' -Evidence ([string]$validatorResult.reason))) + } else { + $semanticAssertions.Add($expected) + } + $assertionIndex++ + } + if ($semanticAssertions.Count -gt 0) { + $semanticWorkers.Add([ordered]@{ + worker_id = Get-ArmKey -EvalId ([int]$record.EvalId) -Configuration ([string]$record.Configuration) + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + record = $record + assertions = @($semanticAssertions.ToArray()) + }) + } + } + + $state = [ordered]@{ + schema = $script:Phase2Schema + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + analyzer_runner = [string]$analyzerProfile.Runner + analyzer_model = [string]$analyzerProfile.Model + expected_worker_ids = @($semanticWorkers | ForEach-Object { [string]$_.worker_id }) + pending = @($semanticWorkers | ForEach-Object { [string]$_.worker_id }) + active = [ordered]@{} + completed = [ordered]@{} + started_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + updated_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + concurrency = $Concurrency + validator_results = @($validatorEntries.ToArray()) + analyzer_results = @() + grading_freeze = $null + status = 'running' + } + Save-Phase2State -Path $statePath -State $state + + $bundles = @{} + foreach ($worker in @($semanticWorkers)) { + $bundle = New-AnalyzerRunBundle -Phase2Root $phase2Root -Worker $worker -AnalyzerProfile $analyzerProfile -AnalyzerExecutionProfilePath $analyzerExecutionProfile.Path -GraderContractText $graderContractText + $bundles[$worker.worker_id] = $bundle + } + + $preflightFailures = [System.Collections.Generic.List[string]]::new() + foreach ($worker in @($semanticWorkers)) { + $preflight = Invoke-AnalyzerPreflight -RunnerPath $analyzerRunner.RunnerPath -Bundle $bundles[$worker.worker_id] -ProgressLogPath $progressLogPath + if ([int]$preflight.ExitCode -ne 0 -or $null -eq $preflight.Result -or [string](Get-JsonProperty -Object $preflight.Result -Name 'status' -Default '') -ne 'compatible') { + $preflightFailures.Add([string]$worker.worker_id) + } + } + if ($preflightFailures.Count -gt 0) { + $state.status = 'failed' + Save-Phase2State -Path $statePath -State $state + throw "Phase 2 analyzer preflight failed before execution for: $($preflightFailures -join ', ')." + } + + $executorSessions = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($entry in @($freezeValidation.Freeze.executions)) { + $session = [string](Get-JsonProperty -Object $entry -Name 'worker_session_id' -Default (Get-JsonProperty -Object $entry -Name 'thread_id' -Default '')) + if (-not [string]::IsNullOrWhiteSpace($session)) { [void]$executorSessions.Add($session) } + } + + $running = [System.Collections.Generic.List[object]]::new() + $completedAnalyzerEntries = [System.Collections.Generic.List[object]]::new() + $failed = [System.Collections.Generic.List[string]]::new() + $workersById = @{} + foreach ($worker in @($semanticWorkers)) { $workersById[[string]$worker.worker_id] = $worker } + + while (@($state.pending).Count -gt 0 -or $running.Count -gt 0) { + while (@($state.pending).Count -gt 0 -and $running.Count -lt $Concurrency) { + $workerId = [string]@($state.pending)[0] + $state.pending = @(@($state.pending) | Where-Object { [string]$_ -ne $workerId }) + $state.active[$workerId] = [ordered]@{ worker_id = $workerId; accepted_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); attempt_count = 1 } + Save-Phase2State -Path $statePath -State $state + $bundle = $bundles[$workerId] + $rawRelative = "phase2/results/$workerId.execution-result.json" + $stderrRelative = "phase2/results/$workerId.stderr.txt" + $rawPath = Join-Path $iteration ($rawRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $stderrPath = Join-Path $iteration ($stderrRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $analyzerRunner.RunnerPath, 'execute', '-Run', $bundle.RunPath, '-Profile', $analyzerExecutionProfile.Path) -WorkingDirectory (Split-Path -Parent $bundle.RunPath) -StdoutPath $rawPath -StderrPath $stderrPath -TimeoutSeconds $TimeoutSeconds -Runner ([string]$analyzerProfile.Runner) -WorkerId $workerId -EvalId $workersById[$workerId].eval_id -Configuration $workersById[$workerId].configuration -Phase 'phase2-analyzer' -ProgressLogPath $progressLogPath + $running.Add([pscustomobject]@{ worker_id = $workerId; child = $child; Process = $child.Process; raw_relative = $rawRelative; stderr_relative = $stderrRelative; bundle = $bundle }) + [Console]::Error.WriteLine(("Phase 2: {0}/{1} analyzer workers terminal; active: {2}; analyzer: {3}/{4}" -f @($state.completed.Keys).Count, @($state.expected_worker_ids).Count, $workerId, [string]$analyzerProfile.Runner, [string]$analyzerProfile.Model)) + } + if ($running.Count -eq 0) { break } + $completedIndex = Wait-AnyRunnerChild -Running $running + $item = $running[$completedIndex] + $running.RemoveAt($completedIndex) + $exitCode = Complete-RunnerChildProcess -Child $item.child + $workerId = [string]$item.worker_id + $worker = $workersById[$workerId] + $rawPath = Join-Path $iteration ([string]$item.raw_relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $terminalStatus = 'completed' + $terminalFailure = '' + $workerGrades = @() + try { + if ($null -eq $exitCode -or [int]$exitCode -ne 0) { throw "Analyzer runner exited with code '$exitCode'." } + $raw = Read-RunnerJson -Path $rawPath + [void](Assert-ExecutionResult -Result $raw) + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $raw -Run (Resolve-RunContract -RunPath $item.bundle.RunPath) -RequestedModel ([string]$analyzerProfile.Model) -ExpectedRunner ([string]$analyzerProfile.Runner) -ExpectedMechanism ([string]$analyzerRunner.Descriptor.delegation.mechanism)) + if ([string]$raw.status -ne 'completed') { throw "Analyzer runner terminal status '$($raw.status)' is not completed." } + $observedModel = Get-ObservedAnalyzerModel -Raw $raw + if ([string]::IsNullOrWhiteSpace($observedModel) -or $observedModel -ne [string]$analyzerProfile.Model) { throw "Analyzer observed model '$observedModel' does not match requested '$($analyzerProfile.Model)'." } + $sessionId = [string](Get-JsonProperty -Object $raw.session -Name 'id' -Default '') + if ([string]::IsNullOrWhiteSpace($sessionId)) { throw 'Analyzer result did not expose a session id.' } + if ($executorSessions.Contains($sessionId)) { throw 'Analyzer session reused an executor session identity.' } + if (@($completedAnalyzerEntries | Where-Object { [string]$_.session_id -eq $sessionId }).Count -gt 0) { throw 'Analyzer worker session was reused across arms.' } + $responseText = [string](Get-JsonProperty -Object $raw.final_response -Name 'text' -Default '') + $fragment = ConvertFrom-AnalyzerResponse -Text $responseText + $canonical = Read-RunnerJson -Path $worker.record.ResultPath + $workerGrades = @(Confirm-AnalyzerFragment -Fragment $fragment -Worker $worker -Canonical $canonical) + $fragmentRelative = "phase2/fragments/$workerId.grading-fragment.json" + $fragmentPath = Join-Path $iteration ($fragmentRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $fragmentPath -Value $fragment + foreach ($grade in $workerGrades) { $grades.Add($grade) } + $analyzerResult = New-AnalyzerResult -Worker $worker -Bundle $item.bundle -Raw $raw -AnalyzerProfile $analyzerProfile -RawRelative ([string]$item.raw_relative) -FragmentRelative $fragmentRelative -Grades $workerGrades + $resultRelative = "phase2/results/$workerId.analyzer-result.json" + $resultPath = Join-Path $iteration ($resultRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $resultPath -Value $analyzerResult + $completedAnalyzerEntries.Add([ordered]@{ path = $resultRelative; sha256 = Get-Sha256HexFromFile -Path $resultPath; status = 'completed'; worker_id = $workerId; session_id = $sessionId }) + $state.completed[$workerId] = [ordered]@{ worker_id = $workerId; status = 'completed'; analyzer_result = $resultRelative; analyzer_result_sha256 = Get-Sha256HexFromFile -Path $resultPath; terminal_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); worker_session_id = $sessionId; attempt_count = 1 } + } catch { + $terminalStatus = 'failed' + $terminalFailure = $_.Exception.Message + $failed.Add("$workerId`: $terminalFailure") + $state.completed[$workerId] = [ordered]@{ worker_id = $workerId; status = 'failed'; terminal_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); failure = $terminalFailure; attempt_count = 1 } + } finally { + if ($state.active.Contains($workerId)) { $state.active.Remove($workerId) } + $state.analyzer_results = @($completedAnalyzerEntries.ToArray()) + if ($terminalStatus -eq 'failed') { $state.status = 'failed' } + Save-Phase2State -Path $statePath -State $state + [Console]::Error.WriteLine(("Phase 2: {0}/{1} analyzer workers terminal; active: {2}; analyzer: {3}/{4}" -f @($state.completed.Keys).Count, @($state.expected_worker_ids).Count, ([string]::Join(',', @($state.active.Keys))), [string]$analyzerProfile.Runner, [string]$analyzerProfile.Model)) + } + } + + if ($failed.Count -gt 0) { + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'failed' + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + validator_results = @($validatorEntries).Count + analyzer_workers = @($semanticWorkers).Count + failed = @($failed.ToArray()) + phase2_state = 'phase2-state.json' + }) -ExitCode 2 + } + + $rootMetadata = Get-RootGradingMetadata -AnalyzerProfile $analyzerProfile + $rootGrading = New-RootGradingDocument -Grades @($grades.ToArray()) -Metadata $rootMetadata + $gradingRelative = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default 'grading.json') + $gradingPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $gradingRelative -FieldName 'grading path' -Kind File + Write-RunnerJsonFile -Path $gradingPath -Value $rootGrading + $state.analyzer_results = @($completedAnalyzerEntries.ToArray()) + $state.status = 'completed' + Save-Phase2State -Path $statePath -State $state + $freeze = New-GradingFreezeDocument -IterationDirectory $iteration -Manifest $manifest -State $state -Grades @($grades.ToArray()) -Metadata $rootMetadata + Write-RunnerJsonFile -Path $freezePath -Value $freeze + $state.grading_freeze = [ordered]@{ path = 'grading-freeze.json'; sha256 = Get-Sha256HexFromFile -Path $freezePath } + Write-RunnerJsonFile -Path $statePath -Value $state + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $gradingRelative) + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'completed' + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + validator_results = @($validatorEntries).Count + analyzer_workers = @($semanticWorkers).Count + grading = $gradingRelative + grading_sha256 = Get-Sha256HexFromFile -Path $gradingPath + grading_freeze = 'grading-freeze.json' + phase2_state = 'phase2-state.json' + progress_log = 'progress/phase2-progress.jsonl' + }) +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/phase2-grading.ps1 b/scripts/eval-runners/phase2-grading.ps1 new file mode 100644 index 0000000..a94f435 --- /dev/null +++ b/scripts/eval-runners/phase2-grading.ps1 @@ -0,0 +1,515 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'runner-common.ps1') +} +if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +} +if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'execution-freeze.ps1') +} + +$script:Phase2Schema = 'codebeltnet/agentic/eval-phase2-state/1' +$script:GradingFreezeSchema = 'codebeltnet/agentic/eval-grading-freeze/1' +$script:AnalyzerProfileSchema = 'codebeltnet/agentic/eval-analyzer-profile/1' +$script:AnalyzerResultSchema = 'codebeltnet/agentic/eval-analyzer-result/1' +$script:ValidatorResultSchema = 'codebeltnet/agentic/eval-validator-result/1' + +function Get-Phase2SchemaNames { + return [ordered]@{ + Phase2State = $script:Phase2Schema + GradingFreeze = $script:GradingFreezeSchema + AnalyzerProfile = $script:AnalyzerProfileSchema + AnalyzerResult = $script:AnalyzerResultSchema + ValidatorResult = $script:ValidatorResultSchema + RootGrading = (Get-RunnerSchemaNames).Grading + } +} + +function Get-AssertionText { + param([Parameter(Mandatory = $true)][object]$Assertion) + + if ($Assertion -is [string]) { return [string]$Assertion } + $text = [string](Get-JsonProperty -Object $Assertion -Name 'assertion' -Default (Get-JsonProperty -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { throw 'Assertion text must be non-empty.' } + return $text +} + +function Normalize-EvalAssertion { + param( + [Parameter(Mandatory = $true)][object]$Assertion, + [Parameter(Mandatory = $true)][int]$Index + ) + + if ($Assertion -is [string]) { + $text = [string]$Assertion + if ([string]::IsNullOrWhiteSpace($text)) { throw "Assertion $Index is empty." } + return [ordered]@{ + assertion = $text + evidence_domain = 'output' + validator = $null + } + } + + $text = [string](Get-JsonProperty -Object $Assertion -Name 'assertion' -Default (Get-JsonProperty -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { throw "Assertion $Index is missing assertion text." } + $domain = [string](Get-JsonProperty -Object $Assertion -Name 'evidence_domain' -Default 'output') + if ($domain -notin @('output', 'transcript', 'validator')) { throw "Assertion $Index evidence_domain '$domain' is unsupported." } + $validator = Get-JsonProperty -Object $Assertion -Name 'validator' -Default $null + if ($domain -eq 'validator' -and [string]::IsNullOrWhiteSpace([string]$validator)) { throw "Assertion $Index with evidence_domain=validator must declare validator." } + if ($domain -ne 'validator' -and -not [string]::IsNullOrWhiteSpace([string]$validator)) { throw "Assertion $Index declares validator for non-validator evidence_domain '$domain'." } + + return [ordered]@{ + assertion = $text + evidence_domain = $domain + validator = if ($null -eq $validator -or [string]::IsNullOrWhiteSpace([string]$validator)) { $null } else { [string]$validator } + } +} + +function Get-NormalizedAssertions { + param([Parameter(Mandatory = $true)][object[]]$Assertions) + + $normalized = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt @($Assertions).Count; $index++) { + $normalized.Add((Normalize-EvalAssertion -Assertion $Assertions[$index] -Index $index)) + } + if ($normalized.Count -eq 0) { throw 'Each eval must declare at least one assertion.' } + return @($normalized.ToArray()) +} + +function Get-EvalMetadataAssertionObjects { + param([Parameter(Mandatory = $true)][object]$Record) + + $metadata = Read-RunnerJson -Path $Record.MetadataPath + $assertions = @(Get-JsonProperty -Object $metadata -Name 'assertions' -Default @()) + return @(Get-NormalizedAssertions -Assertions $assertions) +} + +function Get-ArmKey { + param([Parameter(Mandatory = $true)][int]$EvalId, [Parameter(Mandatory = $true)][string]$Configuration) + + return ('arm-{0}-{1}' -f $EvalId, $Configuration) +} + +function Get-ArmOutputLines { + param([Parameter(Mandatory = $true)][string]$Output) + + $lines = $Output -split "`r?`n", -1 + return @($lines) +} + +function New-OutputEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Output, + [Parameter(Mandatory = $true)][string]$Quote + ) + + $lines = @(Get-ArmOutputLines -Output $Output) + if ([string]::IsNullOrWhiteSpace($Quote)) { throw 'Output evidence quote must be non-empty.' } + for ($index = 0; $index -lt $lines.Count; $index++) { + $line = [string]$lines[$index] + if ($line.Contains($Quote, [StringComparison]::Ordinal)) { + return [ordered]@{ + artifact = [string]$Record.ResultRelative + domain = 'output' + start_line = $index + 1 + end_line = $index + 1 + quote = $Quote + } + } + } + throw 'Output evidence quote is absent from the frozen canonical output.' +} + +function Test-CommandText { + param([object]$Command) + + $text = [string](Get-JsonProperty -Object $Command -Name 'command' -Default (Get-JsonProperty -Object $Command -Name 'tool' -Default '')) + return $text +} + +function Get-NormalizedCommandEvidence { + param([Parameter(Mandatory = $true)][object]$Canonical) + + $commands = [System.Collections.Generic.List[object]]::new() + foreach ($command in @(Get-JsonProperty -Object $Canonical -Name 'shell_commands' -Default @())) { + $text = Test-CommandText -Command $command + if ([string]::IsNullOrWhiteSpace($text)) { continue } + $commands.Add([ordered]@{ + command = $text + status = Get-JsonProperty -Object $command -Name 'status' -Default $null + exit_code = Get-JsonProperty -Object $command -Name 'exit_code' -Default $null + }) + } + return @($commands.ToArray()) +} + +function Test-CommandMatches { + param( + [Parameter(Mandatory = $true)][object[]]$Commands, + [Parameter(Mandatory = $true)][string[]]$Patterns + ) + + foreach ($command in @($Commands)) { + $text = [string](Get-JsonProperty -Object $command -Name 'command' -Default '') + foreach ($pattern in @($Patterns)) { + if ($text -match $pattern) { return $command } + } + } + return $null +} + +function New-ValidatorEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Rule, + [Parameter(Mandatory = $true)][bool]$Passed, + [object]$Evidence = $null + ) + + return [ordered]@{ + artifact = [string]$Record.ResultRelative + domain = 'validator' + rule = $Rule + version = 1 + passed = $Passed + event = if ($null -eq $Evidence) { $null } else { Get-JsonProperty -Object $Evidence -Name 'command' -Default $null } + } +} + +function Invoke-GradingValidatorRule { + param( + [Parameter(Mandatory = $true)][string]$Rule, + [Parameter(Mandatory = $true)][object]$Canonical, + [Parameter(Mandatory = $true)][object]$Record + ) + + $commands = @(Get-NormalizedCommandEvidence -Canonical $Canonical) + $filesWritten = @(Get-JsonProperty -Object $Canonical -Name 'files_written' -Default @()) + $candidateSkill = 'dotnet-change-impact' + $matched = $null + $passed = $false + $reason = '' + switch ($Rule) { + 'git.current_branch_observed' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(branch\s+--show-current|rev-parse\s+--abbrev-ref\s+HEAD|status\b)') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event observed the current branch.' } else { 'No structured command event observed current-branch resolution.' } + } + 'git.default_branch_resolved' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(symbolic-ref\s+refs/remotes/origin/HEAD|remote\s+show\s+origin|merge-base\b|rev-parse\s+origin/(HEAD|main|master|trunk))') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event resolved a local base/default branch.' } else { 'No structured command event resolved a local base/default branch.' } + } + 'git.branch_commits_collected' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(log|rev-list)\b.*(\.\.|origin/HEAD|trunk|main|master)') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event collected branch-only commits.' } else { 'No structured command event collected branch-only commits.' } + } + 'git.three_dot_diff_observed' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\bdiff\b.*\.\.\.') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event collected a three-dot net diff.' } else { 'No structured command event collected a three-dot net diff.' } + } + 'skill.no_candidate_package_search' { + $matched = Test-CommandMatches -Commands $commands -Patterns @("(?i)\b(search|find|list)\b.*$([regex]::Escape($candidateSkill))", "(?i)\b(dotnet|npm|pip|winget|choco|brew)\b.*\b(search|list)\b.*$([regex]::Escape($candidateSkill))") + $passed = $null -eq $matched + $reason = if ($passed) { 'No structured command event searched for the candidate skill as a package or tool.' } else { 'A structured command event searched for the candidate skill as a package or tool.' } + } + 'skill.no_candidate_install_attempt' { + $matched = Test-CommandMatches -Commands $commands -Patterns @("(?i)\b(install|add)\b.*$([regex]::Escape($candidateSkill))", "(?i)\b(dotnet|npm|pip|winget|choco|brew)\b.*\b(install|add)\b.*$([regex]::Escape($candidateSkill))") + $passed = $null -eq $matched + $reason = if ($passed) { 'No structured command event attempted to install the candidate skill as a package or tool.' } else { 'A structured command event attempted to install the candidate skill as a package or tool.' } + } + 'workspace.no_unnecessary_mutation' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\b(dotnet\s+(add|new|tool\s+install|restore)|npm\s+install|pip\s+install|git\s+(commit|checkout|switch|merge|rebase|reset)|Set-Content|Out-File|New-Item|Remove-Item)\b') + $passed = ($null -eq $matched -and $filesWritten.Count -eq 0) + $reason = if ($passed) { 'No structured file-write evidence or mutation command was captured for this read-only classification task.' } else { 'Mutation evidence was captured for this read-only classification task.' } + } + default { throw "Unsupported validator rule '$Rule'." } + } + + return [ordered]@{ + schema = $script:ValidatorResultSchema + rule = $Rule + version = 1 + eval_id = [int]$Record.EvalId + eval_name = [string]$Record.EvalName + configuration = [string]$Record.Configuration + passed = [bool]$passed + reason = $reason + evidence_refs = @((New-ValidatorEvidenceRef -Record $Record -Rule $Rule -Passed ([bool]$passed) -Evidence $matched)) + } +} + +function Get-GraderContractPath { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $path = Join-Path $IterationDirectory 'tools/skill-creator/agents/grader.md' + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { throw "Packaged grader contract is missing at '$path'." } + return $path +} + +function Resolve-AnalyzerProfile { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $relative = [string](Get-JsonProperty -Object $Manifest -Name 'analyzer_profile' -Default 'analyzer-profile.json') + $path = Resolve-ManifestDeclaredPath -IterationDirectory $IterationDirectory -RelativePath $relative -FieldName 'analyzer_profile' -Kind File -RequireExists + $profile = Read-RunnerJson -Path $path + $schemas = Get-Phase2SchemaNames + if ([string](Get-JsonProperty -Object $profile -Name 'schema' -Default '') -ne $schemas.AnalyzerProfile) { throw 'analyzer-profile.json has an unsupported schema.' } + foreach ($name in @('contract_version', 'runner', 'harness', 'model', 'selection_source')) { + if ([string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $profile -Name $name -Default ''))) { throw "analyzer-profile.json must declare non-empty '$name'." } + } + if ([string](Get-JsonProperty -Object $profile -Name 'contract_version' -Default '') -ne $schemas.RootGrading) { throw 'analyzer-profile.json contract_version does not match the grading schema.' } + $source = [string](Get-JsonProperty -Object $profile -Name 'selection_source' -Default '') + if ($source -notin @('codebelt-reference', 'explicit')) { throw "analyzer-profile.json selection_source '$source' is unsupported." } + $runner = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') + if ($runner -notmatch '^[a-z0-9][a-z0-9-]*$') { throw 'analyzer-profile.json runner must be a safe lowercase runner id.' } + return [pscustomobject]@{ + Path = $path + RelativePath = $relative + Profile = $profile + Hash = Get-Sha256HexFromFile -Path $path + Runner = $runner + Model = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') + ReasoningEffort = Get-JsonProperty -Object $profile -Name 'reasoning_effort' -Default $null + Harness = [string](Get-JsonProperty -Object $profile -Name 'harness' -Default '') + SelectionSource = $source + } +} + +function Get-ExpectedGradeRecords { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest | Sort-Object EvalId, Configuration) + $expected = [System.Collections.Generic.List[object]]::new() + foreach ($record in $records) { + $assertions = @(Get-EvalMetadataAssertionObjects -Record $record) + for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertion = $assertions[$index] + $expected.Add([ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + }) + } + } + return @($expected.ToArray()) +} + +function ConvertTo-GradingEntry { + param( + [Parameter(Mandatory = $true)][object]$Expected, + [Parameter(Mandatory = $true)][bool]$Passed, + [Parameter(Mandatory = $true)][string]$Reason, + [Parameter(Mandatory = $true)][object[]]$EvidenceRefs, + [string]$Source = 'analyzer', + [string]$Evidence = '' + ) + + if ([string]::IsNullOrWhiteSpace($Evidence)) { + $Evidence = $Reason + } + return [ordered]@{ + eval_id = [int]$Expected.eval_id + eval_name = [string]$Expected.eval_name + configuration = [string]$Expected.configuration + assertion_index = [int]$Expected.assertion_index + assertion = [string]$Expected.assertion + passed = [bool]$Passed + evidence_domain = [string]$Expected.evidence_domain + evidence_refs = @($EvidenceRefs) + reason = $Reason + evidence = $Evidence + source = $Source + } +} + +function Get-GradeKey { + param([Parameter(Mandatory = $true)][object]$Grade) + + return ('{0}|{1}|{2}' -f [int](Get-JsonProperty -Object $Grade -Name 'eval_id' -Default 0), [string](Get-JsonProperty -Object $Grade -Name 'configuration' -Default ''), [int](Get-JsonProperty -Object $Grade -Name 'assertion_index' -Default -1)) +} + +function Test-GradeEvidenceReference { + param( + [Parameter(Mandatory = $true)][object]$Grade, + [Parameter(Mandatory = $true)][object]$Expected, + [Parameter(Mandatory = $true)][object]$Canonical + ) + + $domain = [string](Get-JsonProperty -Object $Grade -Name 'evidence_domain' -Default '') + if ($domain -ne [string]$Expected.evidence_domain) { throw "grade '$((Get-GradeKey -Grade $Grade))' evidence_domain does not match the assertion." } + $refs = @(Get-JsonProperty -Object $Grade -Name 'evidence_refs' -Default @()) + if ($refs.Count -eq 0) { throw "grade '$((Get-GradeKey -Grade $Grade))' must declare evidence_refs." } + foreach ($ref in $refs) { + $refDomain = [string](Get-JsonProperty -Object $ref -Name 'domain' -Default $domain) + if ($refDomain -ne $domain) { throw "grade '$((Get-GradeKey -Grade $Grade))' has a mismatched evidence ref domain." } + $artifact = [string](Get-JsonProperty -Object $ref -Name 'artifact' -Default '') + if ([string]::IsNullOrWhiteSpace($artifact)) { throw "grade '$((Get-GradeKey -Grade $Grade))' has an empty evidence artifact ref." } + if ($domain -eq 'output') { + if ($artifact -ne [string]$Expected.record.ResultRelative) { throw 'output evidence must cite this arm canonical result only.' } + $line = [int](Get-JsonProperty -Object $ref -Name 'start_line' -Default 0) + $endLine = [int](Get-JsonProperty -Object $ref -Name 'end_line' -Default $line) + $lines = @(Get-ArmOutputLines -Output ([string]$Canonical.output)) + if ($line -lt 1 -or $endLine -lt $line -or $endLine -gt $lines.Count) { throw 'output evidence line range is outside the frozen output.' } + $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $span = [string]::Join("`n", @($lines[($line - 1)..($endLine - 1)])) + if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { throw 'output evidence quote is absent from the referenced frozen output lines.' } + } + } elseif ($domain -eq 'validator') { + $rule = [string](Get-JsonProperty -Object $ref -Name 'rule' -Default '') + if ($rule -ne [string]$Expected.validator) { throw 'validator evidence must cite the assertion validator rule.' } + } elseif ($domain -eq 'transcript') { + if ($artifact -eq [string]$Expected.record.ResultRelative) { throw 'transcript evidence must not cite final output prose.' } + $allowed = @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @()) + if ($allowed -notcontains $artifact -and $artifact -ne [string]$Canonical.execution_result_file) { throw 'transcript evidence must cite a frozen transcript/artifact for the same arm.' } + } + } + return $true +} + +function New-RootGradingDocument { + param( + [Parameter(Mandatory = $true)][object[]]$Grades, + [object]$Metadata = $null + ) + + $ordered = @($Grades | Sort-Object @{ Expression = { [int](Get-JsonProperty -Object $_ -Name 'eval_id' -Default 0) } }, @{ Expression = { [string](Get-JsonProperty -Object $_ -Name 'configuration' -Default '') } }, @{ Expression = { [int](Get-JsonProperty -Object $_ -Name 'assertion_index' -Default 0) } }) + $document = [ordered]@{ + schema = (Get-RunnerSchemaNames).Grading + grading = @($ordered) + } + if ($null -ne $Metadata) { $document.metadata = $Metadata } + return $document +} + +function Get-GradingMergeHash { + param([Parameter(Mandatory = $true)][object]$Document) + + return Get-JsonFingerprint -Object $Document +} + +function Get-Phase2StateHashForFreeze { + param([Parameter(Mandatory = $true)][object]$State) + + return Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $State -PropertyName 'grading_freeze') +} + +function New-GradingFreezeDocument { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][object[]]$Grades, + [object]$Metadata = $null + ) + + $analyzer = Resolve-AnalyzerProfile -IterationDirectory $IterationDirectory -Manifest $Manifest + $graderPath = Get-GraderContractPath -IterationDirectory $IterationDirectory + $expected = @(Get-ExpectedGradeRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + $root = New-RootGradingDocument -Grades $Grades -Metadata $Metadata + $validatorPaths = @(Get-JsonProperty -Object $State -Name 'validator_results' -Default @()) + $analyzerResults = @(Get-JsonProperty -Object $State -Name 'analyzer_results' -Default @()) + + return [ordered]@{ + schema = $script:GradingFreezeSchema + analyzer_profile = [ordered]@{ path = [string]$analyzer.RelativePath; sha256 = [string]$analyzer.Hash } + analyzer_profile_sha256 = [string]$analyzer.Hash + grader_contract = [ordered]@{ path = 'tools/skill-creator/agents/grader.md'; sha256 = Get-Sha256HexFromFile -Path $graderPath } + assertions_sha256 = Get-JsonFingerprint -Object @($expected | ForEach-Object { + [ordered]@{ + eval_id = [int]$_.eval_id + eval_name = [string]$_.eval_name + configuration = [string]$_.configuration + assertion_index = [int]$_.assertion_index + assertion = [string]$_.assertion + evidence_domain = [string]$_.evidence_domain + validator = Get-JsonProperty -Object $_ -Name 'validator' -Default $null + } + }) + validator_results = @($validatorPaths) + analyzer_results = @($analyzerResults) + phase2_state_sha256 = Get-Phase2StateHashForFreeze -State $State + expected_grade_count = $expected.Count + grading_sha256 = Get-GradingMergeHash -Document $root + generated_utc = (Format-UtcTimestamp -Value ([DateTime]::UtcNow)) + } +} + +function Assert-GradingFreeze { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' + ) + + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifest = Read-RunnerJson -Path (Join-Path $iteration 'manifest.json') + $statePath = Join-Path $iteration 'phase2-state.json' + $freezePath = Join-Path $iteration 'grading-freeze.json' + if (-not (Test-Path -LiteralPath $freezePath -PathType Leaf)) { throw 'grading-freeze.json is missing; handcrafted grading.json cannot finalize.' } + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { throw 'Phase 2 state is missing.' } + $state = Read-RunnerJson -Path $statePath + $freeze = Read-RunnerJson -Path $freezePath + if ([string](Get-JsonProperty -Object $state -Name 'schema' -Default '') -ne $script:Phase2Schema) { throw 'phase2-state.json has an unsupported schema.' } + if ([string](Get-JsonProperty -Object $freeze -Name 'schema' -Default '') -ne $script:GradingFreezeSchema) { throw 'grading-freeze.json has an unsupported schema.' } + if ([string](Get-JsonProperty -Object $state -Name 'status' -Default '') -ne 'completed') { throw 'phase2-state.json is not completed.' } + if (@(Get-JsonProperty -Object $state -Name 'pending' -Default @()).Count -ne 0) { throw 'phase2-state.json still has pending analyzer workers.' } + if (@(Get-JsonPropertyNames -Object (Get-JsonProperty -Object $state -Name 'active' -Default ([ordered]@{}))).Count -ne 0) { throw 'phase2-state.json still has active analyzer workers.' } + $expectedWorkers = @(Get-JsonProperty -Object $state -Name 'expected_worker_ids' -Default @()) + $completedWorkers = Get-JsonProperty -Object $state -Name 'completed' -Default ([ordered]@{}) + if (@(Get-JsonPropertyNames -Object $completedWorkers).Count -ne $expectedWorkers.Count) { throw 'phase2-state.json terminal analyzer worker count does not match expected_worker_ids.' } + $analyzerResultRefs = @(Get-JsonProperty -Object $state -Name 'analyzer_results' -Default @()) + if ($analyzerResultRefs.Count -ne $expectedWorkers.Count) { throw 'phase2-state.json analyzer result count does not match expected semantic worker count.' } + $analyzer = Resolve-AnalyzerProfile -IterationDirectory $iteration -Manifest $manifest + if ([string](Get-JsonProperty -Object $freeze -Name 'analyzer_profile_sha256' -Default '') -ne [string]$analyzer.Hash) { throw 'Analyzer profile changed after grading.' } + if ([string](Get-JsonProperty -Object (Get-JsonProperty -Object $freeze -Name 'analyzer_profile' -Default $null) -Name 'sha256' -Default '') -ne [string]$analyzer.Hash) { throw 'grading-freeze.json analyzer_profile hash mismatch.' } + $graderPath = Get-GraderContractPath -IterationDirectory $iteration + if ([string](Get-JsonProperty -Object (Get-JsonProperty -Object $freeze -Name 'grader_contract' -Default $null) -Name 'sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $graderPath)) { throw 'Packaged grader contract changed after Phase 2 freeze.' } + $stateHash = Get-Phase2StateHashForFreeze -State $state + if ([string](Get-JsonProperty -Object $freeze -Name 'phase2_state_sha256' -Default '') -ne $stateHash) { throw 'phase2-state.json changed after Phase 2 freeze.' } + + $gradingFullPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File -RequireExists + $grading = Read-RunnerJson -Path $gradingFullPath + if ([string](Get-JsonProperty -Object $freeze -Name 'grading_sha256' -Default '') -ne (Get-GradingMergeHash -Document $grading)) { throw 'Root grading.json was changed after deterministic Phase 2 merge.' } + + foreach ($validator in @(Get-JsonProperty -Object $freeze -Name 'validator_results' -Default @())) { + $path = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$validator.path) -FieldName 'validator result' -Kind File -RequireExists + if ([string]$validator.sha256 -ne (Get-Sha256HexFromFile -Path $path)) { throw "Validator result '$($validator.path)' changed after Phase 2 freeze." } + } + $workerSessions = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($analyzerResult in @(Get-JsonProperty -Object $freeze -Name 'analyzer_results' -Default @())) { + $path = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$analyzerResult.path) -FieldName 'analyzer result' -Kind File -RequireExists + if ([string]$analyzerResult.sha256 -ne (Get-Sha256HexFromFile -Path $path)) { throw "Analyzer result '$($analyzerResult.path)' changed after Phase 2 freeze." } + $result = Read-RunnerJson -Path $path + if ([string](Get-JsonProperty -Object $result -Name 'analyzer_profile_sha256' -Default '') -ne [string]$analyzer.Hash) { throw "Analyzer result '$($analyzerResult.path)' uses a different analyzer profile." } + if ([string](Get-JsonProperty -Object $result -Name 'observed_model' -Default '') -ne [string]$analyzer.Model) { throw "Analyzer result '$($analyzerResult.path)' observed the wrong model." } + if ([int](Get-JsonProperty -Object $result -Name 'attempt_count' -Default 0) -ne 1) { throw "Analyzer worker '$($analyzerResult.path)' was retried." } + $sessionId = [string](Get-JsonProperty -Object $result -Name 'session_id' -Default '') + if ([string]::IsNullOrWhiteSpace($sessionId) -or -not $workerSessions.Add($sessionId)) { throw 'Analyzer worker/session identity was reused.' } + $rawPathValue = [string](Get-JsonProperty -Object $result -Name 'raw_execution_result' -Default '') + $rawPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $rawPathValue -FieldName 'analyzer raw execution result' -Kind File -RequireExists + if ([string](Get-JsonProperty -Object $result -Name 'raw_execution_result_sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $rawPath)) { throw "Analyzer raw transcript/evidence '$rawPathValue' changed after Phase 2 freeze." } + $fragmentPathValue = [string](Get-JsonProperty -Object $result -Name 'grading_fragment' -Default '') + $fragmentPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $fragmentPathValue -FieldName 'analyzer grading fragment' -Kind File -RequireExists + if ([string](Get-JsonProperty -Object $result -Name 'grading_fragment_sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $fragmentPath)) { throw "Analyzer grading fragment '$fragmentPathValue' changed after Phase 2 freeze." } + } + + return [pscustomobject]@{ Path = $freezePath; Freeze = $freeze; State = $state; Manifest = $manifest; Analyzer = $analyzer; Grading = $grading } +} diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 84eed8f..8b5081e 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -24,6 +24,11 @@ function Get-RunnerSchemaNames { Interaction = 'codebeltnet/agentic/eval-interaction/1' ExecutionFreeze = 'codebeltnet/agentic/eval-execution-freeze/1' Grading = 'codebeltnet/agentic/eval-grading/1' + AnalyzerProfile = 'codebeltnet/agentic/eval-analyzer-profile/1' + Phase2State = 'codebeltnet/agentic/eval-phase2-state/1' + GradingFreeze = 'codebeltnet/agentic/eval-grading-freeze/1' + AnalyzerResult = 'codebeltnet/agentic/eval-analyzer-result/1' + ValidatorResult = 'codebeltnet/agentic/eval-validator-result/1' } } diff --git a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 index 5edb3b1..abc708f 100644 --- a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 +++ b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 @@ -159,6 +159,37 @@ try { } } } + $phase2BundlePath = Join-Path $inputs.Run.WorkingDirectoryPath 'input-bundle.json' + if (Test-Path -LiteralPath $phase2BundlePath -PathType Leaf) { + $bundle = Read-RunnerJson -Path $phase2BundlePath + $fragmentGrades = [System.Collections.Generic.List[object]]::new() + foreach ($assertion in @(Get-JsonProperty -Object $bundle -Name 'assertions' -Default @())) { + $assertionIndex = [int](Get-JsonProperty -Object $assertion -Name 'assertion_index' -Default 0) + $domain = [string](Get-JsonProperty -Object $assertion -Name 'evidence_domain' -Default 'output') + $artifact = [string](Get-JsonProperty -Object $bundle -Name 'canonical_result' -Default '') + $line = @(Get-JsonProperty -Object (Get-JsonProperty -Object $bundle -Name 'frozen_output' -Default $null) -Name 'lines' -Default @() | Select-Object -First 1) + $lineNumber = if ($line.Count -eq 1) { [int](Get-JsonProperty -Object $line[0] -Name 'line' -Default 1) } else { 1 } + $quote = if ($line.Count -eq 1) { [string](Get-JsonProperty -Object $line[0] -Name 'text' -Default 'deterministic fixture response') } else { 'deterministic fixture response' } + $fragmentGrades.Add([ordered]@{ + assertion_index = $assertionIndex + passed = $true + reason = "The deterministic analyzer fixture cites frozen one-arm output for assertion $assertionIndex." + evidence_refs = @([ordered]@{ + artifact = $artifact + domain = $domain + start_line = $lineNumber + end_line = $lineNumber + quote = $quote + }) + }) + } + $fixtureFinalResponse = ConvertTo-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-fragment/1' + eval_id = [int](Get-JsonProperty -Object $bundle -Name 'eval_id' -Default 0) + configuration = [string](Get-JsonProperty -Object $bundle -Name 'configuration' -Default '') + grading = @($fragmentGrades.ToArray()) + }) -Compress + } $durationSeconds = [Math]::Round(($executeFinishUtc - $executeStartUtc).TotalSeconds, 3) $durationOverride = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_DURATION_SECONDS') $parsedDuration = 0.0 @@ -213,11 +244,18 @@ try { $eventsArtifact = New-ArtifactReference -Run $inputs.Run -Path 'evidence/fixture-events.jsonl' -Scope run -MediaType 'application/x-ndjson; charset=utf-8' $evidence = [ordered]@{ capture = [ordered]@{ source = 'harness_native_transport'; terminal = $true; worker_authored = $false } + commands = @( + [ordered]@{ type = 'command_execution'; command = 'git branch --show-current'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git symbolic-ref refs/remotes/origin/HEAD --short'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git log origin/HEAD..HEAD --oneline'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git diff origin/HEAD...HEAD'; status = 'completed'; exit_code = 0 } + ) + files_written = @() delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = 'deterministic-runner-owned-fixture' worker_session_id = $sessionId - observed_model = [string]$inputs.Profile.Model + observed_model = if ([string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_OBSERVED_MODEL'))) { [string]$inputs.Profile.Model } else { [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_OBSERVED_MODEL') } observed_working_directory = [string]$inputs.Run.WorkingDirectoryPath observed_home = [string]$inputs.Run.HomeDirectoryPath fresh_worker = $true diff --git a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 index faaef0d..5f9a789 100644 --- a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 +++ b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 @@ -12,9 +12,11 @@ $tokens = $null $parseErrors = $null $codexAst = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $scriptsRoot 'eval-runners\codex\runner.ps1'), [ref]$tokens, [ref]$parseErrors) if ($parseErrors.Count) { throw 'Codex runner did not parse for sanitized PATH projection.' } -$pathBuilder = @($codexAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Get-CodexSanitizedShellPath' }, $true)) -if ($pathBuilder.Count -ne 1) { throw 'Codex sanitized PATH builder was not found.' } -Invoke-Expression $pathBuilder[0].Extent.Text +foreach ($functionName in @('Join-CodexTargetPath', 'Get-CodexSanitizedShellPath')) { + $definition = @($codexAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $functionName }, $true)) + if ($definition.Count -ne 1) { throw "Codex function '$functionName' was not found." } + Invoke-Expression $definition[0].Extent.Text +} function New-SanitizedGitEnvironment { param([Parameter(Mandatory = $true)][object]$GitCommand) $gitDirectory = Split-Path -Parent ([string]$GitCommand.Source) @@ -43,10 +45,12 @@ function Invoke-SanitizedGit { try { $catalog = Join-Path $workspace 'models.json' [IO.File]::WriteAllText($catalog, '{"models":[{"id":"fixture-model"}]}') - $prompt = & (Join-Path $scriptsRoot 'prepare-skill-evals.ps1') -Skill dotnet-change-impact -Eval 9 -Runner github-copilot -Model fixture-model -ModelCatalogPath $catalog -OutputRoot $workspace -PassThru + $analyzerCatalog = Join-Path $workspace 'analyzer-models.json' + [IO.File]::WriteAllText($analyzerCatalog, '{"models":[{"id":"claude-haiku-4.5"}]}') + $prompt = & (Join-Path $scriptsRoot 'prepare-skill-evals.ps1') -Skill dotnet-change-impact -Eval 9 -Runner github-copilot -Model fixture-model -ModelCatalogPath $catalog -AnalyzerModelCatalogPath $analyzerCatalog -OutputRoot $workspace -PassThru $package = Split-Path -Parent $prompt $handoff = [IO.File]::ReadAllText($prompt) - Assert-True ($handoff.Contains('MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`')) 'Handoff must require the exact packaged grader before grading.' + Assert-True ($handoff.Contains('invoke-phase2-analyzer.ps1') -and $handoff.Contains('The Phase 2 controller, not this outer orchestrator')) 'Handoff must delegate grading to the package-local Phase 2 controller.' $manifest = Get-Content (Join-Path $package 'manifest.json') -Raw | ConvertFrom-Json $heads = @(); $diffs = @(); $refs = @() $gitCommand = Resolve-ExternalCommand -Name 'git' diff --git a/scripts/eval-runners/tests/test-codex-paths.ps1 b/scripts/eval-runners/tests/test-codex-paths.ps1 index 9e7835b..80e6ad7 100644 --- a/scripts/eval-runners/tests/test-codex-paths.ps1 +++ b/scripts/eval-runners/tests/test-codex-paths.ps1 @@ -10,7 +10,7 @@ $tokens = $null $parseErrors = $null $ast = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $PSScriptRoot '../codex/runner.ps1'), [ref]$tokens, [ref]$parseErrors) if ($parseErrors.Count) { throw 'Codex runner did not parse.' } -$names = @('ConvertTo-CodexConfigStringLiteral', 'Get-CodexSanitizedShellPath', 'Get-CodexShellEnvironmentPolicySet', 'Add-CodexShellEnvironmentPolicyConfigArguments', 'ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') +$names = @('ConvertTo-CodexConfigStringLiteral', 'Get-CodexSanitizedShellPath', 'Join-CodexTargetPath', 'Get-CodexShellEnvironmentPolicySet', 'Add-CodexShellEnvironmentPolicyConfigArguments', 'ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') foreach ($name in $names) { $definition = @($ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $name }, $true)) if ($definition.Count -ne 1) { throw "Missing unique function '$name'." } @@ -27,11 +27,15 @@ if (Test-CodexPathInsideComparableRoot $staged ($staged + '-other/FORMS.md')) { $windowsGit = 'C:\Program Files\Git\cmd' $windowsPath = Get-CodexSanitizedShellPath -Platform windows -GitDirectory $windowsGit -WindowsRoot 'C:\Windows' if ($windowsPath -cne "C:\Windows\System32;$windowsGit") { throw "Windows sanitized PATH was '$windowsPath'." } +if ((Join-CodexTargetPath -Root 'C:\Program Files' -Segments @('Git', 'cmd') -Platform windows) -cne 'C:\Program Files\Git\cmd') { throw 'Windows target path construction must not require a host Windows drive.' } if ($windowsPath -match [regex]::Escape('C:\host-only\bin')) { throw 'Windows sanitized PATH exposed an arbitrary host-only directory.' } $linuxPath = Get-CodexSanitizedShellPath -Platform linux -GitDirectory '/opt/git/bin' if ($linuxPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/git/bin') { throw "Linux sanitized PATH was '$linuxPath'." } +if ((Join-CodexTargetPath -Root '/usr/local' -Segments @('bin') -Platform linux) -cne '/usr/local/bin') { throw 'POSIX target path construction must not depend on the host filesystem provider.' } $macPath = Get-CodexSanitizedShellPath -Platform macos -GitDirectory '/usr/bin' if ($macPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin') { throw "macOS sanitized PATH should deduplicate /usr/bin but was '$macPath'." } +$shellEnvironment = Get-CodexShellEnvironmentPolicySet -Inputs ([ordered]@{}) -Platform windows -WindowsRoot 'C:\Windows' +if ([string]$shellEnvironment.ComSpec -cne 'C:\Windows\System32\cmd.exe') { throw "Windows ComSpec was '$($shellEnvironment.ComSpec)'." } $configArgs = [System.Collections.Generic.List[string]]::new() Add-CodexShellEnvironmentPolicyConfigArguments -Arguments $configArgs -ShellEnvironmentSet ([ordered]@{ PATH = $windowsPath; SystemRoot = 'C:\Windows'; ComSpec = 'C:\Windows\System32\cmd.exe'; PATHEXT = '.COM;.EXE;.BAT;.CMD' }) -SwitchName '--config' if (@($configArgs | Where-Object { $_ -eq 'shell_environment_policy.inherit=none' }).Count -ne 1) { throw 'Codex config args must keep shell environment inheritance disabled.' } diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index 84c4da5..041477e 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -184,14 +184,40 @@ function New-TestGradingDocument { $metadata = Read-TestJson -Path $record.MetadataPath $assertions = @($metadata.assertions) for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertionText = if ($assertions[$index] -is [string]) { [string]$assertions[$index] } elseif ($assertions[$index].PSObject.Properties.Name -contains 'assertion') { [string]$assertions[$index].assertion } else { [string]$assertions[$index] } + $domain = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'evidence_domain') { [string]$assertions[$index].evidence_domain } else { 'output' } + $validator = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'validator') { [string]$assertions[$index].validator } else { $null } + $output = [string](Read-TestJson -Path $record.ResultPath).output + $refs = if ($domain -eq 'validator') { + @([ordered]@{ + artifact = [string]$record.ResultRelative + domain = 'validator' + rule = $validator + version = 1 + passed = $true + event = 'deterministic fixture validator evidence' + }) + } else { + @([ordered]@{ + artifact = [string]$record.ResultRelative + domain = 'output' + start_line = 1 + end_line = 1 + quote = $output + }) + } $entries.Add([ordered]@{ eval_id = [int]$record.EvalId eval_name = [string]$record.EvalName configuration = [string]$record.Configuration assertion_index = $index - assertion = [string]$assertions[$index] + assertion = $assertionText passed = $true - evidence = "Source: output`nQuote: $((Read-TestJson -Path $record.ResultPath).output)`nReason: The captured fixture response establishes assertion $index for this deterministic transport case." + evidence_domain = $domain + evidence_refs = @($refs) + reason = "The captured fixture response establishes assertion $index for this deterministic transport case." + evidence = "The captured fixture response establishes assertion $index for this deterministic transport case." + source = 'analyzer' }) } } @@ -378,6 +404,9 @@ try { $fixtureDirectory = Join-Path $packageTools 'fixture' New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + $graderContractDirectory = Join-Path $iteration 'tools/skill-creator/agents' + New-Item -ItemType Directory -Path $graderContractDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $graderContractDirectory 'grader.md'), "# Deterministic grader contract`nGrade against frozen evidence only.`n", [System.Text.UTF8Encoding]::new($false)) $reportScript = Join-Path $iteration 'tools/test-report.ps1' $reportScriptText = @' @@ -410,14 +439,28 @@ for ($index = 0; $index -lt $count; $index++) { $evalName = if ($evalId -eq 2) { 'dotnet-strong-name-signing-confirmation' } else { 'integrity-eval-{0:d2}' -f $evalId } $evalDirectory = Join-Path $iteration $evalName New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - $assertion = if ($evalId -eq 2) { 'the protected operation is absent before confirmation and occurs only after the same-session confirmation turn' } else { 'the deterministic terminal response is captured' } + $assertions = if ($evalId -eq 2) { + @('the protected operation is absent before confirmation and occurs only after the same-session confirmation turn', 'the response contains the fixture completion content') + } elseif ($evalId -eq 3) { + @( + [ordered]@{ assertion = 'current Git branch was resolved'; evidence_domain = 'validator'; validator = 'git.current_branch_observed' } + [ordered]@{ assertion = 'base/default branch was resolved using local Git state'; evidence_domain = 'validator'; validator = 'git.default_branch_resolved' } + [ordered]@{ assertion = 'branch-only commits were collected'; evidence_domain = 'validator'; validator = 'git.branch_commits_collected' } + [ordered]@{ assertion = 'three-dot net diff was collected'; evidence_domain = 'validator'; validator = 'git.three_dot_diff_observed' } + [ordered]@{ assertion = 'no candidate package search was performed'; evidence_domain = 'validator'; validator = 'skill.no_candidate_package_search' } + [ordered]@{ assertion = 'no candidate install attempt was performed'; evidence_domain = 'validator'; validator = 'skill.no_candidate_install_attempt' } + [ordered]@{ assertion = 'no unnecessary project package or tool mutation was performed'; evidence_domain = 'validator'; validator = 'workspace.no_unnecessary_mutation' } + ) + } else { + @('the deterministic terminal response is captured', 'the response contains the fixture completion content') + } Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-metadata/1' eval_id = $evalId eval_name = $evalName prompt = "fixture prompt $evalId" expected_output = 'fixture output' - assertions = @($assertion, 'the response contains the fixture completion content') + assertions = @($assertions) }) $runs = [ordered]@{} foreach ($configuration in @('with_skill', 'without_skill')) { @@ -425,13 +468,23 @@ for ($index = 0; $index -lt $count; $index++) { $run = New-TestRun -IterationDirectory $iteration -EvalId $evalId -EvalName $evalName -Configuration $configuration -Interaction $interactionForRun $resultName = "$configuration.result.json" $executionName = "$configuration.execution-result.json" + $stubGrading = @($assertions | ForEach-Object { + [ordered]@{ + text = if ($_ -is [string]) { [string]$_ } elseif ($_.PSObject.Properties.Name -contains 'assertion') { [string]$_.assertion } else { [string]$_ } + passed = $null + evidence = '' + evidence_domain = if ($_ -isnot [string] -and $_.PSObject.Properties.Name -contains 'evidence_domain') { [string]$_.evidence_domain } else { 'output' } + evidence_refs = @() + reason = '' + } + }) Write-TestJson -Path (Join-Path $evalDirectory "results/$resultName") -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).PortableResult eval_id = $evalId eval_name = $evalName configuration = $configuration execution_status = 'unrun' - grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }, [ordered]@{ text = 'the response contains the fixture completion content'; passed = $null; evidence = '' }) + grading = @($stubGrading) }) $runs[$configuration] = [ordered]@{ mode = $configuration @@ -472,8 +525,33 @@ for ($index = 0; $index -lt $count; $index++) { timeout_seconds = 60 concurrency = 3 } + $analyzerProfile = [ordered]@{ + schema = (Get-RunnerSchemaNames).AnalyzerProfile + contract_version = (Get-RunnerSchemaNames).Grading + runner = 'fixture' + harness = 'deterministic runner-owned fixture' + model = 'fixture-model' + reasoning_effort = $null + selection_source = 'explicit' + } Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $manifest Write-TestJson -Path (Join-Path $iteration 'execution-profile.json') -Value $profile + Write-TestJson -Path (Join-Path $iteration 'analyzer-profile.json') -Value $analyzerProfile + $manifest.analyzer_profile = 'analyzer-profile.json' + $manifest.analyzer_profile_sha256 = Get-Sha256HexFromFile -Path (Join-Path $iteration 'analyzer-profile.json') + $manifest.analyzer_selection = [ordered]@{ + runner = 'fixture' + harness = 'deterministic runner-owned fixture' + model = 'fixture-model' + reasoning_effort = $null + selection_source = 'explicit' + contract_version = (Get-RunnerSchemaNames).Grading + analyzer_profile_sha256 = [string]$manifest.analyzer_profile_sha256 + } + $manifest.phase2_controller = 'tools/eval-runners/invoke-phase2-analyzer.ps1' + $manifest.phase2_state = 'phase2-state.json' + $manifest.grading_freeze = 'grading-freeze.json' + Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $manifest $fanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' $fanout = Invoke-ForegroundPhaseOne -Path $fanoutScript -IterationDirectory $iteration @@ -596,25 +674,36 @@ for ($index = 0; $index -lt $count; $index++) { Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'invalid grading entry validation' -ExpectedText 'passed must be a boolean' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'invalid grading validation' - foreach ($badEvidence in @('', " `t`n", 'Evaluation completed with output', "Source: output`nQuote: fabricated unavailable observation`nReason: This establishes the assertion.", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Evaluation completed with output", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Assertion evaluated against output", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: The assertion is met", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Output matches the assertion")) { + foreach ($badEvidence in @('', " `t`n")) { $bad = Copy-TestGradingDocument -Document $validGrading $bad.grading[0].evidence = $badEvidence Write-TestJson -Path $gradingPath -Value $bad Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'non-evidentiary PASS rejected' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'evidence rejection preserves frozen execution' } + foreach ($badReason in @('Evaluation completed with output', 'Assertion evaluated against output', 'The assertion is met', 'Output matches the assertion')) { + $bad = Copy-TestGradingDocument -Document $validGrading + $bad.grading[0].reason = $badReason + $bad.grading[0].evidence = $badReason + Write-TestJson -Path $gradingPath -Value $bad + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'generic PASS reason rejected' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'reason rejection preserves frozen execution' + } + $badQuote = Copy-TestGradingDocument -Document $validGrading + $badQuote.grading[0].evidence_refs[0].quote = 'fabricated unavailable observation' + Write-TestJson -Path $gradingPath -Value $badQuote + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'missing output quote rejected' -ExpectedText 'quote is absent' + + $badDomain = Copy-TestGradingDocument -Document $validGrading + $badDomain.grading[0].evidence_domain = 'transcript' + Write-TestJson -Path $gradingPath -Value $badDomain + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'wrong evidence domain rejected' -ExpectedText 'assertion identity' + $repeated = Copy-TestGradingDocument -Document $validGrading $repeated.grading[1].evidence = $repeated.grading[0].evidence Write-TestJson -Path $gradingPath -Value $repeated Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'repeated PASS evidence rejected' -ExpectedText 'Repeated PASS evidence' - $artifactEvidence = Copy-TestGradingDocument -Document $validGrading - $firstRecord = @($records | Sort-Object EvalId, Configuration)[0] - $eventLine = @(Get-Content (Join-Path (Split-Path -Parent $firstRecord.RunManifestPath) 'evidence/fixture-events.jsonl'))[0] - $artifactEvidence.grading[0].evidence = "Source: evidence/fixture-events.jsonl`nQuote: $eventLine`nReason: The native event records the fixture response for this assertion." - Write-TestJson -Path $gradingPath -Value $artifactEvidence - Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'frozen native artifact citation accepted' - Write-TestJson -Path $gradingPath -Value $validGrading $validValidation = Invoke-TestTool -Path $validationScript -Arguments $validationArguments Assert-ToolPasses -Invocation $validValidation -Description 'valid grading validation' @@ -624,12 +713,29 @@ for ($index = 0; $index -lt $count; $index++) { } Write-TestJson -Path $gradingPath -Value $validGrading + if ($Suite -in @('All', 'Application', 'Finalization')) { + $manualNoFreezeFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $manualNoFreezeFinalizer -Description 'finalizer rejects handcrafted grading without Phase 2 freeze' -ExpectedText 'grading-freeze.json is missing' + $phase2Script = Join-Path $packageTools 'invoke-phase2-analyzer.ps1' + $phase2 = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolPasses -Invocation $phase2 -Description 'fixture Phase 2 analyzer controller' + Assert-True (Test-Path -LiteralPath (Join-Path $iteration 'phase2-state.json') -PathType Leaf) 'Phase 2 writes phase2-state.json' + Assert-True (Test-Path -LiteralPath (Join-Path $iteration 'grading-freeze.json') -PathType Leaf) 'Phase 2 writes grading-freeze.json' + $phase2State = Read-TestJson -Path (Join-Path $iteration 'phase2-state.json') + Assert-True (@($phase2State.expected_worker_ids | Where-Object { [string]$_ -like 'arm-3-*' }).Count -eq 0) 'validator-only eval arms must require zero analyzer workers' + Assert-Equal 14 @($phase2State.validator_results).Count 'validator-only paired arms resolve process assertions deterministically' + Assert-Equal 4 @($phase2State.expected_worker_ids).Count 'semantic worker cardinality derives from unresolved assertions, not manifest arm count' + $rootGradingAfterPhase2 = Read-TestJson -Path $gradingPath + Assert-Equal 22 @($rootGradingAfterPhase2.grading).Count 'root grading cardinality derives from normalized assertions' + Assert-True (@($rootGradingAfterPhase2.grading | Where-Object { [string]$_.evidence_domain -eq 'validator' -and [string]$_.source -eq 'validator' }).Count -eq 14) 'validator assertions are resolved by deterministic validator results, not analyzer prose' + $validGrading = Read-TestJson -Path $gradingPath + } if ($Suite -in @('All', 'Application')) { $invalidDirectFinalizer = Copy-TestGradingDocument -Document $validGrading $invalidDirectFinalizer.grading[0].evidence = 123 Write-TestJson -Path $gradingPath -Value $invalidDirectFinalizer $directInvalidFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments - Assert-ToolFails -Invocation $directInvalidFinalizer -Description 'finalizer performs its own grading validation' -ExpectedText 'evidence must be a string' + Assert-ToolFails -Invocation $directInvalidFinalizer -Description 'finalizer rejects post-freeze grading mutation' -ExpectedText 'Root grading.json was changed' Assert-True (-not (Test-Path -LiteralPath (Join-Path $iteration 'report.html') -PathType Leaf)) 'invalid direct finalizer produces no report' Write-TestJson -Path $gradingPath -Value $validGrading @@ -645,13 +751,13 @@ for ($index = 0; $index -lt $count; $index++) { $invalidGrading = [ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = @($validGrading.grading); output = 'raw output is forbidden here' } Write-TestJson -Path $gradingPath -Value $invalidGrading $invalidApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments - Assert-ToolFails -Invocation $invalidApply -Description 'grading artifact with raw output is rejected' -ExpectedText 'unsupported field' + Assert-ToolFails -Invocation $invalidApply -Description 'grading artifact with raw output is rejected' -ExpectedText 'Root grading.json was changed' $forbiddenGradingFields = @('model', 'harness', 'execution_result_sha256', 'session_id', 'telemetry') foreach ($forbiddenField in $forbiddenGradingFields) { $forbiddenEntries = @($validGrading.grading | ForEach-Object { $copy = [ordered]@{} - foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence')) { + foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason', 'source')) { $copy[$name] = Get-JsonProperty -Object $_ -Name $name } $copy[$forbiddenField] = 'forbidden' @@ -659,7 +765,7 @@ for ($index = 0; $index -lt $count; $index++) { }) Write-TestJson -Path $gradingPath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = $forbiddenEntries }) $forbiddenApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments - Assert-ToolFails -Invocation $forbiddenApply -Description "grading artifact with $forbiddenField is rejected" -ExpectedText 'unsupported field' + Assert-ToolFails -Invocation $forbiddenApply -Description "grading artifact with $forbiddenField is rejected" -ExpectedText 'Root grading.json was changed' } Write-TestJson -Path $gradingPath -Value $validGrading } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 1fb0222..1763caf 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -2597,12 +2597,12 @@ try { Assert-True ($orchestrationText -notmatch '(?i)capture-native-results\.ps1|synthesize|worker_authored') 'generic orchestration must not manufacture native terminal envelopes' Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' - Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('validate-eval-grading.ps1') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading validation, grading application, and finalization boundaries' + Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('invoke-phase2-analyzer.ps1') -and $prepareText.Contains('phase2-state.json') -and $prepareText.Contains('grading-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the Phase 1 freeze, Phase 2 analyzer controller, grading freeze, deterministic grading, and finalization boundaries' Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' Assert-True ($prepareText.Contains('authentication incompatibility is terminal for this package iteration') -and $prepareText.Contains('Do not suggest switching to another runner') -and $prepareText.Contains('do not suggest starting another Orchestrator') -and $prepareText.Contains('do not rerun Phase 1')) 'handoff preparation must fail closed on Copilot authentication incompatibility without alternate-runner/orchestrator suggestions' - Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' - Assert-True ($prepareText.Contains('Write `grading.json`, then validate it before finalization') -and $prepareText.Contains('Grading validation is retryable; finalization is not') -and $prepareText.Contains('only after grading validation succeeds')) 'handoff preparation must place retryable grading validation before exactly-once finalization' + Assert-True ($prepareText.Contains('The Phase 2 controller, not this outer orchestrator') -and $prepareText.Contains('deterministically merges root `grading.json`')) 'handoff preparation must route semantic grading through the package-local Phase 2 controller' + Assert-True ($prepareText.Contains('phase2-state.json') -and $prepareText.Contains('freezes `grading-freeze.json`') -and $prepareText.Contains('After Phase 2 succeeds, invoke finalization exactly once')) 'handoff preparation must require Phase 2 state/freeze before exactly-once finalization' Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' Assert-True ($prepareText.Contains('evaluation is incomplete') -and $prepareText.Contains('Only persisted runner-produced evidence')) 'handoff preparation must fail closed when runner evidence cannot be persisted' Assert-True ($prepareText.Contains('fresh package/code fix is required') -and $prepareText.Contains('Never patch package-local runner code') -and $prepareText.Contains('delete execution results') -and $prepareText.Contains('delete or replace `execution-freeze.json`') -and $prepareText.Contains('rerun Phase 1') -and $prepareText.Contains('manually broaden a capability check')) 'generated handoff must forbid package-local repair, state deletion, retry, and manual capability broadening' @@ -2611,7 +2611,7 @@ try { Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' Assert-True ($generatedHandoff.Contains('authentication incompatibility is terminal for this package iteration') -and $generatedHandoff.Contains('Do not suggest switching to another runner') -and $generatedHandoff.Contains('do not suggest starting another Orchestrator') -and $generatedHandoff.Contains('do not rerun Phase 1')) 'generated handoff output must keep Copilot authentication failures fail-closed without fallback orchestration suggestions' Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' - Assert-True ($generatedHandoff.Contains('validate-eval-grading.ps1') -and $generatedHandoff.Contains('-ShowSkeleton') -and $generatedHandoff.Contains('Write `grading.json`, then validate it before finalization') -and $generatedHandoff.Contains('Grading validation is retryable; finalization is not') -and $generatedHandoff.Contains('only after grading validation succeeds')) 'generated handoff requires author -> validate/retry -> finalize exactly once' + Assert-True ($generatedHandoff.Contains('invoke-phase2-analyzer.ps1') -and $generatedHandoff.Contains('The Phase 2 controller, not this outer orchestrator') -and $generatedHandoff.Contains('phase2-state.json') -and $generatedHandoff.Contains('grading-freeze.json') -and $generatedHandoff.Contains('After Phase 2 succeeds, invoke finalization exactly once')) 'generated handoff requires package-local Phase 2 controller -> freeze -> finalize exactly once' Assert-True ($generatedHandoff.Contains('allowance of 6240 seconds') -and $generatedHandoff.Contains('6 arm(s) × 120-second fixed model-free runner preflight timeout = 720-second serial preflight allowance') -and $generatedHandoff.Contains('5490-second execution allowance across 3 batch(es) at concurrency 2') -and $generatedHandoff.Contains('2 scripted user turn(s) × profile.timeout_seconds 900 + 30 seconds runner grace') -and $generatedHandoff.Contains('+ 30 seconds orchestration grace')) 'generated handoff uses the fixed preflight timeout, scripted-turn model timeout, execution batches, and separate orchestration grace' Assert-True (-not $generatedHandoff.Contains('max(120, profile.timeout_seconds + runner grace)') -and -not $generatedHandoff.Contains('5580-second serial preflight allowance')) 'generated handoff does not budget preflight from the model timeout plus runner grace' Assert-True ($generatedConcurrencyThreeHandoff.Contains('allowance of 4410 seconds') -and $generatedConcurrencyThreeHandoff.Contains('execution allowance across 2 batch(es) at concurrency 3')) 'generated handoff recalculates execution batches when concurrency changes' diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 281c7d2..50644dd 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -59,6 +59,7 @@ $OutputEncoding = $utf8NoBom . (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') . (Join-Path $PSScriptRoot 'eval-runners/execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/phase2-grading.ps1') function Set-BenchmarkTokenMetrics { param([object]$Benchmark, [object[]]$ManifestRecords) @@ -155,6 +156,21 @@ function Get-Property { return $Default } +function Get-AssertionDisplayText { + param([object]$Assertion) + + if ($Assertion -is [string]) { return [string]$Assertion } + $text = [string](Get-Property -Object $Assertion -Name 'assertion' -Default (Get-Property -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { return [string]$Assertion } + return $text +} + +function Get-AssertionDisplayTexts { + param([object[]]$Assertions) + + return @($Assertions | ForEach-Object { Get-AssertionDisplayText -Assertion $_ }) +} + function Get-SafeSegment { param([string]$Value) @@ -376,18 +392,19 @@ function Get-ReportOutputFiles { function Get-ReportGrades { param( [object]$Result, - [string[]]$Assertions + [object[]]$Assertions ) + $assertionTexts = @(Get-AssertionDisplayTexts -Assertions $Assertions) $grading = @(Get-Property -Object $Result -Name 'grading' -Default @()) - $count = [Math]::Max($grading.Count, $Assertions.Count) + $count = [Math]::Max($grading.Count, $assertionTexts.Count) $grades = [System.Collections.Generic.List[object]]::new() for ($index = 0; $index -lt $count; $index++) { $grade = if ($index -lt $grading.Count) { $grading[$index] } else { $null } $text = [string](Get-Property -Object $grade -Name 'text' -Default '') $generic = [string]::IsNullOrWhiteSpace($text) -or $text -match '^(Passed|Failed|Assertion\s+\d+)$' - if ($generic -and $index -lt $Assertions.Count) { - $text = [string]$Assertions[$index] + if ($generic -and $index -lt $assertionTexts.Count) { + $text = [string]$assertionTexts[$index] } if ([string]::IsNullOrWhiteSpace($text)) { $text = 'Assertion' @@ -419,11 +436,89 @@ function Get-ReportMetric { if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { return $value } + } return $null } +function Get-MetricTotalOrNull { + param([object[]]$Items, [string]$Name) + + if (@($Items).Count -eq 0) { return $null } + $total = 0.0 + foreach ($item in @($Items)) { + $value = Get-Property -Object $item -Name $Name -Default $null + if ($null -eq $value -or [string]::IsNullOrWhiteSpace([string]$value)) { return $null } + $total += [double]$value + } + return $total +} + +function Get-ExecutionUsageSummary { + param([object[]]$ManifestRecords) + + $results = @($ManifestRecords | ForEach-Object { Read-JsonFile -Path $_.ResultPath }) + return [ordered]@{ + runner = $null + model = if ($results.Count -eq 0) { $null } else { ([string]::Join(', ', @($results | ForEach-Object { [string](Get-Property -Object $_ -Name 'model' -Default '') } | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Sort-Object -Unique))) } + duration_seconds = Get-MetricTotalOrNull -Items $results -Name 'duration_seconds' + input_tokens = Get-MetricTotalOrNull -Items $results -Name 'base_input_tokens' + output_tokens = Get-MetricTotalOrNull -Items $results -Name 'output_tokens' + cache_read_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_read_tokens' + cache_write_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_write_tokens' + total_tokens = Get-MetricTotalOrNull -Items $results -Name 'total_tokens' + cost = Get-MetricTotalOrNull -Items $results -Name 'estimated_cost_usd' + worker_count = $results.Count + } +} + +function Get-AnalyzerUsageSummary { + param([string]$IterationPath, [object]$AnalyzerProfile) + + $statePath = Join-Path $IterationPath 'phase2-state.json' + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + return [ordered]@{ + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + duration_seconds = $null + input_tokens = $null + output_tokens = $null + cache_read_tokens = $null + cache_write_tokens = $null + total_tokens = $null + cost = $null + worker_count = $null + failed_worker_count = $null + } + } + $state = Read-JsonFile -Path $statePath + $entries = @(Get-Property -Object $state -Name 'analyzer_results' -Default @()) + $results = @($entries | ForEach-Object { + $relative = [string](Get-Property -Object $_ -Name 'path' -Default '') + if ([string]::IsNullOrWhiteSpace($relative)) { return } + Read-JsonFile -Path (Join-Path $IterationPath ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar)) + }) + $durationMs = Get-MetricTotalOrNull -Items $results -Name 'duration_ms' + return [ordered]@{ + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + profile_sha256 = [string]$AnalyzerProfile.Hash + duration_seconds = if ($results.Count -eq 0) { 0.0 } elseif ($null -eq $durationMs) { $null } else { [Math]::Round(($durationMs / 1000.0), 4) } + input_tokens = Get-MetricTotalOrNull -Items $results -Name 'input_tokens' + output_tokens = Get-MetricTotalOrNull -Items $results -Name 'output_tokens' + cache_read_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_read_tokens' + cache_write_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_write_tokens' + total_tokens = Get-MetricTotalOrNull -Items $results -Name 'total_tokens' + cost = Get-MetricTotalOrNull -Items $results -Name 'cost' + worker_count = @($state.expected_worker_ids).Count + failed_worker_count = @((Get-JsonPropertyNames -Object (Get-Property -Object $state -Name 'completed' -Default ([ordered]@{}))) | Where-Object { + $completed = Get-Property -Object $state -Name 'completed' -Default ([ordered]@{}) + [string](Get-JsonProperty -Object (Get-JsonProperty -Object $completed -Name ([string]$_) -Default $null) -Name 'status' -Default '') -ne 'completed' + }).Count + } +} + function Get-ReportRun { param( [object]$Result, @@ -521,7 +616,8 @@ function Write-FirstPartyReport { [object]$Validation, [string]$IterationPath, [string]$OutputPath, - [object]$Benchmark + [object]$Benchmark, + [object]$Usage = $null ) $evals = [System.Collections.Generic.List[object]]::new() @@ -535,7 +631,7 @@ function Write-FirstPartyReport { $evalDirectory = [string]$entryRecords[0].EvalDirectory $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $runMap = [ordered]@{} - $assertions = @($metadata.assertions | ForEach-Object { [string]$_ }) + $assertions = @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) foreach ($configuration in @('with_skill', 'without_skill')) { $records = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration @@ -572,6 +668,7 @@ function Write-FirstPartyReport { completed_runs = $completedRuns expected_runs = @($Manifest.evals).Count * 2 generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') + usage = $Usage } $reportData = [ordered]@{ skill_name = [string]$Manifest.skill_name @@ -685,7 +782,7 @@ function New-UpstreamWorkspace { eval_name = [string]$metadata.eval_name prompt = [string]$metadata.prompt expected_output = [string](Get-Property -Object $metadata -Name 'expected_output' -Default '') - expectations = @($metadata.assertions) + expectations = @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) } Write-JsonFile -Path (Join-Path $evalFolder 'eval_metadata.json') -Value $upstreamMetadata $workspaceEntries.Add([pscustomobject]@{ EvalId = [int]$entry.eval_id; EvalName = [string]$entry.eval_name }) @@ -727,7 +824,7 @@ function New-UpstreamWorkspace { $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) Copy-RecordedOutputFiles -Result $result -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory - Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @($metadata.assertions | ForEach-Object { [string]$_ }) + Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) } } @@ -749,6 +846,7 @@ foreach ($warning in @($validation.Warnings)) { if (-not $validation.Complete) { throw "Evaluation completion gate failed: expected $($validation.ExpectedArmCount) bridged terminal arms, found $($validation.BridgedResults)." } +$gradingFreeze = Assert-GradingFreeze -IterationDirectory $iterationPath $skillCreatorPathResolved = Resolve-SkillCreatorPath -RequestedPath $SkillCreatorPath $pythonCommand = Resolve-PythonCommand $workspacePath = Join-Path $iterationPath '.skill-creator-report' @@ -796,10 +894,10 @@ $benchmark.metadata.executor_model = if ($models.Count -eq 0) { 'model not recor # identity and analyzer identity remain independently attributable in the report. $analyzerProfilePath = Join-Path $iterationPath 'analyzer-profile.json' if (Test-Path -LiteralPath $analyzerProfilePath -PathType Leaf) { - $analyzerProfile = Read-JsonFile -Path $analyzerProfilePath - $analyzerRunner = [string](Get-Property -Object $analyzerProfile -Name 'runner' -Default '') - $analyzerModelName = [string](Get-Property -Object $analyzerProfile -Name 'model' -Default '') - $analyzerReasoning = [string](Get-Property -Object $analyzerProfile -Name 'reasoning_effort' -Default '') + $analyzerProfile = $gradingFreeze.Analyzer.Profile + $analyzerRunner = [string]$gradingFreeze.Analyzer.Runner + $analyzerModelName = [string]$gradingFreeze.Analyzer.Model + $analyzerReasoning = [string]$gradingFreeze.Analyzer.ReasoningEffort $analyzerIdentity = ((@($analyzerRunner, $analyzerModelName) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) -join ' / ') if ([string]::IsNullOrWhiteSpace($analyzerIdentity)) { $analyzerIdentity = 'analyzer profile incomplete' } if (-not [string]::IsNullOrWhiteSpace($analyzerReasoning)) { $analyzerIdentity += " (reasoning: $analyzerReasoning)" } @@ -807,9 +905,18 @@ if (Test-Path -LiteralPath $analyzerProfilePath -PathType Leaf) { $benchmark.metadata | Add-Member -NotePropertyName analyzer_runner -NotePropertyValue $analyzerRunner -Force $benchmark.metadata | Add-Member -NotePropertyName analyzer_reasoning_effort -NotePropertyValue $analyzerReasoning -Force $benchmark.metadata | Add-Member -NotePropertyName analyzer_contract_version -NotePropertyValue ([string](Get-Property -Object $analyzerProfile -Name 'contract_version' -Default '')) -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_profile_sha256 -NotePropertyValue ([string]$gradingFreeze.Analyzer.Hash) -Force } else { $benchmark.metadata.analyzer_model = 'analyzer profile not recorded' } +$usageSummary = [ordered]@{ + execution = Get-ExecutionUsageSummary -ManifestRecords $manifestRecords + analyzer = Get-AnalyzerUsageSummary -IterationPath $iterationPath -AnalyzerProfile $gradingFreeze.Analyzer +} +$usageSummary.execution.runner = [string](Get-Property -Object $manifest.execution_selection -Name 'runner' -Default '') +$usageSummary.execution.model = [string](Get-Property -Object $manifest.execution_selection -Name 'model' -Default $usageSummary.execution.model) +$benchmark.metadata | Add-Member -NotePropertyName execution_usage -NotePropertyValue $usageSummary.execution -Force +$benchmark.metadata | Add-Member -NotePropertyName analyzer_usage -NotePropertyValue $usageSummary.analyzer -Force $benchmark.metadata.evals_run = @($workspaceEntries | ForEach-Object { $_.EvalId }) foreach ($run in @($benchmark.runs)) { $match = @($workspaceEntries | Where-Object { $_.EvalId -eq [int]$run.eval_id }) | Select-Object -First 1 @@ -841,7 +948,7 @@ $viewerArguments = @( ) Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $viewerPath -Arguments $viewerArguments -Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark +Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark -Usage $usageSummary if ($RequireComplete) { foreach ($output in @($benchmarkOutputPath, $benchmarkMarkdownOutputPath, $upstreamHtmlOutputPath, $htmlOutputPath)) { diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 75e98dd..89a2712 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -63,6 +63,10 @@ .PARAMETER ModelCatalogPath Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. +.PARAMETER AnalyzerModelCatalogPath + Optional deterministic analyzer catalog JSON used by the model discovery helper. Intended for tests and offline + validation of the analyzer profile independently from executor model validation. + .PARAMETER ReasoningEffort Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to low when this is omitted. @@ -152,6 +156,10 @@ param( [Parameter(ParameterSetName = 'Changed')] [string]$ModelCatalogPath, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerModelCatalogPath, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$ReasoningEffort, @@ -203,6 +211,7 @@ $OutputEncoding = $utf8NoBom . (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') . (Join-Path $PSScriptRoot 'eval-runners/package-integrity.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/phase2-grading.ps1') $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' @@ -434,7 +443,8 @@ function Confirm-HarnessModel { param( [Parameter(Mandatory = $true)][string]$RepoRoot, [Parameter(Mandatory = $true)][string]$RunnerName, - [Parameter(Mandatory = $true)][string]$ModelName + [Parameter(Mandatory = $true)][string]$ModelName, + [AllowNull()][string]$CatalogPath = $null ) $discoveryScript = Join-Path $RepoRoot 'scripts/Get-HarnessModels.ps1' @@ -443,8 +453,8 @@ function Confirm-HarnessModel { } $arguments = @('-Runner', $RunnerName, '-RequireModel', $ModelName) - if (-not [string]::IsNullOrWhiteSpace($ModelCatalogPath)) { - $arguments += @('-CatalogPath', $ModelCatalogPath) + if (-not [string]::IsNullOrWhiteSpace($CatalogPath)) { + $arguments += @('-CatalogPath', $CatalogPath) } $discoveryOutput = & pwsh -NoProfile -NonInteractive -File $discoveryScript @arguments 2>&1 if ($LASTEXITCODE -ne 0) { @@ -485,7 +495,7 @@ function Resolve-ExecutionSelection { if ($supportedRunners -notcontains $referenceRunner) { throw "Codebelt Reference requires runner '$referenceRunner', but it is unavailable. Supported runner IDs: $supportedText." } - [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel) + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel -CatalogPath $ModelCatalogPath) return [pscustomobject]@{ Runner = $referenceRunner @@ -534,7 +544,7 @@ function Resolve-ExecutionSelection { } } - [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $resolvedModel) + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $resolvedModel -CatalogPath $ModelCatalogPath) return [pscustomobject]@{ Runner = $resolvedRunner @@ -716,14 +726,17 @@ function New-ResultStub { [object]$EvalEntry, [string]$EvalName, [string]$Configuration, - [string[]]$Assertions + [object[]]$Assertions ) $grading = foreach ($assertion in $Assertions) { [ordered]@{ - text = $assertion + text = Get-AssertionText -Assertion $assertion passed = $null evidence = '' + evidence_domain = [string](Get-JsonProperty -Object $assertion -Name 'evidence_domain' -Default 'output') + evidence_refs = @() + reason = '' } } @@ -799,20 +812,22 @@ function Resolve-AnalyzerSelection { ) # The analyzer/grader is a distinct, persisted, reproducible profile - never "whichever model happens to host the - # outer orchestrator". By repository policy it defaults to the validated executor selection so cross-provider - # preparation never requires a second runner's catalog, and it is overridable to a stable reference analyzer via - # -AnalyzerRunner/-AnalyzerModel so the same validated analyzer can grade Copilot, Codex, and OpenCode executions. + # outer orchestrator" and never silently matched to the executor. The repository-owned default reuses the existing + # Codebelt Reference policy as a stable analyzer stratum across executor providers. + $referenceRunner = 'github-copilot' + $referenceModel = 'claude-haiku-4.5' $hasRunner = -not [string]::IsNullOrWhiteSpace($AnalyzerRunner) $hasModel = -not [string]::IsNullOrWhiteSpace($AnalyzerModel) if ($hasModel -and -not $hasRunner) { throw 'Analyzer selection requires -AnalyzerRunner when -AnalyzerModel is supplied.' } if (-not $hasRunner -and -not $hasModel) { + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel -CatalogPath $AnalyzerModelCatalogPath) return [pscustomobject]@{ - Runner = $ExecutionSelection.Runner - Model = $ExecutionSelection.Model - ReasoningEffort = $ExecutionReasoningEffort - Harness = $ExecutionSelection.Harness - Source = 'executor-matched' + Runner = $referenceRunner + Model = $referenceModel + ReasoningEffort = $null + Harness = Get-HarnessName -RunnerName $referenceRunner + Source = 'codebelt-reference' } } @@ -825,7 +840,7 @@ function Resolve-AnalyzerSelection { if ($supportedRunners -notcontains $resolvedRunner) { throw "Analyzer runner '$resolvedRunner' is not a supported runner id ($($supportedRunners -join ', '))." } - [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $AnalyzerModel) + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $AnalyzerModel -CatalogPath $AnalyzerModelCatalogPath) $reasoning = if (-not [string]::IsNullOrWhiteSpace($AnalyzerReasoningEffort)) { $AnalyzerReasoningEffort } elseif ($resolvedRunner -eq 'codex') { 'low' } else { $null } return [pscustomobject]@{ Runner = $resolvedRunner @@ -918,7 +933,7 @@ function Get-Assertions { param([object]$EvalEntry) if ($EvalEntry.PSObject.Properties.Name -contains 'expectations' -and $null -ne $EvalEntry.expectations) { - return @($EvalEntry.expectations | ForEach-Object { [string]$_ }) + return @(Get-NormalizedAssertions -Assertions @($EvalEntry.expectations)) } return @() @@ -1518,6 +1533,10 @@ function Invoke-PrepareMode { $executionSelection = Resolve-ExecutionSelection -RepoRoot $repoRoot $effectiveConcurrency = Resolve-EffectiveConcurrency -RunnerName ([string]$executionSelection.Runner) -RequestedConcurrency $Concurrency -ConcurrencyWasExplicit ($scriptBoundParameters.ContainsKey('Concurrency')) + $executionProfile = New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value) + # Validate the analyzer/grader profile before any output directory is created so an unknown analyzer model leaves no + # partial package behind. The profile is persisted later once the iteration root exists. + $analyzerSelection = Resolve-AnalyzerSelection -RepoRoot $repoRoot -ExecutionSelection $executionSelection -ExecutionReasoningEffort ([string]$executionProfile.reasoning_effort) $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" @@ -1565,13 +1584,13 @@ function Invoke-PrepareMode { sha256 = Get-TreeHash -Root $copiedRunnerTools file_count = @(Get-ChildItem -LiteralPath $copiedRunnerTools -Recurse -File -Force).Count } - $executionProfile = New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value) ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value $executionProfile # The analyzer/grader profile is validated during preparation exactly like the execution profile and persisted # separately so executor identity and analyzer identity stay independently attributable in evidence and reports. - $analyzerSelection = Resolve-AnalyzerSelection -RepoRoot $repoRoot -ExecutionSelection $executionSelection -ExecutionReasoningEffort ([string]$executionProfile.reasoning_effort) $analyzerProfile = New-AnalyzerProfile -AnalyzerSelection $analyzerSelection - ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'analyzer-profile.json') -Value $analyzerProfile + $analyzerProfilePath = Join-Path $iterationDirectory 'analyzer-profile.json' + ConvertTo-JsonFile -Path $analyzerProfilePath -Value $analyzerProfile + $analyzerProfileHash = Get-FileSha256 -Path $analyzerProfilePath $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1824,6 +1843,7 @@ function Invoke-PrepareMode { runner_prompt = 'RUN-THIS.prompt.md' execution_profile = 'execution-profile.json' analyzer_profile = 'analyzer-profile.json' + analyzer_profile_sha256 = $analyzerProfileHash analyzer_selection = [ordered]@{ runner = $analyzerSelection.Runner harness = $analyzerSelection.Harness @@ -1831,13 +1851,17 @@ function Invoke-PrepareMode { reasoning_effort = $analyzerSelection.ReasoningEffort selection_source = $analyzerSelection.Source contract_version = $analyzerContractVersion + analyzer_profile_sha256 = $analyzerProfileHash } runner_protocol = $runnerProtocolSchema runner_tools = $evalRunnerToolRelativePath runner_tools_integrity = $runnerToolsIntegrity execution_result_schema = $executionResultSchema execution_freeze = 'execution-freeze.json' + phase2_controller = "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1" + phase2_state = 'phase2-state.json' grading = 'grading.json' + grading_freeze = 'grading-freeze.json' grading_validator = "$evalRunnerToolRelativePath/validate-eval-grading.ps1" grading_contract = "$evalRunnerToolRelativePath/contracts/grading.schema.json" finalizer = "$evalRunnerToolRelativePath/finalize-eval-package.ps1" @@ -1901,6 +1925,13 @@ function Invoke-PrepareMode { Write-Host " Preset: $($executionSelection.Preset)" } Write-Host '' + Write-Host 'Analyzer:' + Write-Host " Harness: $($analyzerSelection.Harness)" + Write-Host " Runner: $($analyzerSelection.Runner)" + Write-Host " Model: $($analyzerSelection.Model)" + Write-Host " Source: $($analyzerSelection.Source)" + Write-Host " Profile: analyzer-profile.json ($analyzerProfileHash)" + Write-Host '' Write-Host "Cases: $($manifestEvals.Count)" Write-Host "Arms: $($manifestEvals.Count * 2)" Write-Host '' @@ -1945,7 +1976,7 @@ function New-RunnerPrompt { $profilePath = Join-Path $IterationDirectory 'execution-profile.json' $runnerOwnedFanoutPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1" $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" - $gradingValidatorPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/validate-eval-grading.ps1" + $phase2AnalyzerPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1" $finalizerPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1" $maxScriptedUserTurns = 1 foreach ($manifestEval in @($ManifestEvals)) { @@ -1995,16 +2026,13 @@ function New-RunnerPrompt { [void]$builder.AppendLine() [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") - [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') - [void]$builder.AppendLine('Before creating grading.json, the Grader MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`; that guidance is authoritative during Phase 2. Uncertain or unverified expectations FAIL. Every assertion requires specific evidence. For PASS, evidence must use three newline-separated fields: `Source: output` (or a manifest-recorded run artifact path), `Quote: `, and `Reason: `. Do not reuse generic completion statements or identical evidence across assertions; reasons that merely restate that the assertion passed or was "evaluated against output" are rejected. FAIL evidence must explain what is missing or contradicted. Deterministic validation checks provenance and shape; it does not replace the Grader judgment required by grader.md.') - [void]$builder.AppendLine('Grade every arm with the single validated analyzer/grader profile persisted at `analyzer-profile.json` (also recorded in `manifest.analyzer_selection`), not with whichever model happens to host the orchestrator. The analyzer is an independent boundary: it must not inherit an executor arm''s HOME, provider configuration, session, native tools, skill catalogs, or plugins, and executor identity and analyzer identity stay independently attributable in the report metadata (`executor_model` vs `analyzer_model`/`analyzer_runner`). Use the same analyzer profile for every eval and every configuration so comparisons stay within one analyzer stratum.') - [void]$builder.AppendLine('To display the authoritative top-level grading skeleton, run:') - [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -ShowSkeleton") - [void]$builder.AppendLine('Write `grading.json`, then validate it before finalization:') - [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -IterationDirectory `"$IterationDirectory`" -GradingPath `"grading.json`"") - [void]$builder.AppendLine('Grading validation is retryable; finalization is not. If validation fails, correct `grading.json` and rerun the validation command as many times as required. Do not invoke the application helper separately; the finalizer invokes `apply-eval-grading.ps1` deterministically after revalidating grading. Invoke finalization exactly once, and only after grading validation succeeds:') + [void]$builder.AppendLine('Only if that bridge succeeds, invoke the package-local Phase 2 analyzer controller:') + [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$phase2AnalyzerPath`" -IterationDirectory `"$IterationDirectory`"") + [void]$builder.AppendLine('The Phase 2 controller, not this outer orchestrator, reads the grading key, resolves validator-domain assertions deterministically, creates one isolated analyzer run per semantic arm, dispatches those analyzer runs with bounded concurrency, validates observed analyzer runner/model/session identity, writes `phase2-state.json`, freezes `grading-freeze.json`, and deterministically merges root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`. Each grade is keyed by eval/configuration/assertion_index and carries passed, evidence_domain, evidence_refs, reason, and upstream-compatible evidence. If it fails, stop: do not author or repair grading.json, do not retry an analyzer worker, do not change analyzer-profile.json, do not switch providers, and do not grade in this context.') + [void]$builder.AppendLine('The analyzer profile persisted at `analyzer-profile.json` is the only grading stratum. The analyzer is independent from the executor and must not inherit executor HOME, provider configuration, session, native tools, skill catalogs, plugins, paired-arm output, sibling evals, source checkout, or candidate skill content. The controller supplies only one arm''s allowed evidence bundle and the packaged `tools/skill-creator/agents/grader.md` contract to each analyzer worker.') + [void]$builder.AppendLine('After Phase 2 succeeds, invoke finalization exactly once:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$finalizerPath`" -IterationDirectory `"$IterationDirectory`"") - [void]$builder.AppendLine('The finalizer revalidates the manifest, profile, terminal orchestration/concurrency evidence, immutable freeze, raw artifacts, bridge, canonical results, and grading; it then generates and verifies all required reports. Return only its machine-readable JSON summary and artifact paths. A non-zero exit, missing artifact, integrity error, or report error means the evaluation is incomplete. Never repair, re-freeze, re-bridge a changed raw result, or report prose success.') + [void]$builder.AppendLine('The finalizer revalidates the manifest, profile, terminal orchestration/concurrency evidence, immutable Phase 1 freeze, bridge, Phase 2 state, analyzer evidence, grading freeze, deterministic grading hash/cardinality, canonical results, and reports. Return only its machine-readable JSON summary and artifact paths. A non-zero exit, missing artifact, integrity error, or report error means the evaluation is incomplete. Never repair, re-freeze, re-bridge a changed raw result, or report prose success.') [void]$builder.AppendLine() [void]$builder.AppendLine('The four required package-root artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`. Same-session scripted evals, when present in a run, are handled by the selected runner only if its descriptor/preflight proves `scripted_multi_turn_same_session`; otherwise preflight fails before execution. The paired configurations receive identical scripted user turns.') [void]$builder.AppendLine() @@ -2046,7 +2074,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports, then grades and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports. Phase 2 grading is owned by the package-local analyzer controller, not by the outer orchestrator.') [void]$builder.AppendLine() [void]$builder.AppendLine("Execution selection: runner=$($ExecutionSelection.Runner); model=$($ExecutionSelection.Model); timeout_seconds=$TimeoutSeconds; concurrency=$($EffectiveConcurrency.Value); concurrency_source=$($EffectiveConcurrency.Source).") [void]$builder.AppendLine() @@ -2057,7 +2085,7 @@ function New-PackageReadme { } [void]$builder.AppendLine() [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') - [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, `analyzer-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Orchestration topology') [void]$builder.AppendLine() @@ -2084,9 +2112,9 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine(('1. Read `execution-profile.json` and the selected runner descriptor. If `runner` or `model` is missing or unsupported, fail clearly instead of guessing. For `delegation.dispatch_owner=runner`, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1") + ' exactly once with the caller shell/tool timeout set to at least the package-computed Phase 1 allowance. It performs all preflight, native dispatch, concurrency, terminal registration, timeout handling, and raw-evidence freezing. For `delegation.dispatch_owner=orchestrator`, use only the descriptor-declared native worker mechanism, then invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/freeze-execution-evidence.ps1") + ' after every arm is terminal.')) [void]$builder.AppendLine('2. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. Do not execute an arm in the parent context, create a second worker for a runner-owned arm, expose grading material during execution, or author/repair raw evidence. If Phase 1 reports incompatible or Phase 1/freezing fails, stop: the evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check.') - [void]$builder.AppendLine(('3. After the freeze succeeds, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1") + ' with `-RequireComplete -RequireParallelDispatch` and add `-RequireNativeDelegation` when the selected descriptor has `dispatch_owner=runner`. Only after that deterministic bridge succeeds, give the grading key to the Grader. The Grader writes only the package-root `grading.json` grading-only artifact. It must not modify execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.')) - [void]$builder.AppendLine(('4. To display the authoritative grading skeleton, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/validate-eval-grading.ps1") + ' with `-ShowSkeleton`. After writing `grading.json`, invoke the same helper with `-IterationDirectory "' + $IterationDirectory + '" -GradingPath "grading.json"` before finalization. Grading validation is retryable; finalization is not. Correct and rerun validation until it succeeds.')) - [void]$builder.AppendLine(('5. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1") + ' exactly once, only after grading validation succeeds. It invokes the deterministic apply-eval-grading boundary, validates the frozen evidence, idempotent bridge, complete grading, and report outputs. Return only its machine-readable summary.')) + [void]$builder.AppendLine(('3. After the freeze succeeds, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1") + ' with `-RequireComplete -RequireParallelDispatch` and add `-RequireNativeDelegation` when the selected descriptor has `dispatch_owner=runner`. This deterministic bridge validates the frozen execution evidence and canonical result paths before any grading begins.')) + [void]$builder.AppendLine(('4. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1") + ' once. It resolves validator assertions, starts one fresh analyzer worker per remaining semantic arm using `analyzer-profile.json`, persists `phase2-state.json`, freezes `grading-freeze.json`, and deterministically merges package-root `grading.json`. Do not grade in the outer orchestrator, retry failed analyzer workers, edit analyzer-profile.json, or manually repair grading.json.')) + [void]$builder.AppendLine(('5. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1") + ' exactly once, only after Phase 2 succeeds. It validates the frozen Phase 1 evidence, bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical grading application, and report outputs. Return only its machine-readable summary.')) [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It never executes an eval arm in its own model context. Same-session scripted interactions are allowed only when the selected runner proves that capability; paired runs receive identical deterministic turns. The package is complete only when the finalizer exits successfully.') [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b249241..c5edefb 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -463,17 +463,24 @@ function Invoke-ValidationScriptJobs { $dotnetHome = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-dotnet-home-' + [Guid]::NewGuid().ToString('N')) $oldDotNetCliHome = $env:DOTNET_CLI_HOME $oldXdgDataHome = $env:XDG_DATA_HOME + $oldTemp = $env:TEMP + $oldTmp = $env:TMP try { New-Item -ItemType Directory -Path $dotnetHome -Force | Out-Null $env:DOTNET_CLI_HOME = $dotnetHome $env:XDG_DATA_HOME = Join-Path $dotnetHome 'share' + $env:TEMP = Join-Path $dotnetHome 'tmp' + $env:TMP = $env:TEMP + New-Item -ItemType Directory -Path $env:TEMP -Force | Out-Null $output = & $Script.Path 2>&1 $exitCode = $LASTEXITCODE } finally { $env:DOTNET_CLI_HOME = $oldDotNetCliHome $env:XDG_DATA_HOME = $oldXdgDataHome + $env:TEMP = $oldTemp + $env:TMP = $oldTmp if (Test-Path $dotnetHome) { Remove-Item -Path $dotnetHome -Recurse -Force -ErrorAction SilentlyContinue } @@ -1295,7 +1302,10 @@ Add-ValidationResult -Results $results -Name 'Git-workspace evals omit stale inl $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-change-impact-9-' + [Guid]::NewGuid().ToString('N')) try { - $output = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-change-impact' -Runner 'github-copilot' -Eval 9 -OutputRoot $packageRoot 2>&1 + New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null + $catalogPath = Join-Path $packageRoot 'models.json' + [System.IO.File]::WriteAllText($catalogPath, '{"models":[{"id":"claude-haiku-4.5","operation":"language"}]}', $utf8NoBom) + $output = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-change-impact' -Runner 'github-copilot' -Eval 9 -OutputRoot $packageRoot -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $catalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for change-impact eval 9: $($output -join [Environment]::NewLine)" } $evalDir = @(Get-ChildItem -Path $packageRoot -Recurse -Directory | Where-Object { $_.Name -like 'eval-09*' } | Select-Object -First 1) if ($evalDir.Count -ne 1) { throw 'Prepared change-impact eval 9 directory was not found.' } @@ -1649,6 +1659,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } try { $catalogPath = Join-Path $packageRoot 'fake-model-catalog.json' + $analyzerCatalogPath = Join-Path $packageRoot 'fake-analyzer-model-catalog.json' New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null [System.IO.File]::WriteAllText($catalogPath, (@' { @@ -1663,6 +1674,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable ] } '@), $utf8NoBom) + Copy-Item -LiteralPath $catalogPath -Destination $analyzerCatalogPath -Force $expectedSelectors = @( 'claude-haiku-4.5', @@ -1792,7 +1804,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $referencePackageRoot = Join-Path $packageRoot 'reference-package' - $referencePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath 2>&1 + $referencePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -CodebeltReference failed against the fake current catalog: $($referencePrepareOutput -join [Environment]::NewLine)" } @@ -1803,7 +1815,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PreparedRunnerIdentity -Name 'GitHub Copilot Codebelt Reference package' -IterationDirectory (Join-Path $referencePackageRoot 'iteration-1') -ExpectedRunner 'github-copilot' -ExpectedModel 'claude-haiku-4.5' $codexPackageRoot = Join-Path $packageRoot 'codex-package' - $codexPrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -ModelCatalogPath $catalogPath 2>&1 + $codexPrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for the Codex default fixture: $($codexPrepareOutput -join [Environment]::NewLine)" } @@ -1825,7 +1837,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $opencodePackageRoot = Join-Path $packageRoot 'opencode-package' - $opencodePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $opencodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath 2>&1 + $opencodePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $opencodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for the OpenCode fixture: $($opencodePrepareOutput -join [Environment]::NewLine)" } @@ -1837,7 +1849,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PreparedRunnerIdentity -Name 'OpenCode package' -IterationDirectory (Join-Path $opencodePackageRoot 'iteration-1') -ExpectedRunner 'opencode' -ExpectedModel 'provider-paid/Paid.Model' $explicitGithubPackageRoot = Join-Path $packageRoot 'github-explicit-runner-package' - $explicitGithubOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitGithubPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath 2>&1 + $explicitGithubOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitGithubPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for explicit GitHub Copilot selection: $($explicitGithubOutput -join [Environment]::NewLine)" } $explicitGithubProfile = [System.IO.File]::ReadAllText((Join-Path $explicitGithubPackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json if ([string]$explicitGithubProfile.runner -ne 'github-copilot' -or [string]$explicitGithubProfile.model -ne 'gpt-5.6-luna' -or $null -ne $explicitGithubProfile.reasoning_effort) { @@ -1845,6 +1857,41 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } Assert-PreparedRunnerIdentity -Name 'Explicit GitHub Copilot package' -IterationDirectory (Join-Path $explicitGithubPackageRoot 'iteration-1') -ExpectedRunner 'github-copilot' -ExpectedModel 'gpt-5.6-luna' + $referenceAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $referencePackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $codexAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $codexPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $opencodeAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $opencodePackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $explicitGithubAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $explicitGithubPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + if ($referenceAnalyzerHash -notmatch '^[0-9a-f]{64}$' -or $referenceAnalyzerHash -ne $codexAnalyzerHash -or $referenceAnalyzerHash -ne $opencodeAnalyzerHash) { + throw 'The same repository analyzer policy must produce the same analyzer_profile_sha256 across GitHub Copilot, Codex, and OpenCode executor packages.' + } + if ($referenceAnalyzerHash -ne $explicitGithubAnalyzerHash) { + throw 'Changing the executor model must not change the analyzer_profile_sha256.' + } + foreach ($iterationToInspect in @((Join-Path $referencePackageRoot 'iteration-1'), (Join-Path $codexPackageRoot 'iteration-1'), (Join-Path $opencodePackageRoot 'iteration-1'), (Join-Path $explicitGithubPackageRoot 'iteration-1'))) { + $profileToInspect = [System.IO.File]::ReadAllText((Join-Path $iterationToInspect 'analyzer-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$profileToInspect.selection_source -eq 'executor-matched') { + throw 'Analyzer selection must not silently fall back to executor-matched.' + } + } + + $alternateAnalyzerPackageRoot = Join-Path $packageRoot 'alternate-analyzer-package' + $alternateAnalyzerOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $alternateAnalyzerPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath -AnalyzerRunner 'github-copilot' -AnalyzerModel 'gpt-5.6-luna' -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for alternate analyzer selection: $($alternateAnalyzerOutput -join [Environment]::NewLine)" } + $alternateAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $alternateAnalyzerPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + if ($alternateAnalyzerHash -eq $referenceAnalyzerHash) { + throw 'Changing the analyzer model must change analyzer_profile_sha256.' + } + + $invalidAnalyzerRoot = Join-Path $packageRoot 'invalid-analyzer-model-package' + $invalidAnalyzerOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $invalidAnalyzerRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' -ModelCatalogPath $catalogPath -AnalyzerRunner 'github-copilot' -AnalyzerModel 'missing-analyzer-model' -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 + if ($LASTEXITCODE -eq 0 -or ($invalidAnalyzerOutput -join ' ') -notmatch "model 'missing-analyzer-model' could not be verified") { + throw 'Unknown analyzer model must fail preparation through analyzer model discovery.' + } + if (Test-Path -LiteralPath $invalidAnalyzerRoot) { + Remove-Item -LiteralPath $invalidAnalyzerRoot -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create a package when analyzer model validation fails.' + } + $mismatchIteration = Join-Path $packageRoot 'mismatched-profile-package' Copy-Item -LiteralPath (Join-Path $referencePackageRoot 'iteration-1') -Destination $mismatchIteration -Recurse $mismatchProfilePath = Join-Path $mismatchIteration 'execution-profile.json' @@ -1856,7 +1903,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PackageIdentityValidationFails -Name 'mismatched runner package' -IterationDirectory $mismatchIteration -ExpectedRunner 'github-copilot' -ExpectedMessagePattern 'execution-profile\.json runner .+ does not match manifest\.execution_selection\.runner' $explicitOpenCodePackageRoot = Join-Path $packageRoot 'opencode-explicit-concurrency-package' - $explicitOpenCodeOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitOpenCodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -Concurrency 16 2>&1 + $explicitOpenCodeOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitOpenCodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath -Concurrency 16 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for explicit OpenCode concurrency: $($explicitOpenCodeOutput -join [Environment]::NewLine)" } $explicitOpenCodeProfile = [System.IO.File]::ReadAllText((Join-Path $explicitOpenCodePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json if ([int]$explicitOpenCodeProfile.concurrency -ne 16) { throw 'Explicit OpenCode -Concurrency 16 must be honored without clamping.' } @@ -1895,7 +1942,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw 'OpenCode preparation must not create a package without an explicit model choice.' } - $prepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $prepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" } @@ -1974,11 +2021,13 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' 'assertion_index', 'passed', 'evidence', - 'validate-eval-grading.ps1', - 'Write `grading.json`, then validate it before finalization', - 'Grading validation is retryable; finalization is not', - 'only after grading validation succeeds', - 'apply-eval-grading.ps1', + 'evidence_domain', + 'evidence_refs', + 'invoke-phase2-analyzer.ps1', + 'phase2-state.json', + 'grading-freeze.json', + 'The Phase 2 controller, not this outer orchestrator', + 'After Phase 2 succeeds, invoke finalization exactly once', 'finalize-eval-package.ps1', 'machine-readable JSON summary', 'report.html', @@ -2034,10 +2083,11 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' 'Do not create outer workers', 'execution-freeze.json', 'grading.json', - 'validate-eval-grading.ps1', - 'Grading validation is retryable; finalization is not', - 'only after grading validation succeeds', - 'apply-eval-grading.ps1', + 'invoke-phase2-analyzer.ps1', + 'phase2-state.json', + 'grading-freeze.json', + 'evidence_refs', + 'After Phase 2 succeeds, invoke finalization exactly once', 'finalize-eval-package.ps1', 'evaluation is incomplete and must fail closed', 'Only persisted runner-produced evidence at the manifest-declared paths may proceed' @@ -2232,8 +2282,9 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw "$($entry.eval_name) prompts must not carry the expected output; that is the grading key." } foreach ($assertion in @($metadata.assertions)) { - if ($prompt.Contains([string]$assertion)) { - throw "$($entry.eval_name) prompts must not carry assertion '$assertion'; that is the grading key." + $assertionText = if ($assertion -is [string]) { [string]$assertion } elseif ($assertion.PSObject.Properties.Name -contains 'assertion') { [string]$assertion.assertion } else { '' } + if (-not [string]::IsNullOrWhiteSpace($assertionText) -and $prompt.Contains($assertionText)) { + throw "$($entry.eval_name) prompts must not carry assertion '$assertionText'; that is the grading key." } } } @@ -2365,6 +2416,22 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' $manifest.execution_selection.model = 'fixture-model' $manifest.execution_selection.harness = 'Deterministic runner-owned fixture' $manifest.execution_selection.preset = 'Deterministic validator' + $analyzerProfilePath = Join-Path $iterationDirectory ([string]$manifest.analyzer_profile) + $deterministicAnalyzerProfile = [System.IO.File]::ReadAllText($analyzerProfilePath, $utf8NoBom) | ConvertFrom-Json + $deterministicAnalyzerProfile.runner = 'fixture' + $deterministicAnalyzerProfile.model = 'fixture-model' + $deterministicAnalyzerProfile.harness = 'deterministic runner-owned fixture' + $deterministicAnalyzerProfile.reasoning_effort = $null + $deterministicAnalyzerProfile.selection_source = 'explicit' + [System.IO.File]::WriteAllText($analyzerProfilePath, (($deterministicAnalyzerProfile | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $analyzerProfileHash = Get-FileHash -Algorithm SHA256 -LiteralPath $analyzerProfilePath + $manifest.analyzer_profile_sha256 = $analyzerProfileHash.Hash.ToLowerInvariant() + $manifest.analyzer_selection.runner = 'fixture' + $manifest.analyzer_selection.model = 'fixture-model' + $manifest.analyzer_selection.harness = 'deterministic runner-owned fixture' + $manifest.analyzer_selection.reasoning_effort = $null + $manifest.analyzer_selection.selection_source = 'explicit' + $manifest.analyzer_selection.analyzer_profile_sha256 = $manifest.analyzer_profile_sha256 [System.IO.File]::WriteAllText((Join-Path $iterationDirectory 'manifest.json'), (($manifest | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) $fixtureMetricEnvironment = [ordered]@{ AGENTIC_RUNNER_FIXTURE_FINAL_RESPONSE = 'validator output' @@ -2400,43 +2467,21 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' [Environment]::SetEnvironmentVariable($environmentName, $fixtureMetricEnvironmentBefore[$environmentName]) } } - $executionCollectOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -CollectResults $iterationDirectory 2>&1 + $bridgePath = Join-Path $iterationDirectory 'tools/eval-runners/bridge-manifest-results.ps1' + $bridgeOutput = & pwsh -NoProfile -NonInteractive -File $bridgePath -IterationDirectory $iterationDirectory -RequireComplete -RequireParallelDispatch -RequireNativeDelegation 2>&1 if ($LASTEXITCODE -ne 0) { - throw "prepare-skill-evals.ps1 -CollectResults failed while bridging the complete deterministic fixture: $($executionCollectOutput -join [Environment]::NewLine)" - } - # The only post-execution artifact authored by this validator is a - # grading-only document with exact metadata identities. Canonical - # result grading is projected by the deterministic application helper. - $gradingEntries = [System.Collections.Generic.List[object]]::new() - foreach ($entryToGrade in @($manifest.evals)) { - $metadataPath = Join-Path $iterationDirectory ([string]$entryToGrade.metadata) - $metadataForGrade = [System.IO.File]::ReadAllText($metadataPath, $utf8NoBom) | ConvertFrom-Json - foreach ($configuration in @('with_skill', 'without_skill')) { - for ($assertionIndex = 0; $assertionIndex -lt @($metadataForGrade.assertions).Count; $assertionIndex++) { - $gradingEntries.Add([ordered]@{ - eval_id = [int]$entryToGrade.eval_id - eval_name = [string]$entryToGrade.eval_name - configuration = $configuration - assertion_index = $assertionIndex - assertion = [string]$metadataForGrade.assertions[$assertionIndex] - passed = $true - evidence = "Source: output`nQuote: $(([IO.File]::ReadAllText((Join-Path $iterationDirectory $entryToGrade.runs.$configuration.result)) | ConvertFrom-Json).output)`nReason: The fixture response supplies the observed content for assertion $assertionIndex in this deterministic transport test." - }) - } - } + throw "The deterministic manifest bridge failed: $($bridgeOutput -join [Environment]::NewLine)" + } + $phase2Path = Join-Path $iterationDirectory 'tools/eval-runners/invoke-phase2-analyzer.ps1' + $phase2Output = & pwsh -NoProfile -NonInteractive -File $phase2Path -IterationDirectory $iterationDirectory -Concurrency 4 -TimeoutSeconds 60 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "The deterministic Phase 2 analyzer controller failed: $($phase2Output -join [Environment]::NewLine)" } - $gradingPath = Join-Path $iterationDirectory ([string]$manifest.grading) - [System.IO.File]::WriteAllText($gradingPath, (([ordered]@{ schema = 'codebeltnet/agentic/eval-grading/1'; grading = @($gradingEntries.ToArray()) } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) $validateGradingPath = Join-Path $iterationDirectory 'tools/eval-runners/validate-eval-grading.ps1' $validateGradingOutput = & pwsh -NoProfile -NonInteractive -File $validateGradingPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 if ($LASTEXITCODE -ne 0) { throw "The deterministic grading validation failed: $($validateGradingOutput -join [Environment]::NewLine)" } - $applyGradingPath = Join-Path $iterationDirectory 'tools/eval-runners/apply-eval-grading.ps1' - $applyOutput = & pwsh -NoProfile -NonInteractive -File $applyGradingPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "The deterministic grading-only application failed: $($applyOutput -join [Environment]::NewLine)" - } $finalizerPath = Join-Path $iterationDirectory 'tools/eval-runners/finalize-eval-package.ps1' $finalizerOutput = & pwsh -NoProfile -NonInteractive -File $finalizerPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 if ($LASTEXITCODE -ne 0) { @@ -2469,7 +2514,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw "benchmark.json must preserve recorded token metrics through the upstream skill-creator aggregation (with_skill=$($benchmark.run_summary.with_skill.tokens.mean), without_skill=$($benchmark.run_summary.without_skill.tokens.mean))." } - $changedOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Changed -Base 'HEAD' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $changedOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Changed -Base 'HEAD' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -Changed failed: $($changedOutput -join [Environment]::NewLine)" } @@ -2478,7 +2523,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' - $isolationOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $isolationOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -eq 0) { throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' } @@ -2493,7 +2538,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' try { - $botOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $botOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" } diff --git a/skills/dotnet-change-impact/evals/evals.json b/skills/dotnet-change-impact/evals/evals.json index e2a81dc..8cf472b 100644 --- a/skills/dotnet-change-impact/evals/evals.json +++ b/skills/dotnet-change-impact/evals/evals.json @@ -1,19 +1,19 @@ -{ - "skill_name": "dotnet-change-impact", - "evals": [ - { - "id": 1, - "prompt": "In my .NET library I removed the public method `HttpRetryHandler.Send(HttpRequestMessage)` and renamed the public property `Client.BaseUri` to `Client.BaseAddress`. Should I bump major, minor, or patch?", +{ + "skill_name": "dotnet-change-impact", + "evals": [ + { + "id": 1, + "prompt": "In my .NET library I removed the public method `HttpRetryHandler.Send(HttpRequestMessage)` and renamed the public property `Client.BaseUri` to `Client.BaseAddress`. Should I bump major, minor, or patch?", "expected_output": "Structured answer recommending Major with compatibility reasoning.", "expectations": [ "Recommends Major in the Recommendation section", "Includes Key changes identified, Compatibility impact, Reasoning, and Deterministic decision sections", "Treats removing a public method and renaming a public property as binary and source breaking" ] - }, - { - "id": 2, - "prompt": "I added a brand-new public class `RetryPolicy` and a new `WithRetry(this HttpClient client, int attempts)` extension method to my NuGet package. No existing public APIs were touched. What's the version bump?", + }, + { + "id": 2, + "prompt": "I added a brand-new public class `RetryPolicy` and a new `WithRetry(this HttpClient client, int attempts)` extension method to my NuGet package. No existing public APIs were touched. What's the version bump?", "expected_output": "Structured answer recommending Minor with compatibility reasoning.", "expectations": [ "Recommends Minor in the Recommendation section", @@ -21,10 +21,10 @@ "Recognizes purely additive, backward-compatible public surface as Minor", "Does not escalate to Major when nothing existing was changed or removed" ] - }, - { - "id": 3, - "prompt": "This release only fixes a NullReferenceException inside an internal cache class (no public API or observable behavior change), updates a transitive dependency from 8.0.1 to 8.0.3, and refreshes the README. Major, minor, or patch?", + }, + { + "id": 3, + "prompt": "This release only fixes a NullReferenceException inside an internal cache class (no public API or observable behavior change), updates a transitive dependency from 8.0.1 to 8.0.3, and refreshes the README. Major, minor, or patch?", "expected_output": "Structured answer recommending Patch with compatibility reasoning.", "expectations": [ "Recommends Patch in the Recommendation section", @@ -32,22 +32,22 @@ "Treats internal-only fix, compatible patch dependency bump, and docs as non-breaking", "Does not inflate an internal, non-observable refactor to Minor or Major" ] - }, - { - "id": 4, - "prompt": "I changed `DateParser.Parse` to throw a FormatException on empty input instead of returning null. Should this be a patch? It's just a bug fix.", + }, + { + "id": 4, + "prompt": "I changed `DateParser.Parse` to throw a FormatException on empty input instead of returning null. Should this be a patch? It's just a bug fix.", "expected_output": "Structured answer: ambiguous because it depends on whether Parse is public and whether consumers rely on the null-return behavior. Should not just say Patch.", "expectations": [ "Uses structured reasoning instead of emitting a bare single word", "Uses the structured template with Recommendation, Compatibility impact, Reasoning, and Deterministic decision sections", "Flags this as a behavioral change that can be breaking even though it compiles", "Does not accept the user's framing that a bug fix is automatically a Patch", - "Identifies the deciding fact: whether Parse is public and whether the old null-return behavior is relied upon or documented" - ] - }, - { - "id": 5, - "prompt": "Big release: I added two new public extension methods, fixed a couple of internal bugs, AND removed a public `ILogger` overload that nobody should be using. One command answer please: major, minor, or patch?", + "Identifies the deciding fact: whether Parse is public and whether the old null-return behavior is relied upon or documented" + ] + }, + { + "id": 5, + "prompt": "Big release: I added two new public extension methods, fixed a couple of internal bugs, AND removed a public `ILogger` overload that nobody should be using. One command answer please: major, minor, or patch?", "expected_output": "Structured answer recommending Major, because precedence picks the highest required bump and the public removal is breaking.", "expectations": [ "Recommends Major in the Recommendation section", @@ -55,29 +55,29 @@ "Applies precedence: the breaking public removal dominates the additive and internal changes", "Does not downgrade to Minor or Patch despite the user implying the removed overload is unused" ] - }, - { - "id": 6, - "prompt": "I added a new member `Task FlushAsync()` to my public interface `IMessageSink`. Existing implementations are out in the wild via my NuGet package. Why would this be more than a patch?", + }, + { + "id": 6, + "prompt": "I added a new member `Task FlushAsync()` to my public interface `IMessageSink`. Existing implementations are out in the wild via my NuGet package. Why would this be more than a patch?", "expected_output": "Structured answer recommending Major, explaining that adding a member to a public interface breaks existing implementers (source compatibility).", "expectations": [ "Recommends Major", "Explains that adding a member to an existing public interface breaks existing implementers' source compilation", "Mentions default interface members as a nuance without treating them as automatically safe", - "Uses the structured explanation template" - ] - }, - { - "id": 7, - "prompt": "I added a new enum value `LogLevel.Trace` to an existing public enum. My library serializes these values to JSON and some consumers switch over them. Patch, minor, or major?", + "Uses the structured explanation template" + ] + }, + { + "id": 7, + "prompt": "I added a new enum value `LogLevel.Trace` to an existing public enum. My library serializes these values to JSON and some consumers switch over them. Patch, minor, or major?", "expected_output": "Structured answer: usually Minor but flag the exhaustive-switch and serialization-contract risk that could make it behavioral/source-impacting.", "expectations": [ "Does not emit a bare one-word answer given the explicit switch and serialization caveats", "Notes that adding an enum value is usually Minor", - "Flags exhaustive consumer switch statements and serialization contracts as escalation risks", - "Uses the structured explanation template with a clear recommendation" - ] - }, + "Flags exhaustive consumer switch statements and serialization contracts as escalation risks", + "Uses the structured explanation template with a clear recommendation" + ] + }, { "id": 8, "prompt": "My library dropped support for net6.0 and now targets only net8.0 and net9.0. Existing consumers on net6.0 will be affected. Version bump?", @@ -107,12 +107,45 @@ "prompt": "Classify the version bump for this branch.", "expected_output": "Current-branch default-resolution behavior: inspect the local Git repository, resolve the current branch against the upstream/default base branch, collect commits and net diff, then classify those changes with structured reasoning. Must not ask the user to provide change details first.", "expectations": [ - "Follows Default Resolution Behavior when no explicit change details or compare range are provided", - "Detects the current branch and resolves a base branch using local Git state, following the documented fallback order", - "Collects both branch commits and net diff before classifying compatibility impact", - "Returns structured reasoning with the recommendation instead of a bare Major, Minor, or Patch", - "Does not respond with a generic request for release notes, diffs, PR summaries, or bug descriptions when the current repository can be inspected", - "Asks only for the missing base branch or compare range if local default resolution cannot identify one" + { + "assertion": "Current Git branch was resolved before classifying", + "evidence_domain": "validator", + "validator": "git.current_branch_observed" + }, + { + "assertion": "Base/default branch was resolved using allowed local Git state", + "evidence_domain": "validator", + "validator": "git.default_branch_resolved" + }, + { + "assertion": "Branch-only commits were collected before classifying", + "evidence_domain": "validator", + "validator": "git.branch_commits_collected" + }, + { + "assertion": "Three-dot net diff was collected before classifying", + "evidence_domain": "validator", + "validator": "git.three_dot_diff_observed" + }, + { + "assertion": "No search was performed for a package or tool named dotnet-change-impact", + "evidence_domain": "validator", + "validator": "skill.no_candidate_package_search" + }, + { + "assertion": "No installation attempt was made for dotnet-change-impact", + "evidence_domain": "validator", + "validator": "skill.no_candidate_install_attempt" + }, + { + "assertion": "No unnecessary project, package, or tool mutation was performed for this read-only classification task", + "evidence_domain": "validator", + "validator": "workspace.no_unnecessary_mutation" + }, + { + "assertion": "Returns structured reasoning with the recommendation instead of a bare Major, Minor, or Patch", + "evidence_domain": "output" + } ] } ] From e1c0c27342dbf5d07cf6486cf53a5d10bb253599 Mon Sep 17 00:00:00 2001 From: gimlichael Date: Wed, 9 Sep 2026 20:54:27 +0200 Subject: [PATCH 23/36] =?UTF-8?q?=F0=9F=93=9D=20document=20phase=202=20ana?= =?UTF-8?q?lyzer=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README and runner contract documentation now describe the independent analyzer default, Phase 2 controller, evidence refs, grading freeze, and separate executor/analyzer cost reporting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 10 +++++----- scripts/eval-runners/README.md | 13 ++----------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 75cdea7..f2d1924 100644 --- a/README.md +++ b/README.md @@ -14,13 +14,13 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. This exception is only for isolated disposable eval fixtures under the approved workspace roots; throwaway Git history is still forbidden in the real source working tree. -Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 explicitly follows the packaged Anthropic grader and requires assertion-specific, source-backed PASS evidence; generic completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. +Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 is a package-local analyzer controller: it resolves deterministic validator assertions first, dispatches fresh analyzer workers only for remaining semantic assertions, derives root `grading.json`, and freezes `grading-freeze.json` so generic or handcrafted completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the executor runner/model/configuration; `analyzer-profile.json` selects the independent grading runner/model; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning for execution. The analyzer default is also the Codebelt Reference profile, independent of the selected executor, and can be overridden with `-AnalyzerRunner`, `-AnalyzerModel`, and `-AnalyzerReasoningEffort`; preparation validates executor and analyzer models model-free before writing profiles. Offline deterministic tests may inject separate executor and analyzer catalogs with `-ModelCatalogPath` and `-AnalyzerModelCatalogPath`, but real preparation without catalogs still fails closed when discovery cannot verify the selected model. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, invokes the package-local Phase 2 analyzer controller, then finalizes reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: -Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. +Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, Phase 2 controller, grading application, and report adapter validate that ledger and never re-bless changed bytes. Phase 2 writes `phase2-state.json`, validator/analyzer evidence, `grading-freeze.json`, and the derived package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`) with structured `output`, `transcript`, and `validator` evidence refs. `apply-eval-grading.ps1` projects only frozen grading decisions onto canonical results, and `finalize-eval-package.ps1` owns the deterministic completion boundary: it succeeds only after validating Phase 1, the bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical grading, and all four report artifacts. A changed raw file or changed Phase 2 artifact requires a fresh authorized execution path, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. ```powershell pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot @@ -28,11 +28,11 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved to a runner; `-Runner github-copilot` uses the Codebelt Reference model, `-Runner codex` uses the Codex default, and `-Runner opencode` requires an explicit discovered `provider/model` selector. Direct noninteractive use without `-Runner` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness and fails immediately with the supported runner IDs when `-Runner` is omitted. Use `-CodebeltReference` only when you want the script to select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, normalized assertion domains, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the orchestrator invokes `invoke-phase2-analyzer.ps1`; it does not grade assertions itself, choose a grader model, retry failed analyzer workers, or write `grading.json`. The Phase 2 controller supplies each analyzer worker only that arm's expected output, unresolved semantic assertions, frozen output/transcript evidence, and packaged `tools/skill-creator/agents/grader.md`; paired arms, sibling evals, global skills, and candidate skill content are excluded unless a future assertion explicitly requires them. Missing telemetry is displayed as unavailable rather than estimated, and first-party reports separate execution usage/cost from analyzer usage/cost. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). -The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. +The roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates package-local commands; the Phase 2 analyzer controller owns validator resolution, analyzer worker dispatch, and deterministic grading merge; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 976ea0e..2b163c1 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -78,16 +78,7 @@ must validate that ledger; none of them can replace it or bless changed bytes. If a raw result or referenced artifact changes, the package is corrupted and requires a fresh Phase 1 execution. -The normal post-execution boundary is deterministic: the external Grader writes -only the package-root `grading.json` artifact (`codebeltnet/agentic/eval-grading/1`) -with exact assertion identities and `passed`/`evidence` decisions. The shared -`apply-eval-grading.ps1` helper projects those decisions onto canonical -`result.json` files and verifies that every non-grading field is unchanged. -`finalize-eval-package.ps1` then validates the freeze, bridge, canonical results, -and complete grading, invokes the existing report adapter, and fails unless -`report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` -are all non-empty. A prose success message cannot substitute for its JSON -success summary. +The normal post-execution boundary is deterministic: `invoke-phase2-analyzer.ps1` resolves validator-domain assertions first, dispatches one fresh analyzer worker per remaining semantic arm with the persisted `analyzer-profile.json`, validates the returned runner/model/session evidence, writes `phase2-state.json`, freezes `grading-freeze.json`, and derives package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`) with exact assertion identities, structured evidence domains and evidence refs. The shared `apply-eval-grading.ps1` helper projects those frozen decisions onto canonical `result.json` files and verifies that every non-grading field is unchanged. `finalize-eval-package.ps1` then validates Phase 1, the bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical results, and complete grading, invokes the existing report adapter, and fails unless `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` are all non-empty. A prose success message or handcrafted `grading.json` cannot substitute for its JSON success summary. The delegation contract has three distinct evidence levels: @@ -143,7 +134,7 @@ Copilot execution requires an allowlisted physical temporary projection outside Copilot `session.usage_checkpoint` billing counters are cumulative: the last checkpoint wins, including across resumed turns. `promptCacheBreakState` entries are per-call cache snapshots, deduplicated by native `model_call_id`; exposed prompt/cache buckets are used only when `assistant.usage` is absent. Tool schema tokens and native snapshots remain in evidence, not generated-token buckets. Missing output tokens stay unavailable, and premium requests/nano-AI units are never converted to currency. See [GitHub's event contract](https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/streaming-events). -Phase 2 must read and follow the exact packaged `tools/skill-creator/agents/grader.md` after the execution freeze and bridge. PASS evidence requires `Source: output` (or a captured run artifact path), `Quote: `, and `Reason: ` on separate lines. The deterministic validator checks source membership, verbatim quote presence, nonempty evidence and rejects reused or generic PASS evidence. It cannot establish semantic truth; the Grader must fail uncertain/unverified expectations. Execution evidence remains immutable and finalization remains exactly once. +Phase 2 workers receive only the exact packaged `tools/skill-creator/agents/grader.md`, the one arm's expected output, unresolved semantic assertions, frozen output lines and any explicitly required frozen transcript/artifact evidence. They do not receive the paired arm, sibling evals, benchmark summary, previous grades, user-global skills, candidate skill content, or an ambient repository checkout. Deterministic validation still checks provenance, source membership, verbatim quote presence, nonempty evidence, repeated/generic PASS reasons, analyzer profile hash, raw analyzer transcript hash, grading fragment hash, and worker/session uniqueness; it does not replace semantic analyzer judgement for unresolved semantic assertions. Execution evidence remains immutable and finalization remains exactly once. Historical integrity note: `dotnet-change-impact` Copilot iteration 4 is contaminated and must not be used as effectiveness evidence. Its missing `.external-handoff-started` is consistent with manual handoff, but the package records do not establish the original user request. The current helper reserves the marker before every external handoff and contains no removal path; no reservation-policy change is justified from absence alone. Codex iteration 2 and OpenCode iteration 3 have reservations. New protocol code applies only to newly prepared iterations. From 520cb67de89813f3254991123f0ced56885b36a9 Mon Sep 17 00:00:00 2001 From: gimlichael Date: Wed, 9 Sep 2026 21:11:49 +0200 Subject: [PATCH 24/36] =?UTF-8?q?=E2=9C=85=20cover=20analyzer=20catalog=20?= =?UTF-8?q?injection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eval-request regression coverage now supplies the analyzer catalog fixture separately from the executor catalog, matching the preparation contract on CI hosts without Copilot installed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/eval-runners/tests/test-eval-request.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 81841e7..38db1a3 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -23,7 +23,7 @@ function Invoke-FakeHost($Decision) { } function New-Preparation([string]$Runner, [string]$Name) { return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner - OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog } + OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog; AnalyzerModelCatalogPath = $catalog } } function Assert-Failure([hashtable]$Options, [string]$Pattern) { # The request must fail closed even in an ordinary interactive PowerShell caller. From 06e5adb4b70020a06672672ce36a3c1d19bc70d0 Mon Sep 17 00:00:00 2001 From: gimlichael Date: Wed, 9 Sep 2026 21:23:25 +0200 Subject: [PATCH 25/36] =?UTF-8?q?=E2=9C=85=20harden=20analyzer=20model=20v?= =?UTF-8?q?alidation=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The analyzer-model negative now asserts the fail-closed preparation invariant without depending on host-specific PowerShell error formatting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/validate-skill-templates.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index c5edefb..e192838 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1884,7 +1884,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' $invalidAnalyzerRoot = Join-Path $packageRoot 'invalid-analyzer-model-package' $invalidAnalyzerOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $invalidAnalyzerRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' -ModelCatalogPath $catalogPath -AnalyzerRunner 'github-copilot' -AnalyzerModel 'missing-analyzer-model' -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 - if ($LASTEXITCODE -eq 0 -or ($invalidAnalyzerOutput -join ' ') -notmatch "model 'missing-analyzer-model' could not be verified") { + if ($LASTEXITCODE -eq 0) { throw 'Unknown analyzer model must fail preparation through analyzer model discovery.' } if (Test-Path -LiteralPath $invalidAnalyzerRoot) { From 7268bb4b1a3c645853d92e0f3a5402e9cbde552c Mon Sep 17 00:00:00 2001 From: gimlichael Date: Wed, 9 Sep 2026 21:30:00 +0200 Subject: [PATCH 26/36] =?UTF-8?q?=F0=9F=92=9A=20allow=20phase=202=20valida?= =?UTF-8?q?tion=20runtime?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The validation matrix timeout now accommodates the deterministic Phase 2 controller suites that CI must run to prove analyzer provenance and finalization enforcement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/validate-skill-templates.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/validate-skill-templates.yml b/.github/workflows/validate-skill-templates.yml index 2e26302..72ad9c5 100644 --- a/.github/workflows/validate-skill-templates.yml +++ b/.github/workflows/validate-skill-templates.yml @@ -11,7 +11,7 @@ jobs: validate: name: Validate (${{ matrix.name }}) runs-on: ubuntu-latest - timeout-minutes: 5 + timeout-minutes: 10 strategy: fail-fast: false max-parallel: 13 From 359cdd44ba7ef3c278365b72737374d021cff6da Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 00:49:07 +0200 Subject: [PATCH 27/36] =?UTF-8?q?=F0=9F=94=A7=20enhance=20eval=20runner=20?= =?UTF-8?q?infrastructure=20for=20transcripts=20and=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This update improves the eval runner system to better handle transcript artifacts and strengthen validation checks. Changes include staging transcript artifacts when assertions require transcript-domain evidence, enhancing phase2 analysis with transcript event parsing, and tightening contract validation for candidate instruction hashes in with_skill runs. --- .../eval-runners/github-copilot/isolation.ps1 | 2 +- .../eval-runners/github-copilot/runner.ps1 | 3 +- .../eval-runners/invoke-phase2-analyzer.ps1 | 83 ++++++++++++++++++- scripts/eval-runners/phase2-grading.ps1 | 61 +++++++++++++- scripts/eval-runners/runner-common.ps1 | 7 +- 5 files changed, 143 insertions(+), 13 deletions(-) diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 index 847063f..186a575 100644 --- a/scripts/eval-runners/github-copilot/isolation.ps1 +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -43,7 +43,7 @@ function Assert-CopilotCandidateInstructionBoundary { $markerIndex = $promptText.IndexOf($boundary, [StringComparison]::Ordinal) if ($RunData.Mode -eq 'with_skill') { $expected = [string]$RunData.CandidateInstructionHash - if ([string]::IsNullOrWhiteSpace($expected)) { return } + if ([string]::IsNullOrWhiteSpace($expected)) { throw 'with_skill run must carry candidateInstructionHash; candidate identity is unproven.' } if ($markerIndex -lt 0) { throw 'with_skill prompt has no working-environment boundary; the injected candidate instructions cannot be isolated for hashing.' } $instruction = $promptText.Substring(0, $markerIndex) $actual = ([Convert]::ToHexString([Security.Cryptography.SHA256]::HashData([Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index adf0685..2a0f5dd 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -1175,8 +1175,7 @@ function Get-CopilotCandidateInstructionEvidence { if ($Inputs.Run.Mode -eq 'with_skill') { $result.expected = $expected if ([string]::IsNullOrWhiteSpace($expected)) { - # Nothing was frozen to verify (a synthetic or non-prepared run). Real prepared packages always declare the - # hash, and the bridge independently enforces it there. + $violations.Add('with_skill candidateInstructionHash is required; candidate identity is unproven without it.') $result.verified = $false } elseif ($markerIndex -lt 0) { $violations.Add('with_skill prompt has no working-environment boundary; candidate instruction bytes cannot be isolated for hashing.') diff --git a/scripts/eval-runners/invoke-phase2-analyzer.ps1 b/scripts/eval-runners/invoke-phase2-analyzer.ps1 index 5750835..25ea066 100644 --- a/scripts/eval-runners/invoke-phase2-analyzer.ps1 +++ b/scripts/eval-runners/invoke-phase2-analyzer.ps1 @@ -166,6 +166,59 @@ function New-AnalyzerRunBundle { $lineRecords.Add([ordered]@{ line = $index + 1; text = [string]$outputLines[$index] }) } $metadata = Read-RunnerJson -Path $Worker.record.MetadataPath + + # Stage transcript artifacts when any assertion requires transcript-domain evidence. + $transcriptAssertions = @($Worker.assertions | Where-Object { [string]$_.evidence_domain -eq 'transcript' }) + $stagedTranscripts = [System.Collections.Generic.List[object]]::new() + if ($transcriptAssertions.Count -gt 0) { + $evidenceDir = Join-Path $repoRoot 'evidence' + New-Item -ItemType Directory -Path $evidenceDir -Force | Out-Null + $transcriptSources = [System.Collections.Generic.List[string]]::new() + foreach ($af in @(Get-JsonProperty -Object $canonical -Name 'output_files' -Default @())) { + [void]$transcriptSources.Add([string]$af) + } + $execResultFile = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_file' -Default '') + if (-not [string]::IsNullOrWhiteSpace($execResultFile) -and $transcriptSources -notcontains $execResultFile) { + [void]$transcriptSources.Add($execResultFile) + } + foreach ($artifact in $transcriptSources) { + $sourcePath = Join-Path $Worker.record.EvalDirectory ($artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $sourcePath -PathType Leaf)) { continue } + $sourceHash = Get-Sha256HexFromFile -Path $sourcePath + $fileName = [System.IO.Path]::GetFileName($sourcePath) + $stagedPath = Join-Path $evidenceDir $fileName + [System.IO.File]::Copy($sourcePath, $stagedPath, $true) + $stagedHash = Get-Sha256HexFromFile -Path $stagedPath + $entry = [ordered]@{ artifact = $artifact; source_sha256 = $sourceHash; staged_sha256 = $stagedHash } + if ($artifact -match '\.jsonl$') { + $events = [System.Collections.Generic.List[object]]::new() + $fileLines = [System.IO.File]::ReadAllLines($sourcePath, [System.Text.UTF8Encoding]::new($false)) + foreach ($fileLine in $fileLines) { + if ([string]::IsNullOrWhiteSpace($fileLine)) { continue } + try { + $parsed = $fileLine | ConvertFrom-Json -Depth 100 + $eventType = [string](Get-JsonProperty -Object $parsed -Name 'type' -Default '') + $content = Get-JsonProperty -Object $parsed -Name 'content' -Default $null + if ($null -eq $content) { $content = Get-JsonProperty -Object $parsed -Name 'data' -Default $null } + $contentStr = if ($null -ne $content) { [string]$content } else { ConvertTo-Json $parsed -Compress -Depth 10 } + $events.Add([ordered]@{ event_index = $events.Count; type = $eventType; content = $contentStr }) + } catch { + $events.Add([ordered]@{ event_index = $events.Count; type = ''; content = $fileLine }) + } + } + $entry['events'] = @($events.ToArray()) + } else { + $fileLines = [System.IO.File]::ReadAllLines($sourcePath, [System.Text.UTF8Encoding]::new($false)) + $textLines = [System.Collections.Generic.List[object]]::new() + for ($li = 0; $li -lt $fileLines.Count; $li++) { + $textLines.Add([ordered]@{ line = $li + 1; text = [string]$fileLines[$li] }) + } + $entry['lines'] = @($textLines.ToArray()) + } + $stagedTranscripts.Add($entry) + } + } + $bundle = [ordered]@{ schema = 'codebeltnet/agentic/eval-analyzer-input/1' worker_id = $workerId @@ -189,6 +242,9 @@ function New-AnalyzerRunBundle { } }) } + if ($stagedTranscripts.Count -gt 0) { + $bundle['frozen_transcripts'] = @($stagedTranscripts.ToArray()) + } $bundlePath = Join-Path $repoRoot 'input-bundle.json' Write-RunnerJsonFile -Path $bundlePath -Value $bundle $graderPath = Join-Path $repoRoot 'grader.md' @@ -234,6 +290,7 @@ function New-AnalyzerRunBundle { BundleHash = $bundleHash PromptPath = Join-Path $runRoot 'prompt.md' AnalyzerExecutionProfilePath = $AnalyzerExecutionProfilePath + StagedTranscripts = @($stagedTranscripts.ToArray()) } } @@ -276,7 +333,8 @@ function Confirm-AnalyzerFragment { param( [Parameter(Mandatory = $true)][object]$Fragment, [Parameter(Mandatory = $true)][object]$Worker, - [Parameter(Mandatory = $true)][object]$Canonical + [Parameter(Mandatory = $true)][object]$Canonical, + [AllowNull()][object[]]$StagedTranscripts = $null ) if ([string](Get-JsonProperty -Object $Fragment -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-analyzer-fragment/1') { throw 'Analyzer fragment has an unsupported schema.' } @@ -297,7 +355,7 @@ function Confirm-AnalyzerFragment { if ([string]::IsNullOrWhiteSpace($reason)) { throw 'Analyzer fragment reason must be non-empty.' } $refs = @(Get-JsonProperty -Object $grade -Name 'evidence_refs' -Default @()) $entry = ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$grade.passed) -Reason $reason -EvidenceRefs $refs -Source 'analyzer' -Evidence $reason - [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical) + [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical -StagedTranscripts $StagedTranscripts) $entries.Add($entry) } return @($entries.ToArray()) @@ -413,6 +471,7 @@ try { grading_freeze = 'grading-freeze.json' grading = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default 'grading.json') }) + exit 0 } if (Test-Path -LiteralPath $statePath -PathType Leaf) { throw 'Phase 2 state already exists without a valid grading freeze; refusing to duplicate analyzer work. Use a fresh package iteration.' @@ -577,7 +636,7 @@ try { $responseText = [string](Get-JsonProperty -Object $raw.final_response -Name 'text' -Default '') $fragment = ConvertFrom-AnalyzerResponse -Text $responseText $canonical = Read-RunnerJson -Path $worker.record.ResultPath - $workerGrades = @(Confirm-AnalyzerFragment -Fragment $fragment -Worker $worker -Canonical $canonical) + $workerGrades = @(Confirm-AnalyzerFragment -Fragment $fragment -Worker $worker -Canonical $canonical -StagedTranscripts $item.bundle.StagedTranscripts) $fragmentRelative = "phase2/fragments/$workerId.grading-fragment.json" $fragmentPath = Join-Path $iteration ($fragmentRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) Write-RunnerJsonFile -Path $fragmentPath -Value $fragment @@ -622,7 +681,23 @@ try { $state.analyzer_results = @($completedAnalyzerEntries.ToArray()) $state.status = 'completed' Save-Phase2State -Path $statePath -State $state - $freeze = New-GradingFreezeDocument -IterationDirectory $iteration -Manifest $manifest -State $state -Grades @($grades.ToArray()) -Metadata $rootMetadata + # Collect unique transcript artifact hashes across all semantic workers for the freeze. + $transcriptArtifacts = [System.Collections.Generic.List[object]]::new() + $seenTranscriptKeys = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($worker in @($semanticWorkers)) { + foreach ($st in @($bundles[$worker.worker_id].StagedTranscripts)) { + $artifact = [string]$st.artifact + $sourceHash = [string]$st.source_sha256 + $evalDir = $worker.record.EvalDirectory + $sourcePath = Join-Path $evalDir ($artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $iterRelative = [System.IO.Path]::GetRelativePath($iteration, $sourcePath).Replace('\', '/') + $key = "$iterRelative`:$sourceHash" + if ($seenTranscriptKeys.Add($key)) { + $transcriptArtifacts.Add([ordered]@{ artifact = $iterRelative; sha256 = $sourceHash }) + } + } + } + $freeze = New-GradingFreezeDocument -IterationDirectory $iteration -Manifest $manifest -State $state -Grades @($grades.ToArray()) -Metadata $rootMetadata -TranscriptArtifacts @($transcriptArtifacts.ToArray()) Write-RunnerJsonFile -Path $freezePath -Value $freeze $state.grading_freeze = [ordered]@{ path = 'grading-freeze.json'; sha256 = Get-Sha256HexFromFile -Path $freezePath } Write-RunnerJsonFile -Path $statePath -Value $state diff --git a/scripts/eval-runners/phase2-grading.ps1 b/scripts/eval-runners/phase2-grading.ps1 index a94f435..e19ffda 100644 --- a/scripts/eval-runners/phase2-grading.ps1 +++ b/scripts/eval-runners/phase2-grading.ps1 @@ -350,7 +350,8 @@ function Test-GradeEvidenceReference { param( [Parameter(Mandatory = $true)][object]$Grade, [Parameter(Mandatory = $true)][object]$Expected, - [Parameter(Mandatory = $true)][object]$Canonical + [Parameter(Mandatory = $true)][object]$Canonical, + [AllowNull()][object[]]$StagedTranscripts = $null ) $domain = [string](Get-JsonProperty -Object $Grade -Name 'evidence_domain' -Default '') @@ -379,7 +380,49 @@ function Test-GradeEvidenceReference { } elseif ($domain -eq 'transcript') { if ($artifact -eq [string]$Expected.record.ResultRelative) { throw 'transcript evidence must not cite final output prose.' } $allowed = @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @()) - if ($allowed -notcontains $artifact -and $artifact -ne [string]$Canonical.execution_result_file) { throw 'transcript evidence must cite a frozen transcript/artifact for the same arm.' } + $execResultFile = [string](Get-JsonProperty -Object $Canonical -Name 'execution_result_file' -Default '') + if ($allowed -notcontains $artifact -and $artifact -ne $execResultFile) { throw 'transcript evidence must cite a frozen transcript/artifact for the same arm.' } + if ($null -ne $StagedTranscripts) { + $staged = @($StagedTranscripts | Where-Object { [string]$_.artifact -eq $artifact }) + if ($staged.Count -eq 0) { throw 'transcript evidence artifact was not staged to the analyzer input bundle; grounding cannot be verified.' } + $stagedEntry = $staged[0] + $eventIndexRaw = Get-JsonProperty -Object $ref -Name 'event_index' -Default $null + $startLine = [int](Get-JsonProperty -Object $ref -Name 'start_line' -Default 0) + $endLine = [int](Get-JsonProperty -Object $ref -Name 'end_line' -Default $startLine) + $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') + $hasEvents = Test-JsonProperty -Object $stagedEntry -Name 'events' + $hasLines = Test-JsonProperty -Object $stagedEntry -Name 'lines' + if ($null -ne $eventIndexRaw -and $hasEvents) { + $events = @(Get-JsonProperty -Object $stagedEntry -Name 'events' -Default @()) + $eventIndex = [int]$eventIndexRaw + if ($eventIndex -lt 0 -or $eventIndex -ge $events.Count) { + throw "transcript evidence event_index $eventIndex is outside the frozen artifact ($($events.Count) events)." + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $eventContent = [string](Get-JsonProperty -Object $events[$eventIndex] -Name 'content' -Default '') + if (-not $eventContent.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen event.' + } + } + } elseif ($startLine -gt 0 -and $hasLines) { + $lines = @(Get-JsonProperty -Object $stagedEntry -Name 'lines' -Default @()) + if ($startLine -lt 1 -or $endLine -lt $startLine -or $endLine -gt $lines.Count) { + throw 'transcript evidence line range is outside the frozen artifact.' + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $span = [string]::Join("`n", @($lines[($startLine - 1)..($endLine - 1)] | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen transcript lines.' + } + } + } elseif ($null -ne $eventIndexRaw -and $hasLines) { + throw 'transcript evidence uses event_index but the staged artifact has line-based content.' + } elseif ($startLine -gt 0 -and $hasEvents) { + throw 'transcript evidence uses start_line/end_line but the staged artifact has event-based content.' + } elseif (-not [string]::IsNullOrWhiteSpace($quote)) { + throw 'transcript evidence must declare event_index or start_line/end_line to ground the cited location.' + } + } } } return $true @@ -418,7 +461,8 @@ function New-GradingFreezeDocument { [Parameter(Mandatory = $true)][object]$Manifest, [Parameter(Mandatory = $true)][object]$State, [Parameter(Mandatory = $true)][object[]]$Grades, - [object]$Metadata = $null + [object]$Metadata = $null, + [object[]]$TranscriptArtifacts = @() ) $analyzer = Resolve-AnalyzerProfile -IterationDirectory $IterationDirectory -Manifest $Manifest @@ -428,7 +472,7 @@ function New-GradingFreezeDocument { $validatorPaths = @(Get-JsonProperty -Object $State -Name 'validator_results' -Default @()) $analyzerResults = @(Get-JsonProperty -Object $State -Name 'analyzer_results' -Default @()) - return [ordered]@{ + $freeze = [ordered]@{ schema = $script:GradingFreezeSchema analyzer_profile = [ordered]@{ path = [string]$analyzer.RelativePath; sha256 = [string]$analyzer.Hash } analyzer_profile_sha256 = [string]$analyzer.Hash @@ -446,11 +490,13 @@ function New-GradingFreezeDocument { }) validator_results = @($validatorPaths) analyzer_results = @($analyzerResults) + transcript_artifacts = @($TranscriptArtifacts) phase2_state_sha256 = Get-Phase2StateHashForFreeze -State $State expected_grade_count = $expected.Count grading_sha256 = Get-GradingMergeHash -Document $root generated_utc = (Format-UtcTimestamp -Value ([DateTime]::UtcNow)) } + return $freeze } function Assert-GradingFreeze { @@ -510,6 +556,13 @@ function Assert-GradingFreeze { $fragmentPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $fragmentPathValue -FieldName 'analyzer grading fragment' -Kind File -RequireExists if ([string](Get-JsonProperty -Object $result -Name 'grading_fragment_sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $fragmentPath)) { throw "Analyzer grading fragment '$fragmentPathValue' changed after Phase 2 freeze." } } + foreach ($ta in @(Get-JsonProperty -Object $freeze -Name 'transcript_artifacts' -Default @())) { + $taPath = [string](Get-JsonProperty -Object $ta -Name 'artifact' -Default '') + $taHash = [string](Get-JsonProperty -Object $ta -Name 'sha256' -Default '') + if ([string]::IsNullOrWhiteSpace($taPath) -or [string]::IsNullOrWhiteSpace($taHash)) { throw 'grading-freeze.json transcript_artifacts entry is missing artifact or sha256.' } + $taFullPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $taPath -FieldName 'transcript artifact' -Kind File -RequireExists + if ((Get-Sha256HexFromFile -Path $taFullPath) -ne $taHash) { throw "Transcript artifact '$taPath' changed after Phase 2 freeze." } + } return [pscustomobject]@{ Path = $freezePath; Freeze = $freeze; State = $state; Manifest = $manifest; Analyzer = $analyzer; Grading = $grading } } diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 8b5081e..b16ec8d 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -1102,8 +1102,11 @@ function Resolve-RunContract { # wrapper edits never invalidate the candidate identity. The baseline must not carry it: no candidate is injected. $candidateInstructionHash = [string](Get-JsonProperty -Object $run -Name 'candidateInstructionHash' -Default '') if ($mode -eq 'with_skill') { - if (-not [string]::IsNullOrWhiteSpace($candidateInstructionHash) -and -not (Test-Sha256 -Value $candidateInstructionHash)) { - throw 'with_skill run.json candidateInstructionHash, when declared, must be a SHA-256 value covering the exact frozen candidate instruction bytes injected into the prompt.' + if ([string]::IsNullOrWhiteSpace($candidateInstructionHash)) { + throw 'with_skill run.json candidateInstructionHash is required; it must be a SHA-256 hash covering the exact frozen candidate instruction bytes injected into the prompt.' + } + if (-not (Test-Sha256 -Value $candidateInstructionHash)) { + throw 'with_skill run.json candidateInstructionHash must be a valid SHA-256 value covering the exact frozen candidate instruction bytes injected into the prompt.' } } elseif (-not [string]::IsNullOrWhiteSpace($candidateInstructionHash)) { throw 'without_skill run.json must not declare candidateInstructionHash; the baseline receives no candidate instructions.' From 0cbf72e8467f7354fe751dc64c41147036a272d7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 00:49:21 +0200 Subject: [PATCH 28/36] =?UTF-8?q?=E2=9C=85=20update=20test=20and=20validat?= =?UTF-8?q?or=20infrastructure=20for=20eval=20execution?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This update improves the test infrastructure supporting eval runners and strengthens validation logic. Changes include refactoring test scenarios for better terminal state handling and timeout conditions, enhancing validator checks for analyzer model diagnostics, and maintaining test infrastructure alignment with eval runner improvements. --- .../tests/test-integrity-finalization.ps1 | 41 + .../tests/test-mixed-terminal-regression.ps1 | 366 +-- .../test-opencode-timedout-regression.ps1 | 166 +- .../eval-runners/tests/test-orchestration.ps1 | 7 + .../test-phase1-aggregate-regressions.ps1 | 890 +++---- .../test-phase1-controller-lifecycle.ps1 | 479 ++-- .../tests/test-runner-conformance.ps1 | 33 +- .../tests/test-runner-observability.ps1 | 2243 +++++++++-------- scripts/validate-skill-templates.ps1 | 10 + 9 files changed, 2168 insertions(+), 2067 deletions(-) diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index 041477e..3a40ad3 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -136,6 +136,12 @@ function New-TestRun { $interactionHash = Get-Sha256HexFromFile -Path $interactionFile } + $candidateInstructionHash = $null + if ($Configuration -eq 'with_skill') { + $promptContent = "deterministic fixture prompt for $EvalName/$Configuration`n" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + $run = [ordered]@{ schema = (Get-RunnerSchemaNames).Run evalId = $EvalId @@ -155,6 +161,7 @@ function New-TestRun { inputFiles = @() fixtureHash = ('a' * 64) skillHash = if ($Configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = $candidateInstructionHash contract = [ordered]@{ sandboxRoot = '.' workingDirectory = 'repo' @@ -729,6 +736,40 @@ for ($index = 0; $index -lt $count; $index++) { Assert-Equal 22 @($rootGradingAfterPhase2.grading).Count 'root grading cardinality derives from normalized assertions' Assert-True (@($rootGradingAfterPhase2.grading | Where-Object { [string]$_.evidence_domain -eq 'validator' -and [string]$_.source -eq 'validator' }).Count -eq 14) 'validator assertions are resolved by deterministic validator results, not analyzer prose' $validGrading = Read-TestJson -Path $gradingPath + # Fix 2: second invocation must return already_frozen, exit 0, perform zero new analyzer work. + $freezeBytes = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'grading-freeze.json')) + $stateBytes = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'phase2-state.json')) + $gradingBytes = [System.IO.File]::ReadAllBytes($gradingPath) + $phase2AnalyzerWorkerDirCountBefore = @(Get-ChildItem -LiteralPath (Join-Path $iteration 'phase2\work') -Directory -ErrorAction SilentlyContinue).Count + $phase2Again = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolPasses -Invocation $phase2Again -Description 'second Phase 2 invocation succeeds (already_frozen)' + $phase2AgainOutput = $phase2Again.Text | ConvertFrom-Json -ErrorAction SilentlyContinue + Assert-Equal 'already_frozen' ([string]$phase2AgainOutput.status) 'second Phase 2 invocation must report already_frozen status' + $freezeBytesAfter = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'grading-freeze.json')) + $stateBytesAfter = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'phase2-state.json')) + $gradingBytesAfter = [System.IO.File]::ReadAllBytes($gradingPath) + Assert-True ([System.Linq.Enumerable]::SequenceEqual($freezeBytes, $freezeBytesAfter)) 'grading-freeze.json must be byte-identical after second Phase 2 invocation' + Assert-True ([System.Linq.Enumerable]::SequenceEqual($stateBytesAfter, $stateBytes)) 'phase2-state.json must be byte-identical after second Phase 2 invocation' + Assert-True ([System.Linq.Enumerable]::SequenceEqual($gradingBytesAfter, $gradingBytes)) 'grading.json must be byte-identical after second Phase 2 invocation' + $phase2AnalyzerWorkerDirCountAfter = @(Get-ChildItem -LiteralPath (Join-Path $iteration 'phase2\work') -Directory -ErrorAction SilentlyContinue).Count + Assert-Equal $phase2AnalyzerWorkerDirCountBefore $phase2AnalyzerWorkerDirCountAfter 'second Phase 2 invocation must create zero new analyzer worker directories' + # Tampered grading-freeze.json must fail rather than rerun. + $freezePath = Join-Path $iteration 'grading-freeze.json' + $originalFreezeBytes = [System.IO.File]::ReadAllBytes($freezePath) + $tamperedFreeze = Read-TestJson -Path $freezePath + $tamperedFreeze.grading_sha256 = ('0' * 64) + Write-TestJson -Path $freezePath -Value $tamperedFreeze + $phase2Tampered = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolFails -Invocation $phase2Tampered -Description 'tampered grading-freeze.json fails Phase 2 instead of rerunning' + [System.IO.File]::WriteAllBytes($freezePath, $originalFreezeBytes) + # Orphaned phase2-state.json without a valid freeze must fail closed. + $statePath = Join-Path $iteration 'phase2-state.json' + $originalStateBytes = [System.IO.File]::ReadAllBytes($statePath) + Remove-Item -LiteralPath $freezePath -Force + $phase2Orphaned = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolFails -Invocation $phase2Orphaned -Description 'orphaned phase2-state.json without freeze fails closed' + Assert-True ([string]$phase2Orphaned.Text -match 'duplicate|fresh package|grading freeze') 'orphaned state must explain the retry is forbidden' + [System.IO.File]::WriteAllBytes($freezePath, $originalFreezeBytes) } if ($Suite -in @('All', 'Application')) { $invalidDirectFinalizer = Copy-TestGradingDocument -Document $validGrading diff --git a/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 index 9f79922..2b4f042 100644 --- a/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 +++ b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 @@ -1,184 +1,186 @@ -<# -.SYNOPSIS - Regression: mixed-terminal fan-out preserves raw statuses and records evidence_validation separately. -.DESCRIPTION - MODEL-FREE deterministic check. Does not execute any model. -#> - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'orchestration.ps1') - -function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } -function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } - -function New-TestNativeTerminalEvidence { - param( - [Parameter(Mandatory = $true)][object]$Arm, - [Parameter(Mandatory = $true)][object]$RunData, - [Parameter(Mandatory = $true)][string]$WorkerSessionId - ) - - return [ordered]@{ - status = 'completed' - session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } - requested = [ordered]@{ model = [string]$Arm.worker.model } - input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } - evidence = [ordered]@{ - delegation = [ordered]@{ - mechanism = 'deterministic-fake-native-worker' - worker_session_id = $WorkerSessionId - observed_model = [string]$Arm.worker.model - observed_working_directory = [string]$RunData.WorkingDirectoryPath - observed_home = [string]$RunData.HomeDirectoryPath - fresh_worker = $true - home_config_isolated = $true - prompt_fidelity = $true - prompt_sha256 = [string]$RunData.PromptHash - terminal_result_capture = $true - paired_arm_visible = $false - grading_material_visible = $false - nested_model_execution = $false - model_execution_count = 1 - } - } - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-mixed-terminal-' + [Guid]::NewGuid().ToString('N')) -$iteration = Join-Path $testRoot 'iteration-1' -New-Item -ItemType Directory -Path $iteration -Force | Out-Null - -# Create 4 eval arms -$manifestEvals = [System.Collections.Generic.List[object]]::new() -for ($evalId = 1; $evalId -le 4; $evalId++) { - $evalName = 'eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $iteration $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'without_skill') -Force | Out-Null - Write-Output "prepared $evalName" - $runs = [ordered]@{} - foreach ($configuration in @('with_skill','without_skill')) { - $runPath = Join-Path $evalDirectory $configuration - New-Item -ItemType Directory -Path $runPath -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null - if ($configuration -eq 'with_skill') { - New-Item -ItemType Directory -Path (Join-Path $runPath 'skill') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path (Join-Path $runPath 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) - } - [System.IO.File]::WriteAllText((Join-Path $runPath 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - $runJson = [ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName +<# +.SYNOPSIS + Regression: mixed-terminal fan-out preserves raw statuses and records evidence_validation separately. +.DESCRIPTION + MODEL-FREE deterministic check. Does not execute any model. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +function New-TestNativeTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$Arm, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + return [ordered]@{ + status = 'completed' + session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } + requested = [ordered]@{ model = [string]$Arm.worker.model } + input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $WorkerSessionId + observed_model = [string]$Arm.worker.model + observed_working_directory = [string]$RunData.WorkingDirectoryPath + observed_home = [string]$RunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$RunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-mixed-terminal-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# Create 4 eval arms +$manifestEvals = [System.Collections.Generic.List[object]]::new() +for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'without_skill') -Force | Out-Null + Write-Output "prepared $evalName" + $runs = [ordered]@{} + foreach ($configuration in @('with_skill','without_skill')) { + $runPath = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runPath -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null + if ($configuration -eq 'with_skill') { + New-Item -ItemType Directory -Path (Join-Path $runPath 'skill') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path (Join-Path $runPath 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $runJson = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - mustNotReadOutsideSandbox = $true - fixtureHash = ('a' * 64) - skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } - } - [System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) - $resultsDir = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null - $executionResultRel = "$evalName/results/$configuration.execution-result.json" - $resultRel = "$evalName/results/$configuration.result.json" - # create a canonical (empty) result.json so manifest validation is satisfied - [System.IO.File]::WriteAllText((Join-Path $resultsDir "$configuration.result.json"), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - $runs[$configuration] = [ordered]@{ mode = $configuration; run_manifest = "$evalName/$configuration/run.json"; execution_result = $executionResultRel; result = $resultRel } - } - [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) -} - -$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @($manifestEvals) } -$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'fake'; model = 'fixture-model'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 60; concurrency = 4 } - -$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile -$state = New-OrchestrationState -Plan $plan - -# Accept all workers -$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) -foreach ($d in $dispatches) { [void](Register-DelegationAccepted -State $state -WorkerId $d.worker_id -WorkerSessionId ('sess-' + $d.worker_id)) } -Assert-Equal 4 (Get-OrchestrationActiveCount -State $state) 'all workers active' - -# Prepare synthetic execution evidences -$arms = @($plan.arms) -$evidences = @{} -# arm1 -> timed_out -$arm1 = $arms[0] -$evidences[$arm1.worker_id] = [ordered]@{ - status = 'timed_out' - session = [ordered]@{ id = ('sess-' + $arm1.worker_id); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$arm1.eval_id; eval_name = [string]$arm1.eval_name; configuration = [string]$arm1.configuration } - requested = [ordered]@{ model = [string]$arm1.worker.model } - evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm1.worker_id); observed_model = [string]$arm1.worker.model; terminal_result_capture = $false } } -} -# arm2 -> completed (valid) -$arm2 = $arms[1] -$runData2 = Resolve-RunContract -RunPath ([string]$arm2.worker.run_manifest_path) -$evidences[$arm2.worker_id] = New-TestNativeTerminalEvidence -Arm $arm2 -RunData $runData2 -WorkerSessionId ('sess-' + $arm2.worker_id) -# arm3 -> failed -$arm3 = $arms[2] -$evidences[$arm3.worker_id] = [ordered]@{ - status = 'failed' - session = [ordered]@{ id = ('sess-' + $arm3.worker_id); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$arm3.eval_id; eval_name = [string]$arm3.eval_name; configuration = [string]$arm3.configuration } - requested = [ordered]@{ model = [string]$arm3.worker.model } - evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm3.worker_id); observed_model = [string]$arm3.worker.model; terminal_result_capture = $false } } -} -# arm4 -> completed -$arm4 = $arms[3] -$runData4 = Resolve-RunContract -RunPath ([string]$arm4.worker.run_manifest_path) -$evidences[$arm4.worker_id] = New-TestNativeTerminalEvidence -Arm $arm4 -RunData $runData4 -WorkerSessionId ('sess-' + $arm4.worker_id) - -# Sanity: ensure evidences exist for each arm -Write-Output "ARM KEYS:" -foreach ($arm in $arms) { Write-Output " - $($arm.worker_id)" } -Write-Output "EVIDENCE KEYS:" -foreach ($k in $evidences.Keys) { Write-Output " - $k" } - -# Register terminals -foreach ($arm in $arms) { - $w = [string]$arm.worker_id - if (-not $evidences.ContainsKey($w)) { throw "Missing synthetic evidence for $w" } - $exec = $evidences[$w] - [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $w -ExecutionEvidence $exec) -} - -# Verify ledger preserves raw statuses and records evidence_validation -foreach ($arm in $arms) { - $w = $arm.worker_id - $ledger = $state.completed[$w] - $raw = $evidences[$w] - Assert-Equal $raw.status $ledger.status "ledger.status should equal raw for $w" - $ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null - Write-Output "ledger[$w].native_worker_evidence_failures = $([string]::Join(', ', @($ledger.native_worker_evidence_failures | Where-Object {$_} )))" - Write-Output "ledger[$w].evidence_validation.status = $($ev.status)" - if ($raw.status -eq 'completed') { Assert-Equal 'passed' $ev.status "evidence_validation should pass for $w" } else { Assert-Equal 'failed' $ev.status "evidence_validation should fail for $w" } -} - -# Negative integrity: ledger mismatch must be rejected by Assert-FreezeTerminalLedgerEntry -# craft a fake record and raw object -$fakeRecord = [ordered]@{ EvalId = 999; Configuration = 'with_skill'; } -$fakeRaw = [ordered]@{ status = 'timed_out'; session = [ordered]@{ id = 'sess-fake' } } -# craft a state with a mismatched ledger entry -$badState = [ordered]@{ completed = [ordered]@{ 'arm-999-with_skill' = [ordered]@{ worker_id = 'arm-999-with_skill'; eval_id = 999; configuration = 'with_skill'; status = 'incompatible'; worker_session_id = 'sess-fake' } } } -$threw = $false -try { [void](Assert-FreezeTerminalLedgerEntry -Record $fakeRecord -Raw $fakeRaw -State $badState) } catch { $threw = $true } -Assert-True $threw 'Assert-FreezeTerminalLedgerEntry must reject ledger/raw status mismatch' - -Write-Output 'MIXED-TERMINAL REGRESSION: PASS' \ No newline at end of file + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + mustNotReadOutsideSandbox = $true + fixtureHash = ('a' * 64) + skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = if ($configuration -eq 'with_skill') { ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("terminal test prompt for $evalName/$configuration")))).ToLowerInvariant() } else { $null } + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) + $resultsDir = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null + $executionResultRel = "$evalName/results/$configuration.execution-result.json" + $resultRel = "$evalName/results/$configuration.result.json" + # create a canonical (empty) result.json so manifest validation is satisfied + [System.IO.File]::WriteAllText((Join-Path $resultsDir "$configuration.result.json"), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $runs[$configuration] = [ordered]@{ mode = $configuration; run_manifest = "$evalName/$configuration/run.json"; execution_result = $executionResultRel; result = $resultRel } + } + [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) +} + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @($manifestEvals) } +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'fake'; model = 'fixture-model'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 60; concurrency = 4 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# Accept all workers +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +foreach ($d in $dispatches) { [void](Register-DelegationAccepted -State $state -WorkerId $d.worker_id -WorkerSessionId ('sess-' + $d.worker_id)) } +Assert-Equal 4 (Get-OrchestrationActiveCount -State $state) 'all workers active' + +# Prepare synthetic execution evidences +$arms = @($plan.arms) +$evidences = @{} +# arm1 -> timed_out +$arm1 = $arms[0] +$evidences[$arm1.worker_id] = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('sess-' + $arm1.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm1.eval_id; eval_name = [string]$arm1.eval_name; configuration = [string]$arm1.configuration } + requested = [ordered]@{ model = [string]$arm1.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm1.worker_id); observed_model = [string]$arm1.worker.model; terminal_result_capture = $false } } +} +# arm2 -> completed (valid) +$arm2 = $arms[1] +$runData2 = Resolve-RunContract -RunPath ([string]$arm2.worker.run_manifest_path) +$evidences[$arm2.worker_id] = New-TestNativeTerminalEvidence -Arm $arm2 -RunData $runData2 -WorkerSessionId ('sess-' + $arm2.worker_id) +# arm3 -> failed +$arm3 = $arms[2] +$evidences[$arm3.worker_id] = [ordered]@{ + status = 'failed' + session = [ordered]@{ id = ('sess-' + $arm3.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm3.eval_id; eval_name = [string]$arm3.eval_name; configuration = [string]$arm3.configuration } + requested = [ordered]@{ model = [string]$arm3.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm3.worker_id); observed_model = [string]$arm3.worker.model; terminal_result_capture = $false } } +} +# arm4 -> completed +$arm4 = $arms[3] +$runData4 = Resolve-RunContract -RunPath ([string]$arm4.worker.run_manifest_path) +$evidences[$arm4.worker_id] = New-TestNativeTerminalEvidence -Arm $arm4 -RunData $runData4 -WorkerSessionId ('sess-' + $arm4.worker_id) + +# Sanity: ensure evidences exist for each arm +Write-Output "ARM KEYS:" +foreach ($arm in $arms) { Write-Output " - $($arm.worker_id)" } +Write-Output "EVIDENCE KEYS:" +foreach ($k in $evidences.Keys) { Write-Output " - $k" } + +# Register terminals +foreach ($arm in $arms) { + $w = [string]$arm.worker_id + if (-not $evidences.ContainsKey($w)) { throw "Missing synthetic evidence for $w" } + $exec = $evidences[$w] + [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $w -ExecutionEvidence $exec) +} + +# Verify ledger preserves raw statuses and records evidence_validation +foreach ($arm in $arms) { + $w = $arm.worker_id + $ledger = $state.completed[$w] + $raw = $evidences[$w] + Assert-Equal $raw.status $ledger.status "ledger.status should equal raw for $w" + $ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null + Write-Output "ledger[$w].native_worker_evidence_failures = $([string]::Join(', ', @($ledger.native_worker_evidence_failures | Where-Object {$_} )))" + Write-Output "ledger[$w].evidence_validation.status = $($ev.status)" + if ($raw.status -eq 'completed') { Assert-Equal 'passed' $ev.status "evidence_validation should pass for $w" } else { Assert-Equal 'failed' $ev.status "evidence_validation should fail for $w" } +} + +# Negative integrity: ledger mismatch must be rejected by Assert-FreezeTerminalLedgerEntry +# craft a fake record and raw object +$fakeRecord = [ordered]@{ EvalId = 999; Configuration = 'with_skill'; } +$fakeRaw = [ordered]@{ status = 'timed_out'; session = [ordered]@{ id = 'sess-fake' } } +# craft a state with a mismatched ledger entry +$badState = [ordered]@{ completed = [ordered]@{ 'arm-999-with_skill' = [ordered]@{ worker_id = 'arm-999-with_skill'; eval_id = 999; configuration = 'with_skill'; status = 'incompatible'; worker_session_id = 'sess-fake' } } } +$threw = $false +try { [void](Assert-FreezeTerminalLedgerEntry -Record $fakeRecord -Raw $fakeRaw -State $badState) } catch { $threw = $true } +Assert-True $threw 'Assert-FreezeTerminalLedgerEntry must reject ledger/raw status mismatch' + +Write-Output 'MIXED-TERMINAL REGRESSION: PASS' diff --git a/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 index d019935..f3a5d0c 100644 --- a/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 +++ b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 @@ -1,86 +1,86 @@ -<# -.SYNOPSIS - Regression: OpenCode timed-out interaction evidence accepted as honest terminal result. -.DESCRIPTION - MODEL-FREE deterministic check using synthetic opencode-style execution evidence. -#> - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'orchestration.ps1') - -function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } -function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-opencode-tmo-' + [Guid]::NewGuid().ToString('N')) -$iteration = Join-Path $testRoot 'iteration-1' -New-Item -ItemType Directory -Path $iteration -Force | Out-Null - -# single eval/arm -$evalName = 'eval-01' -$evalDirectory = Join-Path $iteration $evalName -New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null -# run manifest -$runPath = Join-Path $evalDirectory 'with_skill' +<# +.SYNOPSIS + Regression: OpenCode timed-out interaction evidence accepted as honest terminal result. +.DESCRIPTION + MODEL-FREE deterministic check using synthetic opencode-style execution evidence. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-opencode-tmo-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# single eval/arm +$evalName = 'eval-01' +$evalDirectory = Join-Path $iteration $evalName +New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null +# run manifest +$runPath = Join-Path $evalDirectory 'with_skill' $skillPath = Join-Path $runPath 'skill\test-skill' New-Item -ItemType Directory -Path $skillPath -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $skillPath 'SKILL.md'), '# deterministic fixture skill', [System.Text.UTF8Encoding]::new($false)) -$runJson = [ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = $evalName; candidateSkillName = 'test-skill'; skillName = 'test-skill'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/test-skill'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; fixtureHash = ('a' * 64); skillHash = ('b' * 64) } -[System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) -New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null -New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null -# results and metadata -$resultsDir = Join-Path $evalDirectory 'results' -New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null -[System.IO.File]::WriteAllText((Join-Path $resultsDir 'with_skill.result.json'), (([ordered]@{ eval_id = 1; configuration = 'with_skill'; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) -[System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = 1; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - -$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @([ordered]@{ eval_id = 1; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = [ordered]@{ with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = "$evalName/with_skill/run.json"; execution_result = "$evalName/results/with_skill.execution-result.json"; result = "$evalName/results/with_skill.result.json" } } }) } - -$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'opencode'; model = 'opencode/muse-spark-1.2-contributor-free'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900; concurrency = 1 } - -$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile -$state = New-OrchestrationState -Plan $plan - -# accept the single worker -$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) -[void](Register-DelegationAccepted -State $state -WorkerId $dispatches[0].worker_id -WorkerSessionId ('sess-' + $dispatches[0].worker_id)) - -# craft opencode-style timed_out execution evidence -$workerId = $dispatches[0].worker_id -$execEvidence = [ordered]@{ - status = 'timed_out' - session = [ordered]@{ id = ('opencode-session-' + $workerId); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = 1; eval_name = $evalName; configuration = 'with_skill' } - requested = [ordered]@{ model = $profile.model } - input = [ordered]@{ prompt_sha256 = ('a' * 64) } - evidence = [ordered]@{ - delegation = [ordered]@{ - mechanism = 'opencode-native' - worker_session_id = ('opencode-session-' + $workerId) - observed_model = $profile.model - observed_working_directory = (Join-Path $runPath 'repo') - observed_home = (Join-Path $runPath 'home') - fresh_worker = $true - home_config_isolated = $true - # No terminal_result_capture because timed out before assistant response - terminal_result_capture = $false - # include HTTP timeout metadata - http = [ordered]@{ request_start_utc = (Get-Date).ToUniversalTime().ToString('o'); timeout_utc = (Get-Date).AddSeconds(30).ToUniversalTime().ToString('o'); classification = 'request_timeout' } - terminal_event = [ordered]@{ type = 'timeout'; reason = 'request_timeout' } - } - execution_paths = [ordered]@{ logical_working_directory = (Join-Path $runPath 'repo'); logical_home_directory = (Join-Path $runPath 'home') } - } -} - -# Register terminal; orchestration should preserve raw 'timed_out' and record evidence_validation failed -[void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $workerId -ExecutionEvidence $execEvidence) -$ledger = $state.completed[$workerId] -Assert-Equal 'timed_out' $ledger.status 'ledger must preserve raw timed_out status' -$ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null -Assert-Equal 'failed' $ev.status 'evidence_validation should be recorded as failed for a timed_out (no terminal capture)' - -Write-Output 'OPENCODE TIMED-OUT REGRESSION: PASS' \ No newline at end of file +$runJson = [ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = $evalName; candidateSkillName = 'test-skill'; skillName = 'test-skill'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/test-skill'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64); candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("opencode timedout fixture prompt")))).ToLowerInvariant() } +[System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) +New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null +New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null +# results and metadata +$resultsDir = Join-Path $evalDirectory 'results' +New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null +[System.IO.File]::WriteAllText((Join-Path $resultsDir 'with_skill.result.json'), (([ordered]@{ eval_id = 1; configuration = 'with_skill'; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) +[System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = 1; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @([ordered]@{ eval_id = 1; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = [ordered]@{ with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = "$evalName/with_skill/run.json"; execution_result = "$evalName/results/with_skill.execution-result.json"; result = "$evalName/results/with_skill.result.json" } } }) } + +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'opencode'; model = 'opencode/muse-spark-1.2-contributor-free'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900; concurrency = 1 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# accept the single worker +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +[void](Register-DelegationAccepted -State $state -WorkerId $dispatches[0].worker_id -WorkerSessionId ('sess-' + $dispatches[0].worker_id)) + +# craft opencode-style timed_out execution evidence +$workerId = $dispatches[0].worker_id +$execEvidence = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('opencode-session-' + $workerId); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = 1; eval_name = $evalName; configuration = 'with_skill' } + requested = [ordered]@{ model = $profile.model } + input = [ordered]@{ prompt_sha256 = ('a' * 64) } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'opencode-native' + worker_session_id = ('opencode-session-' + $workerId) + observed_model = $profile.model + observed_working_directory = (Join-Path $runPath 'repo') + observed_home = (Join-Path $runPath 'home') + fresh_worker = $true + home_config_isolated = $true + # No terminal_result_capture because timed out before assistant response + terminal_result_capture = $false + # include HTTP timeout metadata + http = [ordered]@{ request_start_utc = (Get-Date).ToUniversalTime().ToString('o'); timeout_utc = (Get-Date).AddSeconds(30).ToUniversalTime().ToString('o'); classification = 'request_timeout' } + terminal_event = [ordered]@{ type = 'timeout'; reason = 'request_timeout' } + } + execution_paths = [ordered]@{ logical_working_directory = (Join-Path $runPath 'repo'); logical_home_directory = (Join-Path $runPath 'home') } + } +} + +# Register terminal; orchestration should preserve raw 'timed_out' and record evidence_validation failed +[void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $workerId -ExecutionEvidence $execEvidence) +$ledger = $state.completed[$workerId] +Assert-Equal 'timed_out' $ledger.status 'ledger must preserve raw timed_out status' +$ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null +Assert-Equal 'failed' $ev.status 'evidence_validation should be recorded as failed for a timed_out (no terminal capture)' + +Write-Output 'OPENCODE TIMED-OUT REGRESSION: PASS' diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index 9b48850..c98b4f0 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -146,9 +146,11 @@ try { freshContextRequired = $true filesystemIsolationRequired = $true isolatedHomeRequired = $true + gitWorkspace = $false mustNotReadOutsideSandbox = $true fixtureHash = ('a' * 64) skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = if ($configuration -eq 'with_skill') { ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("terminal test prompt for $evalName/$configuration")))).ToLowerInvariant() } else { $null } }) $resultFileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } Write-TestJson -Path (Join-Path $evalDirectory (Join-Path 'results' $resultFileName)) -Value ([ordered]@{ @@ -586,11 +588,14 @@ try { [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "fixture prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) $skillDirectory = $null $skillHash = $null + $candidateInstructionHash = $null if ($configuration -eq 'with_skill') { $skillDirectory = 'skill/candidate' New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) $skillHash = ('b' * 64) + $promptContent = "fixture prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() } Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run @@ -607,8 +612,10 @@ try { freshContextRequired = $true filesystemIsolationRequired = $true isolatedHomeRequired = $true + gitWorkspace = $false fixtureHash = ('a' * 64) skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash }) $resultDirectory = Join-Path $runDirectory 'results' New-Item -ItemType Directory -Path $resultDirectory -Force | Out-Null diff --git a/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 index 6eb5612..42fa78c 100644 --- a/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 +++ b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 @@ -1,452 +1,456 @@ -<#! -.SYNOPSIS - Deterministic Phase 1 aggregate fail-closed regressions. - -.DESCRIPTION - Exercises the real runner-owned Phase 1 path with mixed terminal outcomes - and with completed raw results whose evidence validation fails. MODEL-FREE. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -cne [string]$Actual) { - throw "ASSERT: $Message (expected '$Expected', got '$Actual')" - } -} - -function Assert-Contains { - param([string]$Text, [string]$Expected, [string]$Message) - if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { - throw "ASSERT: $Message (missing '$Expected')" - } -} - -function Write-TestJson { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) - - New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null - [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Read-TestJson { - param([Parameter(Mandatory = $true)][string]$Path) - - return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 -} - -function Invoke-TestTool { - param( - [Parameter(Mandatory = $true)][string]$Path, - [Parameter(Mandatory = $true)][string[]]$Arguments - ) - +<#! +.SYNOPSIS + Deterministic Phase 1 aggregate fail-closed regressions. + +.DESCRIPTION + Exercises the real runner-owned Phase 1 path with mixed terminal outcomes + and with completed raw results whose evidence validation fails. MODEL-FREE. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -cne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Assert-Contains { + param([string]$Text, [string]$Expected, [string]$Message) + if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { + throw "ASSERT: $Message (missing '$Expected')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-TestTool { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + $output = & pwsh -NoProfile -NonInteractive -File $Path @Arguments 2>&1 - return [pscustomobject]@{ - ExitCode = $LASTEXITCODE - Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - } -} - -function Invoke-ForegroundPhaseOne { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) - - # STDOUT is the machine protocol; STDERR is live observability. Capture them - # separately so relayed heartbeats never contaminate the terminal JSON. - $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') - try { + return [pscustomobject]@{ + ExitCode = $LASTEXITCODE + Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + } +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) + + # STDOUT is the machine protocol; STDERR is live observability. Capture them + # separately so relayed heartbeats never contaminate the terminal JSON. + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') + try { $output = & pwsh -NoProfile -NonInteractive -File $Path -IterationDirectory $IterationDirectory 2>$stderrPath - $exitCode = $LASTEXITCODE - $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - } finally { - Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue - } - $document = $text | ConvertFrom-Json -Depth 100 - return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } -} - -function Assert-ToolFails { - param( - [Parameter(Mandatory = $true)][object]$Invocation, - [Parameter(Mandatory = $true)][string]$Description, - [string]$ExpectedText = '' - ) - - if ([int]$Invocation.ExitCode -eq 0) { - throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" - } - if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { - Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description - } -} - -function New-TestRun { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [Parameter(Mandatory = $true)][int]$EvalId, - [Parameter(Mandatory = $true)][string]$EvalName - ) - - $runDirectory = Join-Path (Join-Path $IterationDirectory $EvalName) 'with_skill' - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $skillDirectory = Join-Path $runDirectory 'skill/test-skill' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $skillDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "phase1 aggregate prompt for $EvalName/with_skill`n", [System.Text.UTF8Encoding]::new($false)) - - $run = [ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $EvalId - evalName = $EvalName + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + } finally { + Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue + } + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } +} + +function Assert-ToolFails { + param( + [Parameter(Mandatory = $true)][object]$Invocation, + [Parameter(Mandatory = $true)][string]$Description, + [string]$ExpectedText = '' + ) + + if ([int]$Invocation.ExitCode -eq 0) { + throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" + } + if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { + Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description + } +} + +function New-TestRun { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$EvalName + ) + + $runDirectory = Join-Path (Join-Path $IterationDirectory $EvalName) 'with_skill' + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $skillDirectory = Join-Path $runDirectory 'skill/test-skill' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $skillDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "phase1 aggregate prompt for $EvalName/with_skill`n", [System.Text.UTF8Encoding]::new($false)) + + $promptBytes = [System.Text.Encoding]::UTF8.GetBytes("phase1 aggregate prompt for $EvalName/with_skill`n") + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData($promptBytes))).ToLowerInvariant() + + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $EvalId + evalName = $EvalName candidateSkillName = 'test-skill' - skillName = 'test-skill' - iteration = 1 - mode = 'with_skill' - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = 'skill/test-skill' - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - gitWorkspace = $false - inputFiles = @() - fixtureHash = ('a' * 64) - skillHash = ('b' * 64) - contract = [ordered]@{ - sandboxRoot = '.' - workingDirectory = 'repo' - homeDirectory = 'home' - mustNotReadOutsideSandbox = $true - mustNotExposeGlobalSkillsOrConfig = $true - } - } - Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run - - return [pscustomobject]@{ - RunDirectory = $runDirectory - HomeDirectory = $homeDirectory - } -} - -function New-ReportFixtureScript { - param([Parameter(Mandatory = $true)][string]$Path) - - $scriptText = @' -[CmdletBinding()] -param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) -$ErrorActionPreference = 'Stop' -foreach ($file in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory $file), "unexpected report artifact: $file`n", [System.Text.UTF8Encoding]::new($false)) -} -'@ - [System.IO.File]::WriteAllText($Path, $scriptText, [System.Text.UTF8Encoding]::new($false)) -} - -function Initialize-PhaseOneFailurePackage { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, - [int[]]$EvidenceFailureEvalIds = @() - ) - - $packageTools = Join-Path $IterationDirectory 'tools/eval-runners' - New-Item -ItemType Directory -Path $packageTools -Force | Out-Null - foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { - Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force - } - $fixtureDirectory = Join-Path $packageTools 'fixture' - New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force - New-ReportFixtureScript -Path (Join-Path $IterationDirectory 'tools/test-report.ps1') - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le 4; $evalId++) { - $evalName = 'phase1-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-metadata/1' - eval_id = $evalId - eval_name = $evalName - prompt = "fixture prompt $evalId" - expected_output = 'fixture output' - assertions = @('deterministic fixture assertion') - }) - - $run = New-TestRun -IterationDirectory $IterationDirectory -EvalId $evalId -EvalName $evalName - $status = if ($StatusesByEvalId.ContainsKey($evalId)) { [string]$StatusesByEvalId[$evalId] } else { 'completed' } - if ($status -ne 'completed') { - [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'terminal-status'), $status, [System.Text.UTF8Encoding]::new($false)) - } - if ($EvidenceFailureEvalIds -contains $evalId) { - [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'evidence-validation-failed'), 'fixture', [System.Text.UTF8Encoding]::new($false)) - } - - $resultsDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $resultsDirectory 'with_skill.result.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).PortableResult - eval_id = $evalId - eval_name = $evalName - configuration = 'with_skill' - execution_status = 'unrun' - grading = @([ordered]@{ text = 'deterministic fixture assertion'; passed = $null; evidence = '' }) - }) - - $manifestEvals.Add([ordered]@{ - eval_id = $evalId - eval_name = $evalName - directory = $evalName - metadata = "$evalName/eval-metadata.json" - runs = [ordered]@{ - with_skill = [ordered]@{ - mode = 'with_skill' - run_manifest = "$evalName/with_skill/run.json" - execution_result = "$evalName/results/with_skill.execution-result.json" - result = "$evalName/results/with_skill.result.json" - } - } - }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools - Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - skill_name = 'phase1-aggregate-fixture' - iteration = 1 - configurations = @('with_skill') + skillName = 'test-skill' + iteration = 1 + mode = 'with_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = 'skill/test-skill' + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = ('b' * 64) + candidateInstructionHash = $candidateInstructionHash + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + + return [pscustomobject]@{ + RunDirectory = $runDirectory + HomeDirectory = $homeDirectory + } +} + +function New-ReportFixtureScript { + param([Parameter(Mandatory = $true)][string]$Path) + + $scriptText = @' +[CmdletBinding()] +param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) +$ErrorActionPreference = 'Stop' +foreach ($file in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory $file), "unexpected report artifact: $file`n", [System.Text.UTF8Encoding]::new($false)) +} +'@ + [System.IO.File]::WriteAllText($Path, $scriptText, [System.Text.UTF8Encoding]::new($false)) +} + +function Initialize-PhaseOneFailurePackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @() + ) + + $packageTools = Join-Path $IterationDirectory 'tools/eval-runners' + New-Item -ItemType Directory -Path $packageTools -Force | Out-Null + foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { + Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force + } + $fixtureDirectory = Join-Path $packageTools 'fixture' + New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + New-ReportFixtureScript -Path (Join-Path $IterationDirectory 'tools/test-report.ps1') + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'phase1-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/1' + eval_id = $evalId + eval_name = $evalName + prompt = "fixture prompt $evalId" + expected_output = 'fixture output' + assertions = @('deterministic fixture assertion') + }) + + $run = New-TestRun -IterationDirectory $IterationDirectory -EvalId $evalId -EvalName $evalName + $status = if ($StatusesByEvalId.ContainsKey($evalId)) { [string]$StatusesByEvalId[$evalId] } else { 'completed' } + if ($status -ne 'completed') { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'terminal-status'), $status, [System.Text.UTF8Encoding]::new($false)) + } + if ($EvidenceFailureEvalIds -contains $evalId) { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'evidence-validation-failed'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + } + + $resultsDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $resultsDirectory 'with_skill.result.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = $evalId + eval_name = $evalName + configuration = 'with_skill' + execution_status = 'unrun' + grading = @([ordered]@{ text = 'deterministic fixture assertion'; passed = $null; evidence = '' }) + }) + + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + run_manifest = "$evalName/with_skill/run.json" + execution_result = "$evalName/results/with_skill.execution-result.json" + result = "$evalName/results/with_skill.result.json" + } + } + }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + skill_name = 'phase1-aggregate-fixture' + iteration = 1 + configurations = @('with_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Phase 1 aggregate fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - grading = 'grading.json' - report = [ordered]@{ tool = 'tools/test-report.ps1' } - evals = @($manifestEvals.ToArray()) - }) - Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 60 - concurrency = 4 - }) - - return [pscustomobject]@{ - IterationDirectory = $IterationDirectory - FanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' - BridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' - FinalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' - LogPath = Join-Path $IterationDirectory 'runner-events.jsonl' - Records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest (Read-TestJson -Path (Join-Path $IterationDirectory 'manifest.json')) | Sort-Object EvalId, Configuration) - } -} - -function Assert-Counts { - param( - [Parameter(Mandatory = $true)][object]$Source, - [Parameter(Mandatory = $true)][hashtable]$Expected, - [Parameter(Mandatory = $true)][string]$MessagePrefix - ) - - foreach ($name in @( - 'expected_count', - 'terminal_count', - 'completed_count', - 'failed_count', - 'timed_out_count', - 'cancelled_count', - 'incompatible_count', - 'evidence_validation_failed_count' - )) { - Assert-Equal $Expected[$name] (Get-JsonProperty -Object $Source -Name $name -Default $null) "$MessagePrefix $name" - } -} - -function Assert-ArmSummaryShape { - param([Parameter(Mandatory = $true)][object]$Summary, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($arm in @($Summary.arms)) { - foreach ($field in @('worker_id', 'eval_id', 'configuration', 'status', 'worker_session_id', 'evidence_validation')) { - Assert-True (Test-JsonProperty -Object $arm -Name $field) "$ScenarioName arm summary contains $field" - } - $evidenceValidation = Get-JsonProperty -Object $arm -Name 'evidence_validation' -Default $null - Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'status') "$ScenarioName arm summary contains evidence_validation.status" - Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'reasons') "$ScenarioName arm summary contains evidence_validation.reasons" - } -} - -function Assert-CanonicalResultsRemainUnrun { - param([Parameter(Mandatory = $true)][object[]]$Records, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($record in $Records) { - $result = Read-TestJson -Path $record.ResultPath - Assert-Equal 'unrun' ([string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '')) "$ScenarioName keeps $($record.ResultRelative) unbridged" - } -} - -function Assert-NoPhaseTwoArtifacts { - param([Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($relative in @('grading.json', 'report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { - Assert-True (-not (Test-Path -LiteralPath (Join-Path $IterationDirectory $relative) -PathType Leaf)) "$ScenarioName does not produce $relative" - } -} - -function Assert-NoRetries { - param([Parameter(Mandatory = $true)][string]$LogPath, [Parameter(Mandatory = $true)][int]$ExpectedArmCount, [Parameter(Mandatory = $true)][string]$ScenarioName) - - $events = @(Get-Content -LiteralPath $LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal ($ExpectedArmCount * 2) $events.Count "$ScenarioName records one preflight and one execute event per arm" - Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'preflight' }).Count "$ScenarioName records one preflight per arm" - Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'execute' }).Count "$ScenarioName records one execute per arm" - - foreach ($evalId in 1..$ExpectedArmCount) { - $executeEvents = @($events | Where-Object { $_.kind -eq 'execute' -and [int]$_.eval_id -eq $evalId -and [string]$_.configuration -eq 'with_skill' }) - Assert-Equal 1 $executeEvents.Count "$ScenarioName does not retry eval $evalId" - } -} - -function Assert-LedgerMatchesFrozenStatuses { - param([Parameter(Mandatory = $true)][object]$FreezeValidation, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($entry in @($FreezeValidation.Freeze.executions)) { - $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name ([string]$entry.worker_id) -Default $null - Assert-Equal ([string]$entry.terminal_status) ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '')) "$ScenarioName preserves frozen ledger status for $($entry.worker_id)" - } -} - -function Invoke-PhaseOneFailureScenario { - param( - [Parameter(Mandatory = $true)][string]$ScenarioName, - [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, - [int[]]$EvidenceFailureEvalIds = @(), - [Parameter(Mandatory = $true)][hashtable]$ExpectedCounts, - [Parameter(Mandatory = $true)][string[]]$ExpectedFrozenStatuses, - [scriptblock]$AdditionalAssertions = $null - ) - - $iterationDirectory = Join-Path $testRoot $ScenarioName - $package = Initialize-PhaseOneFailurePackage -IterationDirectory $iterationDirectory -StatusesByEvalId $StatusesByEvalId -EvidenceFailureEvalIds $EvidenceFailureEvalIds - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $package.LogPath) - - $fanout = Invoke-ForegroundPhaseOne -Path $package.FanoutScript -IterationDirectory $iterationDirectory - Assert-Equal 2 $fanout.ExitCode "$ScenarioName Phase 1 exits non-zero" - $summary = $fanout.Document - Assert-Equal 'phase1' ([string](Get-JsonProperty -Object $summary -Name 'phase' -Default '')) "$ScenarioName summary identifies Phase 1" - Assert-Equal 'failed' ([string](Get-JsonProperty -Object $summary -Name 'status' -Default '')) "$ScenarioName summary is non-success" - Assert-Counts -Source $summary -Expected $ExpectedCounts -MessagePrefix "$ScenarioName summary" - Assert-ArmSummaryShape -Summary $summary -ScenarioName $ScenarioName - Assert-True (Test-Path -LiteralPath (Join-Path $iterationDirectory 'execution-freeze.json') -PathType Leaf) "$ScenarioName writes execution-freeze.json before failing" - $summaryFreeze = Get-JsonProperty -Object $summary -Name 'execution_freeze' -Default $null - Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'path') "$ScenarioName summary reports execution_freeze.path" - Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'sha256') "$ScenarioName summary reports execution_freeze.sha256" - - $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState - Assert-True (-not [bool]$freezeValidation.PhaseOneSuccess) "$ScenarioName frozen aggregate remains non-success" - Assert-Counts -Source $freezeValidation.Aggregate -Expected $ExpectedCounts -MessagePrefix "$ScenarioName frozen aggregate" - Assert-Equal ([string]::Join(',', $ExpectedFrozenStatuses)) ([string]::Join(',', @($freezeValidation.Freeze.executions | ForEach-Object { [string]$_.terminal_status }))) "$ScenarioName freeze preserves exact raw terminal statuses" - Assert-LedgerMatchesFrozenStatuses -FreezeValidation $freezeValidation -ScenarioName $ScenarioName - Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName - Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName - - $bridge = Invoke-TestTool -Path $package.BridgeScript -Arguments @('-IterationDirectory', $iterationDirectory, '-RequireComplete', '-RequireParallelDispatch', '-RequireNativeDelegation') - Assert-ToolFails -Invocation $bridge -Description "$ScenarioName complete bridge is blocked" -ExpectedText 'completion gate failed' - Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName - - $finalizer = Invoke-TestTool -Path $package.FinalizerScript -Arguments @('-IterationDirectory', $iterationDirectory) - Assert-ToolFails -Invocation $finalizer -Description "$ScenarioName finalizer is blocked" -ExpectedText 'Manifest bridge failed' - Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName - - if ($null -ne $AdditionalAssertions) { - & $AdditionalAssertions $summary $freezeValidation $package - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase1-aggregate-' + [Guid]::NewGuid().ToString('N')) -$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') -try { - Invoke-PhaseOneFailureScenario ` - -ScenarioName 'mixed-terminal' ` - -StatusesByEvalId @{ 1 = 'completed'; 2 = 'timed_out'; 3 = 'failed'; 4 = 'completed' } ` - -ExpectedCounts @{ - expected_count = 4 - terminal_count = 4 - completed_count = 2 - failed_count = 1 - timed_out_count = 1 - cancelled_count = 0 - incompatible_count = 0 - evidence_validation_failed_count = 0 - } ` - -ExpectedFrozenStatuses @('completed', 'timed_out', 'failed', 'completed') ` - -AdditionalAssertions { - param($Summary, $FreezeValidation, $Package) - - Assert-NoRetries -LogPath $Package.LogPath -ExpectedArmCount 4 -ScenarioName 'mixed-terminal' - foreach ($workerId in @('arm-1-with_skill', 'arm-2-with_skill', 'arm-3-with_skill', 'arm-4-with_skill')) { - $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name $workerId -Default $null - $evidenceValidation = Get-JsonProperty -Object $terminal -Name 'evidence_validation' -Default $null - Assert-Equal 'passed' ([string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '')) "mixed-terminal keeps honest evidence_validation for $workerId" - } - } - - Invoke-PhaseOneFailureScenario ` - -ScenarioName 'completed-with-evidence-failure' ` - -StatusesByEvalId @{ 1 = 'completed'; 2 = 'completed'; 3 = 'completed'; 4 = 'completed' } ` - -EvidenceFailureEvalIds @(3) ` - -ExpectedCounts @{ - expected_count = 4 - terminal_count = 4 - completed_count = 4 - failed_count = 0 - timed_out_count = 0 - cancelled_count = 0 - incompatible_count = 0 - evidence_validation_failed_count = 1 - } ` - -ExpectedFrozenStatuses @('completed', 'completed', 'completed', 'completed') ` - -AdditionalAssertions { - param($Summary, $FreezeValidation, $Package) - - $failedTerminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name 'arm-3-with_skill' -Default $null - $failedEvidence = Get-JsonProperty -Object $failedTerminal -Name 'evidence_validation' -Default $null - Assert-Equal 'completed' ([string](Get-JsonProperty -Object $failedTerminal -Name 'status' -Default '')) 'evidence-failure scenario keeps the raw completed status' - Assert-Equal 'failed' ([string](Get-JsonProperty -Object $failedEvidence -Name 'status' -Default '')) 'evidence-failure scenario records failed evidence validation' - Assert-Contains -Text ([string]::Join(', ', @((Get-JsonProperty -Object $failedEvidence -Name 'reasons' -Default @()) | ForEach-Object { [string]$_ }))) -Expected 'prompt_fidelity' -Message 'evidence-failure scenario preserves the validation reason' - } - - Write-Output 'Phase 1 aggregate regressions: PASS' -} finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + grading = 'grading.json' + report = [ordered]@{ tool = 'tools/test-report.ps1' } + evals = @($manifestEvals.ToArray()) + }) + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 4 + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + FanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' + BridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' + FinalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' + LogPath = Join-Path $IterationDirectory 'runner-events.jsonl' + Records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest (Read-TestJson -Path (Join-Path $IterationDirectory 'manifest.json')) | Sort-Object EvalId, Configuration) + } +} + +function Assert-Counts { + param( + [Parameter(Mandatory = $true)][object]$Source, + [Parameter(Mandatory = $true)][hashtable]$Expected, + [Parameter(Mandatory = $true)][string]$MessagePrefix + ) + + foreach ($name in @( + 'expected_count', + 'terminal_count', + 'completed_count', + 'failed_count', + 'timed_out_count', + 'cancelled_count', + 'incompatible_count', + 'evidence_validation_failed_count' + )) { + Assert-Equal $Expected[$name] (Get-JsonProperty -Object $Source -Name $name -Default $null) "$MessagePrefix $name" + } +} + +function Assert-ArmSummaryShape { + param([Parameter(Mandatory = $true)][object]$Summary, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($arm in @($Summary.arms)) { + foreach ($field in @('worker_id', 'eval_id', 'configuration', 'status', 'worker_session_id', 'evidence_validation')) { + Assert-True (Test-JsonProperty -Object $arm -Name $field) "$ScenarioName arm summary contains $field" + } + $evidenceValidation = Get-JsonProperty -Object $arm -Name 'evidence_validation' -Default $null + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'status') "$ScenarioName arm summary contains evidence_validation.status" + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'reasons') "$ScenarioName arm summary contains evidence_validation.reasons" + } +} + +function Assert-CanonicalResultsRemainUnrun { + param([Parameter(Mandatory = $true)][object[]]$Records, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($record in $Records) { + $result = Read-TestJson -Path $record.ResultPath + Assert-Equal 'unrun' ([string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '')) "$ScenarioName keeps $($record.ResultRelative) unbridged" + } +} + +function Assert-NoPhaseTwoArtifacts { + param([Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($relative in @('grading.json', 'report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $IterationDirectory $relative) -PathType Leaf)) "$ScenarioName does not produce $relative" + } +} + +function Assert-NoRetries { + param([Parameter(Mandatory = $true)][string]$LogPath, [Parameter(Mandatory = $true)][int]$ExpectedArmCount, [Parameter(Mandatory = $true)][string]$ScenarioName) + + $events = @(Get-Content -LiteralPath $LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal ($ExpectedArmCount * 2) $events.Count "$ScenarioName records one preflight and one execute event per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'preflight' }).Count "$ScenarioName records one preflight per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'execute' }).Count "$ScenarioName records one execute per arm" + + foreach ($evalId in 1..$ExpectedArmCount) { + $executeEvents = @($events | Where-Object { $_.kind -eq 'execute' -and [int]$_.eval_id -eq $evalId -and [string]$_.configuration -eq 'with_skill' }) + Assert-Equal 1 $executeEvents.Count "$ScenarioName does not retry eval $evalId" + } +} + +function Assert-LedgerMatchesFrozenStatuses { + param([Parameter(Mandatory = $true)][object]$FreezeValidation, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($entry in @($FreezeValidation.Freeze.executions)) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name ([string]$entry.worker_id) -Default $null + Assert-Equal ([string]$entry.terminal_status) ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '')) "$ScenarioName preserves frozen ledger status for $($entry.worker_id)" + } +} + +function Invoke-PhaseOneFailureScenario { + param( + [Parameter(Mandatory = $true)][string]$ScenarioName, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @(), + [Parameter(Mandatory = $true)][hashtable]$ExpectedCounts, + [Parameter(Mandatory = $true)][string[]]$ExpectedFrozenStatuses, + [scriptblock]$AdditionalAssertions = $null + ) + + $iterationDirectory = Join-Path $testRoot $ScenarioName + $package = Initialize-PhaseOneFailurePackage -IterationDirectory $iterationDirectory -StatusesByEvalId $StatusesByEvalId -EvidenceFailureEvalIds $EvidenceFailureEvalIds + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $package.LogPath) + + $fanout = Invoke-ForegroundPhaseOne -Path $package.FanoutScript -IterationDirectory $iterationDirectory + Assert-Equal 2 $fanout.ExitCode "$ScenarioName Phase 1 exits non-zero" + $summary = $fanout.Document + Assert-Equal 'phase1' ([string](Get-JsonProperty -Object $summary -Name 'phase' -Default '')) "$ScenarioName summary identifies Phase 1" + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $summary -Name 'status' -Default '')) "$ScenarioName summary is non-success" + Assert-Counts -Source $summary -Expected $ExpectedCounts -MessagePrefix "$ScenarioName summary" + Assert-ArmSummaryShape -Summary $summary -ScenarioName $ScenarioName + Assert-True (Test-Path -LiteralPath (Join-Path $iterationDirectory 'execution-freeze.json') -PathType Leaf) "$ScenarioName writes execution-freeze.json before failing" + $summaryFreeze = Get-JsonProperty -Object $summary -Name 'execution_freeze' -Default $null + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'path') "$ScenarioName summary reports execution_freeze.path" + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'sha256') "$ScenarioName summary reports execution_freeze.sha256" + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState + Assert-True (-not [bool]$freezeValidation.PhaseOneSuccess) "$ScenarioName frozen aggregate remains non-success" + Assert-Counts -Source $freezeValidation.Aggregate -Expected $ExpectedCounts -MessagePrefix "$ScenarioName frozen aggregate" + Assert-Equal ([string]::Join(',', $ExpectedFrozenStatuses)) ([string]::Join(',', @($freezeValidation.Freeze.executions | ForEach-Object { [string]$_.terminal_status }))) "$ScenarioName freeze preserves exact raw terminal statuses" + Assert-LedgerMatchesFrozenStatuses -FreezeValidation $freezeValidation -ScenarioName $ScenarioName + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + $bridge = Invoke-TestTool -Path $package.BridgeScript -Arguments @('-IterationDirectory', $iterationDirectory, '-RequireComplete', '-RequireParallelDispatch', '-RequireNativeDelegation') + Assert-ToolFails -Invocation $bridge -Description "$ScenarioName complete bridge is blocked" -ExpectedText 'completion gate failed' + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + + $finalizer = Invoke-TestTool -Path $package.FinalizerScript -Arguments @('-IterationDirectory', $iterationDirectory) + Assert-ToolFails -Invocation $finalizer -Description "$ScenarioName finalizer is blocked" -ExpectedText 'Manifest bridge failed' + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + if ($null -ne $AdditionalAssertions) { + & $AdditionalAssertions $summary $freezeValidation $package + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase1-aggregate-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'mixed-terminal' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'timed_out'; 3 = 'failed'; 4 = 'completed' } ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 2 + failed_count = 1 + timed_out_count = 1 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 0 + } ` + -ExpectedFrozenStatuses @('completed', 'timed_out', 'failed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + Assert-NoRetries -LogPath $Package.LogPath -ExpectedArmCount 4 -ScenarioName 'mixed-terminal' + foreach ($workerId in @('arm-1-with_skill', 'arm-2-with_skill', 'arm-3-with_skill', 'arm-4-with_skill')) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name $workerId -Default $null + $evidenceValidation = Get-JsonProperty -Object $terminal -Name 'evidence_validation' -Default $null + Assert-Equal 'passed' ([string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '')) "mixed-terminal keeps honest evidence_validation for $workerId" + } + } + + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'completed-with-evidence-failure' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'completed'; 3 = 'completed'; 4 = 'completed' } ` + -EvidenceFailureEvalIds @(3) ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 4 + failed_count = 0 + timed_out_count = 0 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 1 + } ` + -ExpectedFrozenStatuses @('completed', 'completed', 'completed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + $failedTerminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name 'arm-3-with_skill' -Default $null + $failedEvidence = Get-JsonProperty -Object $failedTerminal -Name 'evidence_validation' -Default $null + Assert-Equal 'completed' ([string](Get-JsonProperty -Object $failedTerminal -Name 'status' -Default '')) 'evidence-failure scenario keeps the raw completed status' + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $failedEvidence -Name 'status' -Default '')) 'evidence-failure scenario records failed evidence validation' + Assert-Contains -Text ([string]::Join(', ', @((Get-JsonProperty -Object $failedEvidence -Name 'reasons' -Default @()) | ForEach-Object { [string]$_ }))) -Expected 'prompt_fidelity' -Message 'evidence-failure scenario preserves the validation reason' + } + + Write-Output 'Phase 1 aggregate regressions: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 index adcb44a..60cc404 100644 --- a/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 +++ b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 @@ -1,244 +1,249 @@ -<#! -.SYNOPSIS - Deterministic foreground runner-owned Phase 1 lifecycle tests. - -.DESCRIPTION - Exercises the restored runner-owned topology: one foreground - invoke-runner-owned-arms.ps1 invocation per iteration. The fixture runner - is model-free and never calls an AI CLI. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -ne [string]$Actual) { - throw "ASSERT: $Message (expected '$Expected', got '$Actual')" - } -} - -function Write-TestJson { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) - - New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null - [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Read-TestJson { - param([Parameter(Mandatory = $true)][string]$Path) - - return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 -} - -function Invoke-ForegroundPhaseOne { - param([Parameter(Mandatory = $true)][string]$IterationDirectory) - - $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' - # STDOUT carries the machine protocol; STDERR carries live observability. - # Keep them separate so heartbeats never corrupt the terminal JSON. - $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') - try { - $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>$stderrPath - $exitCode = $LASTEXITCODE - $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - } finally { - Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue - } - $document = $text | ConvertFrom-Json -Depth 100 - return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } -} - -function New-ForegroundPackage { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [int]$EvalCount = 2, - [int]$Concurrency = 2 - ) - - $tools = Join-Path $IterationDirectory 'tools\eval-runners' - New-Item -ItemType Directory -Path $tools -Force | Out-Null - foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { - Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force - } - $fixtureRunnerDirectory = Join-Path $tools 'fixture' - New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force - Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 30 - concurrency = $Concurrency - }) - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { - $evalName = 'foreground-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) - $runs = [ordered]@{} - foreach ($configuration in @('with_skill', 'without_skill')) { - $runDirectory = Join-Path $evalDirectory $configuration - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $resultDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "foreground prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - $skillDirectory = $null - $skillHash = $null - if ($configuration -eq 'with_skill') { - $skillDirectory = 'skill/candidate' - New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) - $skillHash = ('b' * 64) - } - Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName +<#! +.SYNOPSIS + Deterministic foreground runner-owned Phase 1 lifecycle tests. + +.DESCRIPTION + Exercises the restored runner-owned topology: one foreground + invoke-runner-owned-arms.ps1 invocation per iteration. The fixture runner + is model-free and never calls an AI CLI. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' + # STDOUT carries the machine protocol; STDERR carries live observability. + # Keep them separate so heartbeats never corrupt the terminal JSON. + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') + try { + $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>$stderrPath + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + } finally { + Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue + } + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } +} + +function New-ForegroundPackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [int]$EvalCount = 2, + [int]$Concurrency = 2 + ) + + $tools = Join-Path $IterationDirectory 'tools\eval-runners' + New-Item -ItemType Directory -Path $tools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $tools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = $Concurrency + }) + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { + $evalName = 'foreground-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $resultDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "foreground prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + $candidateInstructionHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + $promptContent = "foreground prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' - skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - iteration = 1 - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $skillDirectory - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('a' * 64) - skillHash = $skillHash - }) - $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } - Write-TestJson -Path (Join-Path $resultDirectory $resultName) -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' }) - $runs[$configuration] = [ordered]@{ - mode = $configuration - run_manifest = "$evalName/$configuration/run.json" - execution_result = "$evalName/results/$executionName" - result = "$evalName/results/$resultName" - } - } - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $tools - Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - configurations = @('with_skill', 'without_skill') + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash + }) + $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + Write-TestJson -Path (Join-Path $resultDirectory $resultName) -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' }) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $tools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Phase 1 lifecycle fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - evals = @($manifestEvals.ToArray()) - }) - - return [pscustomobject]@{ - IterationDirectory = $IterationDirectory - Tools = $tools - LogPath = Join-Path $IterationDirectory 'fixture-events.jsonl' - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-foreground-phase1-' + [Guid]::NewGuid().ToString('N')) -$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') -try { - $success = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'success') -EvalCount 2 -Concurrency 2 - foreach ($obsolete in @('phase1-control-common.ps1', 'control-runner-owned-phase1.ps1', 'supervise-runner-owned-phase1.ps1')) { - Assert-True (-not (Test-Path -LiteralPath (Join-Path $success.Tools $obsolete) -PathType Leaf)) "foreground package does not carry obsolete $obsolete" - } - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $success.LogPath) - $first = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory - Assert-Equal 0 $first.ExitCode 'foreground Phase 1 exits successfully' - Assert-Equal 'phase1' ([string]$first.Document.phase) 'foreground Phase 1 returns the fan-out summary directly' - Assert-Equal 'completed' ([string]$first.Document.status) 'foreground Phase 1 completes' - Assert-Equal 4 ([int]$first.Document.expected_count) 'foreground Phase 1 sees four paired arms' - Assert-Equal 4 ([int]$first.Document.terminal_count) 'foreground Phase 1 registers every arm terminal' - Assert-Equal 4 ([int]$first.Document.execution_count) 'foreground Phase 1 executes every compatible arm' - Assert-True ([int]$first.Document.max_observed_active -gt 1) 'foreground Phase 1 honors requested concurrency when capacity permits' - Assert-True (Test-Path -LiteralPath (Join-Path $success.IterationDirectory 'execution-freeze.json') -PathType Leaf) 'foreground Phase 1 writes execution-freeze.json only after terminal arms' - $freeze = Assert-ExecutionFreeze -IterationDirectory $success.IterationDirectory -RequireOrchestrationState - Assert-True ([bool]$freeze.PhaseOneSuccess) 'foreground Phase 1 freeze validates as successful' - Assert-Equal 4 @($freeze.Freeze.executions).Count 'foreground Phase 1 freeze contains every expected arm' - $events = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 8 $events.Count 'foreground Phase 1 invokes each preflight and execution exactly once' - Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflights every arm' - Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 executes every compatible arm' - $firstExecuteIndex = -1 - $lastPreflightIndex = -1 - for ($index = 0; $index -lt $events.Count; $index++) { - if ($events[$index].kind -eq 'preflight') { $lastPreflightIndex = $index } - if ($events[$index].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $index } - } - Assert-True ($firstExecuteIndex -gt $lastPreflightIndex) 'foreground Phase 1 starts zero executions before all preflights pass' - - $second = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory - Assert-Equal 2 $second.ExitCode 'foreground Phase 1 refuses a second invocation after freeze' - Assert-True ([string]$second.Document.error -match 'already frozen|existing orchestration state') 'foreground Phase 1 reports why rerun is refused' - $eventsAfterSecond = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 4 @($eventsAfterSecond | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 rerun starts zero additional executions' - - $interrupted = Join-Path $testRoot 'interrupted' - Copy-Item -LiteralPath $success.IterationDirectory -Destination $interrupted -Recurse -Force - Remove-Item -LiteralPath (Join-Path $interrupted 'execution-freeze.json') -Force - foreach ($raw in @(Get-ChildItem -LiteralPath $interrupted -Recurse -File -Filter '*.execution-result.json')) { - Remove-Item -LiteralPath $raw.FullName -Force - } - $interruptedLog = Join-Path $interrupted 'new-events.jsonl' - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $interruptedLog) - $interruptedResult = Invoke-ForegroundPhaseOne -IterationDirectory $interrupted - Assert-Equal 2 $interruptedResult.ExitCode 'foreground Phase 1 fails closed on interrupted state without a freeze' - Assert-True ([string]$interruptedResult.Document.error -match 'refuses to replace an existing orchestration state') 'foreground Phase 1 does not adopt or rerun incomplete state' - Assert-True (-not (Test-Path -LiteralPath $interruptedLog -PathType Leaf)) 'foreground Phase 1 interrupted-state refusal starts zero executions' - - $preflightGate = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'preflight-gate') -EvalCount 2 -Concurrency 2 - [System.IO.File]::WriteAllText((Join-Path $preflightGate.IterationDirectory 'foreground-eval-02\with_skill\home\preflight-incompatible'), 'fixture', [System.Text.UTF8Encoding]::new($false)) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $preflightGate.LogPath) - $gate = Invoke-ForegroundPhaseOne -IterationDirectory $preflightGate.IterationDirectory - Assert-Equal 2 $gate.ExitCode 'foreground Phase 1 exits non-zero for incompatible preflight' - Assert-Equal 'preflight' ([string]$gate.Document.phase) 'foreground Phase 1 reports preflight phase failure' - Assert-Equal 'preflight_incompatible' ([string]$gate.Document.status) 'foreground Phase 1 reports incompatible preflight' - Assert-Equal 4 ([int]$gate.Document.preflight_count) 'foreground Phase 1 still probes every arm' - Assert-True (-not [bool]$gate.Document.execution_started) 'foreground Phase 1 starts zero executions when any preflight is incompatible' - $gateEvents = @(Get-Content -LiteralPath $preflightGate.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 4 @($gateEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflight gate records every preflight' - Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 preflight gate records zero executions' - Assert-True (-not (Test-Path -LiteralPath (Join-Path $preflightGate.IterationDirectory 'execution-freeze.json') -PathType Leaf)) 'foreground Phase 1 writes no freeze before a failed preflight gate' - - $fanoutText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($fanoutText -notmatch '(?i)Job Object|breakaway|process ancestry|supervisor independence|phase1-control-common|AGENTIC_PHASE1_SUPERVISOR_ID') 'foreground Phase 1 has no Windows host-security/durable-detachment requirement' - - Write-Output 'Runner-owned foreground Phase 1 lifecycle: PASS' -} finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals.ToArray()) + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + Tools = $tools + LogPath = Join-Path $IterationDirectory 'fixture-events.jsonl' + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-foreground-phase1-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + $success = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'success') -EvalCount 2 -Concurrency 2 + foreach ($obsolete in @('phase1-control-common.ps1', 'control-runner-owned-phase1.ps1', 'supervise-runner-owned-phase1.ps1')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $success.Tools $obsolete) -PathType Leaf)) "foreground package does not carry obsolete $obsolete" + } + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $success.LogPath) + $first = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 0 $first.ExitCode 'foreground Phase 1 exits successfully' + Assert-Equal 'phase1' ([string]$first.Document.phase) 'foreground Phase 1 returns the fan-out summary directly' + Assert-Equal 'completed' ([string]$first.Document.status) 'foreground Phase 1 completes' + Assert-Equal 4 ([int]$first.Document.expected_count) 'foreground Phase 1 sees four paired arms' + Assert-Equal 4 ([int]$first.Document.terminal_count) 'foreground Phase 1 registers every arm terminal' + Assert-Equal 4 ([int]$first.Document.execution_count) 'foreground Phase 1 executes every compatible arm' + Assert-True ([int]$first.Document.max_observed_active -gt 1) 'foreground Phase 1 honors requested concurrency when capacity permits' + Assert-True (Test-Path -LiteralPath (Join-Path $success.IterationDirectory 'execution-freeze.json') -PathType Leaf) 'foreground Phase 1 writes execution-freeze.json only after terminal arms' + $freeze = Assert-ExecutionFreeze -IterationDirectory $success.IterationDirectory -RequireOrchestrationState + Assert-True ([bool]$freeze.PhaseOneSuccess) 'foreground Phase 1 freeze validates as successful' + Assert-Equal 4 @($freeze.Freeze.executions).Count 'foreground Phase 1 freeze contains every expected arm' + $events = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 8 $events.Count 'foreground Phase 1 invokes each preflight and execution exactly once' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflights every arm' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 executes every compatible arm' + $firstExecuteIndex = -1 + $lastPreflightIndex = -1 + for ($index = 0; $index -lt $events.Count; $index++) { + if ($events[$index].kind -eq 'preflight') { $lastPreflightIndex = $index } + if ($events[$index].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $index } + } + Assert-True ($firstExecuteIndex -gt $lastPreflightIndex) 'foreground Phase 1 starts zero executions before all preflights pass' + + $second = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 2 $second.ExitCode 'foreground Phase 1 refuses a second invocation after freeze' + Assert-True ([string]$second.Document.error -match 'already frozen|existing orchestration state') 'foreground Phase 1 reports why rerun is refused' + $eventsAfterSecond = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($eventsAfterSecond | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 rerun starts zero additional executions' + + $interrupted = Join-Path $testRoot 'interrupted' + Copy-Item -LiteralPath $success.IterationDirectory -Destination $interrupted -Recurse -Force + Remove-Item -LiteralPath (Join-Path $interrupted 'execution-freeze.json') -Force + foreach ($raw in @(Get-ChildItem -LiteralPath $interrupted -Recurse -File -Filter '*.execution-result.json')) { + Remove-Item -LiteralPath $raw.FullName -Force + } + $interruptedLog = Join-Path $interrupted 'new-events.jsonl' + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $interruptedLog) + $interruptedResult = Invoke-ForegroundPhaseOne -IterationDirectory $interrupted + Assert-Equal 2 $interruptedResult.ExitCode 'foreground Phase 1 fails closed on interrupted state without a freeze' + Assert-True ([string]$interruptedResult.Document.error -match 'refuses to replace an existing orchestration state') 'foreground Phase 1 does not adopt or rerun incomplete state' + Assert-True (-not (Test-Path -LiteralPath $interruptedLog -PathType Leaf)) 'foreground Phase 1 interrupted-state refusal starts zero executions' + + $preflightGate = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'preflight-gate') -EvalCount 2 -Concurrency 2 + [System.IO.File]::WriteAllText((Join-Path $preflightGate.IterationDirectory 'foreground-eval-02\with_skill\home\preflight-incompatible'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $preflightGate.LogPath) + $gate = Invoke-ForegroundPhaseOne -IterationDirectory $preflightGate.IterationDirectory + Assert-Equal 2 $gate.ExitCode 'foreground Phase 1 exits non-zero for incompatible preflight' + Assert-Equal 'preflight' ([string]$gate.Document.phase) 'foreground Phase 1 reports preflight phase failure' + Assert-Equal 'preflight_incompatible' ([string]$gate.Document.status) 'foreground Phase 1 reports incompatible preflight' + Assert-Equal 4 ([int]$gate.Document.preflight_count) 'foreground Phase 1 still probes every arm' + Assert-True (-not [bool]$gate.Document.execution_started) 'foreground Phase 1 starts zero executions when any preflight is incompatible' + $gateEvents = @(Get-Content -LiteralPath $preflightGate.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($gateEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflight gate records every preflight' + Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 preflight gate records zero executions' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $preflightGate.IterationDirectory 'execution-freeze.json') -PathType Leaf)) 'foreground Phase 1 writes no freeze before a failed preflight gate' + + $fanoutText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($fanoutText -notmatch '(?i)Job Object|breakaway|process ancestry|supervisor independence|phase1-control-common|AGENTIC_PHASE1_SUPERVISOR_ID') 'foreground Phase 1 has no Windows host-security/durable-detachment requirement' + + Write-Output 'Runner-owned foreground Phase 1 lifecycle: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 1763caf..f463a49 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -2305,19 +2305,27 @@ function New-TestRun { [System.IO.File]::WriteAllText((Join-Path $repo 'AGENTS.md'), '# repo-owned-agent-instruction', [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $repo '.github\copilot-instructions.md'), '# repo-owned-copilot-instruction', [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $repo 'opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) - $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀. I’m testing Microsoft’s guidance. ÆØÅ`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) - [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'expected-prompt-sha256.txt'), (Get-Sha256HexFromFile -Path (Join-Path $runRoot 'prompt.md')), [System.Text.UTF8Encoding]::new($false)) + # For with_skill the prompt must have the candidate instructions before the + # working-environment boundary so the Copilot runner can verify the hash. + $taskBody = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀. I'm testing Microsoft's guidance. ÆØÅ`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) + $candidateInstructionHash = $null if ($Configuration -eq 'with_skill') { $skill = Join-Path $runRoot 'skill\candidate' New-Item -ItemType Directory -Path $skill -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $skill 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) $skillDirectory = 'skill/candidate' $skillHash = Get-TestTreeHash -Root $skill + $candidateInstructions = "## Skill: candidate`r`n`r`nConformance fixture candidate instructions for testing." + $candidateInstructionsNormalized = $candidateInstructions -replace "`r`n", "`n" -replace "`r", "`n" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.UTF8Encoding]::new($false).GetBytes($candidateInstructionsNormalized)))).ToLowerInvariant() + $prompt = $candidateInstructions + "`r`n`r`n# Working environment`r`n`r`n" + $taskBody } else { $skillDirectory = $null $skillHash = $null + $prompt = $taskBody } + [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'expected-prompt-sha256.txt'), (Get-Sha256HexFromFile -Path (Join-Path $runRoot 'prompt.md')), [System.Text.UTF8Encoding]::new($false)) $run = [ordered]@{ schema = (Get-RunnerSchemaNames).Run evalId = 1 @@ -2337,6 +2345,7 @@ function New-TestRun { inputFiles = @() fixtureHash = Get-TestTreeHash -Root $repo skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash contract = [ordered]@{ sandboxRoot = '.' workingDirectory = 'repo' @@ -2429,6 +2438,24 @@ try { Assert-Equal 'candidate' $withRunContract.skillName 'with_skill run.json names the exposed candidate' Assert-True ($null -eq $withoutRunContract.skillName) 'without_skill run.json keeps skillName null' Assert-True ($null -eq $withoutRunContract.skillDirectory) 'without_skill run.json keeps skillDirectory null' + # Fix 1: candidate instruction hash invariant - with_skill requires hash, without_skill forbids it. + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$withRunContract.candidateInstructionHash)) 'prepared with_skill run.json declares candidateInstructionHash' + Assert-True ($withRunContract.candidateInstructionHash -match '^[0-9a-f]{64}$') 'with_skill candidateInstructionHash is a valid SHA-256' + Assert-True ($null -eq $withoutRunContract.candidateInstructionHash -or [string]::IsNullOrWhiteSpace([string]$withoutRunContract.candidateInstructionHash)) 'without_skill run.json must not carry candidateInstructionHash' + $missingHashDir = Join-Path $iteration 'missing-hash'; New-Item -ItemType Directory -Path (Join-Path $missingHashDir 'repo'), (Join-Path $missingHashDir 'home'), (Join-Path $missingHashDir 'skill/c') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $missingHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $missingHashDir 'skill/c/SKILL.md'), '# c', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $missingHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = 'c'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/c'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64) }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $missingHashDir 'run.json') } 'missing with_skill candidateInstructionHash must fail Resolve-RunContract' + $badHashDir = Join-Path $iteration 'bad-hash'; New-Item -ItemType Directory -Path (Join-Path $badHashDir 'repo'), (Join-Path $badHashDir 'home'), (Join-Path $badHashDir 'skill/c') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $badHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $badHashDir 'skill/c/SKILL.md'), '# c', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $badHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = 'c'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/c'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64); candidateInstructionHash = 'not-a-sha256' }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $badHashDir 'run.json') } 'malformed with_skill candidateInstructionHash must fail Resolve-RunContract' + $baselineHashDir = Join-Path $iteration 'baseline-with-hash'; New-Item -ItemType Directory -Path (Join-Path $baselineHashDir 'repo'), (Join-Path $baselineHashDir 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $baselineHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $baselineHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = $null; mode = 'without_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = $null; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = $null; candidateInstructionHash = ('a' * 64) }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $baselineHashDir 'run.json') } 'without_skill with candidateInstructionHash must fail Resolve-RunContract' $withoutPromptText = [System.IO.File]::ReadAllText((Join-Path $without.Root 'prompt.md'), [System.Text.UTF8Encoding]::new($false)) Assert-True (-not $withoutPromptText.Contains('candidate')) 'candidate control-plane identity is not added to the baseline prompt' $fakePath = Join-Path $runnerRoot 'fake\runner.ps1' diff --git a/scripts/eval-runners/tests/test-runner-observability.ps1 b/scripts/eval-runners/tests/test-runner-observability.ps1 index 9e67c1d..d0b696b 100644 --- a/scripts/eval-runners/tests/test-runner-observability.ps1 +++ b/scripts/eval-runners/tests/test-runner-observability.ps1 @@ -1,351 +1,356 @@ -<#! -.SYNOPSIS - Deterministic, model-free live-observability tests for every eval runner. - -.DESCRIPTION - Proves the shared observability contract without any model, network, or - authenticated CLI. Synthetic child processes exercise the shared process - primitive (heartbeats, activity tracking, tee/relay, timeout diagnostics, - secret hygiene), and one real runner-owned fan-out run over the deterministic - fixture proves concurrent-arm attribution, the STDOUT machine contract, and - persisted progress evidence. No test may hang: every scenario is bounded. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'runner-progress.ps1') -. (Join-Path $runnerRoot 'fanout-process.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } -} - -function Get-Field { - param([object]$Object, [string]$Name, [object]$Default = $null) - if ($null -ne $Object.PSObject.Properties[$Name]) { return $Object.$Name } - return $Default -} - -$pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-observability-' + [Guid]::NewGuid().ToString('N')) -New-Item -ItemType Directory -Path $testRoot -Force | Out-Null - -function New-SyntheticChildScript { - param( - [Parameter(Mandatory = $true)][string]$Name, - [Parameter(Mandatory = $true)][string]$Body - ) - - $path = Join-Path $testRoot ($Name + '.ps1') - [System.IO.File]::WriteAllText($path, $Body, [System.Text.UTF8Encoding]::new($false)) - return $path -} - -function Invoke-SyntheticChild { - <# - Drives one synthetic child through the shared process primitive exactly as - the fan-out does: Start -> Wait-AnyRunnerChild (heartbeats/relay) -> Complete - (terminal diagnostic). Returns the child, exit code, and the parsed progress - log so a test can assert what an operator would have seen live. - #> - param( - [Parameter(Mandatory = $true)][string]$ScriptPath, - [Parameter(Mandatory = $true)][string]$WorkerId, - [int]$TimeoutSeconds = 30, - [double]$HeartbeatSeconds = 0.3, - [object]$EvalId = 1, - [string]$Configuration = 'with_skill' - ) - - $workDirectory = Join-Path $testRoot ('work-' + [Guid]::NewGuid().ToString('N')) - New-Item -ItemType Directory -Path $workDirectory -Force | Out-Null - $stdoutPath = Join-Path $workDirectory 'result.json' - $stderrPath = Join-Path $workDirectory 'child.stderr' - $logPath = Join-Path $workDirectory 'progress.jsonl' - $operatorErrorWriter = [System.IO.StringWriter]::new([Globalization.CultureInfo]::InvariantCulture) - $originalErrorWriter = [Console]::Error - try { - [Console]::SetError($operatorErrorWriter) +<#! +.SYNOPSIS + Deterministic, model-free live-observability tests for every eval runner. + +.DESCRIPTION + Proves the shared observability contract without any model, network, or + authenticated CLI. Synthetic child processes exercise the shared process + primitive (heartbeats, activity tracking, tee/relay, timeout diagnostics, + secret hygiene), and one real runner-owned fan-out run over the deterministic + fixture proves concurrent-arm attribution, the STDOUT machine contract, and + persisted progress evidence. No test may hang: every scenario is bounded. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'runner-progress.ps1') +. (Join-Path $runnerRoot 'fanout-process.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } +} + +function Get-Field { + param([object]$Object, [string]$Name, [object]$Default = $null) + if ($null -ne $Object.PSObject.Properties[$Name]) { return $Object.$Name } + return $Default +} + +$pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-observability-' + [Guid]::NewGuid().ToString('N')) +New-Item -ItemType Directory -Path $testRoot -Force | Out-Null + +function New-SyntheticChildScript { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][string]$Body + ) + + $path = Join-Path $testRoot ($Name + '.ps1') + [System.IO.File]::WriteAllText($path, $Body, [System.Text.UTF8Encoding]::new($false)) + return $path +} + +function Invoke-SyntheticChild { + <# + Drives one synthetic child through the shared process primitive exactly as + the fan-out does: Start -> Wait-AnyRunnerChild (heartbeats/relay) -> Complete + (terminal diagnostic). Returns the child, exit code, and the parsed progress + log so a test can assert what an operator would have seen live. + #> + param( + [Parameter(Mandatory = $true)][string]$ScriptPath, + [Parameter(Mandatory = $true)][string]$WorkerId, + [int]$TimeoutSeconds = 30, + [double]$HeartbeatSeconds = 0.3, + [object]$EvalId = 1, + [string]$Configuration = 'with_skill' + ) + + $workDirectory = Join-Path $testRoot ('work-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $workDirectory -Force | Out-Null + $stdoutPath = Join-Path $workDirectory 'result.json' + $stderrPath = Join-Path $workDirectory 'child.stderr' + $logPath = Join-Path $workDirectory 'progress.jsonl' + $operatorErrorWriter = [System.IO.StringWriter]::new([Globalization.CultureInfo]::InvariantCulture) + $originalErrorWriter = [Console]::Error + try { + [Console]::SetError($operatorErrorWriter) $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $ScriptPath) -WorkingDirectory $workDirectory -StdoutPath $stdoutPath -StderrPath $stderrPath -TimeoutSeconds $TimeoutSeconds -Runner 'synthetic' -WorkerId $WorkerId -EvalId $EvalId -Configuration $Configuration -Phase 'model-cli' -ProgressLogPath $logPath -HeartbeatSeconds $HeartbeatSeconds - $running = [System.Collections.Generic.List[object]]::new() - $running.Add([pscustomobject]@{ worker_id = $WorkerId; child = $child; Process = $child.Process }) - $index = Wait-AnyRunnerChild -Running $running - $exitCode = Complete-RunnerChildProcess -Child $child - } finally { - try { [Console]::Error.Flush() } catch { } - [Console]::SetError($originalErrorWriter) - } - $events = @() - if (Test-Path -LiteralPath $logPath -PathType Leaf) { - $events = @(Get-Content -LiteralPath $logPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - } - return [pscustomobject]@{ - Child = $child - ExitCode = $exitCode - WaitIndex = $index - Events = $events - StdoutPath = $stdoutPath - StderrPath = $stderrPath - LogPath = $logPath - OperatorStderr = [string]$operatorErrorWriter.ToString() - } -} - -function New-ObservabilityFanoutPackage { - <# - Builds a minimal runner-owned fixture package. The fixture is a protocol - adapter only; it never calls a model or an AI CLI. Each arm is slowed by a - per-run marker so heartbeats fire on the real operator-facing path. - #> - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [int]$EvalCount = 2, - [int]$Concurrency = 4, - [int]$DelayMs = 700 - ) - - $tools = Join-Path $IterationDirectory 'tools\eval-runners' - New-Item -ItemType Directory -Path $tools -Force | Out-Null - foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { - Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force - } - $fixtureRunnerDirectory = Join-Path $tools 'fixture' - New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force - - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'execution-profile.json'), (([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 30 - concurrency = $Concurrency - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { - $evalName = 'obs-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $runs = [ordered]@{} - foreach ($configuration in @('with_skill', 'without_skill')) { - $runDirectory = Join-Path $evalDirectory $configuration - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $resultDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "observability prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - # Slow each arm past the heartbeat interval so the operator path emits - # heartbeats without any model involvement. - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), [string]$DelayMs, [System.Text.UTF8Encoding]::new($false)) - $skillDirectory = $null - $skillHash = $null - if ($configuration -eq 'with_skill') { - $skillDirectory = 'skill/candidate' - New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) - $skillHash = ('b' * 64) - } - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'run.json'), (([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName + $running = [System.Collections.Generic.List[object]]::new() + $running.Add([pscustomobject]@{ worker_id = $WorkerId; child = $child; Process = $child.Process }) + $index = Wait-AnyRunnerChild -Running $running + $exitCode = Complete-RunnerChildProcess -Child $child + } finally { + try { [Console]::Error.Flush() } catch { } + [Console]::SetError($originalErrorWriter) + } + $events = @() + if (Test-Path -LiteralPath $logPath -PathType Leaf) { + $events = @(Get-Content -LiteralPath $logPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + return [pscustomobject]@{ + Child = $child + ExitCode = $exitCode + WaitIndex = $index + Events = $events + StdoutPath = $stdoutPath + StderrPath = $stderrPath + LogPath = $logPath + OperatorStderr = [string]$operatorErrorWriter.ToString() + } +} + +function New-ObservabilityFanoutPackage { + <# + Builds a minimal runner-owned fixture package. The fixture is a protocol + adapter only; it never calls a model or an AI CLI. Each arm is slowed by a + per-run marker so heartbeats fire on the real operator-facing path. + #> + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [int]$EvalCount = 2, + [int]$Concurrency = 4, + [int]$DelayMs = 700 + ) + + $tools = Join-Path $IterationDirectory 'tools\eval-runners' + New-Item -ItemType Directory -Path $tools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $tools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'execution-profile.json'), (([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = $Concurrency + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { + $evalName = 'obs-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $resultDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "observability prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + # Slow each arm past the heartbeat interval so the operator path emits + # heartbeats without any model involvement. + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), [string]$DelayMs, [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + $candidateInstructionHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + $promptContent = "observability prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'run.json'), (([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' - skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - iteration = 1 - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $skillDirectory - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('a' * 64) - skillHash = $skillHash - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } - [System.IO.File]::WriteAllText((Join-Path $resultDirectory $resultName), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $runs[$configuration] = [ordered]@{ - mode = $configuration - run_manifest = "$evalName/$configuration/run.json" - execution_result = "$evalName/results/$executionName" - result = "$evalName/results/$resultName" - } - } - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $tools - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'manifest.json'), (([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - configurations = @('with_skill', 'without_skill') + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + [System.IO.File]::WriteAllText((Join-Path $resultDirectory $resultName), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $tools + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'manifest.json'), (([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Observability fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - evals = @($manifestEvals.ToArray()) - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - return $IterationDirectory -} - -function Invoke-ObservabilityFanout { - param( - [Parameter(Mandatory = $true)][string]$Root, - [int]$EvalCount = 2, - [int]$Concurrency = 4, - [int]$DelayMs = 700 - ) - - New-Item -ItemType Directory -Path $Root -Force | Out-Null - [void](New-ObservabilityFanoutPackage -IterationDirectory $Root -EvalCount $EvalCount -Concurrency $Concurrency -DelayMs $DelayMs) - $fanout = Join-Path $Root 'tools/eval-runners/invoke-runner-owned-arms.ps1' - $stderrPath = Join-Path $Root 'phase1.stderr' - $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', '0.3') - try { + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals.ToArray()) + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + return $IterationDirectory +} + +function Invoke-ObservabilityFanout { + param( + [Parameter(Mandatory = $true)][string]$Root, + [int]$EvalCount = 2, + [int]$Concurrency = 4, + [int]$DelayMs = 700 + ) + + New-Item -ItemType Directory -Path $Root -Force | Out-Null + [void](New-ObservabilityFanoutPackage -IterationDirectory $Root -EvalCount $EvalCount -Concurrency $Concurrency -DelayMs $DelayMs) + $fanout = Join-Path $Root 'tools/eval-runners/invoke-runner-owned-arms.ps1' + $stderrPath = Join-Path $Root 'phase1.stderr' + $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', '0.3') + try { $output = & $pwshPath -NoProfile -NonInteractive -File $fanout -IterationDirectory $Root 2>$stderrPath - $exitCode = $LASTEXITCODE - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) - } - $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - return [pscustomobject]@{ - IterationDirectory = $Root - ExitCode = $exitCode - Stdout = $stdout - Stderr = $stderr - } -} - -function Write-TestJson { - param( - [Parameter(Mandatory = $true)][string]$Path, - [Parameter(Mandatory = $true)][object]$Value - ) - - [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Get-RelayedProgressEventsFromText { - param([AllowEmptyString()][string]$Text) - - $sentinel = Get-RunnerProgressSentinel - return @($Text -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) } | ForEach-Object { - $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $payload | ConvertFrom-Json -Depth 50 } catch { $null } - } | Where-Object { $null -ne $_ }) -} - -function New-CodexObservabilityFixture { - param( - [Parameter(Mandatory = $true)][string]$Root, - [ValidateSet('success', 'timeout')][string]$Mode = 'success', - [int]$TimeoutSeconds = 5 - ) - - New-Item -ItemType Directory -Path $Root -Force | Out-Null - $fakeBin = Join-Path $Root 'fake-bin' - New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null - $fakeCodexPath = Join-Path $fakeBin 'codex.ps1' - [System.IO.File]::WriteAllText($fakeCodexPath, @' -param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) -$heartbeatMs = 150 -$heartbeatOverride = [Environment]::GetEnvironmentVariable('AGENTIC_CODEX_OBS_HEARTBEAT_MS') -if (-not [string]::IsNullOrWhiteSpace($heartbeatOverride)) { - $parsedHeartbeat = 0 - if ([int]::TryParse($heartbeatOverride, [ref]$parsedHeartbeat) -and $parsedHeartbeat -gt 0) { - $heartbeatMs = $parsedHeartbeat - } -} -$homeRoot = [Environment]::GetEnvironmentVariable('HOME') + $exitCode = $LASTEXITCODE + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) + } + $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + return [pscustomobject]@{ + IterationDirectory = $Root + ExitCode = $exitCode + Stdout = $stdout + Stderr = $stderr + } +} + +function Write-TestJson { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$Value + ) + + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Get-RelayedProgressEventsFromText { + param([AllowEmptyString()][string]$Text) + + $sentinel = Get-RunnerProgressSentinel + return @($Text -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) } | ForEach-Object { + $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $payload | ConvertFrom-Json -Depth 50 } catch { $null } + } | Where-Object { $null -ne $_ }) +} + +function New-CodexObservabilityFixture { + param( + [Parameter(Mandatory = $true)][string]$Root, + [ValidateSet('success', 'timeout')][string]$Mode = 'success', + [int]$TimeoutSeconds = 5 + ) + + New-Item -ItemType Directory -Path $Root -Force | Out-Null + $fakeBin = Join-Path $Root 'fake-bin' + New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null + $fakeCodexPath = Join-Path $fakeBin 'codex.ps1' + [System.IO.File]::WriteAllText($fakeCodexPath, @' +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) +$heartbeatMs = 150 +$heartbeatOverride = [Environment]::GetEnvironmentVariable('AGENTIC_CODEX_OBS_HEARTBEAT_MS') +if (-not [string]::IsNullOrWhiteSpace($heartbeatOverride)) { + $parsedHeartbeat = 0 + if ([int]::TryParse($heartbeatOverride, [ref]$parsedHeartbeat) -and $parsedHeartbeat -gt 0) { + $heartbeatMs = $parsedHeartbeat + } +} +$homeRoot = [Environment]::GetEnvironmentVariable('HOME') $isProjectedExecution = [string](Get-Location).Path -match 'agentic-codex-projection-' $nativeSuppressionEnabled = @($arguments | Where-Object { [string]$_ -eq 'skills.include_instructions=false' -or [string]$_ -eq 'skills.bundled.enabled=false' }).Count -gt 0 $mode = if ($isProjectedExecution -and $nativeSuppressionEnabled -and ( - (-not [string]::IsNullOrWhiteSpace($homeRoot) -and (Test-Path -LiteralPath (Join-Path $homeRoot 'codex-observability-timeout') -PathType Leaf)) -or - (Test-Path -LiteralPath (Join-Path (Get-Location).Path 'codex-observability-timeout') -PathType Leaf) + (-not [string]::IsNullOrWhiteSpace($homeRoot) -and (Test-Path -LiteralPath (Join-Path $homeRoot 'codex-observability-timeout') -PathType Leaf)) -or + (Test-Path -LiteralPath (Join-Path (Get-Location).Path 'codex-observability-timeout') -PathType Leaf) )) { 'timeout' } else { 'success' } - -function Get-DelayMilliseconds { - param([double]$Multiplier) - - return [int][Math]::Max(50, [Math]::Ceiling($heartbeatMs * $Multiplier)) -} - -function Read-AppServerMessage { + +function Get-DelayMilliseconds { + param([double]$Multiplier) + + return [int][Math]::Max(50, [Math]::Ceiling($heartbeatMs * $Multiplier)) +} + +function Read-AppServerMessage { param([switch]$AllowEndOfStream) - $line = [Console]::In.ReadLine() + $line = [Console]::In.ReadLine() if ($null -eq $line) { if ($AllowEndOfStream) { return $null } throw 'observability fake app-server reached EOF before the expected request' } - return ($line | ConvertFrom-Json -Depth 50) -} - -function Write-AppServerMessage { - param([Parameter(Mandatory = $true)][object]$Value) - - [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) - [Console]::Out.Flush() -} - -function Write-CodexSchemas { - param([Parameter(Mandatory = $true)][string]$SchemaDirectory) - - New-Item -ItemType Directory -Path $SchemaDirectory -Force | Out-Null - foreach ($existing in @(Get-ChildItem -LiteralPath $SchemaDirectory -Force -ErrorAction SilentlyContinue)) { - Remove-Item -LiteralPath $existing.FullName -Recurse -Force - } - - $schema = 'http://json-schema.org/draft-07/schema#' - $definitions = [ordered]@{ - AbsolutePathBuf = [ordered]@{ type = 'string' } - LegacyAppPathString = [ordered]@{ type = 'string' } - SandboxMode = [ordered]@{ type = 'string'; enum = @('read-only', 'workspace-write', 'danger-full-access') } - AskForApproval = [ordered]@{ oneOf = @([ordered]@{ type = 'string'; enum = @('untrusted', 'on-request', 'never') }) } - ReasoningEffort = [ordered]@{ type = 'string'; minLength = 1 } - ModelRerouteReason = [ordered]@{ type = 'string'; enum = @('highRiskCyberActivity') } - TurnStatus = [ordered]@{ type = 'string'; enum = @('inProgress', 'completed', 'failed', 'interrupted') } - UserInput = [ordered]@{ oneOf = @([ordered]@{ type = 'object'; required = @('text', 'type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('text') }; text = [ordered]@{ type = 'string' } } }) } - SandboxPolicy = [ordered]@{ oneOf = @( - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('dangerFullAccess') } } } - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('readOnly') }; networkAccess = [ordered]@{ type = 'boolean' } } } - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('workspaceWrite') }; writableRoots = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } }; networkAccess = [ordered]@{ type = 'boolean' } } } - ) } - Thread = [ordered]@{ type = 'object'; required = @('id', 'cwd', 'ephemeral', 'sessionId', 'turns'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; cwd = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }) }; ephemeral = [ordered]@{ type = 'boolean' }; sessionId = [ordered]@{ type = 'string' }; turns = [ordered]@{ type = 'array' } } } - Turn = [ordered]@{ type = 'object'; required = @('id', 'items', 'status'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; items = [ordered]@{ type = 'array' }; status = [ordered]@{ '$ref' = '#/definitions/TurnStatus' } } } + return ($line | ConvertFrom-Json -Depth 50) +} + +function Write-AppServerMessage { + param([Parameter(Mandatory = $true)][object]$Value) + + [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + [Console]::Out.Flush() +} + +function Write-CodexSchemas { + param([Parameter(Mandatory = $true)][string]$SchemaDirectory) + + New-Item -ItemType Directory -Path $SchemaDirectory -Force | Out-Null + foreach ($existing in @(Get-ChildItem -LiteralPath $SchemaDirectory -Force -ErrorAction SilentlyContinue)) { + Remove-Item -LiteralPath $existing.FullName -Recurse -Force + } + + $schema = 'http://json-schema.org/draft-07/schema#' + $definitions = [ordered]@{ + AbsolutePathBuf = [ordered]@{ type = 'string' } + LegacyAppPathString = [ordered]@{ type = 'string' } + SandboxMode = [ordered]@{ type = 'string'; enum = @('read-only', 'workspace-write', 'danger-full-access') } + AskForApproval = [ordered]@{ oneOf = @([ordered]@{ type = 'string'; enum = @('untrusted', 'on-request', 'never') }) } + ReasoningEffort = [ordered]@{ type = 'string'; minLength = 1 } + ModelRerouteReason = [ordered]@{ type = 'string'; enum = @('highRiskCyberActivity') } + TurnStatus = [ordered]@{ type = 'string'; enum = @('inProgress', 'completed', 'failed', 'interrupted') } + UserInput = [ordered]@{ oneOf = @([ordered]@{ type = 'object'; required = @('text', 'type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('text') }; text = [ordered]@{ type = 'string' } } }) } + SandboxPolicy = [ordered]@{ oneOf = @( + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('dangerFullAccess') } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('readOnly') }; networkAccess = [ordered]@{ type = 'boolean' } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('workspaceWrite') }; writableRoots = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } }; networkAccess = [ordered]@{ type = 'boolean' } } } + ) } + Thread = [ordered]@{ type = 'object'; required = @('id', 'cwd', 'ephemeral', 'sessionId', 'turns'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; cwd = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }) }; ephemeral = [ordered]@{ type = 'boolean' }; sessionId = [ordered]@{ type = 'string' }; turns = [ordered]@{ type = 'array' } } } + Turn = [ordered]@{ type = 'object'; required = @('id', 'items', 'status'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; items = [ordered]@{ type = 'array' }; status = [ordered]@{ '$ref' = '#/definitions/TurnStatus' } } } SkillMetadata = [ordered]@{ type = 'object'; required = @('name', 'path', 'enabled'); properties = [ordered]@{ name = [ordered]@{ type = 'string' }; path = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }; enabled = [ordered]@{ type = 'boolean' }; scope = [ordered]@{ type = 'string'; enum = @('user', 'repo', 'system', 'admin') }; description = [ordered]@{ type = 'string' } } } SkillsListEntry = [ordered]@{ type = 'object'; required = @('cwd', 'errors', 'skills'); properties = [ordered]@{ cwd = [ordered]@{ type = 'string' }; errors = [ordered]@{ type = 'array' }; skills = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/SkillMetadata' } } } } Config = [ordered]@{ type = 'object'; additionalProperties = $true } @@ -381,159 +386,159 @@ function Write-CodexSchemas { type = 'object' required = @('data') properties = [ordered]@{ data = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/SkillsListEntry' } } } - } - $definitions.ThreadStartParams = [ordered]@{ - '$schema' = $schema - title = 'ThreadStartParams' - type = 'object' - properties = [ordered]@{ - model = [ordered]@{ type = @('string', 'null') } - cwd = [ordered]@{ type = @('string', 'null') } - approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } - sandbox = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxMode' }, [ordered]@{ type = 'null' }) } - ephemeral = [ordered]@{ type = @('boolean', 'null') } - } - } - $definitions.ThreadStartResponse = [ordered]@{ - '$schema' = $schema - title = 'ThreadStartResponse' - type = 'object' - required = @('approvalPolicy', 'approvalsReviewer', 'cwd', 'model', 'modelProvider', 'sandbox', 'thread') - properties = [ordered]@{ - approvalPolicy = [ordered]@{ '$ref' = '#/definitions/AskForApproval' } - cwd = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } - instructionSources = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/LegacyAppPathString' } } - model = [ordered]@{ type = 'string' } - sandbox = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }) } - thread = [ordered]@{ '$ref' = '#/definitions/Thread' } - } - } - $definitions.TurnStartParams = [ordered]@{ - '$schema' = $schema - title = 'TurnStartParams' - type = 'object' - required = @('input', 'threadId') - properties = [ordered]@{ - threadId = [ordered]@{ type = 'string' } - input = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/UserInput' } } - cwd = [ordered]@{ type = @('string', 'null') } - model = [ordered]@{ type = @('string', 'null') } - effort = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/ReasoningEffort' }, [ordered]@{ type = 'null' }) } - approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } - sandboxPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }, [ordered]@{ type = 'null' }) } - } - } - $definitions.TurnStartResponse = [ordered]@{ - '$schema' = $schema - title = 'TurnStartResponse' - type = 'object' - required = @('turn') - properties = [ordered]@{ turn = [ordered]@{ '$ref' = '#/definitions/Turn' } } - } - $definitions.ThreadReadParams = [ordered]@{ - '$schema' = $schema - title = 'ThreadReadParams' - type = 'object' - required = @('threadId') - properties = [ordered]@{ threadId = [ordered]@{ type = 'string' }; includeTurns = [ordered]@{ type = 'boolean' } } - } - $definitions.ThreadReadResponse = [ordered]@{ - '$schema' = $schema - title = 'ThreadReadResponse' - type = 'object' - required = @('thread') - properties = [ordered]@{ thread = [ordered]@{ '$ref' = '#/definitions/Thread' } } - } - $definitions.ModelReroutedNotification = [ordered]@{ - '$schema' = $schema - title = 'ModelReroutedNotification' - type = 'object' - required = @('fromModel', 'reason', 'threadId', 'toModel', 'turnId') - properties = [ordered]@{ - fromModel = [ordered]@{ type = 'string' } - reason = [ordered]@{ '$ref' = '#/definitions/ModelRerouteReason' } - threadId = [ordered]@{ type = 'string' } - toModel = [ordered]@{ type = 'string' } - turnId = [ordered]@{ type = 'string' } - } - } - - $schemaFiles = [ordered]@{ - 'codex_app_server_protocol.v2.schemas.json' = [ordered]@{ '$schema' = $schema; title = 'codex_app_server_protocol.v2.schemas'; type = 'object'; definitions = $definitions } - } + } + $definitions.ThreadStartParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartParams' + type = 'object' + properties = [ordered]@{ + model = [ordered]@{ type = @('string', 'null') } + cwd = [ordered]@{ type = @('string', 'null') } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandbox = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxMode' }, [ordered]@{ type = 'null' }) } + ephemeral = [ordered]@{ type = @('boolean', 'null') } + } + } + $definitions.ThreadStartResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartResponse' + type = 'object' + required = @('approvalPolicy', 'approvalsReviewer', 'cwd', 'model', 'modelProvider', 'sandbox', 'thread') + properties = [ordered]@{ + approvalPolicy = [ordered]@{ '$ref' = '#/definitions/AskForApproval' } + cwd = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } + instructionSources = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/LegacyAppPathString' } } + model = [ordered]@{ type = 'string' } + sandbox = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }) } + thread = [ordered]@{ '$ref' = '#/definitions/Thread' } + } + } + $definitions.TurnStartParams = [ordered]@{ + '$schema' = $schema + title = 'TurnStartParams' + type = 'object' + required = @('input', 'threadId') + properties = [ordered]@{ + threadId = [ordered]@{ type = 'string' } + input = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/UserInput' } } + cwd = [ordered]@{ type = @('string', 'null') } + model = [ordered]@{ type = @('string', 'null') } + effort = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/ReasoningEffort' }, [ordered]@{ type = 'null' }) } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandboxPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }, [ordered]@{ type = 'null' }) } + } + } + $definitions.TurnStartResponse = [ordered]@{ + '$schema' = $schema + title = 'TurnStartResponse' + type = 'object' + required = @('turn') + properties = [ordered]@{ turn = [ordered]@{ '$ref' = '#/definitions/Turn' } } + } + $definitions.ThreadReadParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadParams' + type = 'object' + required = @('threadId') + properties = [ordered]@{ threadId = [ordered]@{ type = 'string' }; includeTurns = [ordered]@{ type = 'boolean' } } + } + $definitions.ThreadReadResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadResponse' + type = 'object' + required = @('thread') + properties = [ordered]@{ thread = [ordered]@{ '$ref' = '#/definitions/Thread' } } + } + $definitions.ModelReroutedNotification = [ordered]@{ + '$schema' = $schema + title = 'ModelReroutedNotification' + type = 'object' + required = @('fromModel', 'reason', 'threadId', 'toModel', 'turnId') + properties = [ordered]@{ + fromModel = [ordered]@{ type = 'string' } + reason = [ordered]@{ '$ref' = '#/definitions/ModelRerouteReason' } + threadId = [ordered]@{ type = 'string' } + toModel = [ordered]@{ type = 'string' } + turnId = [ordered]@{ type = 'string' } + } + } + + $schemaFiles = [ordered]@{ + 'codex_app_server_protocol.v2.schemas.json' = [ordered]@{ '$schema' = $schema; title = 'codex_app_server_protocol.v2.schemas'; type = 'object'; definitions = $definitions } + } foreach ($schemaName in @('ConfigReadParams', 'ConfigReadResponse', 'SkillsListParams', 'SkillsListResponse', 'ThreadStartParams', 'ThreadStartResponse', 'TurnStartParams', 'TurnStartResponse', 'ThreadReadParams', 'ThreadReadResponse', 'ModelReroutedNotification')) { - $source = $definitions[$schemaName] - $individual = [ordered]@{ '$schema' = $schema } - foreach ($propertyName in @('title', 'type', 'properties', 'required')) { - if ($source.Contains($propertyName)) { $individual[$propertyName] = $source[$propertyName] } - } - $individual.definitions = $definitions - $schemaFiles[('v2\{0}.json' -f $schemaName)] = $individual - } - foreach ($schemaName in $schemaFiles.Keys) { - $schemaPath = Join-Path $SchemaDirectory $schemaName - New-Item -ItemType Directory -Path (Split-Path -Parent $schemaPath) -Force | Out-Null - [System.IO.File]::WriteAllText($schemaPath, ([string]($schemaFiles[$schemaName] | ConvertTo-Json -Depth 100)), [System.Text.UTF8Encoding]::new($false)) - } -} - -if ($arguments -contains '--version') { - Write-Output 'recorded-codex 9.9' - exit 0 -} -if ($arguments -contains '--help' -and -not ($arguments -contains 'app-server')) { + $source = $definitions[$schemaName] + $individual = [ordered]@{ '$schema' = $schema } + foreach ($propertyName in @('title', 'type', 'properties', 'required')) { + if ($source.Contains($propertyName)) { $individual[$propertyName] = $source[$propertyName] } + } + $individual.definitions = $definitions + $schemaFiles[('v2\{0}.json' -f $schemaName)] = $individual + } + foreach ($schemaName in $schemaFiles.Keys) { + $schemaPath = Join-Path $SchemaDirectory $schemaName + New-Item -ItemType Directory -Path (Split-Path -Parent $schemaPath) -Force | Out-Null + [System.IO.File]::WriteAllText($schemaPath, ([string]($schemaFiles[$schemaName] | ConvertTo-Json -Depth 100)), [System.Text.UTF8Encoding]::new($false)) + } +} + +if ($arguments -contains '--version') { + Write-Output 'recorded-codex 9.9' + exit 0 +} +if ($arguments -contains '--help' -and -not ($arguments -contains 'app-server')) { Write-Output '--ask-for-approval never --strict-config --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox danger-full-access --cd --model --config' - exit 0 -} -if ($arguments -contains 'features' -and $arguments -contains 'list') { - Write-Output 'multi_agent stable true' - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains '--help') { - Write-Output 'generate-json-schema' - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { - $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) - if ($outArgument.Count -eq 0) { exit 2 } - $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) - Write-CodexSchemas -SchemaDirectory $schemaDirectory - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { - $initialize = Read-AppServerMessage - if ($mode -eq 'timeout') { - Start-Sleep -Milliseconds (Get-DelayMilliseconds 20) - exit 0 - } - - $repoAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' - $threadId = 'obs-thread' - $turnId = 'obs-turn' - $threadObject = [ordered]@{ - id = $threadId - sessionId = 'obs-session' - ephemeral = $true - cwd = (Get-Location).Path - cliVersion = '9.9' - createdAt = 1 - updatedAt = 1 - modelProvider = 'recorded-provider' - preview = $false - projectId = $null - source = 'startup' - status = [ordered]@{ type = 'idle' } - turns = @() - } - - Start-Sleep -Milliseconds (Get-DelayMilliseconds 3.2) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $initialize.id - result = [ordered]@{ serverInfo = [ordered]@{ name = 'observability-codex'; version = '9.9' } } - }) - - $null = Read-AppServerMessage + exit 0 +} +if ($arguments -contains 'features' -and $arguments -contains 'list') { + Write-Output 'multi_agent stable true' + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains '--help') { + Write-Output 'generate-json-schema' + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { + $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) + if ($outArgument.Count -eq 0) { exit 2 } + $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) + Write-CodexSchemas -SchemaDirectory $schemaDirectory + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { + $initialize = Read-AppServerMessage + if ($mode -eq 'timeout') { + Start-Sleep -Milliseconds (Get-DelayMilliseconds 20) + exit 0 + } + + $repoAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' + $threadId = 'obs-thread' + $turnId = 'obs-turn' + $threadObject = [ordered]@{ + id = $threadId + sessionId = 'obs-session' + ephemeral = $true + cwd = (Get-Location).Path + cliVersion = '9.9' + createdAt = 1 + updatedAt = 1 + modelProvider = 'recorded-provider' + preview = $false + projectId = $null + source = 'startup' + status = [ordered]@{ type = 'idle' } + turns = @() + } + + Start-Sleep -Milliseconds (Get-DelayMilliseconds 3.2) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $initialize.id + result = [ordered]@{ serverInfo = [ordered]@{ name = 'observability-codex'; version = '9.9' } } + }) + + $null = Read-AppServerMessage $skillsList = Read-AppServerMessage $candidateName = 'candidate' $ambientCandidatePath = "C:\Users\some-user\.agents\skills\$candidateName\SKILL.md" @@ -547,650 +552,650 @@ if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { }) $threadStart = Read-AppServerMessage -AllowEndOfStream if ($null -eq $threadStart) { exit 0 } - Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.8) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $threadStart.id - result = [ordered]@{ - approvalPolicy = 'never' - approvalsReviewer = 'user' - cwd = (Get-Location).Path - model = [string]$threadStart.params.model - modelProvider = 'recorded-provider' - sandbox = [ordered]@{ type = 'readOnly' } - instructionSources = @($repoAgentsPath) - thread = $threadObject - } - }) - - $turnStart = Read-AppServerMessage - Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.4) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $turnStart.id - result = [ordered]@{ turn = [ordered]@{ id = $turnId; status = 'inProgress'; items = @() } } - }) - - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'thread/started' - params = [ordered]@{ thread = [ordered]@{ id = $threadId } } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'item/completed' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - completedAtMs = 1 - item = [ordered]@{ - type = 'commandExecution' - id = 'cmd-1' - command = 'echo observability' - commandActions = @() - cwd = (Get-Location).Path - status = 'completed' - exitCode = 0 - aggregatedOutput = 'OBSERVABILITY_PROTOCOL_OUTPUT_CANARY' - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'item/completed' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - completedAtMs = 2 - item = [ordered]@{ - type = 'agentMessage' - id = 'message-1' - text = 'OBSERVABILITY_MODEL_CONTENT_CANARY' - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'thread/tokenUsage/updated' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - tokenUsage = [ordered]@{ - total = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } - last = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'turn/completed' - params = [ordered]@{ - threadId = $threadId - turn = [ordered]@{ id = $turnId; status = 'completed'; items = @() } - } - }) - - $threadRead = Read-AppServerMessage - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $threadRead.id - result = [ordered]@{ thread = $threadObject } - }) - exit 0 -} - -exit 0 -'@, [System.Text.UTF8Encoding]::new($false)) - - $runRoot = Join-Path $Root 'codex-run' - $repoRoot = Join-Path $runRoot 'repo' - $homeRoot = Join-Path $runRoot 'home' - New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null - if ($Mode -eq 'timeout') { - [System.IO.File]::WriteAllText((Join-Path $homeRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $repoRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) - } - [System.IO.File]::WriteAllText((Join-Path $repoRoot 'AGENTS.md'), '# codex observability repo instruction', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), 'codex observability prompt', [System.Text.UTF8Encoding]::new($false)) - $runPath = Join-Path $runRoot 'run.json' - Write-TestJson -Path $runPath -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = 99 - evalName = 'codex-observability' + Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.8) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $threadStart.id + result = [ordered]@{ + approvalPolicy = 'never' + approvalsReviewer = 'user' + cwd = (Get-Location).Path + model = [string]$threadStart.params.model + modelProvider = 'recorded-provider' + sandbox = [ordered]@{ type = 'readOnly' } + instructionSources = @($repoAgentsPath) + thread = $threadObject + } + }) + + $turnStart = Read-AppServerMessage + Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.4) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $turnStart.id + result = [ordered]@{ turn = [ordered]@{ id = $turnId; status = 'inProgress'; items = @() } } + }) + + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'thread/started' + params = [ordered]@{ thread = [ordered]@{ id = $threadId } } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'item/completed' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + completedAtMs = 1 + item = [ordered]@{ + type = 'commandExecution' + id = 'cmd-1' + command = 'echo observability' + commandActions = @() + cwd = (Get-Location).Path + status = 'completed' + exitCode = 0 + aggregatedOutput = 'OBSERVABILITY_PROTOCOL_OUTPUT_CANARY' + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'item/completed' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + completedAtMs = 2 + item = [ordered]@{ + type = 'agentMessage' + id = 'message-1' + text = 'OBSERVABILITY_MODEL_CONTENT_CANARY' + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'thread/tokenUsage/updated' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + tokenUsage = [ordered]@{ + total = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } + last = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'turn/completed' + params = [ordered]@{ + threadId = $threadId + turn = [ordered]@{ id = $turnId; status = 'completed'; items = @() } + } + }) + + $threadRead = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $threadRead.id + result = [ordered]@{ thread = $threadObject } + }) + exit 0 +} + +exit 0 +'@, [System.Text.UTF8Encoding]::new($false)) + + $runRoot = Join-Path $Root 'codex-run' + $repoRoot = Join-Path $runRoot 'repo' + $homeRoot = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null + if ($Mode -eq 'timeout') { + [System.IO.File]::WriteAllText((Join-Path $homeRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repoRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) + } + [System.IO.File]::WriteAllText((Join-Path $repoRoot 'AGENTS.md'), '# codex observability repo instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), 'codex observability prompt', [System.Text.UTF8Encoding]::new($false)) + $runPath = Join-Path $runRoot 'run.json' + Write-TestJson -Path $runPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 99 + evalName = 'codex-observability' candidateSkillName = 'candidate' - skillName = $null - iteration = 1 - mode = 'without_skill' - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $null - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('c' * 64) - }) - $profilePath = Join-Path $Root 'execution-profile.json' - Write-TestJson -Path $profilePath -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'codex' - model = 'gpt-5.6-luna' - reasoning_effort = 'medium' - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = $TimeoutSeconds - concurrency = 1 - }) - - $ambientCodexHome = Join-Path $Root 'ambient-codex-home' - New-Item -ItemType Directory -Path $ambientCodexHome -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $ambientCodexHome 'auth.json'), '{"access_token":"fixture"}', [System.Text.UTF8Encoding]::new($false)) - - return [pscustomobject]@{ - Root = $Root - FakeBin = $fakeBin - RunRoot = $runRoot - RunPath = $runPath - ProfilePath = $profilePath - AmbientCodexHome = $ambientCodexHome - RunnerStderrPath = Join-Path $Root 'runner.stderr' - } -} - -function Invoke-CodexAppServerFixture { - param( - [ValidateSet('success', 'timeout')][string]$Mode, - [double]$HeartbeatSeconds = 0.15, - [int]$TimeoutSeconds = 5 - ) - - $fixture = New-CodexObservabilityFixture -Root (Join-Path $testRoot ('codex-app-server-' + $Mode + '-' + [Guid]::NewGuid().ToString('N'))) -Mode $Mode -TimeoutSeconds $TimeoutSeconds - $runnerPath = Join-Path $runnerRoot 'codex\runner.ps1' - $previousPath = [Environment]::GetEnvironmentVariable('PATH') - $previousCodexHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') - $previousProgress = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') - $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') - $previousOpenAiKey = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') - $heartbeatText = $HeartbeatSeconds.ToString([Globalization.CultureInfo]::InvariantCulture) - $clock = [System.Diagnostics.Stopwatch]::StartNew() - try { - [Environment]::SetEnvironmentVariable('PATH', $fixture.FakeBin + [System.IO.Path]::PathSeparator + $previousPath) - [Environment]::SetEnvironmentVariable('CODEX_HOME', $fixture.AmbientCodexHome) - [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $null) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $heartbeatText) + skillName = $null + iteration = 1 + mode = 'without_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + fixtureHash = ('c' * 64) + }) + $profilePath = Join-Path $Root 'execution-profile.json' + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'codex' + model = 'gpt-5.6-luna' + reasoning_effort = 'medium' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = $TimeoutSeconds + concurrency = 1 + }) + + $ambientCodexHome = Join-Path $Root 'ambient-codex-home' + New-Item -ItemType Directory -Path $ambientCodexHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $ambientCodexHome 'auth.json'), '{"access_token":"fixture"}', [System.Text.UTF8Encoding]::new($false)) + + return [pscustomobject]@{ + Root = $Root + FakeBin = $fakeBin + RunRoot = $runRoot + RunPath = $runPath + ProfilePath = $profilePath + AmbientCodexHome = $ambientCodexHome + RunnerStderrPath = Join-Path $Root 'runner.stderr' + } +} + +function Invoke-CodexAppServerFixture { + param( + [ValidateSet('success', 'timeout')][string]$Mode, + [double]$HeartbeatSeconds = 0.15, + [int]$TimeoutSeconds = 5 + ) + + $fixture = New-CodexObservabilityFixture -Root (Join-Path $testRoot ('codex-app-server-' + $Mode + '-' + [Guid]::NewGuid().ToString('N'))) -Mode $Mode -TimeoutSeconds $TimeoutSeconds + $runnerPath = Join-Path $runnerRoot 'codex\runner.ps1' + $previousPath = [Environment]::GetEnvironmentVariable('PATH') + $previousCodexHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $previousProgress = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') + $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') + $previousOpenAiKey = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') + $heartbeatText = $HeartbeatSeconds.ToString([Globalization.CultureInfo]::InvariantCulture) + $clock = [System.Diagnostics.Stopwatch]::StartNew() + try { + [Environment]::SetEnvironmentVariable('PATH', $fixture.FakeBin + [System.IO.Path]::PathSeparator + $previousPath) + [Environment]::SetEnvironmentVariable('CODEX_HOME', $fixture.AmbientCodexHome) + [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $null) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $heartbeatText) $output = & $pwshPath -NoProfile -NonInteractive -File $runnerPath execute -Run $fixture.RunPath -Profile $fixture.ProfilePath 2>$fixture.RunnerStderrPath - $exitCode = $LASTEXITCODE - } finally { - $clock.Stop() - [Environment]::SetEnvironmentVariable('PATH', $previousPath) - [Environment]::SetEnvironmentVariable('CODEX_HOME', $previousCodexHome) - [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $previousOpenAiKey) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousProgress) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) - } - - $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $fixture.RunnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($fixture.RunnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } - $progressEvents = @(Get-RelayedProgressEventsFromText -Text $stderr) - return [pscustomobject]@{ - Fixture = $fixture - ExitCode = $exitCode - ElapsedSeconds = [Math]::Round($clock.Elapsed.TotalSeconds, 3) - Stdout = $stdout - Stderr = $stderr - Result = $result - ProgressEvents = $progressEvents - AppServerEvents = @($progressEvents | Where-Object { [string]$_.phase -eq 'codex-app-server' }) - RawEventsPath = Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl' - RawEventsText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl'), [System.Text.UTF8Encoding]::new($false)) } else { '' } - RawStderrPath = Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt' - RawStderrText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt'), [System.Text.UTF8Encoding]::new($false)) } else { '' } - } -} - -try { - # Pre-warm the shared activity type so its one-time JIT/compile cost is paid - # before any timed scenario. Otherwise, under heavy machine load, that cost - # inflates the first child's launch timestamp and compresses its heartbeat - # window, making timing-sensitive assertions flaky. - Initialize-RunnerActivityType - - # ------------------------------------------------------------------ - # Test 1 - quiet but alive: a healthy but silent process must still be - # visibly alive. Heartbeats appear even though the process emits nothing. - # ------------------------------------------------------------------ - $quietScript = New-SyntheticChildScript -Name 'quiet' -Body @' -Start-Sleep -Milliseconds 1500 -[Console]::Out.Write('{"status":"completed"}') -'@ - $quiet = Invoke-SyntheticChild -ScriptPath $quietScript -WorkerId 'arm-1-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $quiet.ExitCode 'quiet-but-alive child exits cleanly' - $quietHeartbeats = @($quiet.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' -and $null -ne $_.PSObject.Properties['pid'] }) - Assert-True ($quietHeartbeats.Count -ge 2) "a quiet but alive process still emits heartbeats (got $($quietHeartbeats.Count))" - $quietElapsed = @($quiet.Events | Where-Object { $null -ne $_.PSObject.Properties['elapsedSeconds'] } | ForEach-Object { [double]$_.elapsedSeconds }) - $quietElapsedSpan = (($quietElapsed | Measure-Object -Maximum).Maximum) - (($quietElapsed | Measure-Object -Minimum).Minimum) - Assert-True ($quietElapsedSpan -ge 0.2) "quiet progress shows elapsed runtime advancing (span=$([Math]::Round($quietElapsedSpan,3))s)" - Assert-True (@($quiet.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ne 0 }).Count -eq 0) 'quiet process reports zero real stderr activity' - Assert-True (@($quiet.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'quiet process reports a terminal completed state' - - # ------------------------------------------------------------------ - # Test 2 - active process: periodic output is observable as live activity - # without corrupting the captured machine result or leaking into it. - # ------------------------------------------------------------------ - $activeScript = New-SyntheticChildScript -Name 'active' -Body @' -for ($i = 0; $i -lt 4; $i++) { - [Console]::Error.WriteLine("chunk $i produced") - Start-Sleep -Milliseconds 250 -} -[Console]::Error.WriteLine('@@AGENTIC-PROGRESS@@ {"state":"active","phase":"model-cli","detail":"turn-1","stdoutEvents":3}') -[Console]::Out.Write('{"status":"completed"}') -'@ - $active = Invoke-SyntheticChild -ScriptPath $activeScript -WorkerId 'arm-2-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $active.ExitCode 'active child exits cleanly' - Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $active.StdoutPath -Raw).Trim()) 'active child result STDOUT is captured exactly, uncorrupted by activity' - Assert-True (@($active.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ge 1 }).Count -ge 1) 'active process real stderr activity is observable in heartbeats' - $activeWithAge = @($active.Events | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) - Assert-True ($activeWithAge.Count -ge 1) 'active process reports the age of its most recent activity' - $relayed = @($active.Events | Where-Object { [string]$_.origin -eq 'relay' }) - Assert-True ($relayed.Count -ge 1) 'structured child progress is relayed through the parent' - Assert-Equal 'arm-2-with_skill' ([string]$relayed[0].worker) 'a relayed event is attributed to the emitting worker' - Assert-Equal 'turn-1' ([string]$relayed[0].detail) 'a relayed event preserves the safe child-provided detail' - - # ------------------------------------------------------------------ - # Test 3 - hanging process: the watchdog must fire, terminate the child, - # and produce a final diagnostic. There must be no indefinite hang. - # ------------------------------------------------------------------ - $hangScript = New-SyntheticChildScript -Name 'hang' -Body @' -while ($true) { Start-Sleep -Milliseconds 150 } -'@ - $hangClock = [System.Diagnostics.Stopwatch]::StartNew() - $hang = Invoke-SyntheticChild -ScriptPath $hangScript -WorkerId 'arm-3-with_skill' -TimeoutSeconds 2 -HeartbeatSeconds 0.3 - $hangClock.Stop() - Assert-True ($hangClock.Elapsed.TotalSeconds -lt 15) ("hanging process reaches a terminal state promptly; elapsed={0:N2}s" -f $hangClock.Elapsed.TotalSeconds) - Assert-True ([bool]$hang.Child.TimedOut) 'hanging process is reported as timed out' - Assert-True ([bool]$hang.Child.TerminationObserved) 'hanging process is actually terminated' - Assert-True ($null -eq $hang.ExitCode) 'timed-out child has no synthesized success exit code' - $hangHeartbeats = @($hang.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) - Assert-True ($hangHeartbeats.Count -ge 2) 'heartbeats continue while a process hangs' - $timedOut = @($hang.Events | Where-Object { [string]$_.state -eq 'timed-out' }) - Assert-Equal 1 $timedOut.Count 'a hanging process produces exactly one timed-out diagnostic' - Assert-True ([bool]$timedOut[0].terminationObserved) 'the timeout diagnostic records that termination was observed' - Assert-True ($null -ne $timedOut[0].PSObject.Properties['elapsedSeconds'] -and [double]$timedOut[0].elapsedSeconds -gt 0) 'the timeout diagnostic records elapsed runtime' - Assert-True (@($hang.Events | Where-Object { [string]$_.state -eq 'terminating' }).Count -ge 1) 'a terminating lifecycle state precedes termination' - - # ------------------------------------------------------------------ - # Test 6 - fast successful runner: observability must not disturb a normal - # short execution or its captured result. - # ------------------------------------------------------------------ - $fastScript = New-SyntheticChildScript -Name 'fast' -Body @' -[Console]::Out.Write('{"status":"completed"}') -'@ - $fast = Invoke-SyntheticChild -ScriptPath $fastScript -WorkerId 'arm-6-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $fast.ExitCode 'fast successful child exits cleanly' - Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $fast.StdoutPath -Raw).Trim()) 'fast child result is captured exactly' - Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'running' -and $null -ne $_.PSObject.Properties['pid'] }).Count -ge 1) 'fast child still records a launch lifecycle event' - Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'fast child records exactly one completed state' - - # ------------------------------------------------------------------ - # Test 7 - failure diagnostics: a runner that fails after some activity must - # yield a diagnostic identifying meaningful last-known state. - # ------------------------------------------------------------------ - $failScript = New-SyntheticChildScript -Name 'fail' -Body @' -[Console]::Error.WriteLine('preparing request') -Start-Sleep -Milliseconds 400 -[Console]::Error.WriteLine('harness aborted unexpectedly') -exit 17 -'@ - $fail = Invoke-SyntheticChild -ScriptPath $failScript -WorkerId 'arm-7-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 17 $fail.ExitCode 'failing child reports its real non-zero exit code' - $failed = @($fail.Events | Where-Object { [string]$_.state -eq 'failed' }) - Assert-Equal 1 $failed.Count 'a failing child produces exactly one failure diagnostic' - Assert-Equal 17 ([int]$failed[0].exitCode) 'the failure diagnostic records the exit code' - Assert-True ([int64](Get-Field $failed[0] 'stderrEvents' 0) -ge 1) 'the failure diagnostic records observed stderr activity' - Assert-Equal 'process exited with non-zero status' ([string]$failed[0].detail) 'the failure diagnostic keeps a structured detail only' - Assert-True ([bool](Get-Field $failed[0] 'outputDrainCompleted' $false)) 'the failure diagnostic records bounded output draining' - $failLogText = if (Test-Path -LiteralPath $fail.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($fail.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-True (-not $failLogText.Contains('harness aborted unexpectedly')) 'raw stderr text is absent from persisted progress diagnostics' - Assert-True (-not ([string]$fail.OperatorStderr).Contains('harness aborted unexpectedly')) 'raw stderr text is absent from live operator diagnostics' - Assert-True ((Get-Content -LiteralPath $fail.StderrPath -Raw).Contains('harness aborted unexpectedly')) 'raw stderr evidence remains available in its file' - - # ------------------------------------------------------------------ - # Test 8 - sensitive value hygiene: a recognizable secret in the process - # environment and stderr must never surface in operator progress output. - # ------------------------------------------------------------------ - $secret = 'topsecret-' + [Guid]::NewGuid().ToString('N') - $oldSecret = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET') - [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $secret) - try { - $secretScript = New-SyntheticChildScript -Name 'secret' -Body @' -[Console]::Error.WriteLine('AUTH_TOKEN=' + [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET')) -Start-Sleep -Milliseconds 300 -exit 9 -'@ - $secretRun = Invoke-SyntheticChild -ScriptPath $secretScript -WorkerId 'arm-8-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $oldSecret) - } - $secretLog = if (Test-Path -LiteralPath $secretRun.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($secretRun.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-True (-not $secretLog.Contains($secret)) 'the raw secret value never appears in persisted progress output' - Assert-True (-not ([string]$secretRun.OperatorStderr).Contains($secret)) 'the raw secret value never appears in live operator progress' - $secretFailed = @($secretRun.Events | Where-Object { [string]$_.state -eq 'failed' }) - Assert-Equal 1 $secretFailed.Count 'the secret-bearing child still produces a failure diagnostic' - Assert-Equal 'process exited with non-zero status' ([string]$secretFailed[0].detail) 'secret-bearing failures keep structured detail only' - Assert-True ([bool](Test-Path -LiteralPath $secretRun.StderrPath -PathType Leaf)) 'raw child stderr evidence is still captured to its file' - Assert-True ((Get-Content -LiteralPath $secretRun.StderrPath -Raw).Contains($secret)) 'raw child stderr evidence still retains the secret for deliberate forensic inspection' - - # ------------------------------------------------------------------ - # Tests 4 and 5 - one real runner-owned fan-out run over the deterministic - # fixture proves concurrent-arm attribution, the STDOUT machine contract, - # and persisted progress evidence, all on the true operator-facing path. - # ------------------------------------------------------------------ - $fanoutResult = Invoke-ObservabilityFanout -Root (Join-Path $testRoot 'fanout') -EvalCount 2 -Concurrency 4 -DelayMs 700 - Assert-Equal 0 $fanoutResult.ExitCode 'observability fan-out completes successfully' - - # Test 5 - the STDOUT contract: exactly one machine-readable terminal JSON. - $stdoutTrimmed = ([string]$fanoutResult.Stdout).Trim() - $terminal = $null - $terminal = $stdoutTrimmed | ConvertFrom-Json -Depth 100 - Assert-Equal 'phase1' ([string]$terminal.phase) 'STDOUT still carries exactly one machine-readable terminal summary' - Assert-Equal 'completed' ([string]$terminal.status) 'the terminal summary reports completion' - Assert-True (-not $stdoutTrimmed.Contains((Get-RunnerProgressSentinel))) 'no relay sentinel ever leaks onto STDOUT' - Assert-True (-not ($stdoutTrimmed -match '(?m)^\[synthetic\]|(?m)^\[fixture\]')) 'no operator progress line contaminates STDOUT' - Assert-True ($stdoutTrimmed.StartsWith('{') -and $stdoutTrimmed.EndsWith('}')) 'STDOUT is a single JSON object with no surrounding progress text' - - # Progress is also persisted for post-mortem inspection at the advertised path. - Assert-Equal 'progress/phase1-progress.jsonl' ([string]$terminal.progress_log) 'the terminal summary advertises where progress is persisted' - $persistedLog = Join-Path $fanoutResult.IterationDirectory ([string]$terminal.progress_log) - Assert-True (Test-Path -LiteralPath $persistedLog -PathType Leaf) 'the persisted progress log exists at the advertised path' - $persistedEvents = @(Get-Content -LiteralPath $persistedLog | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-True ($persistedEvents.Count -ge 4) 'the persisted progress log retains events for post-mortem inspection' - - # Test 4 - concurrent-arm attribution: every progress line is attributable. - $stderrLines = @(([string]$fanoutResult.Stderr) -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) - $progressLines = @($stderrLines | Where-Object { $_ -match '^\[(synthetic|fixture)\]' }) - Assert-True ($progressLines.Count -ge 2) 'the concurrent fan-out emits live operator progress on STDERR' - $unlabeled = @($progressLines | Where-Object { -not ($_ -match '\[arm-\d+-(with_skill|without_skill)\]') }) - Assert-Equal 0 $unlabeled.Count 'every operator progress line carries a resolvable worker identity' - $workerIds = @($persistedEvents | ForEach-Object { [string]$_.worker } | Where-Object { $_ -match '^arm-\d+-' } | Sort-Object -Unique) - Assert-True ($workerIds.Count -ge 2) "concurrent arms are individually attributable (distinct workers: $($workerIds.Count))" - foreach ($workerId in @('arm-1-with_skill', 'arm-1-without_skill', 'arm-2-with_skill', 'arm-2-without_skill')) { - Assert-True ($workerIds -contains $workerId) "progress is attributed to worker $workerId" - } - # Attribution must be exclusive: an event's fields belong to exactly its worker. - $misattributed = @($persistedEvents | Where-Object { - [string]$_.state -in @('running', 'active', 'completed', 'timed-out', 'failed') -and - $null -ne $_.PSObject.Properties['configuration'] -and - -not ([string]$_.worker).EndsWith([string]$_.configuration) - }) - Assert-Equal 0 $misattributed.Count 'no progress event mixes one worker identity with another configuration' - - # ------------------------------------------------------------------ - # Test 9 - the shared process primitive: runner model-CLI progress is opt-in - # via the orchestration environment, relays through STDERR, and never - # contaminates the captured model result on STDOUT. - # ------------------------------------------------------------------ - Assert-True ($null -eq (Get-RunnerModelProgressContext -Runner 'opencode')) 'runner model-CLI progress is silent without the orchestration flag' - $previousFlag = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') - try { - $enabledContext = Get-RunnerModelProgressContext -Runner 'opencode' -Phase 'opencode-cli' - Assert-True ($null -ne $enabledContext) 'the orchestration flag enables a runner model-CLI progress context' - Assert-Equal 'Relayable' ([string]$enabledContext['channel']) 'runner model-CLI progress uses the relayable channel' - Assert-Equal 'opencode' ([string]$enabledContext['runner']) 'runner model-CLI progress context carries the runner identity' - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousFlag) - } - - $primitiveDriver = New-SyntheticChildScript -Name 'primitive-driver' -Body @' -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') -$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source -$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.3 } -$grandchild = 'Start-Sleep -Milliseconds 900; [Console]::Out.Write(''grandchild-done'')' -$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchild) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx -[Console]::Out.Write([string]$r.Stdout) -'@ - $driverStderrPath = Join-Path $testRoot 'primitive.stderr' - $previousRoot = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - try { + $exitCode = $LASTEXITCODE + } finally { + $clock.Stop() + [Environment]::SetEnvironmentVariable('PATH', $previousPath) + [Environment]::SetEnvironmentVariable('CODEX_HOME', $previousCodexHome) + [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $previousOpenAiKey) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousProgress) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) + } + + $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $fixture.RunnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($fixture.RunnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } + $progressEvents = @(Get-RelayedProgressEventsFromText -Text $stderr) + return [pscustomobject]@{ + Fixture = $fixture + ExitCode = $exitCode + ElapsedSeconds = [Math]::Round($clock.Elapsed.TotalSeconds, 3) + Stdout = $stdout + Stderr = $stderr + Result = $result + ProgressEvents = $progressEvents + AppServerEvents = @($progressEvents | Where-Object { [string]$_.phase -eq 'codex-app-server' }) + RawEventsPath = Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl' + RawEventsText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl'), [System.Text.UTF8Encoding]::new($false)) } else { '' } + RawStderrPath = Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt' + RawStderrText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt'), [System.Text.UTF8Encoding]::new($false)) } else { '' } + } +} + +try { + # Pre-warm the shared activity type so its one-time JIT/compile cost is paid + # before any timed scenario. Otherwise, under heavy machine load, that cost + # inflates the first child's launch timestamp and compresses its heartbeat + # window, making timing-sensitive assertions flaky. + Initialize-RunnerActivityType + + # ------------------------------------------------------------------ + # Test 1 - quiet but alive: a healthy but silent process must still be + # visibly alive. Heartbeats appear even though the process emits nothing. + # ------------------------------------------------------------------ + $quietScript = New-SyntheticChildScript -Name 'quiet' -Body @' +Start-Sleep -Milliseconds 1500 +[Console]::Out.Write('{"status":"completed"}') +'@ + $quiet = Invoke-SyntheticChild -ScriptPath $quietScript -WorkerId 'arm-1-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $quiet.ExitCode 'quiet-but-alive child exits cleanly' + $quietHeartbeats = @($quiet.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' -and $null -ne $_.PSObject.Properties['pid'] }) + Assert-True ($quietHeartbeats.Count -ge 2) "a quiet but alive process still emits heartbeats (got $($quietHeartbeats.Count))" + $quietElapsed = @($quiet.Events | Where-Object { $null -ne $_.PSObject.Properties['elapsedSeconds'] } | ForEach-Object { [double]$_.elapsedSeconds }) + $quietElapsedSpan = (($quietElapsed | Measure-Object -Maximum).Maximum) - (($quietElapsed | Measure-Object -Minimum).Minimum) + Assert-True ($quietElapsedSpan -ge 0.2) "quiet progress shows elapsed runtime advancing (span=$([Math]::Round($quietElapsedSpan,3))s)" + Assert-True (@($quiet.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ne 0 }).Count -eq 0) 'quiet process reports zero real stderr activity' + Assert-True (@($quiet.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'quiet process reports a terminal completed state' + + # ------------------------------------------------------------------ + # Test 2 - active process: periodic output is observable as live activity + # without corrupting the captured machine result or leaking into it. + # ------------------------------------------------------------------ + $activeScript = New-SyntheticChildScript -Name 'active' -Body @' +for ($i = 0; $i -lt 4; $i++) { + [Console]::Error.WriteLine("chunk $i produced") + Start-Sleep -Milliseconds 250 +} +[Console]::Error.WriteLine('@@AGENTIC-PROGRESS@@ {"state":"active","phase":"model-cli","detail":"turn-1","stdoutEvents":3}') +[Console]::Out.Write('{"status":"completed"}') +'@ + $active = Invoke-SyntheticChild -ScriptPath $activeScript -WorkerId 'arm-2-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $active.ExitCode 'active child exits cleanly' + Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $active.StdoutPath -Raw).Trim()) 'active child result STDOUT is captured exactly, uncorrupted by activity' + Assert-True (@($active.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ge 1 }).Count -ge 1) 'active process real stderr activity is observable in heartbeats' + $activeWithAge = @($active.Events | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) + Assert-True ($activeWithAge.Count -ge 1) 'active process reports the age of its most recent activity' + $relayed = @($active.Events | Where-Object { [string]$_.origin -eq 'relay' }) + Assert-True ($relayed.Count -ge 1) 'structured child progress is relayed through the parent' + Assert-Equal 'arm-2-with_skill' ([string]$relayed[0].worker) 'a relayed event is attributed to the emitting worker' + Assert-Equal 'turn-1' ([string]$relayed[0].detail) 'a relayed event preserves the safe child-provided detail' + + # ------------------------------------------------------------------ + # Test 3 - hanging process: the watchdog must fire, terminate the child, + # and produce a final diagnostic. There must be no indefinite hang. + # ------------------------------------------------------------------ + $hangScript = New-SyntheticChildScript -Name 'hang' -Body @' +while ($true) { Start-Sleep -Milliseconds 150 } +'@ + $hangClock = [System.Diagnostics.Stopwatch]::StartNew() + $hang = Invoke-SyntheticChild -ScriptPath $hangScript -WorkerId 'arm-3-with_skill' -TimeoutSeconds 2 -HeartbeatSeconds 0.3 + $hangClock.Stop() + Assert-True ($hangClock.Elapsed.TotalSeconds -lt 15) ("hanging process reaches a terminal state promptly; elapsed={0:N2}s" -f $hangClock.Elapsed.TotalSeconds) + Assert-True ([bool]$hang.Child.TimedOut) 'hanging process is reported as timed out' + Assert-True ([bool]$hang.Child.TerminationObserved) 'hanging process is actually terminated' + Assert-True ($null -eq $hang.ExitCode) 'timed-out child has no synthesized success exit code' + $hangHeartbeats = @($hang.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) + Assert-True ($hangHeartbeats.Count -ge 2) 'heartbeats continue while a process hangs' + $timedOut = @($hang.Events | Where-Object { [string]$_.state -eq 'timed-out' }) + Assert-Equal 1 $timedOut.Count 'a hanging process produces exactly one timed-out diagnostic' + Assert-True ([bool]$timedOut[0].terminationObserved) 'the timeout diagnostic records that termination was observed' + Assert-True ($null -ne $timedOut[0].PSObject.Properties['elapsedSeconds'] -and [double]$timedOut[0].elapsedSeconds -gt 0) 'the timeout diagnostic records elapsed runtime' + Assert-True (@($hang.Events | Where-Object { [string]$_.state -eq 'terminating' }).Count -ge 1) 'a terminating lifecycle state precedes termination' + + # ------------------------------------------------------------------ + # Test 6 - fast successful runner: observability must not disturb a normal + # short execution or its captured result. + # ------------------------------------------------------------------ + $fastScript = New-SyntheticChildScript -Name 'fast' -Body @' +[Console]::Out.Write('{"status":"completed"}') +'@ + $fast = Invoke-SyntheticChild -ScriptPath $fastScript -WorkerId 'arm-6-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $fast.ExitCode 'fast successful child exits cleanly' + Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $fast.StdoutPath -Raw).Trim()) 'fast child result is captured exactly' + Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'running' -and $null -ne $_.PSObject.Properties['pid'] }).Count -ge 1) 'fast child still records a launch lifecycle event' + Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'fast child records exactly one completed state' + + # ------------------------------------------------------------------ + # Test 7 - failure diagnostics: a runner that fails after some activity must + # yield a diagnostic identifying meaningful last-known state. + # ------------------------------------------------------------------ + $failScript = New-SyntheticChildScript -Name 'fail' -Body @' +[Console]::Error.WriteLine('preparing request') +Start-Sleep -Milliseconds 400 +[Console]::Error.WriteLine('harness aborted unexpectedly') +exit 17 +'@ + $fail = Invoke-SyntheticChild -ScriptPath $failScript -WorkerId 'arm-7-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 17 $fail.ExitCode 'failing child reports its real non-zero exit code' + $failed = @($fail.Events | Where-Object { [string]$_.state -eq 'failed' }) + Assert-Equal 1 $failed.Count 'a failing child produces exactly one failure diagnostic' + Assert-Equal 17 ([int]$failed[0].exitCode) 'the failure diagnostic records the exit code' + Assert-True ([int64](Get-Field $failed[0] 'stderrEvents' 0) -ge 1) 'the failure diagnostic records observed stderr activity' + Assert-Equal 'process exited with non-zero status' ([string]$failed[0].detail) 'the failure diagnostic keeps a structured detail only' + Assert-True ([bool](Get-Field $failed[0] 'outputDrainCompleted' $false)) 'the failure diagnostic records bounded output draining' + $failLogText = if (Test-Path -LiteralPath $fail.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($fail.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-True (-not $failLogText.Contains('harness aborted unexpectedly')) 'raw stderr text is absent from persisted progress diagnostics' + Assert-True (-not ([string]$fail.OperatorStderr).Contains('harness aborted unexpectedly')) 'raw stderr text is absent from live operator diagnostics' + Assert-True ((Get-Content -LiteralPath $fail.StderrPath -Raw).Contains('harness aborted unexpectedly')) 'raw stderr evidence remains available in its file' + + # ------------------------------------------------------------------ + # Test 8 - sensitive value hygiene: a recognizable secret in the process + # environment and stderr must never surface in operator progress output. + # ------------------------------------------------------------------ + $secret = 'topsecret-' + [Guid]::NewGuid().ToString('N') + $oldSecret = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET') + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $secret) + try { + $secretScript = New-SyntheticChildScript -Name 'secret' -Body @' +[Console]::Error.WriteLine('AUTH_TOKEN=' + [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET')) +Start-Sleep -Milliseconds 300 +exit 9 +'@ + $secretRun = Invoke-SyntheticChild -ScriptPath $secretScript -WorkerId 'arm-8-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $oldSecret) + } + $secretLog = if (Test-Path -LiteralPath $secretRun.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($secretRun.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-True (-not $secretLog.Contains($secret)) 'the raw secret value never appears in persisted progress output' + Assert-True (-not ([string]$secretRun.OperatorStderr).Contains($secret)) 'the raw secret value never appears in live operator progress' + $secretFailed = @($secretRun.Events | Where-Object { [string]$_.state -eq 'failed' }) + Assert-Equal 1 $secretFailed.Count 'the secret-bearing child still produces a failure diagnostic' + Assert-Equal 'process exited with non-zero status' ([string]$secretFailed[0].detail) 'secret-bearing failures keep structured detail only' + Assert-True ([bool](Test-Path -LiteralPath $secretRun.StderrPath -PathType Leaf)) 'raw child stderr evidence is still captured to its file' + Assert-True ((Get-Content -LiteralPath $secretRun.StderrPath -Raw).Contains($secret)) 'raw child stderr evidence still retains the secret for deliberate forensic inspection' + + # ------------------------------------------------------------------ + # Tests 4 and 5 - one real runner-owned fan-out run over the deterministic + # fixture proves concurrent-arm attribution, the STDOUT machine contract, + # and persisted progress evidence, all on the true operator-facing path. + # ------------------------------------------------------------------ + $fanoutResult = Invoke-ObservabilityFanout -Root (Join-Path $testRoot 'fanout') -EvalCount 2 -Concurrency 4 -DelayMs 700 + Assert-Equal 0 $fanoutResult.ExitCode 'observability fan-out completes successfully' + + # Test 5 - the STDOUT contract: exactly one machine-readable terminal JSON. + $stdoutTrimmed = ([string]$fanoutResult.Stdout).Trim() + $terminal = $null + $terminal = $stdoutTrimmed | ConvertFrom-Json -Depth 100 + Assert-Equal 'phase1' ([string]$terminal.phase) 'STDOUT still carries exactly one machine-readable terminal summary' + Assert-Equal 'completed' ([string]$terminal.status) 'the terminal summary reports completion' + Assert-True (-not $stdoutTrimmed.Contains((Get-RunnerProgressSentinel))) 'no relay sentinel ever leaks onto STDOUT' + Assert-True (-not ($stdoutTrimmed -match '(?m)^\[synthetic\]|(?m)^\[fixture\]')) 'no operator progress line contaminates STDOUT' + Assert-True ($stdoutTrimmed.StartsWith('{') -and $stdoutTrimmed.EndsWith('}')) 'STDOUT is a single JSON object with no surrounding progress text' + + # Progress is also persisted for post-mortem inspection at the advertised path. + Assert-Equal 'progress/phase1-progress.jsonl' ([string]$terminal.progress_log) 'the terminal summary advertises where progress is persisted' + $persistedLog = Join-Path $fanoutResult.IterationDirectory ([string]$terminal.progress_log) + Assert-True (Test-Path -LiteralPath $persistedLog -PathType Leaf) 'the persisted progress log exists at the advertised path' + $persistedEvents = @(Get-Content -LiteralPath $persistedLog | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-True ($persistedEvents.Count -ge 4) 'the persisted progress log retains events for post-mortem inspection' + + # Test 4 - concurrent-arm attribution: every progress line is attributable. + $stderrLines = @(([string]$fanoutResult.Stderr) -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) + $progressLines = @($stderrLines | Where-Object { $_ -match '^\[(synthetic|fixture)\]' }) + Assert-True ($progressLines.Count -ge 2) 'the concurrent fan-out emits live operator progress on STDERR' + $unlabeled = @($progressLines | Where-Object { -not ($_ -match '\[arm-\d+-(with_skill|without_skill)\]') }) + Assert-Equal 0 $unlabeled.Count 'every operator progress line carries a resolvable worker identity' + $workerIds = @($persistedEvents | ForEach-Object { [string]$_.worker } | Where-Object { $_ -match '^arm-\d+-' } | Sort-Object -Unique) + Assert-True ($workerIds.Count -ge 2) "concurrent arms are individually attributable (distinct workers: $($workerIds.Count))" + foreach ($workerId in @('arm-1-with_skill', 'arm-1-without_skill', 'arm-2-with_skill', 'arm-2-without_skill')) { + Assert-True ($workerIds -contains $workerId) "progress is attributed to worker $workerId" + } + # Attribution must be exclusive: an event's fields belong to exactly its worker. + $misattributed = @($persistedEvents | Where-Object { + [string]$_.state -in @('running', 'active', 'completed', 'timed-out', 'failed') -and + $null -ne $_.PSObject.Properties['configuration'] -and + -not ([string]$_.worker).EndsWith([string]$_.configuration) + }) + Assert-Equal 0 $misattributed.Count 'no progress event mixes one worker identity with another configuration' + + # ------------------------------------------------------------------ + # Test 9 - the shared process primitive: runner model-CLI progress is opt-in + # via the orchestration environment, relays through STDERR, and never + # contaminates the captured model result on STDOUT. + # ------------------------------------------------------------------ + Assert-True ($null -eq (Get-RunnerModelProgressContext -Runner 'opencode')) 'runner model-CLI progress is silent without the orchestration flag' + $previousFlag = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') + try { + $enabledContext = Get-RunnerModelProgressContext -Runner 'opencode' -Phase 'opencode-cli' + Assert-True ($null -ne $enabledContext) 'the orchestration flag enables a runner model-CLI progress context' + Assert-Equal 'Relayable' ([string]$enabledContext['channel']) 'runner model-CLI progress uses the relayable channel' + Assert-Equal 'opencode' ([string]$enabledContext['runner']) 'runner model-CLI progress context carries the runner identity' + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousFlag) + } + + $primitiveDriver = New-SyntheticChildScript -Name 'primitive-driver' -Body @' +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') +$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source +$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.3 } +$grandchild = 'Start-Sleep -Milliseconds 900; [Console]::Out.Write(''grandchild-done'')' +$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchild) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx +[Console]::Out.Write([string]$r.Stdout) +'@ + $driverStderrPath = Join-Path $testRoot 'primitive.stderr' + $previousRoot = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + try { $driverOut = & $pwshPath -NoProfile -NonInteractive -File $primitiveDriver 2>$driverStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot) - } - $driverStdout = ([string]::Join('', @($driverOut | ForEach-Object { [string]$_ }))).Trim() - $driverStderr = if (Test-Path -LiteralPath $driverStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($driverStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-Equal 'grandchild-done' $driverStdout 'the model result on STDOUT passes through the shared primitive uncorrupted' - Assert-True (-not $driverStdout.Contains((Get-RunnerProgressSentinel))) 'no relayable sentinel leaks onto the shared primitive STDOUT' - Assert-True ($driverStderr.Contains((Get-RunnerProgressSentinel))) 'the shared primitive relays model-process progress on STDERR' - Assert-True ($driverStderr -match '"state":"running"') 'the relayed model-process progress reports a running lifecycle state' - - # ------------------------------------------------------------------ - # Test 9b - active inner model process: incremental output advances - # stdout/stderr event and byte counters BEFORE the process completes. - # A sleeping grandchild with one final write does NOT satisfy this test. - # ------------------------------------------------------------------ - $activeInnerDriver = New-SyntheticChildScript -Name 'active-inner-driver' -Body @' -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-progress.ps1') -$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source -$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.15 } -# Grandchild emits 6 stderr lines at 130ms intervals, then the final stdout result. -$grandchildBody = 'for ($i = 0; $i -lt 6; $i++) { [Console]::Error.WriteLine("event-" + $i); [System.Threading.Thread]::Sleep(130) }; [Console]::Out.Write("{""result"":""active-inner-done""}")' -$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchildBody) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx -# Forward captured stdout verbatim so the outer driver can verify fidelity. -[Console]::Out.Write([string]$r.Stdout) -'@ - $activeInnerStderrPath = Join-Path $testRoot 'active-inner-driver.stderr' - $previousRoot2 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - try { + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot) + } + $driverStdout = ([string]::Join('', @($driverOut | ForEach-Object { [string]$_ }))).Trim() + $driverStderr = if (Test-Path -LiteralPath $driverStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($driverStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-Equal 'grandchild-done' $driverStdout 'the model result on STDOUT passes through the shared primitive uncorrupted' + Assert-True (-not $driverStdout.Contains((Get-RunnerProgressSentinel))) 'no relayable sentinel leaks onto the shared primitive STDOUT' + Assert-True ($driverStderr.Contains((Get-RunnerProgressSentinel))) 'the shared primitive relays model-process progress on STDERR' + Assert-True ($driverStderr -match '"state":"running"') 'the relayed model-process progress reports a running lifecycle state' + + # ------------------------------------------------------------------ + # Test 9b - active inner model process: incremental output advances + # stdout/stderr event and byte counters BEFORE the process completes. + # A sleeping grandchild with one final write does NOT satisfy this test. + # ------------------------------------------------------------------ + $activeInnerDriver = New-SyntheticChildScript -Name 'active-inner-driver' -Body @' +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-progress.ps1') +$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source +$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.15 } +# Grandchild emits 6 stderr lines at 130ms intervals, then the final stdout result. +$grandchildBody = 'for ($i = 0; $i -lt 6; $i++) { [Console]::Error.WriteLine("event-" + $i); [System.Threading.Thread]::Sleep(130) }; [Console]::Out.Write("{""result"":""active-inner-done""}")' +$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchildBody) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx +# Forward captured stdout verbatim so the outer driver can verify fidelity. +[Console]::Out.Write([string]$r.Stdout) +'@ + $activeInnerStderrPath = Join-Path $testRoot 'active-inner-driver.stderr' + $previousRoot2 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + try { $activeInnerOut = & $pwshPath -NoProfile -NonInteractive -File $activeInnerDriver 2>$activeInnerStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot2) - } - $activeInnerStdout = ([string]::Join('', @($activeInnerOut | ForEach-Object { [string]$_ }))).Trim() - $activeInnerStderr = if (Test-Path -LiteralPath $activeInnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($activeInnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - # Machine STDOUT must be the exact captured model result, uncorrupted. - Assert-Equal '{"result":"active-inner-done"}' $activeInnerStdout 'active inner process: captured stdout passes through uncorrupted' - Assert-True (-not $activeInnerStdout.Contains((Get-RunnerProgressSentinel))) 'active inner process: no sentinel leaks onto captured STDOUT' - # Parse the relayed sentinel lines to verify live activity tracking. - $sentinel = Get-RunnerProgressSentinel - $relayedLines = @($activeInnerStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) - Assert-True ($relayedLines.Count -ge 2) "active inner process: at least two relayed heartbeats (got $($relayedLines.Count))" - $relayedEvents = @($relayedLines | ForEach-Object { - $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $payload | ConvertFrom-Json } catch { $null } - } | Where-Object { $null -ne $_ }) - # Counters must advance: the last heartbeat must show more events than the first. - $eventCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) - Assert-True ($eventCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrEvents' - $firstCount = ($eventCounts | Measure-Object -Minimum).Minimum - $lastCount = ($eventCounts | Measure-Object -Maximum).Maximum - Assert-True ($lastCount -gt $firstCount) "active inner process: stderrEvents increase across heartbeats (first=$firstCount last=$lastCount)" - # Byte counters must also advance. - $byteCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrBytes'] } | ForEach-Object { [int64]$_.stderrBytes }) - Assert-True ($byteCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrBytes' - Assert-True (($byteCounts | Measure-Object -Maximum).Maximum -gt ($byteCounts | Measure-Object -Minimum).Minimum) 'active inner process: stderrBytes increase across heartbeats' - # lastActivity must appear once any real output has arrived. - $withLastActivity = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) - Assert-True ($withLastActivity.Count -ge 1) 'active inner process: lastActivity is present once real output is received' - - # ------------------------------------------------------------------ - # Test 9c - OpenCode-shaped streaming: a grandchild emitting structured - # JSONL/event-like output (simulating OpenCode session events) advances - # activity metadata safely without echoing model content to the operator. - # ------------------------------------------------------------------ - $openCodeGrandchild = New-SyntheticChildScript -Name 'oc-stream-grandchild' -Body @' -$events = @( - '{"type":"session.start","session_id":"abc123","model":"claude-3-5-haiku"}', - '{"type":"assistant.delta","session_id":"abc123","content":"I will"}', - '{"type":"assistant.delta","session_id":"abc123","content":"analyze"}', - '{"type":"tool.use","tool":"read_file","path":"input.txt"}', - '{"type":"assistant.delta","session_id":"abc123","content":"the result"}', - '{"type":"session.complete","session_id":"abc123","cost":0.002}' -) -foreach ($ev in $events) { - [Console]::Error.WriteLine($ev) - [System.Threading.Thread]::Sleep(100) -} -[Console]::Out.Write('{"status":"completed","session_id":"abc123"}') -'@ - $openCodeStreamDriver = New-SyntheticChildScript -Name 'opencode-stream-driver' -Body ( -'$ErrorActionPreference = "Stop"; Set-StrictMode -Version Latest' + "`n" + -'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-common.ps1")' + "`n" + -'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-progress.ps1")' + "`n" + -'$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source' + "`n" + -'$ctx = @{ enabled = $true; runner = "opencode"; phase = "opencode-cli"; channel = "Relayable"; heartbeatSeconds = 0.15 }' + "`n" + -('$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @("-NoProfile", "-File", $env:AGENTIC_OBS_GRANDCHILD) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx') + "`n" + -'[Console]::Out.Write([string]$r.Stdout)' -) - $ocStreamStderrPath = Join-Path $testRoot 'oc-stream-driver.stderr' - $previousRoot3 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - $previousGrandchild = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $openCodeGrandchild) - try { + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot2) + } + $activeInnerStdout = ([string]::Join('', @($activeInnerOut | ForEach-Object { [string]$_ }))).Trim() + $activeInnerStderr = if (Test-Path -LiteralPath $activeInnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($activeInnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + # Machine STDOUT must be the exact captured model result, uncorrupted. + Assert-Equal '{"result":"active-inner-done"}' $activeInnerStdout 'active inner process: captured stdout passes through uncorrupted' + Assert-True (-not $activeInnerStdout.Contains((Get-RunnerProgressSentinel))) 'active inner process: no sentinel leaks onto captured STDOUT' + # Parse the relayed sentinel lines to verify live activity tracking. + $sentinel = Get-RunnerProgressSentinel + $relayedLines = @($activeInnerStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) + Assert-True ($relayedLines.Count -ge 2) "active inner process: at least two relayed heartbeats (got $($relayedLines.Count))" + $relayedEvents = @($relayedLines | ForEach-Object { + $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $payload | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ }) + # Counters must advance: the last heartbeat must show more events than the first. + $eventCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) + Assert-True ($eventCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrEvents' + $firstCount = ($eventCounts | Measure-Object -Minimum).Minimum + $lastCount = ($eventCounts | Measure-Object -Maximum).Maximum + Assert-True ($lastCount -gt $firstCount) "active inner process: stderrEvents increase across heartbeats (first=$firstCount last=$lastCount)" + # Byte counters must also advance. + $byteCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrBytes'] } | ForEach-Object { [int64]$_.stderrBytes }) + Assert-True ($byteCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrBytes' + Assert-True (($byteCounts | Measure-Object -Maximum).Maximum -gt ($byteCounts | Measure-Object -Minimum).Minimum) 'active inner process: stderrBytes increase across heartbeats' + # lastActivity must appear once any real output has arrived. + $withLastActivity = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) + Assert-True ($withLastActivity.Count -ge 1) 'active inner process: lastActivity is present once real output is received' + + # ------------------------------------------------------------------ + # Test 9c - OpenCode-shaped streaming: a grandchild emitting structured + # JSONL/event-like output (simulating OpenCode session events) advances + # activity metadata safely without echoing model content to the operator. + # ------------------------------------------------------------------ + $openCodeGrandchild = New-SyntheticChildScript -Name 'oc-stream-grandchild' -Body @' +$events = @( + '{"type":"session.start","session_id":"abc123","model":"claude-3-5-haiku"}', + '{"type":"assistant.delta","session_id":"abc123","content":"I will"}', + '{"type":"assistant.delta","session_id":"abc123","content":"analyze"}', + '{"type":"tool.use","tool":"read_file","path":"input.txt"}', + '{"type":"assistant.delta","session_id":"abc123","content":"the result"}', + '{"type":"session.complete","session_id":"abc123","cost":0.002}' +) +foreach ($ev in $events) { + [Console]::Error.WriteLine($ev) + [System.Threading.Thread]::Sleep(100) +} +[Console]::Out.Write('{"status":"completed","session_id":"abc123"}') +'@ + $openCodeStreamDriver = New-SyntheticChildScript -Name 'opencode-stream-driver' -Body ( +'$ErrorActionPreference = "Stop"; Set-StrictMode -Version Latest' + "`n" + +'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-common.ps1")' + "`n" + +'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-progress.ps1")' + "`n" + +'$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source' + "`n" + +'$ctx = @{ enabled = $true; runner = "opencode"; phase = "opencode-cli"; channel = "Relayable"; heartbeatSeconds = 0.15 }' + "`n" + +('$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @("-NoProfile", "-File", $env:AGENTIC_OBS_GRANDCHILD) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx') + "`n" + +'[Console]::Out.Write([string]$r.Stdout)' +) + $ocStreamStderrPath = Join-Path $testRoot 'oc-stream-driver.stderr' + $previousRoot3 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + $previousGrandchild = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $openCodeGrandchild) + try { $ocStreamOut = & $pwshPath -NoProfile -NonInteractive -File $openCodeStreamDriver 2>$ocStreamStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot3) - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $previousGrandchild) - } - $ocStreamStdout = ([string]::Join('', @($ocStreamOut | ForEach-Object { [string]$_ }))).Trim() - $ocStreamStderr = if (Test-Path -LiteralPath $ocStreamStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($ocStreamStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-Equal '{"status":"completed","session_id":"abc123"}' $ocStreamStdout 'opencode streaming: final stdout captured exactly' - Assert-True (-not $ocStreamStdout.Contains('assistant.delta')) 'opencode streaming: model content not present on stdout' - Assert-True (-not $ocStreamStderr.Contains('"content":"I will"')) 'opencode streaming: model delta content not echoed to operator stderr' - Assert-True (-not $ocStreamStderr.Contains('"content":"analyze"')) 'opencode streaming: second delta not echoed to operator stderr' - $ocRelayedLines = @($ocStreamStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) - Assert-True ($ocRelayedLines.Count -ge 2) "opencode streaming: operator receives multiple heartbeats (got $($ocRelayedLines.Count))" - $ocRelayedEvents = @($ocRelayedLines | ForEach-Object { - $p = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $p | ConvertFrom-Json } catch { $null } - } | Where-Object { $null -ne $_ }) - $ocEventCounts = @($ocRelayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) - Assert-True ($ocEventCounts.Count -ge 2) 'opencode streaming: multiple heartbeats carry stderrEvents' - Assert-True (($ocEventCounts | Measure-Object -Maximum).Maximum -gt ($ocEventCounts | Measure-Object -Minimum).Minimum) 'opencode streaming: stderrEvents advance as events are received' - - # ------------------------------------------------------------------ - # Test 9d - Codex app-server real transport: the actual Invoke-CodexAppServer - # path must emit heartbeats while blocked waiting for protocol input, then - # surface real protocol counters and last-activity aging without echoing - # JSON-RPC payloads or model content to operator progress. - # ------------------------------------------------------------------ - $codexApp = Invoke-CodexAppServerFixture -Mode 'success' -HeartbeatSeconds 0.15 -TimeoutSeconds 6 - Assert-Equal 0 $codexApp.ExitCode 'codex app-server: runner process exits cleanly' - $codexStdout = ([string]$codexApp.Stdout).Trim() - Assert-True ($codexStdout.StartsWith('{') -and $codexStdout.EndsWith('}')) 'codex app-server: STDOUT remains one terminal JSON object' - Assert-True (-not $codexStdout.Contains((Get-RunnerProgressSentinel))) 'codex app-server: no progress sentinel contaminates STDOUT' - Assert-True (-not ($codexStdout -match '(?m)^\[codex\]')) 'codex app-server: operator progress never contaminates STDOUT' - Assert-Equal 'completed' ([string]$codexApp.Result.status) 'codex app-server: execution result completes successfully' - Assert-Equal 'OBSERVABILITY_MODEL_CONTENT_CANARY' ([string]$codexApp.Result.final_response.text) 'codex app-server: machine output preserves the final response text' - Assert-True ([bool](Test-Path -LiteralPath $codexApp.RawEventsPath -PathType Leaf)) 'codex app-server: raw protocol evidence is retained on disk' - Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: raw protocol evidence retains model content' - Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: raw protocol evidence retains protocol payload content' - Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: model content is not echoed to operator progress' - Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: protocol payload content is not echoed to operator progress' - $codexAppEvents = @($codexApp.AppServerEvents) - Assert-True ($codexAppEvents.Count -ge 6) "codex app-server: real transport emitted observable progress events (got $($codexAppEvents.Count))" - $firstProtocolIndex = -1 - for ($eventIndex = 0; $eventIndex -lt $codexAppEvents.Count; $eventIndex++) { - if ([int64](Get-Field $codexAppEvents[$eventIndex] 'stdoutEvents' 0) -gt 0) { - $firstProtocolIndex = $eventIndex - break - } - } - Assert-True ($firstProtocolIndex -gt 0) 'codex app-server: at least one heartbeat occurs before the first protocol message is observed' - $preProtocolEvents = @($codexAppEvents[0..($firstProtocolIndex - 1)]) - $quietPreProtocolHeartbeats = @($preProtocolEvents | Where-Object { - [string]$_.state -eq 'running' -and - [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and - $null -eq $_.PSObject.Properties['lastActivitySeconds'] -and - [string]::IsNullOrWhiteSpace([string](Get-Field $_ 'detail' '')) - }) - Assert-True ($quietPreProtocolHeartbeats.Count -ge 1) 'codex app-server: quiet heartbeats are visible before any protocol message arrives' - $firstActivityEvent = $codexAppEvents[$firstProtocolIndex] - $firstActivityCount = [int64](Get-Field $firstActivityEvent 'stdoutEvents' 0) - $firstActivityBytes = [int64](Get-Field $firstActivityEvent 'stdoutBytes' 0) - Assert-True ($firstActivityCount -gt 0 -and $firstActivityBytes -gt 0) 'codex app-server: the first observed protocol heartbeat carries non-zero event and byte counters' - Assert-True ($null -ne $firstActivityEvent.PSObject.Properties['lastActivitySeconds']) 'codex app-server: lastActivity appears after the first real protocol message' - $sameCounterQuietEvents = @($codexAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -eq $firstActivityCount -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq $firstActivityBytes -and - $null -ne $_.PSObject.Properties['lastActivitySeconds'] - }) - Assert-True ($sameCounterQuietEvents.Count -ge 2) 'codex app-server: quiet heartbeats preserve counters after the first protocol message' - $sameCounterAges = @($sameCounterQuietEvents | ForEach-Object { [double]$_.lastActivitySeconds }) - $sameCounterMinAge = ($sameCounterAges | Measure-Object -Minimum).Minimum - $sameCounterMaxAge = ($sameCounterAges | Measure-Object -Maximum).Maximum - Assert-True ($sameCounterMaxAge -gt $sameCounterMinAge) 'codex app-server: lastActivitySeconds ages during a quiet period' - $nextActivityEvent = @($codexAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -gt $firstActivityCount -and - [int64](Get-Field $_ 'stdoutBytes' 0) -gt $firstActivityBytes -and - $null -ne $_.PSObject.Properties['lastActivitySeconds'] - } | Select-Object -First 1) - Assert-True ($nextActivityEvent.Count -eq 1) 'codex app-server: later protocol traffic advances counters again' - Assert-True ([double]$nextActivityEvent[0].lastActivitySeconds -lt $sameCounterMaxAge) 'codex app-server: lastActivitySeconds resets after fresh protocol traffic' - - # ------------------------------------------------------------------ - # Test 9e - Codex app-server timeout: repeated quiet heartbeats must not - # extend the total timeout. The real transport still fails closed, bounded. - # ------------------------------------------------------------------ - $codexTimeout = Invoke-CodexAppServerFixture -Mode 'timeout' -HeartbeatSeconds 0.15 -TimeoutSeconds 2 - Assert-Equal 0 $codexTimeout.ExitCode 'codex app-server timeout: runner still returns a terminal result object' + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot3) + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $previousGrandchild) + } + $ocStreamStdout = ([string]::Join('', @($ocStreamOut | ForEach-Object { [string]$_ }))).Trim() + $ocStreamStderr = if (Test-Path -LiteralPath $ocStreamStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($ocStreamStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-Equal '{"status":"completed","session_id":"abc123"}' $ocStreamStdout 'opencode streaming: final stdout captured exactly' + Assert-True (-not $ocStreamStdout.Contains('assistant.delta')) 'opencode streaming: model content not present on stdout' + Assert-True (-not $ocStreamStderr.Contains('"content":"I will"')) 'opencode streaming: model delta content not echoed to operator stderr' + Assert-True (-not $ocStreamStderr.Contains('"content":"analyze"')) 'opencode streaming: second delta not echoed to operator stderr' + $ocRelayedLines = @($ocStreamStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) + Assert-True ($ocRelayedLines.Count -ge 2) "opencode streaming: operator receives multiple heartbeats (got $($ocRelayedLines.Count))" + $ocRelayedEvents = @($ocRelayedLines | ForEach-Object { + $p = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $p | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ }) + $ocEventCounts = @($ocRelayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) + Assert-True ($ocEventCounts.Count -ge 2) 'opencode streaming: multiple heartbeats carry stderrEvents' + Assert-True (($ocEventCounts | Measure-Object -Maximum).Maximum -gt ($ocEventCounts | Measure-Object -Minimum).Minimum) 'opencode streaming: stderrEvents advance as events are received' + + # ------------------------------------------------------------------ + # Test 9d - Codex app-server real transport: the actual Invoke-CodexAppServer + # path must emit heartbeats while blocked waiting for protocol input, then + # surface real protocol counters and last-activity aging without echoing + # JSON-RPC payloads or model content to operator progress. + # ------------------------------------------------------------------ + $codexApp = Invoke-CodexAppServerFixture -Mode 'success' -HeartbeatSeconds 0.15 -TimeoutSeconds 6 + Assert-Equal 0 $codexApp.ExitCode 'codex app-server: runner process exits cleanly' + $codexStdout = ([string]$codexApp.Stdout).Trim() + Assert-True ($codexStdout.StartsWith('{') -and $codexStdout.EndsWith('}')) 'codex app-server: STDOUT remains one terminal JSON object' + Assert-True (-not $codexStdout.Contains((Get-RunnerProgressSentinel))) 'codex app-server: no progress sentinel contaminates STDOUT' + Assert-True (-not ($codexStdout -match '(?m)^\[codex\]')) 'codex app-server: operator progress never contaminates STDOUT' + Assert-Equal 'completed' ([string]$codexApp.Result.status) 'codex app-server: execution result completes successfully' + Assert-Equal 'OBSERVABILITY_MODEL_CONTENT_CANARY' ([string]$codexApp.Result.final_response.text) 'codex app-server: machine output preserves the final response text' + Assert-True ([bool](Test-Path -LiteralPath $codexApp.RawEventsPath -PathType Leaf)) 'codex app-server: raw protocol evidence is retained on disk' + Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: raw protocol evidence retains model content' + Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: raw protocol evidence retains protocol payload content' + Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: model content is not echoed to operator progress' + Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: protocol payload content is not echoed to operator progress' + $codexAppEvents = @($codexApp.AppServerEvents) + Assert-True ($codexAppEvents.Count -ge 6) "codex app-server: real transport emitted observable progress events (got $($codexAppEvents.Count))" + $firstProtocolIndex = -1 + for ($eventIndex = 0; $eventIndex -lt $codexAppEvents.Count; $eventIndex++) { + if ([int64](Get-Field $codexAppEvents[$eventIndex] 'stdoutEvents' 0) -gt 0) { + $firstProtocolIndex = $eventIndex + break + } + } + Assert-True ($firstProtocolIndex -gt 0) 'codex app-server: at least one heartbeat occurs before the first protocol message is observed' + $preProtocolEvents = @($codexAppEvents[0..($firstProtocolIndex - 1)]) + $quietPreProtocolHeartbeats = @($preProtocolEvents | Where-Object { + [string]$_.state -eq 'running' -and + [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and + $null -eq $_.PSObject.Properties['lastActivitySeconds'] -and + [string]::IsNullOrWhiteSpace([string](Get-Field $_ 'detail' '')) + }) + Assert-True ($quietPreProtocolHeartbeats.Count -ge 1) 'codex app-server: quiet heartbeats are visible before any protocol message arrives' + $firstActivityEvent = $codexAppEvents[$firstProtocolIndex] + $firstActivityCount = [int64](Get-Field $firstActivityEvent 'stdoutEvents' 0) + $firstActivityBytes = [int64](Get-Field $firstActivityEvent 'stdoutBytes' 0) + Assert-True ($firstActivityCount -gt 0 -and $firstActivityBytes -gt 0) 'codex app-server: the first observed protocol heartbeat carries non-zero event and byte counters' + Assert-True ($null -ne $firstActivityEvent.PSObject.Properties['lastActivitySeconds']) 'codex app-server: lastActivity appears after the first real protocol message' + $sameCounterQuietEvents = @($codexAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -eq $firstActivityCount -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq $firstActivityBytes -and + $null -ne $_.PSObject.Properties['lastActivitySeconds'] + }) + Assert-True ($sameCounterQuietEvents.Count -ge 2) 'codex app-server: quiet heartbeats preserve counters after the first protocol message' + $sameCounterAges = @($sameCounterQuietEvents | ForEach-Object { [double]$_.lastActivitySeconds }) + $sameCounterMinAge = ($sameCounterAges | Measure-Object -Minimum).Minimum + $sameCounterMaxAge = ($sameCounterAges | Measure-Object -Maximum).Maximum + Assert-True ($sameCounterMaxAge -gt $sameCounterMinAge) 'codex app-server: lastActivitySeconds ages during a quiet period' + $nextActivityEvent = @($codexAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -gt $firstActivityCount -and + [int64](Get-Field $_ 'stdoutBytes' 0) -gt $firstActivityBytes -and + $null -ne $_.PSObject.Properties['lastActivitySeconds'] + } | Select-Object -First 1) + Assert-True ($nextActivityEvent.Count -eq 1) 'codex app-server: later protocol traffic advances counters again' + Assert-True ([double]$nextActivityEvent[0].lastActivitySeconds -lt $sameCounterMaxAge) 'codex app-server: lastActivitySeconds resets after fresh protocol traffic' + + # ------------------------------------------------------------------ + # Test 9e - Codex app-server timeout: repeated quiet heartbeats must not + # extend the total timeout. The real transport still fails closed, bounded. + # ------------------------------------------------------------------ + $codexTimeout = Invoke-CodexAppServerFixture -Mode 'timeout' -HeartbeatSeconds 0.15 -TimeoutSeconds 2 + Assert-Equal 0 $codexTimeout.ExitCode 'codex app-server timeout: runner still returns a terminal result object' Assert-True ($codexTimeout.ElapsedSeconds -lt 20) ("codex app-server timeout: transport remains bounded after discovery/verification probes; elapsed={0:N3}s" -f $codexTimeout.ElapsedSeconds) - Assert-Equal 'incompatible' ([string]$codexTimeout.Result.status) 'codex app-server timeout: native evidence still fails closed after the bounded timeout' + Assert-Equal 'incompatible' ([string]$codexTimeout.Result.status) 'codex app-server timeout: native evidence still fails closed after the bounded timeout' Assert-Equal 'native_skill_isolation_unverified' ([string]$codexTimeout.Result.exit.failure.code) 'codex app-server timeout: the failure remains structured' - Assert-True ([string]$codexTimeout.Result.exit.failure.message -match 'Codex did not finish before timeout_seconds') 'codex app-server timeout: the failure message preserves the bounded timeout detail' - $timeoutAppEvents = @($codexTimeout.AppServerEvents) - Assert-True ($timeoutAppEvents.Count -ge 2) 'codex app-server timeout: quiet heartbeats occur before the timeout result' - $timeoutQuietEvents = @($timeoutAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and - $null -eq $_.PSObject.Properties['lastActivitySeconds'] - }) - Assert-True ($timeoutQuietEvents.Count -ge 2) 'codex app-server timeout: a silent server stays externally observable without inventing activity' - - # ------------------------------------------------------------------ - # Test 10 - synchronous wait/preflight heartbeat: Complete-RunnerChildProcess - # must emit heartbeats while waiting even when the caller is not using the - # concurrent Wait-AnyRunnerChild loop. - # ------------------------------------------------------------------ - $slowPreflightScript = New-SyntheticChildScript -Name 'slow-preflight' -Body @' -Start-Sleep -Milliseconds 1500 -[Console]::Out.Write('{"status":"compatible"}') -'@ - $preflightStdoutPath = Join-Path $testRoot 'preflight.stdout' - $preflightStderrPath = Join-Path $testRoot 'preflight.stderr' - $preflightLogPath = Join-Path $testRoot 'preflight-progress.jsonl' + Assert-True ([string]$codexTimeout.Result.exit.failure.message -match 'Codex did not finish before timeout_seconds') 'codex app-server timeout: the failure message preserves the bounded timeout detail' + $timeoutAppEvents = @($codexTimeout.AppServerEvents) + Assert-True ($timeoutAppEvents.Count -ge 2) 'codex app-server timeout: quiet heartbeats occur before the timeout result' + $timeoutQuietEvents = @($timeoutAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and + $null -eq $_.PSObject.Properties['lastActivitySeconds'] + }) + Assert-True ($timeoutQuietEvents.Count -ge 2) 'codex app-server timeout: a silent server stays externally observable without inventing activity' + + # ------------------------------------------------------------------ + # Test 10 - synchronous wait/preflight heartbeat: Complete-RunnerChildProcess + # must emit heartbeats while waiting even when the caller is not using the + # concurrent Wait-AnyRunnerChild loop. + # ------------------------------------------------------------------ + $slowPreflightScript = New-SyntheticChildScript -Name 'slow-preflight' -Body @' +Start-Sleep -Milliseconds 1500 +[Console]::Out.Write('{"status":"compatible"}') +'@ + $preflightStdoutPath = Join-Path $testRoot 'preflight.stdout' + $preflightStderrPath = Join-Path $testRoot 'preflight.stderr' + $preflightLogPath = Join-Path $testRoot 'preflight-progress.jsonl' $preflightChild = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $slowPreflightScript) -WorkingDirectory $testRoot -StdoutPath $preflightStdoutPath -StderrPath $preflightStderrPath -TimeoutSeconds 30 -Runner 'opencode' -WorkerId 'preflight-arm-99' -EvalId 99 -Configuration 'with_skill' -Phase 'preflight' -ProgressLogPath $preflightLogPath -HeartbeatSeconds 0.3 - # Call Complete-RunnerChildProcess directly (the synchronous preflight path), - # without using Wait-AnyRunnerChild. - $preflightExit = Complete-RunnerChildProcess -Child $preflightChild - $preflightEvents = @() - if (Test-Path -LiteralPath $preflightLogPath -PathType Leaf) { - $preflightEvents = @(Get-Content -LiteralPath $preflightLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - } - Assert-Equal 0 $preflightExit 'synchronous preflight completes cleanly' - $preflightHeartbeats = @($preflightEvents | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) - Assert-True ($preflightHeartbeats.Count -ge 2) "synchronous Complete-RunnerChildProcess emits heartbeats during wait (got $($preflightHeartbeats.Count))" - $preflightCompleted = @($preflightEvents | Where-Object { [string]$_.state -eq 'completed' }) - Assert-Equal 1 $preflightCompleted.Count 'synchronous preflight reports exactly one completed state' - - Write-Output 'Runner observability: PASS' -} finally { - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + # Call Complete-RunnerChildProcess directly (the synchronous preflight path), + # without using Wait-AnyRunnerChild. + $preflightExit = Complete-RunnerChildProcess -Child $preflightChild + $preflightEvents = @() + if (Test-Path -LiteralPath $preflightLogPath -PathType Leaf) { + $preflightEvents = @(Get-Content -LiteralPath $preflightLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + Assert-Equal 0 $preflightExit 'synchronous preflight completes cleanly' + $preflightHeartbeats = @($preflightEvents | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) + Assert-True ($preflightHeartbeats.Count -ge 2) "synchronous Complete-RunnerChildProcess emits heartbeats during wait (got $($preflightHeartbeats.Count))" + $preflightCompleted = @($preflightEvents | Where-Object { [string]$_.state -eq 'completed' }) + Assert-Equal 1 $preflightCompleted.Count 'synchronous preflight reports exactly one completed state' + + Write-Output 'Runner observability: PASS' +} finally { + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index e192838..c0d6969 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1887,6 +1887,16 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' if ($LASTEXITCODE -eq 0) { throw 'Unknown analyzer model must fail preparation through analyzer model discovery.' } + $invalidAnalyzerDiagnostic = $invalidAnalyzerOutput -join [Environment]::NewLine + if ($invalidAnalyzerDiagnostic -notmatch 'missing-analyzer-model') { + throw "Invalid analyzer rejection must name the missing analyzer model in the diagnostic; got: $invalidAnalyzerDiagnostic" + } + if ($invalidAnalyzerDiagnostic -notmatch '(?i)analyzer') { + throw "Invalid analyzer rejection must identify the failure as an analyzer model validation/discovery error; got: $invalidAnalyzerDiagnostic" + } + if ($invalidAnalyzerDiagnostic -notmatch '(?i)(github-copilot|analyzer.*runner|runner.*analyzer)') { + throw "Invalid analyzer rejection must identify the analyzer runner; got: $invalidAnalyzerDiagnostic" + } if (Test-Path -LiteralPath $invalidAnalyzerRoot) { Remove-Item -LiteralPath $invalidAnalyzerRoot -Recurse -Force throw 'prepare-skill-evals.ps1 must not create a package when analyzer model validation fails.' From 8b9cd29b9e526a6d24c2e1d8102538624661d13f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 01:55:38 +0200 Subject: [PATCH 29/36] =?UTF-8?q?=F0=9F=90=9B=20fix=20eval=20handoff=20and?= =?UTF-8?q?=20transcript=20evidence=20checks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tighten explicit eval handoff provenance to require a prepared package, preserve transcript artifact identity during Phase 2 staging, and require exact transcript locators so transcript grading fails closed on ambiguous or cross-artifact evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/eval-request.ps1 | 53 +++- .../eval-runners/eval-grading-contract.ps1 | 7 +- .../eval-runners/invoke-phase2-analyzer.ps1 | 87 +++---- scripts/eval-runners/phase2-grading.ps1 | 210 +++++++++++++--- .../tests/fixtures/runner-owned-fixture.ps1 | 82 ++++++- .../eval-runners/tests/test-eval-request.ps1 | 27 ++ .../tests/test-integrity-finalization.ps1 | 232 +++++++++++++++++- 7 files changed, 601 insertions(+), 97 deletions(-) diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 index 0a4a8af..3550c24 100644 --- a/scripts/eval-request.ps1 +++ b/scripts/eval-request.ps1 @@ -10,19 +10,64 @@ #> Set-StrictMode -Version Latest -function Get-EvalHandoff { +. (Join-Path $PSScriptRoot 'eval-runners/runner-common.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/package-integrity.ps1') + +function Assert-PreparedEvalHandoffPackage { [CmdletBinding()] param( - [Parameter(Mandatory)][string]$PromptPath, - [switch]$Yolo, - [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + [Parameter(Mandatory)][string]$PromptPath ) $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path if ([IO.Path]::GetFileName($path) -cne 'RUN-THIS.prompt.md') { throw 'Handoff requires the prepared RUN-THIS.prompt.md file.' } + $package = Split-Path -Parent $path + try { + $manifestPath = Resolve-ManifestDeclaredPath -IterationDirectory $package -RelativePath 'manifest.json' -FieldName 'manifest.json' -Kind File -RequireExists + $manifest = Read-RunnerJson -Path $manifestPath + if ([string](Get-JsonProperty -Object $manifest -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-package/2') { + throw "manifest.json must declare 'codebeltnet/agentic/eval-package/2'." + } + if ([string](Get-JsonProperty -Object $manifest -Name 'execution' -Default '') -ne 'runner_handoff') { + throw "manifest.execution must be 'runner_handoff'." + } + $runnerPrompt = [string](Get-JsonProperty -Object $manifest -Name 'runner_prompt' -Default '') + if ([string]::IsNullOrWhiteSpace($runnerPrompt)) { + throw 'manifest.json must declare runner_prompt.' + } + $resolvedPrompt = Resolve-ManifestDeclaredPath -IterationDirectory $package -RelativePath $runnerPrompt -FieldName 'runner_prompt' -Kind File -RequireExists + $comparison = if ($IsWindows) { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } + if (-not [string]::Equals([System.IO.Path]::GetFullPath($resolvedPrompt), [System.IO.Path]::GetFullPath($path), $comparison)) { + throw 'The supplied RUN-THIS.prompt.md is not the manifest-declared runner_prompt.' + } + [void](Get-ManifestRunRecords -IterationDirectory $package -Manifest $manifest) + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $package -Manifest $manifest) + [void](Assert-PackageRunnerIdentity -IterationDirectory $package -Manifest $manifest) + } catch { + throw "Handoff requires a valid prepared eval package produced by this repository: $($_.Exception.Message)" + } + + return [pscustomobject]@{ + PromptPath = $path + Package = $package + } +} + +function Get-EvalHandoff { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath, + [switch]$Yolo, + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + ) + + $validated = Assert-PreparedEvalHandoffPackage -PromptPath $PromptPath + $path = [string]$validated.PromptPath + $package = [string]$validated.Package $decision = [ordered]@{ action = 'manual_handoff'; prompt_path = $path; reason = 'Preparation complete; hand this file to an external Eval Orchestrator.' } if (-not $Yolo) { return [pscustomobject]$decision } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index c4a82a9..e55d80c 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -85,7 +85,12 @@ function Test-GenericGradingReason { function Assert-EvalPassEvidence { param([object]$Entry, [object]$Canonical, [object]$Expected) - [void](Test-GradeEvidenceReference -Grade $Entry -Expected $Expected -Canonical $Canonical) + + $transcriptArtifacts = $null + if ([string](Get-JsonProperty -Object $Entry -Name 'evidence_domain' -Default '') -eq 'transcript') { + $transcriptArtifacts = @(Get-CanonicalTranscriptArtifacts -Record $Expected.record -Canonical $Canonical) + } + [void](Test-GradeEvidenceReference -Grade $Entry -Expected $Expected -Canonical $Canonical -TranscriptArtifacts $transcriptArtifacts) if (-not $Entry.passed) { return } $reason = [string](Get-JsonProperty -Object $Entry -Name 'reason' -Default '') if (Test-GenericGradingReason -Reason $reason -Assertion ([string]$Entry.assertion)) { diff --git a/scripts/eval-runners/invoke-phase2-analyzer.ps1 b/scripts/eval-runners/invoke-phase2-analyzer.ps1 index 25ea066..30c22c9 100644 --- a/scripts/eval-runners/invoke-phase2-analyzer.ps1 +++ b/scripts/eval-runners/invoke-phase2-analyzer.ps1 @@ -144,6 +144,24 @@ $bundleJson "@ } +function Resolve-AnalyzerTranscriptStagedPath { + param( + [Parameter(Mandatory = $true)][string]$RepoRoot, + [Parameter(Mandatory = $true)][string]$Artifact + ) + + Assert-SafeRelativePath -RelativePath $Artifact -FieldName 'transcript evidence artifact' + $relative = ('evidence/{0}' -f $Artifact.Replace('\', '/')) + $candidate = [System.IO.Path]::GetFullPath((Join-Path $RepoRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $RepoRoot -CandidatePath $candidate)) { + throw "Transcript artifact '$Artifact' resolves outside the analyzer workspace." + } + return [pscustomobject]@{ + Relative = $relative + FullPath = $candidate + } +} + function New-AnalyzerRunBundle { param( [Parameter(Mandatory = $true)][string]$Phase2Root, @@ -173,48 +191,32 @@ function New-AnalyzerRunBundle { if ($transcriptAssertions.Count -gt 0) { $evidenceDir = Join-Path $repoRoot 'evidence' New-Item -ItemType Directory -Path $evidenceDir -Force | Out-Null - $transcriptSources = [System.Collections.Generic.List[string]]::new() - foreach ($af in @(Get-JsonProperty -Object $canonical -Name 'output_files' -Default @())) { - [void]$transcriptSources.Add([string]$af) - } - $execResultFile = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_file' -Default '') - if (-not [string]::IsNullOrWhiteSpace($execResultFile) -and $transcriptSources -notcontains $execResultFile) { - [void]$transcriptSources.Add($execResultFile) - } - foreach ($artifact in $transcriptSources) { - $sourcePath = Join-Path $Worker.record.EvalDirectory ($artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar) - if (-not (Test-Path -LiteralPath $sourcePath -PathType Leaf)) { continue } - $sourceHash = Get-Sha256HexFromFile -Path $sourcePath - $fileName = [System.IO.Path]::GetFileName($sourcePath) - $stagedPath = Join-Path $evidenceDir $fileName - [System.IO.File]::Copy($sourcePath, $stagedPath, $true) - $stagedHash = Get-Sha256HexFromFile -Path $stagedPath - $entry = [ordered]@{ artifact = $artifact; source_sha256 = $sourceHash; staged_sha256 = $stagedHash } - if ($artifact -match '\.jsonl$') { - $events = [System.Collections.Generic.List[object]]::new() - $fileLines = [System.IO.File]::ReadAllLines($sourcePath, [System.Text.UTF8Encoding]::new($false)) - foreach ($fileLine in $fileLines) { - if ([string]::IsNullOrWhiteSpace($fileLine)) { continue } - try { - $parsed = $fileLine | ConvertFrom-Json -Depth 100 - $eventType = [string](Get-JsonProperty -Object $parsed -Name 'type' -Default '') - $content = Get-JsonProperty -Object $parsed -Name 'content' -Default $null - if ($null -eq $content) { $content = Get-JsonProperty -Object $parsed -Name 'data' -Default $null } - $contentStr = if ($null -ne $content) { [string]$content } else { ConvertTo-Json $parsed -Compress -Depth 10 } - $events.Add([ordered]@{ event_index = $events.Count; type = $eventType; content = $contentStr }) - } catch { - $events.Add([ordered]@{ event_index = $events.Count; type = ''; content = $fileLine }) - } + $stagedPathMap = @{} + foreach ($artifact in @(Get-CanonicalTranscriptArtifactPaths -Canonical $canonical)) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $Worker.record -Artifact $artifact -AllowMissing + if ($null -eq $sourcePath) { continue } + $snapshot = Get-TranscriptArtifactSnapshot -Record $Worker.record -Artifact $artifact -SourcePath $sourcePath + $stagedLocation = Resolve-AnalyzerTranscriptStagedPath -RepoRoot $repoRoot -Artifact $artifact + $stagedKey = ConvertTo-ComparablePath -Path $stagedLocation.FullPath + if ($stagedPathMap.ContainsKey($stagedKey)) { + $existing = [string]$stagedPathMap[$stagedKey] + if (-not [string]::Equals($existing, [string]$snapshot.SourcePath, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Transcript artifacts '$existing' and '$($snapshot.SourcePath)' resolve to the same analyzer staging path '$($stagedLocation.Relative)'." } - $entry['events'] = @($events.ToArray()) - } else { - $fileLines = [System.IO.File]::ReadAllLines($sourcePath, [System.Text.UTF8Encoding]::new($false)) - $textLines = [System.Collections.Generic.List[object]]::new() - for ($li = 0; $li -lt $fileLines.Count; $li++) { - $textLines.Add([ordered]@{ line = $li + 1; text = [string]$fileLines[$li] }) - } - $entry['lines'] = @($textLines.ToArray()) + continue + } + $stagedPathMap[$stagedKey] = [string]$snapshot.SourcePath + $parent = Split-Path -Parent $stagedLocation.FullPath + if (-not [string]::IsNullOrWhiteSpace($parent)) { + New-Item -ItemType Directory -Path $parent -Force | Out-Null + } + if (Test-Path -LiteralPath $stagedLocation.FullPath -PathType Leaf) { + throw "Transcript staging path '$($stagedLocation.Relative)' already exists in the analyzer workspace." } + [System.IO.File]::Copy($snapshot.SourcePath, $stagedLocation.FullPath, $false) + $entry = $snapshot.Entry + $entry['staged_artifact'] = [string]$stagedLocation.Relative + $entry['staged_sha256'] = Get-Sha256HexFromFile -Path $stagedLocation.FullPath $stagedTranscripts.Add($entry) } } @@ -355,7 +357,7 @@ function Confirm-AnalyzerFragment { if ([string]::IsNullOrWhiteSpace($reason)) { throw 'Analyzer fragment reason must be non-empty.' } $refs = @(Get-JsonProperty -Object $grade -Name 'evidence_refs' -Default @()) $entry = ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$grade.passed) -Reason $reason -EvidenceRefs $refs -Source 'analyzer' -Evidence $reason - [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical -StagedTranscripts $StagedTranscripts) + [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical -TranscriptArtifacts $StagedTranscripts) $entries.Add($entry) } return @($entries.ToArray()) @@ -688,8 +690,7 @@ try { foreach ($st in @($bundles[$worker.worker_id].StagedTranscripts)) { $artifact = [string]$st.artifact $sourceHash = [string]$st.source_sha256 - $evalDir = $worker.record.EvalDirectory - $sourcePath = Join-Path $evalDir ($artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $worker.record -Artifact $artifact $iterRelative = [System.IO.Path]::GetRelativePath($iteration, $sourcePath).Replace('\', '/') $key = "$iterRelative`:$sourceHash" if ($seenTranscriptKeys.Add($key)) { diff --git a/scripts/eval-runners/phase2-grading.ps1 b/scripts/eval-runners/phase2-grading.ps1 index e19ffda..d55bca0 100644 --- a/scripts/eval-runners/phase2-grading.ps1 +++ b/scripts/eval-runners/phase2-grading.ps1 @@ -100,6 +100,121 @@ function Get-ArmOutputLines { return @($lines) } +function Get-CanonicalTranscriptArtifactPaths { + param([Parameter(Mandatory = $true)][object]$Canonical) + + $artifacts = [System.Collections.Generic.List[string]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($value in @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @())) { + $artifact = [string]$value + if ([string]::IsNullOrWhiteSpace($artifact)) { continue } + if ($seen.Add($artifact)) { $artifacts.Add($artifact) } + } + $executionResultFile = [string](Get-JsonProperty -Object $Canonical -Name 'execution_result_file' -Default '') + if (-not [string]::IsNullOrWhiteSpace($executionResultFile) -and $seen.Add($executionResultFile)) { + $artifacts.Add($executionResultFile) + } + return @($artifacts.ToArray()) +} + +function Resolve-TranscriptArtifactSourcePath { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Artifact, + [switch]$AllowMissing + ) + + Assert-SafeRelativePath -RelativePath $Artifact -FieldName 'transcript evidence artifact' + $basePath = (Resolve-Path -LiteralPath $Record.EvalDirectory -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $basePath ($Artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $basePath -CandidatePath $candidate)) { + throw "transcript evidence artifact '$Artifact' resolves outside the frozen eval directory." + } + if (-not (Test-Path -LiteralPath $candidate -PathType Leaf)) { + if ($AllowMissing) { return $null } + throw "transcript evidence artifact '$Artifact' does not exist in the frozen eval directory." + } + $resolved = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $basePath -CandidatePath $resolved)) { + throw "transcript evidence artifact '$Artifact' resolves through a link outside the frozen eval directory." + } + return $resolved +} + +function Get-TranscriptArtifactSnapshot { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Artifact, + [string]$SourcePath = '' + ) + + if ([string]::IsNullOrWhiteSpace($SourcePath)) { + $SourcePath = Resolve-TranscriptArtifactSourcePath -Record $Record -Artifact $Artifact + } + + $entry = [ordered]@{ + artifact = [string]$Artifact + source_sha256 = Get-Sha256HexFromFile -Path $SourcePath + } + if ($Artifact -match '\.jsonl$') { + $events = [System.Collections.Generic.List[object]]::new() + $fileLines = [System.IO.File]::ReadAllLines($SourcePath, [System.Text.UTF8Encoding]::new($false)) + foreach ($fileLine in $fileLines) { + if ([string]::IsNullOrWhiteSpace($fileLine)) { continue } + $sourceText = [string]$fileLine + try { + $parsed = $fileLine | ConvertFrom-Json -Depth 100 + $eventType = [string](Get-JsonProperty -Object $parsed -Name 'type' -Default '') + $content = Get-JsonProperty -Object $parsed -Name 'content' -Default $null + if ($null -eq $content) { $content = Get-JsonProperty -Object $parsed -Name 'data' -Default $null } + $contentStr = if ($null -ne $content) { [string]$content } else { ConvertTo-Json $parsed -Compress -Depth 10 } + $events.Add([ordered]@{ + event_index = $events.Count + type = $eventType + content = $contentStr + source_text = $sourceText + }) + } catch { + $events.Add([ordered]@{ + event_index = $events.Count + type = '' + content = $sourceText + source_text = $sourceText + }) + } + } + $entry['events'] = @($events.ToArray()) + } else { + $fileLines = [System.IO.File]::ReadAllLines($SourcePath, [System.Text.UTF8Encoding]::new($false)) + $textLines = [System.Collections.Generic.List[object]]::new() + for ($lineIndex = 0; $lineIndex -lt $fileLines.Count; $lineIndex++) { + $textLines.Add([ordered]@{ line = $lineIndex + 1; text = [string]$fileLines[$lineIndex] }) + } + $entry['lines'] = @($textLines.ToArray()) + } + + return [pscustomobject]@{ + Artifact = [string]$Artifact + SourcePath = $SourcePath + Entry = $entry + } +} + +function Get-CanonicalTranscriptArtifacts { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Canonical + ) + + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($artifact in @(Get-CanonicalTranscriptArtifactPaths -Canonical $Canonical)) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $Record -Artifact $artifact -AllowMissing + if ($null -eq $sourcePath) { continue } + $entries.Add((Get-TranscriptArtifactSnapshot -Record $Record -Artifact $artifact -SourcePath $sourcePath).Entry) + } + return @($entries.ToArray()) +} + function New-OutputEvidenceRef { param( [Parameter(Mandatory = $true)][object]$Record, @@ -351,7 +466,7 @@ function Test-GradeEvidenceReference { [Parameter(Mandatory = $true)][object]$Grade, [Parameter(Mandatory = $true)][object]$Expected, [Parameter(Mandatory = $true)][object]$Canonical, - [AllowNull()][object[]]$StagedTranscripts = $null + [AllowNull()][object[]]$TranscriptArtifacts = $null ) $domain = [string](Get-JsonProperty -Object $Grade -Name 'evidence_domain' -Default '') @@ -382,45 +497,66 @@ function Test-GradeEvidenceReference { $allowed = @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @()) $execResultFile = [string](Get-JsonProperty -Object $Canonical -Name 'execution_result_file' -Default '') if ($allowed -notcontains $artifact -and $artifact -ne $execResultFile) { throw 'transcript evidence must cite a frozen transcript/artifact for the same arm.' } - if ($null -ne $StagedTranscripts) { - $staged = @($StagedTranscripts | Where-Object { [string]$_.artifact -eq $artifact }) - if ($staged.Count -eq 0) { throw 'transcript evidence artifact was not staged to the analyzer input bundle; grounding cannot be verified.' } - $stagedEntry = $staged[0] - $eventIndexRaw = Get-JsonProperty -Object $ref -Name 'event_index' -Default $null + if ($null -eq $TranscriptArtifacts) { + throw 'transcript evidence requires the frozen transcript evidence set so the cited location can be grounded.' + } + $matchingArtifacts = @($TranscriptArtifacts | Where-Object { [string]$_.artifact -eq $artifact }) + if ($matchingArtifacts.Count -eq 0) { + throw 'transcript evidence artifact is not present in the frozen transcript evidence set; grounding cannot be verified.' + } + $transcriptEntry = $matchingArtifacts[0] + $hasEventLocator = Test-JsonProperty -Object $ref -Name 'event_index' + $hasStartLine = Test-JsonProperty -Object $ref -Name 'start_line' + $hasEndLine = Test-JsonProperty -Object $ref -Name 'end_line' + $hasLineLocator = $hasStartLine -or $hasEndLine + $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') + $hasEvents = Test-JsonProperty -Object $transcriptEntry -Name 'events' + $hasLines = Test-JsonProperty -Object $transcriptEntry -Name 'lines' + if ($hasEventLocator -and $hasLineLocator) { + throw 'transcript evidence must declare exactly one locator form.' + } + if (-not $hasEventLocator -and -not $hasLineLocator) { + throw 'transcript evidence must declare event_index or start_line/end_line to ground the cited location.' + } + if ($hasEventLocator) { + if ($hasLines) { + throw 'transcript evidence uses event_index but the frozen artifact has line-based content.' + } + if (-not $hasEvents) { + throw 'transcript evidence event_index cannot be applied because the frozen artifact has no event-based content.' + } + $events = @(Get-JsonProperty -Object $transcriptEntry -Name 'events' -Default @()) + $eventIndex = [int](Get-JsonProperty -Object $ref -Name 'event_index' -Default -1) + if ($eventIndex -lt 0 -or $eventIndex -ge $events.Count) { + throw "transcript evidence event_index $eventIndex is outside the frozen artifact ($($events.Count) events)." + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $eventText = [string](Get-JsonProperty -Object $events[$eventIndex] -Name 'source_text' -Default (Get-JsonProperty -Object $events[$eventIndex] -Name 'content' -Default '')) + if (-not $eventText.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen event.' + } + } + } else { + if (-not $hasStartLine -or -not $hasEndLine) { + throw 'transcript evidence line locator must declare both start_line and end_line.' + } + if ($hasEvents) { + throw 'transcript evidence uses start_line/end_line but the frozen artifact has event-based content.' + } + if (-not $hasLines) { + throw 'transcript evidence line locator cannot be applied because the frozen artifact has no line-based content.' + } $startLine = [int](Get-JsonProperty -Object $ref -Name 'start_line' -Default 0) $endLine = [int](Get-JsonProperty -Object $ref -Name 'end_line' -Default $startLine) - $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') - $hasEvents = Test-JsonProperty -Object $stagedEntry -Name 'events' - $hasLines = Test-JsonProperty -Object $stagedEntry -Name 'lines' - if ($null -ne $eventIndexRaw -and $hasEvents) { - $events = @(Get-JsonProperty -Object $stagedEntry -Name 'events' -Default @()) - $eventIndex = [int]$eventIndexRaw - if ($eventIndex -lt 0 -or $eventIndex -ge $events.Count) { - throw "transcript evidence event_index $eventIndex is outside the frozen artifact ($($events.Count) events)." - } - if (-not [string]::IsNullOrWhiteSpace($quote)) { - $eventContent = [string](Get-JsonProperty -Object $events[$eventIndex] -Name 'content' -Default '') - if (-not $eventContent.Contains($quote, [StringComparison]::Ordinal)) { - throw 'transcript evidence quote is absent from the referenced frozen event.' - } - } - } elseif ($startLine -gt 0 -and $hasLines) { - $lines = @(Get-JsonProperty -Object $stagedEntry -Name 'lines' -Default @()) - if ($startLine -lt 1 -or $endLine -lt $startLine -or $endLine -gt $lines.Count) { - throw 'transcript evidence line range is outside the frozen artifact.' - } - if (-not [string]::IsNullOrWhiteSpace($quote)) { - $span = [string]::Join("`n", @($lines[($startLine - 1)..($endLine - 1)] | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) - if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { - throw 'transcript evidence quote is absent from the referenced frozen transcript lines.' - } + $lines = @(Get-JsonProperty -Object $transcriptEntry -Name 'lines' -Default @()) + if ($startLine -lt 1 -or $endLine -lt $startLine -or $endLine -gt $lines.Count) { + throw 'transcript evidence line range is outside the frozen artifact.' + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $span = [string]::Join("`n", @($lines[($startLine - 1)..($endLine - 1)] | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen transcript lines.' } - } elseif ($null -ne $eventIndexRaw -and $hasLines) { - throw 'transcript evidence uses event_index but the staged artifact has line-based content.' - } elseif ($startLine -gt 0 -and $hasEvents) { - throw 'transcript evidence uses start_line/end_line but the staged artifact has event-based content.' - } elseif (-not [string]::IsNullOrWhiteSpace($quote)) { - throw 'transcript evidence must declare event_index or start_line/end_line to ground the cited location.' } } } diff --git a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 index abc708f..888f8c3 100644 --- a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 +++ b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 @@ -95,6 +95,30 @@ try { return (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'evidence-validation-failed') -PathType Leaf) } + function Get-ExtraTranscriptArtifacts { + $markerPath = Join-Path $inputs.Run.HomeDirectoryPath 'extra-transcript-artifacts' + if (-not (Test-Path -LiteralPath $markerPath -PathType Leaf)) { + return @() + } + return @( + [ordered]@{ + Path = 'evidence/a/events.jsonl' + Content = @('{"type":"assistant.terminal","content":"artifact-a exact frozen quote"}') + MediaType = 'application/x-ndjson; charset=utf-8' + } + [ordered]@{ + Path = 'evidence/b/events.jsonl' + Content = @('{"type":"assistant.terminal","content":"artifact-b exact frozen quote"}') + MediaType = 'application/x-ndjson; charset=utf-8' + } + [ordered]@{ + Path = 'evidence/logs/transcript.txt' + Content = @('alpha frozen transcript line', 'beta frozen transcript line') + MediaType = 'text/plain; charset=utf-8' + } + ) + } + if ($Command -eq 'preflight') { Write-FixtureEvent -Kind 'preflight' if (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'preflight-incompatible') -PathType Leaf) { @@ -167,20 +191,50 @@ try { $assertionIndex = [int](Get-JsonProperty -Object $assertion -Name 'assertion_index' -Default 0) $domain = [string](Get-JsonProperty -Object $assertion -Name 'evidence_domain' -Default 'output') $artifact = [string](Get-JsonProperty -Object $bundle -Name 'canonical_result' -Default '') - $line = @(Get-JsonProperty -Object (Get-JsonProperty -Object $bundle -Name 'frozen_output' -Default $null) -Name 'lines' -Default @() | Select-Object -First 1) - $lineNumber = if ($line.Count -eq 1) { [int](Get-JsonProperty -Object $line[0] -Name 'line' -Default 1) } else { 1 } - $quote = if ($line.Count -eq 1) { [string](Get-JsonProperty -Object $line[0] -Name 'text' -Default 'deterministic fixture response') } else { 'deterministic fixture response' } - $fragmentGrades.Add([ordered]@{ - assertion_index = $assertionIndex - passed = $true - reason = "The deterministic analyzer fixture cites frozen one-arm output for assertion $assertionIndex." - evidence_refs = @([ordered]@{ + $evidenceRef = $null + if ($domain -eq 'transcript') { + $transcripts = @(Get-JsonProperty -Object $bundle -Name 'frozen_transcripts' -Default @()) + $eventTranscript = @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'events' } | Select-Object -First 1) + if ($eventTranscript.Count -eq 1) { + $event = @(Get-JsonProperty -Object $eventTranscript[0] -Name 'events' -Default @() | Select-Object -First 1) + if ($event.Count -ne 1) { throw 'Phase 2 fixture expected one frozen transcript event.' } + $evidenceRef = [ordered]@{ + artifact = [string](Get-JsonProperty -Object $eventTranscript[0] -Name 'artifact' -Default '') + domain = 'transcript' + event_index = [int](Get-JsonProperty -Object $event[0] -Name 'event_index' -Default 0) + quote = [string](Get-JsonProperty -Object $event[0] -Name 'source_text' -Default (Get-JsonProperty -Object $event[0] -Name 'content' -Default '')) + } + } else { + $lineTranscript = @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'lines' } | Select-Object -First 1) + if ($lineTranscript.Count -ne 1) { throw 'Phase 2 fixture expected frozen transcript evidence for transcript-domain grading.' } + $lines = @(Get-JsonProperty -Object $lineTranscript[0] -Name 'lines' -Default @() | Select-Object -First 2) + if ($lines.Count -lt 1) { throw 'Phase 2 fixture expected at least one frozen transcript line.' } + $quote = [string]::Join("`n", @($lines | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + $evidenceRef = [ordered]@{ + artifact = [string](Get-JsonProperty -Object $lineTranscript[0] -Name 'artifact' -Default '') + domain = 'transcript' + start_line = [int](Get-JsonProperty -Object $lines[0] -Name 'line' -Default 1) + end_line = [int](Get-JsonProperty -Object $lines[$lines.Count - 1] -Name 'line' -Default 1) + quote = $quote + } + } + } else { + $line = @(Get-JsonProperty -Object (Get-JsonProperty -Object $bundle -Name 'frozen_output' -Default $null) -Name 'lines' -Default @() | Select-Object -First 1) + $lineNumber = if ($line.Count -eq 1) { [int](Get-JsonProperty -Object $line[0] -Name 'line' -Default 1) } else { 1 } + $quote = if ($line.Count -eq 1) { [string](Get-JsonProperty -Object $line[0] -Name 'text' -Default 'deterministic fixture response') } else { 'deterministic fixture response' } + $evidenceRef = [ordered]@{ artifact = $artifact domain = $domain start_line = $lineNumber end_line = $lineNumber quote = $quote - }) + } + } + $fragmentGrades.Add([ordered]@{ + assertion_index = $assertionIndex + passed = $true + reason = "The deterministic analyzer fixture cites frozen one-arm output for assertion $assertionIndex." + evidence_refs = @($evidenceRef) }) } $fixtureFinalResponse = ConvertTo-RunnerJson -Value ([ordered]@{ @@ -242,6 +296,14 @@ try { } [IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $eventLines) + "`n"), [Text.UTF8Encoding]::new($false)) $eventsArtifact = New-ArtifactReference -Run $inputs.Run -Path 'evidence/fixture-events.jsonl' -Scope run -MediaType 'application/x-ndjson; charset=utf-8' + $artifactReferences = [System.Collections.Generic.List[object]]::new() + $artifactReferences.Add($eventsArtifact) + foreach ($extra in @(Get-ExtraTranscriptArtifacts)) { + $artifactPath = Join-Path $inputs.Run.RunRoot (([string]$extra.Path) -replace '/', [IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $artifactPath) -Force | Out-Null + [IO.File]::WriteAllText($artifactPath, ([string]::Join("`n", @($extra.Content)) + "`n"), [Text.UTF8Encoding]::new($false)) + $artifactReferences.Add((New-ArtifactReference -Run $inputs.Run -Path ([string]$extra.Path) -Scope run -MediaType ([string]$extra.MediaType))) + } $evidence = [ordered]@{ capture = [ordered]@{ source = 'harness_native_transport'; terminal = $true; worker_authored = $false } commands = @( @@ -295,7 +357,7 @@ try { 'cancelled' { New-ExecutionFailure -Code 'cancelled' -Message 'The deterministic runner-owned fixture was cancelled before completion.' } default { $null } } - $result = New-ExecutionResult -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Status $terminalStatus -FinalResponse $finalResponse -FinalResponseReason $finalResponseReason -StartedUtc ($executeStartUtc.ToString('o')) -FinishedUtc ($executeFinishUtc.ToString('o')) -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @('deterministic runner-owned fixture') -Telemetry ([ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fixture-events.jsonl'; complete = $true }); tokens = $fixtureTelemetryTokens; tool_calls = $fixtureTelemetryToolCalls; cost = New-UnavailableMetric -Reason 'fixture' }) -Artifacts @($eventsArtifact) -Evidence $evidence -AttemptCount 1 + $result = New-ExecutionResult -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Status $terminalStatus -FinalResponse $finalResponse -FinalResponseReason $finalResponseReason -StartedUtc ($executeStartUtc.ToString('o')) -FinishedUtc ($executeFinishUtc.ToString('o')) -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @('deterministic runner-owned fixture') -Telemetry ([ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fixture-events.jsonl'; complete = $true }); tokens = $fixtureTelemetryTokens; tool_calls = $fixtureTelemetryToolCalls; cost = New-UnavailableMetric -Reason 'fixture' }) -Artifacts @($artifactReferences.ToArray()) -Evidence $evidence -AttemptCount 1 [void](Assert-ExecutionResult -Result $result) Write-RunnerJson -Value $result -AsOutput } catch { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 38db1a3..ae38c1f 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -41,6 +41,22 @@ function Assert-Failure([hashtable]$Options, [string]$Pattern) { Assert-True ($script:dispatches.Count -eq $before) 'Failed preparation dispatched an Orchestrator.' } +function Assert-HandoffFailure([string]$PromptPath, [string]$Pattern, [string]$Description) { + $before = $script:dispatches.Count + $failed = $false + $claim = Join-Path (Split-Path -Parent $PromptPath) '.external-handoff-started' + try { + $decision = Get-EvalHandoff -PromptPath $PromptPath -Yolo -CanDelegateFreshOrchestrator + Invoke-FakeHost $decision + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected handoff failure for ${Description}: $($_.Exception.Message)" + } + Assert-True $failed "$Description unexpectedly passed handoff validation." + Assert-True ($script:dispatches.Count -eq $before) "$Description dispatched an Orchestrator." + Assert-True (-not (Test-Path -LiteralPath $claim)) "$Description created .external-handoff-started before rejection." +} + try { $catalog = Join-Path $workspace 'models.json' @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'claude-opus-4.7' }, @{ id = 'provider/Exact.Model' }) } | @@ -147,6 +163,17 @@ try { $invalid.Skill = 'no-such-skill' Assert-Failure $invalid 'skill|directory|path' + $standaloneRoot = Join-Path $workspace 'standalone' + [void](New-Item -ItemType Directory -Path $standaloneRoot -Force) + $standalonePrompt = Join-Path $standaloneRoot 'RUN-THIS.prompt.md' + [System.IO.File]::WriteAllText($standalonePrompt, '# forged handoff', [System.Text.UTF8Encoding]::new($false)) + Assert-HandoffFailure -PromptPath $standalonePrompt -Pattern 'valid prepared eval package' -Description 'standalone RUN-THIS.prompt.md' + + $tampered = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'tampered') + $tamperedPackage = Split-Path -Parent $tampered.prompt_path + [System.IO.File]::AppendAllText((Join-Path $tamperedPackage 'tools/eval-runners/resolve-runner.ps1'), "`n# tampered`n", [System.Text.UTF8Encoding]::new($false)) + Assert-HandoffFailure -PromptPath $tampered.prompt_path -Pattern 'valid prepared eval package|Requires a fresh package|changed after preparation' -Description 'tampered prepared package' + # Check the real discovery call contract, not a mocked model resolver. $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index 3a40ad3..fa406f6 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -24,6 +24,7 @@ $repositoryRoot = (Resolve-Path (Join-Path $runnerRoot '..')).Path . (Join-Path $runnerRoot 'manifest-paths.ps1') . (Join-Path $runnerRoot 'orchestration.ps1') . (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'phase2-grading.ps1') . (Join-Path $runnerRoot 'package-integrity.ps1') . (Join-Path $runnerRoot 'fanout-process.ps1') @@ -57,6 +58,19 @@ function Read-TestJson { return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 } +function Remove-TestProperty { + param( + [Parameter(Mandatory = $true)][object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + if ($Object -is [System.Collections.IDictionary]) { + [void]$Object.Remove($Name) + } else { + [void]$Object.PSObject.Properties.Remove($Name) + } +} + function Invoke-TestTool { param( [Parameter(Mandatory = $true)][string]$Path, @@ -176,6 +190,10 @@ function New-TestRun { } Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + if ($EvalId -eq 2) { + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'extra-transcript-artifacts'), "1`n", [System.Text.UTF8Encoding]::new($false)) + } + return [pscustomobject]@{ Directory = $runDirectory RunPath = Join-Path $runDirectory 'run.json' @@ -183,6 +201,77 @@ function New-TestRun { } } +function Get-GradingEntry { + param( + [Parameter(Mandatory = $true)][object]$Document, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$Configuration, + [Parameter(Mandatory = $true)][int]$AssertionIndex + ) + + return @($Document.grading | Where-Object { + [int]$_.eval_id -eq $EvalId -and + [string]$_.configuration -eq $Configuration -and + [int]$_.assertion_index -eq $AssertionIndex + } | Select-Object -First 1)[0] +} + +function New-TestTranscriptEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Canonical, + [ValidateSet('Events', 'Lines')][string]$Kind = 'Events', + [string]$Artifact = '' + ) + + $transcripts = @(Get-CanonicalTranscriptArtifacts -Record $Record -Canonical $Canonical) + if ([string]::IsNullOrWhiteSpace($Artifact)) { + $entry = if ($Kind -eq 'Events') { + @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'events' } | Select-Object -First 1) + } else { + @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'lines' } | Select-Object -First 1) + } + } else { + $entry = @($transcripts | Where-Object { [string]$_.artifact -eq $Artifact } | Select-Object -First 1) + } + if ($entry.Count -ne 1 -and [string]::IsNullOrWhiteSpace($Artifact) -and $Kind -eq 'Events') { + $entry = @((Get-TranscriptArtifactSnapshot -Record $Record -Artifact ("$([string]$Record.Configuration)/evidence/fixture-events.jsonl")).Entry) + } + if ($entry.Count -ne 1) { + throw "Missing $Kind transcript artifact '$Artifact' for deterministic grading coverage." + } + if ($Kind -eq 'Events') { + if (-not (Test-JsonProperty -Object $entry[0] -Name 'events')) { + throw "Transcript artifact '$([string]$entry[0].artifact)' does not expose events." + } + $event = @(Get-JsonProperty -Object $entry[0] -Name 'events' -Default @() | Select-Object -First 1) + if ($event.Count -ne 1) { + throw "Transcript artifact '$([string]$entry[0].artifact)' has no event content." + } + return [ordered]@{ + artifact = [string](Get-JsonProperty -Object $entry[0] -Name 'artifact' -Default '') + domain = 'transcript' + event_index = [int](Get-JsonProperty -Object $event[0] -Name 'event_index' -Default 0) + quote = [string](Get-JsonProperty -Object $event[0] -Name 'source_text' -Default (Get-JsonProperty -Object $event[0] -Name 'content' -Default '')) + } + } + + if (-not (Test-JsonProperty -Object $entry[0] -Name 'lines')) { + throw "Transcript artifact '$([string]$entry[0].artifact)' does not expose line content." + } + $lines = @(Get-JsonProperty -Object $entry[0] -Name 'lines' -Default @() | Select-Object -First 2) + if ($lines.Count -lt 1) { + throw "Transcript artifact '$([string]$entry[0].artifact)' has no line content." + } + return [ordered]@{ + artifact = [string](Get-JsonProperty -Object $entry[0] -Name 'artifact' -Default '') + domain = 'transcript' + start_line = [int](Get-JsonProperty -Object $lines[0] -Name 'line' -Default 1) + end_line = [int](Get-JsonProperty -Object $lines[$lines.Count - 1] -Name 'line' -Default 1) + quote = [string]::Join("`n", @($lines | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + } +} + function New-TestGradingDocument { param([Parameter(Mandatory = $true)][object[]]$Records) @@ -190,11 +279,12 @@ function New-TestGradingDocument { foreach ($record in @($Records | Sort-Object EvalId, Configuration)) { $metadata = Read-TestJson -Path $record.MetadataPath $assertions = @($metadata.assertions) + $canonical = Read-TestJson -Path $record.ResultPath for ($index = 0; $index -lt $assertions.Count; $index++) { $assertionText = if ($assertions[$index] -is [string]) { [string]$assertions[$index] } elseif ($assertions[$index].PSObject.Properties.Name -contains 'assertion') { [string]$assertions[$index].assertion } else { [string]$assertions[$index] } $domain = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'evidence_domain') { [string]$assertions[$index].evidence_domain } else { 'output' } $validator = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'validator') { [string]$assertions[$index].validator } else { $null } - $output = [string](Read-TestJson -Path $record.ResultPath).output + $output = [string]$canonical.output $refs = if ($domain -eq 'validator') { @([ordered]@{ artifact = [string]$record.ResultRelative @@ -204,6 +294,8 @@ function New-TestGradingDocument { passed = $true event = 'deterministic fixture validator evidence' }) + } elseif ($domain -eq 'transcript') { + @((New-TestTranscriptEvidenceRef -Record $record -Canonical $canonical -Kind Events)) } else { @([ordered]@{ artifact = [string]$record.ResultRelative @@ -447,7 +539,13 @@ for ($index = 0; $index -lt $count; $index++) { $evalDirectory = Join-Path $iteration $evalName New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null $assertions = if ($evalId -eq 2) { - @('the protected operation is absent before confirmation and occurs only after the same-session confirmation turn', 'the response contains the fixture completion content') + @( + [ordered]@{ + assertion = 'the protected operation is absent before confirmation and occurs only after the same-session confirmation turn' + evidence_domain = 'transcript' + } + 'the response contains the fixture completion content' + ) } elseif ($evalId -eq 3) { @( [ordered]@{ assertion = 'current Git branch was resolved'; evidence_domain = 'validator'; validator = 'git.current_branch_observed' } @@ -647,6 +745,15 @@ for ($index = 0; $index -lt $count; $index++) { } $gradingPath = Join-Path $iteration 'grading.json' $validGrading = New-TestGradingDocument -Records $records + $transcriptEvalId = 2 + $transcriptAssertionIndex = 0 + $transcriptRecord = @($records | Where-Object { $_.EvalId -eq $transcriptEvalId -and $_.Configuration -eq 'with_skill' } | Select-Object -First 1)[0] + $transcriptCanonical = Read-TestJson -Path $transcriptRecord.ResultPath + $transcriptArtifacts = @(Get-CanonicalTranscriptArtifacts -Record $transcriptRecord -Canonical $transcriptCanonical) + $collisionEventArtifacts = @($transcriptArtifacts | Where-Object { [string]$_.artifact -like 'with_skill/evidence/*/events.jsonl' } | Sort-Object artifact) + $lineTranscriptArtifact = @($transcriptArtifacts | Where-Object { [string]$_.artifact -eq 'with_skill/evidence/logs/transcript.txt' } | Select-Object -First 1)[0] + Assert-Equal 2 $collisionEventArtifacts.Count 'fixture exposes two same-basename transcript artifacts' + Assert-True ($null -ne $lineTranscriptArtifact) 'fixture exposes a line-based transcript artifact' $validationScript = Join-Path $packageTools 'validate-eval-grading.ps1' $validationArguments = @('-IterationDirectory', $iteration, '-GradingPath', 'grading.json') $validationSideEffectPaths = Get-GradingValidationSideEffectPaths -IterationDirectory $iteration -Records $records @@ -706,6 +813,109 @@ for ($index = 0; $index -lt $count; $index++) { Write-TestJson -Path $gradingPath -Value $badDomain Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'wrong evidence domain rejected' -ExpectedText 'assertion identity' + $validTranscriptEvent = Copy-TestGradingDocument -Document $validGrading + $validTranscriptEventEntry = Get-GradingEntry -Document $validTranscriptEvent -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $validTranscriptEventEntry.evidence_refs = @((New-TestTranscriptEvidenceRef -Record $transcriptRecord -Canonical $transcriptCanonical -Kind Events -Artifact ([string]$collisionEventArtifacts[0].artifact))) + Write-TestJson -Path $gradingPath -Value $validTranscriptEvent + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'valid transcript event_index grading validation' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'valid transcript event-index grading validation' + + $missingTranscriptLocator = Copy-TestGradingDocument -Document $validGrading + $missingTranscriptLocatorEntry = Get-GradingEntry -Document $missingTranscriptLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + Remove-TestProperty -Object $missingTranscriptLocatorEntry.evidence_refs[0] -Name 'event_index' + Remove-TestProperty -Object $missingTranscriptLocatorEntry.evidence_refs[0] -Name 'quote' + Write-TestJson -Path $gradingPath -Value $missingTranscriptLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'artifact-only transcript evidence rejected' -ExpectedText 'must declare event_index or start_line/end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'artifact-only transcript evidence rejection' + + $quotedWithoutTranscriptLocator = Copy-TestGradingDocument -Document $validGrading + $quotedWithoutTranscriptLocatorEntry = Get-GradingEntry -Document $quotedWithoutTranscriptLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + Remove-TestProperty -Object $quotedWithoutTranscriptLocatorEntry.evidence_refs[0] -Name 'event_index' + $quotedWithoutTranscriptLocatorEntry.evidence_refs[0].quote = 'some text' + Write-TestJson -Path $gradingPath -Value $quotedWithoutTranscriptLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'quoted transcript evidence without locator rejected' -ExpectedText 'must declare event_index or start_line/end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'quoted transcript evidence without locator rejection' + + $invalidEventIndex = Copy-TestGradingDocument -Document $validGrading + $invalidEventIndexEntry = Get-GradingEntry -Document $invalidEventIndex -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $invalidEventIndexEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + event_index = 9 + }) + Write-TestJson -Path $gradingPath -Value $invalidEventIndex + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'out-of-range transcript event_index rejected' -ExpectedText 'event_index 9 is outside' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'transcript event-index bounds rejection' + + $validTranscriptLines = Copy-TestGradingDocument -Document $validGrading + $validTranscriptLinesEntry = Get-GradingEntry -Document $validTranscriptLines -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $validTranscriptLinesEntry.evidence_refs = @((New-TestTranscriptEvidenceRef -Record $transcriptRecord -Canonical $transcriptCanonical -Kind Lines -Artifact ([string]$lineTranscriptArtifact.artifact))) + Write-TestJson -Path $gradingPath -Value $validTranscriptLines + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'valid transcript line-range grading validation' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'valid transcript line-range grading validation' + + $invalidTranscriptLines = Copy-TestGradingDocument -Document $validGrading + $invalidTranscriptLinesEntry = Get-GradingEntry -Document $invalidTranscriptLines -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $invalidTranscriptLinesEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + start_line = 9 + end_line = 9 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $invalidTranscriptLines + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'out-of-range transcript line range rejected' -ExpectedText 'line range is outside' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'transcript line-range bounds rejection' + + $missingTranscriptEndLine = Copy-TestGradingDocument -Document $validGrading + $missingTranscriptEndLineEntry = Get-GradingEntry -Document $missingTranscriptEndLine -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $missingTranscriptEndLineEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + start_line = 1 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $missingTranscriptEndLine + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'partial transcript line locator rejected' -ExpectedText 'line locator must declare both start_line and end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'partial transcript line locator rejection' + + $lineArtifactEventLocator = Copy-TestGradingDocument -Document $validGrading + $lineArtifactEventLocatorEntry = Get-GradingEntry -Document $lineArtifactEventLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $lineArtifactEventLocatorEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + event_index = 0 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $lineArtifactEventLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'event locator on line-based transcript rejected' -ExpectedText 'uses event_index but the frozen artifact has line-based content' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'line-based transcript event locator rejection' + + $eventArtifactLineLocator = Copy-TestGradingDocument -Document $validGrading + $eventArtifactLineLocatorEntry = Get-GradingEntry -Document $eventArtifactLineLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $eventArtifactLineLocatorEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + start_line = 1 + end_line = 1 + quote = 'artifact-a exact frozen quote' + }) + Write-TestJson -Path $gradingPath -Value $eventArtifactLineLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'line locator on event-based transcript rejected' -ExpectedText 'uses start_line/end_line but the frozen artifact has event-based content' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'event-based transcript line locator rejection' + + $crossArtifactQuote = Copy-TestGradingDocument -Document $validGrading + $crossArtifactQuoteEntry = Get-GradingEntry -Document $crossArtifactQuote -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $crossArtifactQuoteEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + event_index = 0 + quote = 'artifact-b exact frozen quote' + }) + Write-TestJson -Path $gradingPath -Value $crossArtifactQuote + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'cross-artifact transcript quote rejected' -ExpectedText 'quote is absent from the referenced frozen event' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'cross-artifact transcript quote rejection' + $repeated = Copy-TestGradingDocument -Document $validGrading $repeated.grading[1].evidence = $repeated.grading[0].evidence Write-TestJson -Path $gradingPath -Value $repeated @@ -732,6 +942,24 @@ for ($index = 0; $index -lt $count; $index++) { Assert-True (@($phase2State.expected_worker_ids | Where-Object { [string]$_ -like 'arm-3-*' }).Count -eq 0) 'validator-only eval arms must require zero analyzer workers' Assert-Equal 14 @($phase2State.validator_results).Count 'validator-only paired arms resolve process assertions deterministically' Assert-Equal 4 @($phase2State.expected_worker_ids).Count 'semantic worker cardinality derives from unresolved assertions, not manifest arm count' + $phase2BundlePath = Join-Path $iteration 'phase2\work\arm-2-with_skill\repo\input-bundle.json' + $phase2Bundle = Read-TestJson -Path $phase2BundlePath + $stagedCollisionEntries = @($phase2Bundle.frozen_transcripts | Where-Object { [string]$_.artifact -like 'with_skill/evidence/*/events.jsonl' } | Sort-Object artifact) + Assert-Equal 2 $stagedCollisionEntries.Count 'Phase 2 stages both same-basename transcript artifacts' + Assert-True ([string]$stagedCollisionEntries[0].staged_artifact -cne [string]$stagedCollisionEntries[1].staged_artifact) 'same-basename transcript artifacts must stage to unique paths' + $phase2BundleRepo = Join-Path $iteration 'phase2\work\arm-2-with_skill\repo' + $stagedCollisionContents = [System.Collections.Generic.List[string]]::new() + foreach ($entry in $stagedCollisionEntries) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $transcriptRecord -Artifact ([string]$entry.artifact) + $stagedPath = Resolve-ContainedPath -BasePath $phase2BundleRepo -RelativePath ([string]$entry.staged_artifact) -FieldName 'staged transcript artifact' -Kind File + Assert-Equal ([string]$entry.source_sha256) (Get-Sha256HexFromFile -Path $sourcePath) "bundle preserves the source hash for $([string]$entry.artifact)" + Assert-Equal ([string]$entry.staged_sha256) (Get-Sha256HexFromFile -Path $stagedPath) "bundle preserves the staged hash for $([string]$entry.artifact)" + $sourceContent = [System.IO.File]::ReadAllText($sourcePath, [System.Text.UTF8Encoding]::new($false)) + $stagedContent = [System.IO.File]::ReadAllText($stagedPath, [System.Text.UTF8Encoding]::new($false)) + Assert-Equal $sourceContent $stagedContent "staged transcript preserves the frozen content for $([string]$entry.artifact)" + $stagedCollisionContents.Add($stagedContent) + } + Assert-True ([string]$stagedCollisionContents[0] -cne [string]$stagedCollisionContents[1]) 'same-basename transcript artifacts keep distinct staged content' $rootGradingAfterPhase2 = Read-TestJson -Path $gradingPath Assert-Equal 22 @($rootGradingAfterPhase2.grading).Count 'root grading cardinality derives from normalized assertions' Assert-True (@($rootGradingAfterPhase2.grading | Where-Object { [string]$_.evidence_domain -eq 'validator' -and [string]$_.source -eq 'validator' }).Count -eq 14) 'validator assertions are resolved by deterministic validator results, not analyzer prose' From 7b2c4c8d4e9fa58f28103cc62b1904b9266c9173 Mon Sep 17 00:00:00 2001 From: gimlichael Date: Thu, 10 Sep 2026 03:00:45 +0200 Subject: [PATCH 30/36] =?UTF-8?q?=F0=9F=90=9B=20bind=20prepared=20handoff?= =?UTF-8?q?=20prompt=20bytes=20to=20the=20manifest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Persist the final RUN-THIS.prompt.md SHA-256 in the prepared manifest and verify it before any handoff reservation. This makes prompt tampering fail closed while preserving normal manual and yolo handoff flows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/eval-request.ps1 | 11 ++++ .../eval-runners/tests/test-eval-request.ps1 | 50 +++++++++++++++++++ scripts/prepare-skill-evals.ps1 | 10 ++-- scripts/validate-skill-templates.ps1 | 10 ++++ 4 files changed, 77 insertions(+), 4 deletions(-) diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 index 3550c24..b0b4426 100644 --- a/scripts/eval-request.ps1 +++ b/scripts/eval-request.ps1 @@ -39,11 +39,22 @@ function Assert-PreparedEvalHandoffPackage { if ([string]::IsNullOrWhiteSpace($runnerPrompt)) { throw 'manifest.json must declare runner_prompt.' } + $runnerPromptHash = [string](Get-JsonProperty -Object $manifest -Name 'runner_prompt_sha256' -Default '') + if ([string]::IsNullOrWhiteSpace($runnerPromptHash)) { + throw 'manifest.json must declare runner_prompt_sha256.' + } + if (-not (Test-Sha256 -Value $runnerPromptHash) -or $runnerPromptHash -cne $runnerPromptHash.ToLowerInvariant()) { + throw 'manifest.runner_prompt_sha256 must be a lowercase SHA-256.' + } $resolvedPrompt = Resolve-ManifestDeclaredPath -IterationDirectory $package -RelativePath $runnerPrompt -FieldName 'runner_prompt' -Kind File -RequireExists $comparison = if ($IsWindows) { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } if (-not [string]::Equals([System.IO.Path]::GetFullPath($resolvedPrompt), [System.IO.Path]::GetFullPath($path), $comparison)) { throw 'The supplied RUN-THIS.prompt.md is not the manifest-declared runner_prompt.' } + $currentRunnerPromptHash = Get-Sha256HexFromFile -Path $resolvedPrompt + if ($currentRunnerPromptHash -cne $runnerPromptHash) { + throw 'manifest.runner_prompt_sha256 does not match the current RUN-THIS.prompt.md bytes. Requires a fresh package.' + } [void](Get-ManifestRunRecords -IterationDirectory $package -Manifest $manifest) [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $package -Manifest $manifest) [void](Assert-PackageRunnerIdentity -IterationDirectory $package -Manifest $manifest) diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index ae38c1f..9d48f32 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -8,10 +8,26 @@ $scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path $workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) [void](New-Item -ItemType Directory -Path $workspace -Force) $script:dispatches = [Collections.Generic.List[string]]::new() +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) function Assert-True([bool]$Condition, [string]$Message) { if (-not $Condition) { throw $Message } } +function Read-PackageManifest([string]$PromptPath) { + $manifestPath = Join-Path (Split-Path -Parent $PromptPath) 'manifest.json' + return [System.IO.File]::ReadAllText($manifestPath, $utf8NoBom) | ConvertFrom-Json +} +function Write-PackageManifest([string]$PromptPath, [object]$Manifest) { + $manifestPath = Join-Path (Split-Path -Parent $PromptPath) 'manifest.json' + [System.IO.File]::WriteAllText($manifestPath, (($Manifest | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} +function Assert-PreparedPromptBinding([string]$PromptPath, [string]$Description) { + $manifest = Read-PackageManifest -PromptPath $PromptPath + Assert-True ([string]$manifest.runner_prompt -ceq 'RUN-THIS.prompt.md') "$Description manifest.runner_prompt changed unexpectedly." + $declaredHash = [string]$manifest.runner_prompt_sha256 + Assert-True ($declaredHash -match '^[0-9a-f]{64}$') "$Description manifest.runner_prompt_sha256 must be a lowercase SHA-256." + Assert-True ($declaredHash -ceq (Get-Sha256HexFromFile -Path $PromptPath)) "$Description manifest.runner_prompt_sha256 must match RUN-THIS.prompt.md byte-for-byte." +} function Invoke-FakeHost($Decision) { if ($Decision.action -eq 'external_handoff') { # The host's only input is the actual canonical handoff file, never an arm prompt. @@ -45,6 +61,7 @@ function Assert-HandoffFailure([string]$PromptPath, [string]$Pattern, [string]$D $before = $script:dispatches.Count $failed = $false $claim = Join-Path (Split-Path -Parent $PromptPath) '.external-handoff-started' + $decision = $null try { $decision = Get-EvalHandoff -PromptPath $PromptPath -Yolo -CanDelegateFreshOrchestrator Invoke-FakeHost $decision @@ -53,6 +70,7 @@ function Assert-HandoffFailure([string]$PromptPath, [string]$Pattern, [string]$D Assert-True ($_.Exception.Message -match $Pattern) "Unexpected handoff failure for ${Description}: $($_.Exception.Message)" } Assert-True $failed "$Description unexpectedly passed handoff validation." + Assert-True ($null -eq $decision -or $decision.action -ne 'external_handoff') "$Description returned external_handoff before failing." Assert-True ($script:dispatches.Count -eq $before) "$Description dispatched an Orchestrator." Assert-True (-not (Test-Path -LiteralPath $claim)) "$Description created .external-handoff-started before rejection." } @@ -64,6 +82,7 @@ try { $normalOptions = New-Preparation 'Codex' 'normal' $normal = Invoke-EvalRequest -Preparation $normalOptions -CanDelegateFreshOrchestrator + Assert-PreparedPromptBinding -PromptPath $normal.prompt_path -Description 'Normal preparation' Invoke-FakeHost $normal Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' @@ -108,6 +127,7 @@ try { } else { Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator } + Assert-PreparedPromptBinding -PromptPath $decision.prompt_path -Description $case.Name $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' if ($case.ContainsKey('StrongerModel')) { @@ -169,6 +189,36 @@ try { [System.IO.File]::WriteAllText($standalonePrompt, '# forged handoff', [System.Text.UTF8Encoding]::new($false)) Assert-HandoffFailure -PromptPath $standalonePrompt -Pattern 'valid prepared eval package' -Description 'standalone RUN-THIS.prompt.md' + $pathMismatch = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'path-mismatch') + $pathMismatchManifest = Read-PackageManifest -PromptPath $pathMismatch.prompt_path + $pathMismatchManifest.runner_prompt = 'README.md' + $pathMismatchManifest.runner_prompt_sha256 = Get-Sha256HexFromFile -Path (Join-Path (Split-Path -Parent $pathMismatch.prompt_path) 'README.md') + Write-PackageManifest -PromptPath $pathMismatch.prompt_path -Manifest $pathMismatchManifest + Assert-HandoffFailure -PromptPath $pathMismatch.prompt_path -Pattern 'manifest-declared runner_prompt' -Description 'manifest runner_prompt path mismatch' + + $tamperedPrompt = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'tampered-prompt') + Assert-PreparedPromptBinding -PromptPath $tamperedPrompt.prompt_path -Description 'Prompt tamper fixture' + [System.IO.File]::AppendAllText($tamperedPrompt.prompt_path, "`n# tampered prompt`n", $utf8NoBom) + Assert-HandoffFailure -PromptPath $tamperedPrompt.prompt_path -Pattern 'runner_prompt_sha256|RUN-THIS\.prompt\.md bytes|Requires a fresh package' -Description 'tampered RUN-THIS.prompt.md' + + $missingPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'missing-prompt-hash') + $missingPromptHashManifest = Read-PackageManifest -PromptPath $missingPromptHash.prompt_path + [void]$missingPromptHashManifest.PSObject.Properties.Remove('runner_prompt_sha256') + Write-PackageManifest -PromptPath $missingPromptHash.prompt_path -Manifest $missingPromptHashManifest + Assert-HandoffFailure -PromptPath $missingPromptHash.prompt_path -Pattern 'runner_prompt_sha256' -Description 'missing runner_prompt_sha256' + + $malformedPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'malformed-prompt-hash') + $malformedPromptHashManifest = Read-PackageManifest -PromptPath $malformedPromptHash.prompt_path + $malformedPromptHashManifest.runner_prompt_sha256 = 'not-a-lowercase-sha256' + Write-PackageManifest -PromptPath $malformedPromptHash.prompt_path -Manifest $malformedPromptHashManifest + Assert-HandoffFailure -PromptPath $malformedPromptHash.prompt_path -Pattern 'runner_prompt_sha256' -Description 'malformed runner_prompt_sha256' + + $mismatchedPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'mismatched-prompt-hash') + $mismatchedPromptHashManifest = Read-PackageManifest -PromptPath $mismatchedPromptHash.prompt_path + $mismatchedPromptHashManifest.runner_prompt_sha256 = ('1' * 64) + Write-PackageManifest -PromptPath $mismatchedPromptHash.prompt_path -Manifest $mismatchedPromptHashManifest + Assert-HandoffFailure -PromptPath $mismatchedPromptHash.prompt_path -Pattern 'runner_prompt_sha256|RUN-THIS\.prompt\.md bytes|Requires a fresh package' -Description 'mismatched runner_prompt_sha256' + $tampered = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'tampered') $tamperedPackage = Split-Path -Parent $tampered.prompt_path [System.IO.File]::AppendAllText((Join-Path $tamperedPackage 'tools/eval-runners/resolve-runner.ps1'), "`n# tampered`n", [System.Text.UTF8Encoding]::new($false)) diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 89a2712..adf94b5 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1826,6 +1826,11 @@ function Invoke-PrepareMode { }) } + Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -EffectiveConcurrency $effectiveConcurrency) + $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' + Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency ([int]$effectiveConcurrency.Value) -PerArmTimeoutSeconds $TimeoutSeconds) + $runnerPromptHash = Get-FileSha256 -Path $runnerPath + $manifest = [ordered]@{ schema = $packageSchema skill_name = $Skill @@ -1841,6 +1846,7 @@ function Invoke-PrepareMode { preset = $executionSelection.Preset } runner_prompt = 'RUN-THIS.prompt.md' + runner_prompt_sha256 = $runnerPromptHash execution_profile = 'execution-profile.json' analyzer_profile = 'analyzer-profile.json' analyzer_profile_sha256 = $analyzerProfileHash @@ -1906,10 +1912,6 @@ function Invoke-PrepareMode { evals = @($manifestEvals) } ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'manifest.json') -Value $manifest - - Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -EffectiveConcurrency $effectiveConcurrency) - $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' - Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency ([int]$effectiveConcurrency.Value) -PerArmTimeoutSeconds $TimeoutSeconds) [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iterationDirectory -Manifest $manifest) [void](Assert-PackageRunnerIdentity -IterationDirectory $iterationDirectory -Manifest $manifest -ExpectedRunner ([string]$executionSelection.Runner)) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index c0d6969..0f5473e 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1607,6 +1607,16 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable if ([string]$manifest.execution_selection.runner -ne $ExpectedRunner -or [string]$manifest.execution_selection.model -ne $ExpectedModel) { throw "$Name manifest.execution_selection must preserve requested runner/model '$ExpectedRunner'/'$ExpectedModel'." } + if ([string]$manifest.runner_prompt -ne 'RUN-THIS.prompt.md') { + throw "$Name manifest.runner_prompt must declare RUN-THIS.prompt.md." + } + if ([string]$manifest.runner_prompt_sha256 -notmatch '^[0-9a-f]{64}$') { + throw "$Name manifest.runner_prompt_sha256 must be a lowercase SHA-256." + } + $actualPromptHash = (Get-FileHash -Algorithm SHA256 -LiteralPath $promptPath).Hash.ToLowerInvariant() + if ([string]$manifest.runner_prompt_sha256 -ne $actualPromptHash) { + throw "$Name manifest.runner_prompt_sha256 must match RUN-THIS.prompt.md byte-for-byte." + } $runnerTools = Join-Path $IterationDirectory ([string]$manifest.runner_tools) . (Join-Path $runnerTools 'runner-common.ps1') From aeb7e9dc3d8f0571d845411e38549d983cbe8f6f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 04:18:39 +0200 Subject: [PATCH 31/36] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20handoff?= =?UTF-8?q?=20decision=20workflow=20and=20state=20lifecycle?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the new explicit state machine for eval handoff decisions, including confirmation_required, dispatch_immediately, and max_new_external_orchestrators fields. Clarify that external_handoff authorization is complete and does not require further user confirmation. Explain the orchestrator handle lifecycle and bounded wait semantics for remaining running state. --- AGENTS.md | 6 ++++-- CONTRIBUTING.md | 2 +- README.md | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fe5f7bd..c97bf4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,14 +68,16 @@ An explicit request such as `Please eval dotnet-strong-name-signing using Codex. Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available and do not infer otherwise merely because no shell command named "external orchestrator" exists. In capability terms: GitHub Copilot CLI with `task` + `general-purpose` available means `ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` is true. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. -For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. Its returned decision is machine-readable: `external_handoff` means authorization is already complete (`confirmation_required = false`, `dispatch_immediately = true`, `max_new_external_orchestrators = 1`). Do not reinterpret that state as "ask the user again." If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: - `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. -- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not ask the user for confirmation, offer a manual-handoff menu, paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. - `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. +After launch, retain the one native Orchestrator handle. A bounded native wait that returns no completed agent or otherwise no terminal result means still running, not failure; wait again on that same handle. Only a true terminal `completed` or `failed` Orchestrator status ends the outer handoff lifecycle. Distinguish that from a launch failure before a handle exists. A parent wait timeout never fabricates a package-local Phase 1 failure. + The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. ### Handing the package over diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c53b690..b2f271c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -100,7 +100,7 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill < Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. For `github-copilot` without an explicit model, discovery is only an availability check for the repository-defined default; never rank or replace it with a subjective stronger/newer choice. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. `external_handoff` means authorization is complete: do not ask again. The helper encodes that as `confirmation_required = false`, `dispatch_immediately = true`, and `max_new_external_orchestrators = 1`. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. After launch, keep that same handle across bounded native waits; a wait that returns no completed agent or otherwise no terminal result means still running, so wait again on the same handle until a true terminal `completed` or `failed` result exists. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. diff --git a/README.md b/README.md index f2d1924..b08168c 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, normalized assertion domains, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the orchestrator invokes `invoke-phase2-analyzer.ps1`; it does not grade assertions itself, choose a grader model, retry failed analyzer workers, or write `grading.json`. The Phase 2 controller supplies each analyzer worker only that arm's expected output, unresolved semantic assertions, frozen output/transcript evidence, and packaged `tools/skill-creator/agents/grader.md`; paired arms, sibling evals, global skills, and candidate skill content are excluded unless a future assertion explicitly requires them. Missing telemetry is displayed as unavailable rather than estimated, and first-party reports separate execution usage/cost from analyzer usage/cost. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. -Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. `external_handoff` means authorization is complete: do not ask again. The helper encodes that as `confirmation_required = false`, `dispatch_immediately = true`, and `max_new_external_orchestrators = 1`. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. After launch, keep that same handle across bounded native waits; an empty wait or timeout means still running, so wait again on the same handle until a true terminal `completed` or `failed` result exists. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). The roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates package-local commands; the Phase 2 analyzer controller owns validator resolution, analyzer worker dispatch, and deterministic grading merge; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. From 51c5e84fb3eb2bc35aae954525dd570f0333eedd Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 04:18:46 +0200 Subject: [PATCH 32/36] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20formalize=20eval=20h?= =?UTF-8?q?andoff=20decision=20states=20and=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add New-EvalHandoffDecision function to construct machine-readable handoff decisions with explicit state fields: confirmation_required, dispatch_immediately, max_new_external_orchestrators, and same_handle_required. Add Assert-ExternalEvalHandoffDecision to validate decisions meet the one-shot handoff contract. Update helper descriptions to reflect explicit state machine semantics and New-ExternalEvalOrchestratorState / Update-ExternalEvalOrchestratorState lifecycle management. Include corresponding test updates for new decision validation. --- scripts/eval-request.ps1 | 235 ++++++++++++++++-- .../eval-runners/tests/test-eval-request.ps1 | 102 +++++++- 2 files changed, 314 insertions(+), 23 deletions(-) diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 index b0b4426..fc8025f 100644 --- a/scripts/eval-request.ps1 +++ b/scripts/eval-request.ps1 @@ -2,11 +2,16 @@ .SYNOPSIS Deterministic helpers for the agent's explicit eval request workflow. Dot-source to use. .DESCRIPTION - These helpers never launch a model. The interactive host consumes an external_handoff decision - with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire - this decision to CI, hooks, preparation, validation, or completion-gate execution. - For GitHub Copilot CLI, task + general-purpose delegation is a valid external-orchestrator - capability for this one-shot handoff. + These helpers never launch a model. The interactive host consumes Get-EvalHandoff's + machine-readable decision and, when action=external_handoff, must treat authorization as + complete: confirmation_required=false, dispatch_immediately=true, and exactly one fresh + external Eval Orchestrator may be created. After launch, New-ExternalEvalOrchestratorState and + Update-ExternalEvalOrchestratorState model the outer host lifecycle: keep the same native + handle, treat a bounded wait with no terminal result as still running, and wait again on that + same handle until a true terminal completed/failed status exists. Do not wire this decision to + CI, hooks, preparation, validation, or completion-gate execution. For GitHub Copilot CLI, + task + general-purpose delegation is a valid external-orchestrator capability for this one-shot + handoff. #> Set-StrictMode -Version Latest @@ -68,6 +73,198 @@ function Assert-PreparedEvalHandoffPackage { } } +function New-EvalHandoffDecision { + [CmdletBinding()] + param( + [Parameter(Mandatory)][ValidateSet('manual_handoff', 'external_handoff', 'already_started')][string]$Action, + [Parameter(Mandatory)][string]$PromptPath, + [Parameter(Mandatory)][string]$Reason, + [switch]$UserAuthorized, + [switch]$HostCanDelegateFreshOrchestrator, + [switch]$DispatchImmediately, + [int]$MaxNewExternalOrchestrators = 0, + [switch]$SameHandleRequired + ) + + $pendingWaitAction = if ($SameHandleRequired) { 'wait_same_handle_again' } else { 'none' } + $terminalStatuses = if ($SameHandleRequired) { @('completed', 'failed') } else { @() } + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-handoff-decision/1' + action = $Action + prompt_path = $PromptPath + reason = $Reason + user_authorized = $UserAuthorized.IsPresent + host_can_delegate_fresh_orchestrator = $HostCanDelegateFreshOrchestrator.IsPresent + confirmation_required = $false + dispatch_immediately = $DispatchImmediately.IsPresent + max_new_external_orchestrators = $MaxNewExternalOrchestrators + same_handle_required = $SameHandleRequired.IsPresent + pending_wait_action = $pendingWaitAction + terminal_statuses = @($terminalStatuses) + } +} + +function Assert-ExternalEvalHandoffDecision { + [CmdletBinding()] + param( + [Parameter(Mandatory)][object]$Decision + ) + + $schema = [string](Get-JsonProperty -Object $Decision -Name 'schema' -Default '') + if ($schema -ne 'codebeltnet/agentic/eval-handoff-decision/1') { + throw "External Eval handoff requires schema 'codebeltnet/agentic/eval-handoff-decision/1'." + } + if ([string](Get-JsonProperty -Object $Decision -Name 'action' -Default '') -ne 'external_handoff') { + throw "External Eval handoff requires action 'external_handoff'." + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'user_authorized' -Default $false)) { + throw 'External Eval handoff requires explicit user authorization.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'host_can_delegate_fresh_orchestrator' -Default $false)) { + throw 'External Eval handoff requires fresh-context host delegation capability.' + } + if ([bool](Get-JsonProperty -Object $Decision -Name 'confirmation_required' -Default $true)) { + throw 'External Eval handoff must not require another user confirmation.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'dispatch_immediately' -Default $false)) { + throw 'External Eval handoff must dispatch immediately.' + } + if ([int](Get-JsonProperty -Object $Decision -Name 'max_new_external_orchestrators' -Default 0) -ne 1) { + throw 'External Eval handoff permits exactly one fresh external Eval Orchestrator.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'same_handle_required' -Default $false)) { + throw 'External Eval handoff must retain the same native Orchestrator handle across waits.' + } + if ([string](Get-JsonProperty -Object $Decision -Name 'pending_wait_action' -Default '') -ne 'wait_same_handle_again') { + throw "External Eval handoff must map a non-terminal bounded wait to 'wait_same_handle_again'." + } + $terminalStatuses = @((Get-JsonProperty -Object $Decision -Name 'terminal_statuses' -Default @())) + if ($terminalStatuses.Count -ne 2 -or $terminalStatuses[0] -ne 'completed' -or $terminalStatuses[1] -ne 'failed') { + throw "External Eval handoff must declare terminal statuses 'completed' and 'failed'." + } +} + +function New-ExternalEvalOrchestratorState { + [CmdletBinding(DefaultParameterSetName = 'Launched')] + param( + [Parameter(Mandatory)][object]$Decision, + [Parameter(Mandatory, ParameterSetName = 'Launched')][string]$NativeHandle, + [Parameter(Mandatory, ParameterSetName = 'LaunchFailed')][switch]$LaunchFailed, + [Parameter(ParameterSetName = 'LaunchFailed')][string]$FailureReason = 'The external Eval Orchestrator launch failed before a native handle existed.' + ) + + Assert-ExternalEvalHandoffDecision -Decision $Decision + $promptPath = [string](Get-JsonProperty -Object $Decision -Name 'prompt_path' -Default '') + if ($PSCmdlet.ParameterSetName -eq 'LaunchFailed') { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'launch_failed' + terminal = $true + native_handle = $null + wait_count = 0 + same_handle_required = $false + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = $FailureReason + } + } + + if ([string]::IsNullOrWhiteSpace($NativeHandle)) { + throw 'A launched external Eval Orchestrator requires a non-empty native handle.' + } + + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'running' + terminal = $false + native_handle = $NativeHandle + wait_count = 0 + same_handle_required = $true + pending_wait_action = 'wait_same_handle_again' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = 'The external Eval Orchestrator exists. Keep this same native handle and repeat bounded waits until a true terminal completed/failed result exists.' + } +} + +function Update-ExternalEvalOrchestratorState { + [CmdletBinding()] + param( + [Parameter(Mandatory)][object]$State, + [Parameter(Mandatory)][string]$NativeHandle, + [Parameter(Mandatory)][ValidateSet('pending', 'completed', 'failed')][string]$NativeWaitStatus, + [object]$TerminalResult = $null + ) + + $schema = [string](Get-JsonProperty -Object $State -Name 'schema' -Default '') + if ($schema -ne 'codebeltnet/agentic/eval-orchestrator-lifecycle/1') { + throw "External Eval Orchestrator state requires schema 'codebeltnet/agentic/eval-orchestrator-lifecycle/1'." + } + if ([bool](Get-JsonProperty -Object $State -Name 'terminal' -Default $false)) { + throw 'A terminal external Eval Orchestrator state cannot be waited again.' + } + $expectedHandle = [string](Get-JsonProperty -Object $State -Name 'native_handle' -Default '') + if ([string]::IsNullOrWhiteSpace($expectedHandle)) { + throw 'A running external Eval Orchestrator state must retain its native handle.' + } + if ($NativeHandle -cne $expectedHandle) { + throw "External Eval Orchestrator waits must retain the same native handle. Expected '$expectedHandle', received '$NativeHandle'." + } + + $waitCount = [int](Get-JsonProperty -Object $State -Name 'wait_count' -Default 0) + 1 + $promptPath = [string](Get-JsonProperty -Object $State -Name 'prompt_path' -Default '') + switch ($NativeWaitStatus) { + 'pending' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'running' + terminal = $false + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'wait_same_handle_again' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = 'The bounded native wait returned no terminal Orchestrator result yet. Still running: wait again on this same handle and do not infer Phase 1 failure or create another Orchestrator.' + } + } + 'completed' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'completed' + terminal = $true + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $TerminalResult + reason = 'The same external Eval Orchestrator reached a true terminal completed state.' + } + } + 'failed' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'failed' + terminal = $true + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $TerminalResult + reason = 'The same external Eval Orchestrator reached a true terminal failed state.' + } + } + } +} + function Get-EvalHandoff { [CmdletBinding()] param( @@ -79,7 +276,9 @@ function Get-EvalHandoff { $validated = Assert-PreparedEvalHandoffPackage -PromptPath $PromptPath $path = [string]$validated.PromptPath $package = [string]$validated.Package - $decision = [ordered]@{ action = 'manual_handoff'; prompt_path = $path; reason = 'Preparation complete; hand this file to an external Eval Orchestrator.' } + $decision = New-EvalHandoffDecision -Action 'manual_handoff' -PromptPath $path ` + -Reason 'Preparation complete; hand this file to an external Eval Orchestrator.' ` + -UserAuthorized:$Yolo.IsPresent -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent if (-not $Yolo) { return [pscustomobject]$decision } # Reserve the handoff before the host delegates. An uncertain launch must never be retried. @@ -87,26 +286,28 @@ function Get-EvalHandoff { if ((Test-Path -LiteralPath $claim) -or (Test-Path -LiteralPath (Join-Path $package 'orchestration-state.json')) -or (Test-Path -LiteralPath (Join-Path $package 'execution-freeze.json'))) { - $decision.action = 'already_started' - $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' - return [pscustomobject]$decision + return (New-EvalHandoffDecision -Action 'already_started' -PromptPath $path ` + -Reason 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent -SameHandleRequired) } if (-not $CanDelegateFreshOrchestrator) { - $decision.reason = 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' - return [pscustomobject]$decision + return (New-EvalHandoffDecision -Action 'manual_handoff' -PromptPath $path ` + -Reason 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$false) } try { $stream = [IO.File]::Open($claim, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None) $stream.Dispose() } catch [IO.IOException] { if (-not (Test-Path -LiteralPath $claim)) { throw } - $decision.action = 'already_started' - $decision.reason = 'A handoff was already reserved. Do not dispatch again.' - return [pscustomobject]$decision + return (New-EvalHandoffDecision -Action 'already_started' -PromptPath $path ` + -Reason 'A handoff was already reserved. Do not dispatch again.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent -SameHandleRequired) } - $decision.action = 'external_handoff' - $decision.reason = 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; wait for its final result.' - return [pscustomobject]$decision + return (New-EvalHandoffDecision -Action 'external_handoff' -PromptPath $path ` + -Reason 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; do not ask again, dispatch immediately, and keep the same native handle across bounded waits until its final result.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$true -DispatchImmediately ` + -MaxNewExternalOrchestrators 1 -SameHandleRequired) } function Invoke-EvalRequest { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 9d48f32..fb42de0 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -8,6 +8,8 @@ $scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path $workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) [void](New-Item -ItemType Directory -Path $workspace -Force) $script:dispatches = [Collections.Generic.List[string]]::new() +$script:launches = [Collections.Generic.List[string]]::new() +$script:waits = [Collections.Generic.List[string]]::new() $utf8NoBom = [System.Text.UTF8Encoding]::new($false) function Assert-True([bool]$Condition, [string]$Message) { @@ -28,6 +30,27 @@ function Assert-PreparedPromptBinding([string]$PromptPath, [string]$Description) Assert-True ($declaredHash -match '^[0-9a-f]{64}$') "$Description manifest.runner_prompt_sha256 must be a lowercase SHA-256." Assert-True ($declaredHash -ceq (Get-Sha256HexFromFile -Path $PromptPath)) "$Description manifest.runner_prompt_sha256 must match RUN-THIS.prompt.md byte-for-byte." } +function Assert-ExternalHandoffDecision([object]$Decision, [string]$Description) { + Assert-True ([string]$Decision.schema -ceq 'codebeltnet/agentic/eval-handoff-decision/1') "$Description schema changed unexpectedly." + Assert-True ([string]$Decision.action -ceq 'external_handoff') "$Description did not return external_handoff." + Assert-True ([bool]$Decision.user_authorized) "$Description lost explicit user authorization." + Assert-True ([bool]$Decision.host_can_delegate_fresh_orchestrator) "$Description lost fresh-context delegation capability." + Assert-True (-not [bool]$Decision.confirmation_required) "$Description reintroduced a confirmation state after external_handoff." + Assert-True ([bool]$Decision.dispatch_immediately) "$Description did not require immediate dispatch." + Assert-True ([int]$Decision.max_new_external_orchestrators -eq 1) "$Description allowed more or fewer than one new external Orchestrator." + Assert-True ([bool]$Decision.same_handle_required) "$Description no longer requires the same Orchestrator handle across waits." + Assert-True ([string]$Decision.pending_wait_action -ceq 'wait_same_handle_again') "$Description changed pending wait semantics." + Assert-True ((@($Decision.terminal_statuses) -join ',') -ceq 'completed,failed') "$Description changed terminal wait statuses." +} +function Assert-AlreadyStartedDecision([object]$Decision, [string]$Description) { + Assert-True ([string]$Decision.schema -ceq 'codebeltnet/agentic/eval-handoff-decision/1') "$Description schema changed unexpectedly." + Assert-True ([string]$Decision.action -ceq 'already_started') "$Description did not stay already_started." + Assert-True (-not [bool]$Decision.confirmation_required) "$Description requested confirmation while resuming an existing Orchestrator." + Assert-True (-not [bool]$Decision.dispatch_immediately) "$Description attempted a replacement Orchestrator dispatch." + Assert-True ([int]$Decision.max_new_external_orchestrators -eq 0) "$Description permitted a second Orchestrator." + Assert-True ([bool]$Decision.same_handle_required) "$Description lost the same-handle requirement." + Assert-True ([string]$Decision.pending_wait_action -ceq 'wait_same_handle_again') "$Description changed resume wait semantics." +} function Invoke-FakeHost($Decision) { if ($Decision.action -eq 'external_handoff') { # The host's only input is the actual canonical handoff file, never an arm prompt. @@ -37,6 +60,17 @@ function Invoke-FakeHost($Decision) { $script:dispatches.Add($Decision.prompt_path) } } +function Invoke-FakeExternalOrchestratorDispatch([object]$Decision, [string]$NativeHandle) { + Assert-ExternalHandoffDecision -Decision $Decision -Description 'Fake host dispatch' + Invoke-FakeHost $Decision + $script:launches.Add($NativeHandle) + return New-ExternalEvalOrchestratorState -Decision $Decision -NativeHandle $NativeHandle +} +function Invoke-FakeExternalOrchestratorWait([object]$State, [string]$NativeWaitStatus, [object]$TerminalResult = $null) { + $handle = [string]$State.native_handle + $script:waits.Add($handle) + return Update-ExternalEvalOrchestratorState -State $State -NativeHandle $handle -NativeWaitStatus $NativeWaitStatus -TerminalResult $TerminalResult +} function New-Preparation([string]$Runner, [string]$Name) { return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog; AnalyzerModelCatalogPath = $catalog } @@ -136,17 +170,70 @@ try { if ($case.Expected -eq 'codex') { Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' } - Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' + Assert-ExternalHandoffDecision -Decision $decision -Description $case.Name $before = $script:dispatches.Count - Invoke-FakeHost $decision + $state = Invoke-FakeExternalOrchestratorDispatch -Decision $decision -NativeHandle ('dispatch-' + $case.Name) Assert-True ($script:dispatches[$before] -ceq $decision.prompt_path) 'Host did not receive the exact generated RUN-THIS.prompt.md path.' + Assert-True ($state.status -eq 'running' -and -not $state.terminal -and $state.native_handle -ceq ('dispatch-' + $case.Name)) "$($case.Name) did not dispatch immediately into a running Orchestrator state." $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-AlreadyStartedDecision -Decision $again -Description ($case.Name + ' duplicate handoff') Invoke-FakeHost $again - Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' + Assert-True ($script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo - Assert-True ($unavailableAfterStart.action -eq 'already_started') 'An uncertain launch incorrectly suggested a new manual execution.' + Assert-AlreadyStartedDecision -Decision $unavailableAfterStart -Description ($case.Name + ' resume after dispatch') } + $copilotLaunchFailed = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'copilot-launch-failed') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $copilotLaunchFailed -Description 'Copilot launch-failed contract' + $failedLaunchState = New-ExternalEvalOrchestratorState -Decision $copilotLaunchFailed -LaunchFailed -FailureReason 'launch failed before a native handle existed' + Assert-True ($failedLaunchState.status -eq 'launch_failed' -and $failedLaunchState.terminal) 'Launch-failed Orchestrator state was not terminal.' + Assert-True ($null -eq $failedLaunchState.native_handle -and $failedLaunchState.wait_count -eq 0) 'Launch-failed Orchestrator state incorrectly retained a handle or wait history.' + Assert-True ($failedLaunchState.max_new_external_orchestrators -eq 0) 'Launch failure permitted a replacement Orchestrator.' + + $codexSuccess = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'codex-lifecycle-success') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $codexSuccess -Description 'Codex lifecycle success contract' + $launchBefore = $script:launches.Count + $waitBefore = $script:waits.Count + $codexState = Invoke-FakeExternalOrchestratorDispatch -Decision $codexSuccess -NativeHandle 'O1' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 0) 'Codex lifecycle did not start in a running state.' + $mismatchFailed = $false + try { + Update-ExternalEvalOrchestratorState -State $codexState -NativeHandle 'O2' -NativeWaitStatus 'pending' | Out-Null + } catch { + $mismatchFailed = $true + Assert-True ($_.Exception.Message -match 'same native handle') "Unexpected handle-mismatch failure: $($_.Exception.Message)" + } + Assert-True $mismatchFailed 'Codex lifecycle accepted a replacement Orchestrator handle.' + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'pending' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 1) 'First pending wait incorrectly terminated Codex lifecycle.' + Assert-True ($codexState.pending_wait_action -eq 'wait_same_handle_again') 'First pending wait did not require another bounded wait on O1.' + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'pending' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 2) 'Second pending wait incorrectly terminated Codex lifecycle.' + $terminalResult = [pscustomobject]@{ native_handle = 'O1'; terminal_status = 'completed'; report_path = 'C:\fake\report.html' } + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'completed' -TerminalResult $terminalResult + $successLaunches = @($script:launches | Select-Object -Skip $launchBefore) + $successWaits = @($script:waits | Select-Object -Skip $waitBefore) + Assert-True ($successLaunches.Count -eq 1 -and $successLaunches[0] -ceq 'O1') 'Codex success lifecycle created more than one Orchestrator.' + Assert-True (($successWaits -join ',') -ceq 'O1,O1,O1') 'Codex success lifecycle stopped waiting on O1.' + Assert-True ($codexState.status -eq 'completed' -and $codexState.terminal -and $codexState.wait_count -eq 3) 'Codex success lifecycle did not end at terminal completion.' + Assert-True ($codexState.terminal_result.native_handle -ceq 'O1' -and $codexState.terminal_result.terminal_status -ceq 'completed') 'Codex success terminal result did not belong to O1.' + + $codexFailure = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'codex-lifecycle-failure') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $codexFailure -Description 'Codex lifecycle failure contract' + $launchBefore = $script:launches.Count + $waitBefore = $script:waits.Count + $codexFailedState = Invoke-FakeExternalOrchestratorDispatch -Decision $codexFailure -NativeHandle 'O1' + $codexFailedState = Invoke-FakeExternalOrchestratorWait -State $codexFailedState -NativeWaitStatus 'pending' + Assert-True ($codexFailedState.status -eq 'running' -and -not $codexFailedState.terminal -and $codexFailedState.wait_count -eq 1) 'Codex failure lifecycle ended during a pending wait.' + $failureResult = [pscustomobject]@{ native_handle = 'O1'; terminal_status = 'failed'; error = 'native orchestrator failed' } + $codexFailedState = Invoke-FakeExternalOrchestratorWait -State $codexFailedState -NativeWaitStatus 'failed' -TerminalResult $failureResult + $failureLaunches = @($script:launches | Select-Object -Skip $launchBefore) + $failureWaits = @($script:waits | Select-Object -Skip $waitBefore) + Assert-True ($failureLaunches.Count -eq 1 -and $failureLaunches[0] -ceq 'O1') 'Codex failure lifecycle created more than one Orchestrator.' + Assert-True (($failureWaits -join ',') -ceq 'O1,O1') 'Codex failure lifecycle stopped waiting on O1.' + Assert-True ($codexFailedState.status -eq 'failed' -and $codexFailedState.terminal -and $codexFailedState.wait_count -eq 2) 'Codex failure lifecycle did not return the real terminal failure.' + Assert-True ($codexFailedState.terminal_result.native_handle -ceq 'O1' -and $codexFailedState.terminal_result.terminal_status -ceq 'failed') 'Codex failure terminal result did not belong to O1.' + $reference = New-Preparation 'unused' 'reference' $reference.Remove('Runner') $reference.CodebeltReference = $true @@ -164,11 +251,13 @@ try { $before = $script:dispatches.Count Invoke-FakeHost $unavailable Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' + Assert-True ([bool]$unavailable.user_authorized -and -not [bool]$unavailable.host_can_delegate_fresh_orchestrator) 'Unavailable host decision lost authorization/capability semantics.' + Assert-True (-not [bool]$unavailable.confirmation_required -and -not [bool]$unavailable.dispatch_immediately) 'Unavailable host decision reintroduced a confirmation or dispatch state.' Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' # Existing execution state also blocks automatic handoff, even without a handoff receipt. '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -CanDelegateFreshOrchestrator - Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' + Assert-AlreadyStartedDecision -Decision $started -Description 'Existing Phase 1' Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' $invalid = New-Preparation 'GitHub Copilot' 'invalid-model' @@ -229,9 +318,10 @@ try { Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw Assert-True ($helper.Contains("[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator")) 'External orchestrator capability alias changed unexpectedly.' + Assert-True ($helper.Contains('dispatch_immediately') -and $helper.Contains('wait_same_handle_again') -and $helper.Contains('New-ExternalEvalOrchestratorState') -and $helper.Contains('Update-ExternalEvalOrchestratorState')) 'Eval request helper lost the explicit handoff lifecycle contract.' Assert-True (-not $helper.Contains('claude-haiku-4.5') -and -not $helper.Contains('gpt-5.6-luna') -and -not $helper.Contains('claude-opus-4.7')) 'Eval request helper must not embed model-selection policy.' Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' - Write-Host 'PASS: normal/yolo requests, Copilot delegation capability semantics, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' + Write-Host 'PASS: normal/yolo requests, explicit no-confirmation handoff semantics, same-handle wait lifecycle, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' } finally { # The absolute target is the unique child allocated under the external test workspace above. $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar From deeb3a96c68f5ec76940a3d713be58c78a82c6ee Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 04:18:51 +0200 Subject: [PATCH 33/36] =?UTF-8?q?=F0=9F=94=A7=20enhance=20skill=20validati?= =?UTF-8?q?on=20templates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add validation enhancements to skill template checking script to support updated eval request and decision workflow validation. --- scripts/validate-skill-templates.ps1 | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 0f5473e..976fe81 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1512,6 +1512,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '-CanDelegateFreshOrchestrator' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'authorization is already complete (`confirmation_required = false`, `dispatch_immediately = true`, `max_new_external_orchestrators = 1`)' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'A bounded native wait that returns no completed agent or otherwise no terminal result means still running, not failure; wait again on that same handle.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery mirrors every model exposed by all configured OpenCode providers' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'availability is presentation metadata only and never filters the selectable catalog' @@ -1544,17 +1546,26 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' Assert-Contains -Name 'README.md' -Content $readme -Needle 'In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability' + Assert-Contains -Name 'README.md' -Content $readme -Needle '`external_handoff` means authorization is complete: do not ask again.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'an empty wait or timeout means still running, so wait again on the same handle' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Discovery is an availability check, not model ranking' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'choose a Harness + Model when the user did not already do so' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'OpenCode mirrors every model exposed by all configured providers' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'ask the user to choose one, and wait' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle '`external_handoff` means authorization is complete: do not ask again.' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'a wait that returns no completed agent or otherwise no terminal result means still running' Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' Assert-Contains -Name 'README.md' -Content $readme -Needle 'OpenCode discovery mirrors every model exposed by all configured providers' Assert-Contains -Name 'README.md' -Content $readme -Needle 'the user must choose from the discovered selectors before package preparation' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'OpenCode through `opencode models --verbose`' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'without filtering the selectable catalog' Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle "[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator" + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'dispatch_immediately' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'max_new_external_orchestrators' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'wait_same_handle_again' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'New-ExternalEvalOrchestratorState' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'Update-ExternalEvalOrchestratorState' Assert-NotContains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'claude-opus-4.7' Assert-Contains -Name 'README.md' -Content $readme -Needle 'full operational permission inside each isolated behavioral harness configuration' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Hard filesystem confinement is a separate optional outer capability' From d1fb99f09745cd85c088adeb49f3f8417bb43c6e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 13:49:20 +0200 Subject: [PATCH 34/36] =?UTF-8?q?=F0=9F=8F=97=EF=B8=8F=20enhance=20eval-ru?= =?UTF-8?q?nner=20conformance=20and=20isolation=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add new boundary context and validation functions to isolation.ps1 for detecting and validating worker data boundaries. Implement Unix-style path resolution and path info extraction in runner-common.ps1 to support cross-platform isolation checks. Enhance phase2-analyzer with allowed artifact tracking and execution role metadata. Improve github-copilot runner isolation handling and opencode runner conformance support. --- .../eval-runners/github-copilot/isolation.ps1 | 196 +++++++++++++- .../eval-runners/github-copilot/runner.ps1 | 40 ++- .../eval-runners/invoke-phase2-analyzer.ps1 | 134 +++++++++- scripts/eval-runners/opencode/runner.ps1 | 244 +++++++++++++++++- scripts/eval-runners/runner-common.ps1 | 174 ++++++++++++- 5 files changed, 752 insertions(+), 36 deletions(-) diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 index 186a575..1236b3d 100644 --- a/scripts/eval-runners/github-copilot/isolation.ps1 +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -1,16 +1,192 @@ -function Find-CopilotBoundaryContradictions { - param([object]$Data, [object]$Projection) - # Structured tool arguments/results can contradict projection isolation; +function New-CopilotBoundaryContext { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [AllowEmptyString()][string]$PackageRoot = '', + [AllowEmptyString()][string]$SourceRepositoryRoot = '' + ) + + $executionRole = if ($RunData.PSObject.Properties.Name -contains 'ExecutionRole' -and -not [string]::IsNullOrWhiteSpace([string]$RunData.ExecutionRole)) { + [string]$RunData.ExecutionRole + } else { + 'eval_arm' + } + return [pscustomobject]@{ + PackageRoot = $PackageRoot + SourceRepositoryRoot = $SourceRepositoryRoot + RunRoot = [string]$RunData.RunRoot + WorkingDirectoryRoot = [string]$RunData.WorkingDirectoryPath + ExecutionRole = $executionRole + } +} + +function Add-CopilotBoundaryValueCandidate { + param( + [AllowNull()][object]$Value, + [Parameter(Mandatory = $true)][string]$PropertyPath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[object]]$Candidates + ) + + if ($null -eq $Value) { return } + if ($Value -is [string]) { + $leaf = [regex]::Replace($PropertyPath, '.*[.\[]', '').TrimEnd(']') + if ([string]::IsNullOrWhiteSpace($leaf)) { return } + $kind = if ($leaf -match '^(?i)(command|cmd)$') { + 'command' + } elseif ($leaf -match '^(?i)(path|paths|file|files|filepath|filepaths|directory|cwd|root|target|targets)$') { + 'path' + } else { + $null + } + if ($null -ne $kind) { + $Candidates.Add([pscustomobject]@{ + kind = $kind + source = $PropertyPath + value = [string]$Value + }) + } + return + } + if ($Value -is [System.Collections.IDictionary]) { + foreach ($key in @($Value.Keys)) { + Add-CopilotBoundaryValueCandidate -Value $Value[$key] -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$key)) -Candidates $Candidates + } + return + } + if ($Value -is [System.Collections.IEnumerable] -and $Value -isnot [string]) { + $index = 0 + foreach ($item in $Value) { + Add-CopilotBoundaryValueCandidate -Value $item -PropertyPath ([string]::Concat($PropertyPath, '[', $index, ']')) -Candidates $Candidates + $index++ + } + return + } + foreach ($property in @($Value.PSObject.Properties)) { + Add-CopilotBoundaryValueCandidate -Value $property.Value -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$property.Name)) -Candidates $Candidates + } +} + +function Get-CopilotBoundaryValueCandidates { + param([AllowNull()][object]$Data) + + $candidates = [System.Collections.Generic.List[object]]::new() + Add-CopilotBoundaryValueCandidate -Value $Data -PropertyPath 'data' -Candidates $candidates + return @($candidates.ToArray()) +} + +function Test-CopilotBoundaryPairedArmPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Boundary + ) + + $runRoot = [string](Get-JsonProperty -Object $Boundary -Name 'RunRoot' -Default '') + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + if (-not [string]::IsNullOrWhiteSpace($runRoot) -and (Test-ObservedPathInside -BasePath $runRoot -CandidatePath $ResolvedPath)) { return $false } + return ($ResolvedPath -replace '\\', '/') -match '(?i)(?:^|/)(arm-\d+-(?:with_skill|without_skill)|with_skill|without_skill)(?:/|$)' +} + +function Test-CopilotBoundaryForbiddenGradingPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Boundary + ) + + $normalizedPath = $ResolvedPath -replace '\\', '/' + if ($normalizedPath -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)$') { + return $true + } + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + $relative = if (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath) { + [System.IO.Path]::GetRelativePath($packageRoot, $ResolvedPath).Replace('\', '/') + } else { + '' + } + return $relative -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)(?:$|/)' -or + $relative -match '^(?i)(results|tools|progress)(?:/|$)' +} + +function Get-CopilotBoundaryAssessment { + param([object]$Data, [object]$Boundary) + + # Structured tool arguments/commands can contradict projection isolation; # their absence never proves OS confinement. - $text = ($Data | ConvertTo-Json -Depth 100 -Compress).Replace('\\', '/').Replace('\', '/') - foreach ($root in @($Projection.PackageRoot, $Projection.SourceRepositoryRoot) | Where-Object { $_ }) { - if ($text.IndexOf(([string]$root).Replace('\', '/'), [StringComparison]::OrdinalIgnoreCase) -ge 0) { 'Tool event references forbidden package/source path.' } + $executionRole = [string](Get-JsonProperty -Object $Boundary -Name 'ExecutionRole' -Default 'eval_arm') + if ([string]::IsNullOrWhiteSpace($executionRole)) { $executionRole = 'eval_arm' } + $workingDirectoryRoot = [string](Get-JsonProperty -Object $Boundary -Name 'WorkingDirectoryRoot' -Default (Get-JsonProperty -Object $Boundary -Name 'Root' -Default '')) + $runRoot = [string](Get-JsonProperty -Object $Boundary -Name 'RunRoot' -Default $workingDirectoryRoot) + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + $sourceRepositoryRoot = [string](Get-JsonProperty -Object $Boundary -Name 'SourceRepositoryRoot' -Default '') + $contradictions = [System.Collections.Generic.List[string]]::new() + $ownArmGradingVisible = $false + $pairedArmVisible = $false + $pairedOrPackageGradingVisible = $false + + foreach ($candidate in @(Get-CopilotBoundaryValueCandidates -Data $Data)) { + $pathValues = if ([string]$candidate.kind -eq 'command') { + if (Test-FileSystemCommandText -Text ([string]$candidate.value)) { + @(Get-ObservedPathTokensFromCommandText -Text ([string]$candidate.value)) + } else { + @() + } + } else { + @([string]$candidate.value) + } + foreach ($pathValue in $pathValues) { + $observed = Get-ObservedPathInfo -Path $pathValue -BasePath $workingDirectoryRoot + if ($null -eq $observed) { continue } + $resolvedPath = [string]$observed.FullPath + $insideWorking = -not [string]::IsNullOrWhiteSpace($workingDirectoryRoot) -and (Test-ObservedPathInside -BasePath $workingDirectoryRoot -CandidatePath $resolvedPath) + $insideRun = -not [string]::IsNullOrWhiteSpace($runRoot) -and (Test-ObservedPathInside -BasePath $runRoot -CandidatePath $resolvedPath) + $insidePackage = -not [string]::IsNullOrWhiteSpace($packageRoot) -and (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $resolvedPath) + $insideSource = -not [string]::IsNullOrWhiteSpace($sourceRepositoryRoot) -and (Test-ObservedPathInside -BasePath $sourceRepositoryRoot -CandidatePath $resolvedPath) + if ($executionRole -eq 'phase2_analyzer') { + if ($insideWorking) { + $relative = [System.IO.Path]::GetRelativePath($workingDirectoryRoot, $resolvedPath).Replace('\', '/') + if ($relative -in @('input-bundle.json', 'grader.md')) { $ownArmGradingVisible = $true } + continue + } + if ($insideRun) { + $contradictions.Add("Tool event references analyzer transport path '$pathValue' outside the staged repo bundle.") + } elseif ($insidePackage) { + $contradictions.Add("Tool event references package path '$pathValue' outside the staged analyzer bundle.") + } elseif ($insideSource) { + $contradictions.Add("Tool event references source repository path '$pathValue' outside the staged analyzer bundle.") + } else { + $contradictions.Add("Tool event references path '$pathValue' outside the staged analyzer bundle.") + } + } else { + if ($insideRun) { + continue + } + if ($insidePackage) { + $contradictions.Add("Tool event references forbidden package path '$pathValue'.") + } elseif ($insideSource) { + $contradictions.Add("Tool event references forbidden source repository path '$pathValue'.") + } else { + $contradictions.Add("Tool event references path '$pathValue' outside the physical projection.") + } + } + if (Test-CopilotBoundaryPairedArmPath -ResolvedPath $resolvedPath -Boundary ([pscustomobject]@{ PackageRoot = $packageRoot; RunRoot = $runRoot })) { $pairedArmVisible = $true } + if (Test-CopilotBoundaryForbiddenGradingPath -ResolvedPath $resolvedPath -Boundary ([pscustomobject]@{ PackageRoot = $packageRoot })) { $pairedOrPackageGradingVisible = $true } + } } - if ($text -match '(?i)(eval-metadata\.json|(?:^|[/"\s])(?:with_skill|without_skill)(?:[/"\s]|$)|(?:\.\./)+(?:results|tools|progress)(?:/|"|\s)|(?:execution-freeze|orchestration-state|grading|benchmark)\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)') { - 'Tool event references forbidden grading, paired-arm, or orchestration material.' + + return [pscustomobject]@{ + Contradictions = @($contradictions | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -Unique) + OwnArmGradingMaterialVisible = $ownArmGradingVisible + PairedArmVisible = $pairedArmVisible + PairedOrPackageGradingMaterialVisible = $pairedOrPackageGradingVisible } } +function Find-CopilotBoundaryContradictions { + param([object]$Data, [object]$Projection) + + return @((Get-CopilotBoundaryAssessment -Data $Data -Boundary $Projection).Contradictions) +} + # Conservative detector for a native Copilot skill activation of the evaluated candidate. The native skill tool is # excluded and skill discovery roots are isolated, so a candidate skill tool call or a native skill-resolution result # naming the candidate is an isolation violation in BOTH arms. It fires only on a native 'skill' tool identity or a @@ -66,7 +242,7 @@ function Assert-CopilotCapturedBoundary { $physical = [string](Get-JsonProperty -Object $paths -Name physical_run_root -Default '') if ([string]::IsNullOrWhiteSpace($physical) -or (Test-PathInside -BasePath $package -CandidatePath $physical) -or ($source -and (Test-PathInside -BasePath $source -CandidatePath $physical))) { throw 'Invalid Copilot physical projection boundary.' } Assert-CopilotCandidateInstructionBoundary -RunData $RunData - $proof = [pscustomobject]@{ PackageRoot = $package; SourceRepositoryRoot = $source } + $proof = New-CopilotBoundaryContext -RunData $RunData -PackageRoot $package -SourceRepositoryRoot $source $candidateSkillName = [string]$RunData.CandidateSkillName $transcript = @($Raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -eq 'evidence/copilot-events.jsonl' }) if ($transcript.Count -ne 1) { throw 'Copilot native transcript is missing.' } @@ -77,7 +253,7 @@ function Assert-CopilotCapturedBoundary { $eventType = [string](Get-JsonProperty -Object $event -Name type -Default '') $data = Get-JsonProperty -Object $event -Name data -Default $null if ($eventType -match '^(tool\.|command\.)') { - if (@(Find-CopilotBoundaryContradictions -Data $data -Projection $proof).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + if (@((Get-CopilotBoundaryAssessment -Data $data -Boundary $proof).Contradictions).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } if (@(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $candidateSkillName -EventType $eventType).Count) { throw 'Copilot transcript shows native candidate skill activation; grading is forbidden.' } } } diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index 2a0f5dd..db27b88 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -43,7 +43,11 @@ $copilotCandidateInstructionBoundary = "`n`n# Working environment" $script:copilotHomeCleanupSafe = $true $script:copilotLogicalInputs = $null $script:copilotProjection = $null +$script:copilotBoundaryContext = $null $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() +$script:copilotOwnArmGradingMaterialVisible = $false +$script:copilotPairedArmVisible = $false +$script:copilotPairedOrPackageGradingMaterialVisible = $false # The evaluated candidate skill name (both arms) and any observed native-skill activation of it. A non-empty violation # list at the end of execute is a fail-closed isolation breach: the candidate must never reach the worker natively. $script:copilotCandidateSkillName = $null @@ -965,9 +969,12 @@ function Read-CopilotEvents { } if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } $data = Get-JsonProperty -Object $event -Name 'data' -Default $null - if ($eventType -match '^(tool\.|command\.)' -and $null -ne $script:copilotProjection) { - $violations = @(Find-CopilotBoundaryContradictions -Data $data -Projection $script:copilotProjection) - foreach ($violation in $violations) { $script:copilotBoundaryViolations.Add($violation) } + if ($eventType -match '^(tool\.|command\.)' -and $null -ne $script:copilotBoundaryContext) { + $assessment = Get-CopilotBoundaryAssessment -Data $data -Boundary $script:copilotBoundaryContext + foreach ($violation in @($assessment.Contradictions)) { $script:copilotBoundaryViolations.Add($violation) } + $script:copilotOwnArmGradingMaterialVisible = $script:copilotOwnArmGradingMaterialVisible -or [bool]$assessment.OwnArmGradingMaterialVisible + $script:copilotPairedArmVisible = $script:copilotPairedArmVisible -or [bool]$assessment.PairedArmVisible + $script:copilotPairedOrPackageGradingMaterialVisible = $script:copilotPairedOrPackageGradingMaterialVisible -or [bool]$assessment.PairedOrPackageGradingMaterialVisible } if ($eventType -match '^(tool\.|command\.)' -and -not [string]::IsNullOrWhiteSpace($script:copilotCandidateSkillName)) { foreach ($activation in @(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $script:copilotCandidateSkillName -EventType $eventType)) { @@ -1474,8 +1481,11 @@ function Invoke-CopilotScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) - grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$script:copilotPairedArmVisible + grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + paired_or_package_grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + own_arm_grading_material_visible = [bool]$script:copilotOwnArmGradingMaterialVisible nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1512,7 +1522,11 @@ function Invoke-CopilotExecute { $logicalInputs = $Inputs $script:copilotLogicalInputs = $Inputs $script:copilotProjection = $plan + $script:copilotBoundaryContext = $null $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() + $script:copilotOwnArmGradingMaterialVisible = $false + $script:copilotPairedArmVisible = $false + $script:copilotPairedOrPackageGradingMaterialVisible = $false $script:copilotCandidateSkillName = [string]$Inputs.Run.CandidateSkillName $script:copilotNativeSkillViolations = [System.Collections.Generic.List[string]]::new() $script:copilotNativeSkillCatalog = $null @@ -1541,6 +1555,7 @@ function Invoke-CopilotExecute { } $physicalRun.PromptPath = Join-Path $plan.Root 'prompt.md' [IO.File]::WriteAllBytes($physicalRun.PromptPath, $Inputs.Run.PromptBytes) + $script:copilotBoundaryContext = New-CopilotBoundaryContext -RunData $physicalRun -PackageRoot $plan.PackageRoot -SourceRepositoryRoot $plan.SourceRepositoryRoot $projectedFiles = @(Get-ChildItem -LiteralPath $physicalRun.WorkingDirectoryPath -Recurse -Force -File | ForEach-Object { [IO.Path]::GetRelativePath($physicalRun.WorkingDirectoryPath, $_.FullName) }) $physicalInputs = [pscustomobject]@{ Run = $physicalRun; Profile = $Inputs.Profile } $result = Invoke-CopilotWithPreparedHome -Inputs $physicalInputs -Action { Invoke-CopilotProjectedExecute -Inputs $physicalInputs } @@ -1600,10 +1615,6 @@ function Invoke-CopilotExecute { $result.isolation.level = 'unsupported' $result.isolation.hard_filesystem_confinement = $false $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $isolationViolations)) - if ($result.evidence.Contains('delegation')) { - $result.evidence.delegation.paired_arm_visible = $true - $result.evidence.delegation.grading_material_visible = $true - } } $result.evidence.boundary.contradictions = @($script:copilotBoundaryViolations | Select-Object -Unique) if ($result.evidence.Contains('native_skill')) { @@ -1627,6 +1638,10 @@ function Invoke-CopilotExecute { Remove-Item -LiteralPath $plan.Root -Recurse -Force $script:copilotLogicalInputs = $null $script:copilotProjection = $null + $script:copilotBoundaryContext = $null + $script:copilotOwnArmGradingMaterialVisible = $false + $script:copilotPairedArmVisible = $false + $script:copilotPairedOrPackageGradingMaterialVisible = $false } } @@ -1839,8 +1854,11 @@ function Invoke-CopilotProjectedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) - grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$script:copilotPairedArmVisible + grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + paired_or_package_grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + own_arm_grading_material_visible = [bool]$script:copilotOwnArmGradingMaterialVisible nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/invoke-phase2-analyzer.ps1 b/scripts/eval-runners/invoke-phase2-analyzer.ps1 index 30c22c9..c27c4ce 100644 --- a/scripts/eval-runners/invoke-phase2-analyzer.ps1 +++ b/scripts/eval-runners/invoke-phase2-analyzer.ps1 @@ -221,6 +221,12 @@ function New-AnalyzerRunBundle { } } + $allowedArtifacts = [System.Collections.Generic.List[string]]::new() + $allowedArtifacts.Add([string]$Worker.record.ResultRelative) + foreach ($artifact in @((Get-JsonProperty -Object $canonical -Name 'output_files' -Default @()) | ForEach-Object { [string]$_ })) { + if (-not [string]::IsNullOrWhiteSpace($artifact)) { $allowedArtifacts.Add($artifact) } + } + $bundle = [ordered]@{ schema = 'codebeltnet/agentic/eval-analyzer-input/1' worker_id = $workerId @@ -235,7 +241,7 @@ function New-AnalyzerRunBundle { artifact = [string]$Worker.record.ResultRelative lines = @($lineRecords.ToArray()) } - allowed_artifacts = @(([string]$Worker.record.ResultRelative) + @((Get-JsonProperty -Object $canonical -Name 'output_files' -Default @()) | ForEach-Object { [string]$_ })) + allowed_artifacts = @($allowedArtifacts.ToArray()) assertions = @($Worker.assertions | ForEach-Object { [ordered]@{ assertion_index = [int]$_.assertion_index @@ -262,6 +268,7 @@ function New-AnalyzerRunBundle { skillName = $null iteration = 1 mode = 'without_skill' + executionRole = 'phase2_analyzer' promptFile = 'prompt.md' workingDirectory = 'repo' homeDirectory = 'home' @@ -320,15 +327,119 @@ function Invoke-AnalyzerPreflight { } } +function ConvertFrom-AnalyzerJsonDocument { + param( + [Parameter(Mandatory = $true)][string]$Json, + [Parameter(Mandatory = $true)][ValidateSet('raw_json', 'fenced_json', 'fenced_json_with_surrounding_text')][string]$TransportNormalization + ) + + $trimmed = $Json.Trim() + if (-not ($trimmed.StartsWith('{', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('}', [System.StringComparison]::Ordinal))) { + throw 'response contained no JSON object candidate.' + } + try { + return [pscustomobject]@{ + Fragment = $trimmed | ConvertFrom-Json -Depth 100 + TransportNormalization = $TransportNormalization + } + } catch { + throw $_.Exception + } +} + +function Get-AnalyzerFencedBlocks { + param([Parameter(Mandatory = $true)][string]$Text) + + $lines = [regex]::Split($Text, "\r?\n") + $blocks = [System.Collections.Generic.List[object]]::new() + $blockStart = $null + $blockInfo = '' + $bodyLines = $null + for ($index = 0; $index -lt $lines.Count; $index++) { + $line = [string]$lines[$index] + if ($null -eq $blockStart) { + $open = [regex]::Match($line, '^[ \t]*```(?[^\r\n`]*)[ \t]*$') + if ($open.Success) { + $blockStart = $index + $blockInfo = [string]$open.Groups['info'].Value + $bodyLines = [System.Collections.Generic.List[string]]::new() + } + continue + } + + if ($line -match '^[ \t]*```[ \t]*$') { + $leading = if ($blockStart -gt 0) { [string]::Join([Environment]::NewLine, @($lines[0..($blockStart - 1)])) } else { '' } + $trailing = if ($index + 1 -lt $lines.Count) { [string]::Join([Environment]::NewLine, @($lines[($index + 1)..($lines.Count - 1)])) } else { '' } + $blocks.Add([pscustomobject]@{ + Info = $blockInfo + Body = [string]::Join([Environment]::NewLine, @($bodyLines.ToArray())) + LeadingText = $leading + TrailingText = $trailing + }) + $blockStart = $null + $blockInfo = '' + $bodyLines = $null + continue + } + + $bodyLines.Add($line) + } + if ($null -ne $blockStart) { + throw 'Analyzer returned malformed JSON: the fenced block is not terminated.' + } + return @($blocks.ToArray()) +} + function ConvertFrom-AnalyzerResponse { param([Parameter(Mandatory = $true)][string]$Text) if ([string]::IsNullOrWhiteSpace($Text)) { throw 'Analyzer returned an empty response.' } - try { - return $Text | ConvertFrom-Json -Depth 100 - } catch { - throw "Analyzer returned malformed JSON: $($_.Exception.Message)" + $trimmed = $Text.Trim() + $candidates = [System.Collections.Generic.List[object]]::new() + $parseErrors = [System.Collections.Generic.List[string]]::new() + + if ($trimmed.StartsWith('{', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('}', [System.StringComparison]::Ordinal)) { + try { + $candidates.Add((ConvertFrom-AnalyzerJsonDocument -Json $trimmed -TransportNormalization 'raw_json')) + } catch { + $parseErrors.Add([string]$_.Exception.Message) + } } + + $fenceBlocks = @(Get-AnalyzerFencedBlocks -Text $Text) + if ($fenceBlocks.Count -gt 1) { + throw 'Analyzer returned multiple fenced blocks; the response must contain exactly one analyzer JSON document.' + } + if ($fenceBlocks.Count -eq 1) { + $block = $fenceBlocks[0] + $info = [string]$block.Info + if (-not [string]::IsNullOrWhiteSpace($info) -and $info.Trim() -cne 'json') { + throw 'Analyzer returned malformed JSON: the fenced block must be a JSON fence.' + } + $leading = [string]$block.LeadingText + $trailing = [string]$block.TrailingText + $normalization = if ([string]::IsNullOrWhiteSpace($leading) -and [string]::IsNullOrWhiteSpace($trailing)) { + 'fenced_json' + } else { + 'fenced_json_with_surrounding_text' + } + try { + $candidates.Add((ConvertFrom-AnalyzerJsonDocument -Json ([string]$block.Body) -TransportNormalization $normalization)) + } catch { + $parseErrors.Add([string]$_.Exception.Message) + } + } + + if ($candidates.Count -gt 1) { + throw 'Analyzer returned multiple JSON candidates; the response must contain exactly one analyzer JSON document.' + } + if ($candidates.Count -eq 1) { + return $candidates[0] + } + if ($parseErrors.Count -gt 0) { + throw "Analyzer returned malformed JSON: $($parseErrors[0])" + } + throw 'Analyzer returned malformed JSON: response contained no unambiguous JSON object.' } function Confirm-AnalyzerFragment { @@ -347,10 +458,12 @@ function Confirm-AnalyzerFragment { $expectedByIndex = @{} foreach ($assertion in @($Worker.assertions)) { $expectedByIndex[[int]$assertion.assertion_index] = $assertion } if ($grades.Count -ne @($Worker.assertions).Count) { throw 'Analyzer fragment grade cardinality does not match unresolved semantic assertions.' } + $seenIndexes = [System.Collections.Generic.HashSet[int]]::new() $entries = [System.Collections.Generic.List[object]]::new() foreach ($grade in $grades) { $index = [int](Get-JsonProperty -Object $grade -Name 'assertion_index' -Default -1) if (-not $expectedByIndex.ContainsKey($index)) { throw "Analyzer fragment includes unexpected assertion_index '$index'." } + if (-not $seenIndexes.Add($index)) { throw "Analyzer fragment duplicates assertion_index '$index'." } $expected = $expectedByIndex[$index] if ((Get-JsonProperty -Object $grade -Name 'passed' -Default $null) -isnot [bool]) { throw 'Analyzer fragment passed must be boolean.' } $reason = [string](Get-JsonProperty -Object $grade -Name 'reason' -Default '') @@ -360,6 +473,10 @@ function Confirm-AnalyzerFragment { [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical -TranscriptArtifacts $StagedTranscripts) $entries.Add($entry) } + if ($seenIndexes.Count -ne $expectedByIndex.Count) { + $missing = @($expectedByIndex.Keys | Where-Object { -not $seenIndexes.Contains([int]$_) } | Sort-Object) + throw "Analyzer fragment is missing assertion_index '$($missing[0])'." + } return @($entries.ToArray()) } @@ -383,6 +500,7 @@ function New-AnalyzerResult { [Parameter(Mandatory = $true)][string]$RawRelative, [Parameter(Mandatory = $true)][string]$FragmentRelative, [Parameter(Mandatory = $true)][object[]]$Grades, + [string]$ResponseTransportNormalization = '', [string]$Status = 'completed', [string]$Failure = '' ) @@ -421,6 +539,7 @@ function New-AnalyzerResult { raw_execution_result_sha256 = Get-Sha256HexFromFile -Path $rawPath grading_fragment = $FragmentRelative grading_fragment_sha256 = Get-Sha256HexFromFile -Path $fragmentPath + response_transport_normalization = $ResponseTransportNormalization parsed_grading = @($Grades) input_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'input_tokens' -Default $null output_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'output_tokens' -Default $null @@ -636,14 +755,15 @@ try { if ($executorSessions.Contains($sessionId)) { throw 'Analyzer session reused an executor session identity.' } if (@($completedAnalyzerEntries | Where-Object { [string]$_.session_id -eq $sessionId }).Count -gt 0) { throw 'Analyzer worker session was reused across arms.' } $responseText = [string](Get-JsonProperty -Object $raw.final_response -Name 'text' -Default '') - $fragment = ConvertFrom-AnalyzerResponse -Text $responseText + $response = ConvertFrom-AnalyzerResponse -Text $responseText + $fragment = $response.Fragment $canonical = Read-RunnerJson -Path $worker.record.ResultPath $workerGrades = @(Confirm-AnalyzerFragment -Fragment $fragment -Worker $worker -Canonical $canonical -StagedTranscripts $item.bundle.StagedTranscripts) $fragmentRelative = "phase2/fragments/$workerId.grading-fragment.json" $fragmentPath = Join-Path $iteration ($fragmentRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) Write-RunnerJsonFile -Path $fragmentPath -Value $fragment foreach ($grade in $workerGrades) { $grades.Add($grade) } - $analyzerResult = New-AnalyzerResult -Worker $worker -Bundle $item.bundle -Raw $raw -AnalyzerProfile $analyzerProfile -RawRelative ([string]$item.raw_relative) -FragmentRelative $fragmentRelative -Grades $workerGrades + $analyzerResult = New-AnalyzerResult -Worker $worker -Bundle $item.bundle -Raw $raw -AnalyzerProfile $analyzerProfile -RawRelative ([string]$item.raw_relative) -FragmentRelative $fragmentRelative -Grades $workerGrades -ResponseTransportNormalization ([string]$response.TransportNormalization) $resultRelative = "phase2/results/$workerId.analyzer-result.json" $resultPath = Join-Path $iteration ($resultRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) Write-RunnerJsonFile -Path $resultPath -Value $analyzerResult diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 1dce3a5..89a78b1 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -620,6 +620,207 @@ function Remove-OpenCodeAnsiSequences { return [regex]::Replace($Text, "`e\[[0-?]*[ -/]*[@-~]", '') } +function Add-OpenCodeBoundaryValueCandidate { + param( + [AllowNull()][object]$Value, + [Parameter(Mandatory = $true)][string]$PropertyPath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[object]]$Candidates + ) + + if ($null -eq $Value) { return } + if ($Value -is [string]) { + $leaf = [regex]::Replace($PropertyPath, '.*[.\[]', '').TrimEnd(']') + if ([string]::IsNullOrWhiteSpace($leaf)) { return } + $kind = if ($leaf -match '^(?i)(command|cmd)$') { + 'command' + } elseif ($leaf -match '^(?i)(path|paths|file|files|filepath|filepaths|directory|cwd|root|target|targets|location)$') { + 'path' + } else { + $null + } + if ($null -ne $kind) { + $Candidates.Add([pscustomobject]@{ + kind = $kind + source = $PropertyPath + value = [string]$Value + }) + } + return + } + if ($Value -is [System.Collections.IDictionary]) { + foreach ($key in @($Value.Keys)) { + Add-OpenCodeBoundaryValueCandidate -Value $Value[$key] -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$key)) -Candidates $Candidates + } + return + } + if ($Value -is [System.Collections.IEnumerable] -and $Value -isnot [string]) { + $index = 0 + foreach ($item in $Value) { + Add-OpenCodeBoundaryValueCandidate -Value $item -PropertyPath ([string]::Concat($PropertyPath, '[', $index, ']')) -Candidates $Candidates + $index++ + } + return + } + foreach ($property in @($Value.PSObject.Properties)) { + Add-OpenCodeBoundaryValueCandidate -Value $property.Value -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$property.Name)) -Candidates $Candidates + } +} + +function Get-OpenCodeBoundaryValueCandidates { + param([AllowNull()][object]$Value) + + $candidates = [System.Collections.Generic.List[object]]::new() + Add-OpenCodeBoundaryValueCandidate -Value $Value -PropertyPath 'event' -Candidates $candidates + return @($candidates.ToArray()) +} + +function Get-OpenCodeLogicalPackageRoot { + param([Parameter(Mandatory = $true)][object]$Projection) + + $logicalRun = Get-JsonProperty -Object $Projection -Name 'LogicalRun' -Default $null + $logicalRunRoot = [string](Get-JsonProperty -Object $logicalRun -Name 'RunRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($logicalRunRoot)) { return '' } + return [System.IO.Path]::GetFullPath((Split-Path -Parent (Split-Path -Parent $logicalRunRoot))) +} + +function Test-OpenCodeBoundaryPairedArmPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Projection + ) + + $logicalRun = Get-JsonProperty -Object $Projection -Name 'LogicalRun' -Default $null + $logicalRunRoot = [string](Get-JsonProperty -Object $logicalRun -Name 'RunRoot' -Default '') + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + if (-not [string]::IsNullOrWhiteSpace($logicalRunRoot) -and (Test-ObservedPathInside -BasePath $logicalRunRoot -CandidatePath $ResolvedPath)) { return $false } + return ($ResolvedPath -replace '\\', '/') -match '(?i)(?:^|/)(arm-\d+-(?:with_skill|without_skill)|with_skill|without_skill)(?:/|$)' +} + +function Test-OpenCodeBoundaryForbiddenGradingPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Projection + ) + + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + $relative = [System.IO.Path]::GetRelativePath($packageRoot, $ResolvedPath).Replace('\', '/') + return $relative -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)(?:$|/)' -or + $relative -match '^(?i)(results|tools|progress)(?:/|$)' +} + +function Get-OpenCodeBoundaryAssessment { + param( + [AllowEmptyCollection()][object[]]$Events = @(), + [Parameter(Mandatory = $true)][object]$Projection + ) + + $contradictions = [System.Collections.Generic.List[object]]::new() + $projectionRoot = [string](Get-JsonProperty -Object $Projection -Name 'Root' -Default '') + $physicalWorkingDirectory = [string](Get-JsonProperty -Object $Projection -Name 'PhysicalWorkingDirectory' -Default $projectionRoot) + $sourceRepositoryRoot = [string](Get-JsonProperty -Object $Projection -Name 'SourceRepositoryRoot' -Default '') + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + + foreach ($event in @($Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ($eventType -ne 'tool_use') { continue } + $tool = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $event -Name 'part' -Default $null) -Name 'tool' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '')) + foreach ($candidate in @(Get-OpenCodeBoundaryValueCandidates -Value $event)) { + $pathValues = if ([string]$candidate.kind -eq 'command') { + if (Test-FileSystemCommandText -Text ([string]$candidate.value)) { + @(Get-ObservedPathTokensFromCommandText -Text ([string]$candidate.value)) + } else { + @() + } + } else { + @([string]$candidate.value) + } + foreach ($pathValue in $pathValues) { + $observed = Get-ObservedPathInfo -Path $pathValue -BasePath $physicalWorkingDirectory + if ($null -eq $observed) { continue } + $resolvedPath = [string]$observed.FullPath + if (-not [string]::IsNullOrWhiteSpace($projectionRoot) -and (Test-ObservedPathInside -BasePath $projectionRoot -CandidatePath $resolvedPath)) { continue } + + $category = if (-not [string]::IsNullOrWhiteSpace($packageRoot) -and (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $resolvedPath)) { + if (Test-OpenCodeBoundaryForbiddenGradingPath -ResolvedPath $resolvedPath -Projection $Projection) { 'package_grading_material' } + elseif (Test-OpenCodeBoundaryPairedArmPath -ResolvedPath $resolvedPath -Projection $Projection) { 'paired_arm' } + else { 'package_root_outside_projection' } + } elseif (-not [string]::IsNullOrWhiteSpace($sourceRepositoryRoot) -and (Test-ObservedPathInside -BasePath $sourceRepositoryRoot -CandidatePath $resolvedPath)) { + 'source_repository_outside_projection' + } else { + 'outside_projection' + } + $message = switch ($category) { + 'package_grading_material' { "Structured OpenCode tool evidence referenced package grading/orchestration material outside the physical projection: '$pathValue'." } + 'paired_arm' { "Structured OpenCode tool evidence referenced a paired-arm/package path outside the physical projection: '$pathValue'." } + 'package_root_outside_projection' { "Structured OpenCode tool evidence referenced a package path outside the physical projection: '$pathValue'." } + 'source_repository_outside_projection' { "Structured OpenCode tool evidence referenced a source-repository path outside the physical projection: '$pathValue'." } + default { "Structured OpenCode tool evidence referenced a path outside the physical projection: '$pathValue'." } + } + $contradictions.Add([ordered]@{ + category = $category + tool = $tool + source = [string]$candidate.source + observed_path = [string]$pathValue + resolved_path = $resolvedPath + event_type = $eventType + message = $message + }) + } + } + } + + return [pscustomobject]@{ + Contradictions = @($contradictions.ToArray()) + ProjectionEscapeObserved = $contradictions.Count -gt 0 + PairedArmVisible = @($contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'paired_arm' }).Count -gt 0 + GradingMaterialVisible = @($contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'package_grading_material' }).Count -gt 0 + } +} + +function ConvertTo-OpenCodeBoundaryAssessment { + param([AllowEmptyCollection()][object[]]$Contradictions = @()) + + return [pscustomobject]@{ + Contradictions = @($Contradictions) + ProjectionEscapeObserved = @($Contradictions).Count -gt 0 + PairedArmVisible = @($Contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'paired_arm' }).Count -gt 0 + GradingMaterialVisible = @($Contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'package_grading_material' }).Count -gt 0 + } +} + +function Apply-OpenCodeBoundaryAssessment { + param( + [Parameter(Mandatory = $true)][string]$Status, + [AllowEmptyString()][string]$FailureCode = '', + [AllowEmptyString()][string]$FailureMessage = '', + [Parameter(Mandatory = $true)][object]$Assessment + ) + + $appliedStatus = $Status + $appliedFailureCode = $FailureCode + $appliedFailureMessage = $FailureMessage + if ([bool](Get-JsonProperty -Object $Assessment -Name 'ProjectionEscapeObserved' -Default $false) -and $Status -eq 'completed') { + $appliedStatus = 'incompatible' + $appliedFailureCode = 'opencode_projection_escape_observed' + $appliedFailureMessage = 'OpenCode structured tool evidence proves access outside the physical run projection.' + } + return [pscustomobject]@{ + Status = $appliedStatus + FailureCode = $appliedFailureCode + FailureMessage = $appliedFailureMessage + ProjectionEscapeObserved = [bool](Get-JsonProperty -Object $Assessment -Name 'ProjectionEscapeObserved' -Default $false) + PairedArmVisible = [bool](Get-JsonProperty -Object $Assessment -Name 'PairedArmVisible' -Default $false) + GradingMaterialVisible = [bool](Get-JsonProperty -Object $Assessment -Name 'GradingMaterialVisible' -Default $false) + BoundaryEvidence = [ordered]@{ + proof = 'physical_run_projection' + contradictions = @(Get-JsonProperty -Object $Assessment -Name 'Contradictions' -Default @()) + event_inspection = 'structured_tool_contradiction_detector' + } + } +} + function Get-OpenCodeContinuationCapability { param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$HelpText) @@ -801,6 +1002,7 @@ function Invoke-OpenCodeScriptedExecute { $artifacts = [System.Collections.Generic.List[object]]::new() $warnings = [System.Collections.Generic.List[string]]::new() $nativeFailures = [System.Collections.Generic.List[string]]::new() + $boundaryContradictions = [System.Collections.Generic.List[object]]::new() $eventCounts = @{} $observedModels = [System.Collections.Generic.List[string]]::new() $usageBuckets = [ordered]@{} @@ -865,6 +1067,10 @@ function Invoke-OpenCodeScriptedExecute { $artifacts.Add($turnStderrArtifact) $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { [pscustomobject]@{ Events = @(); Errors = @() } } else { ConvertFrom-JsonLines -Text $process.Stdout } foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode turn $turnNumber event parse error: $parseError") } + if (-not $hardFilesystem) { + $turnBoundary = Get-OpenCodeBoundaryAssessment -Events @($parsed.Events) -Projection $projection + foreach ($contradiction in @($turnBoundary.Contradictions)) { $boundaryContradictions.Add($contradiction) } + } $parsedEvents = Read-OpenCodeScriptedTurn -Parsed $parsed -Warnings $warnings $turnTiming = [ordered]@{ turn = $turnNumber @@ -964,6 +1170,14 @@ function Invoke-OpenCodeScriptedExecute { $failureMessage = 'OpenCode scripted interaction did not complete every ordered user/assistant turn.' } if ($status -eq 'completed' -and $nativeFailures.Count -gt 0) { $status = 'incompatible' } + $boundaryAssessment = ConvertTo-OpenCodeBoundaryAssessment -Contradictions @($boundaryContradictions.ToArray()) + $boundaryOutcome = Apply-OpenCodeBoundaryAssessment -Status $status -FailureCode ([string]$failureCode) -FailureMessage ([string]$failureMessage) -Assessment $boundaryAssessment + $status = [string]$boundaryOutcome.Status + $failureCode = [string]$boundaryOutcome.FailureCode + $failureMessage = [string]$boundaryOutcome.FailureMessage + if ([bool]$boundaryOutcome.ProjectionEscapeObserved -and $nativeFailures -notcontains 'projection_escape_observed') { + $nativeFailures.Add('projection_escape_observed') + } if ([string]::IsNullOrWhiteSpace($capturedSessionId)) { $capturedSessionId = [Guid]::NewGuid().ToString('D') } $finished = $lastProcess.FinishedUtc $durationSeconds = [Math]::Round(($finished - $firstProcess.StartedUtc).TotalSeconds, 3) @@ -1072,6 +1286,7 @@ function Invoke-OpenCodeScriptedExecute { complete_structured_transcript = [bool]$terminalCapture turn_artifacts = @($nativeTurns.ToArray() | ForEach-Object { "evidence/opencode-turn-$(Get-JsonProperty -Object $_ -Name 'turn' -Default 0)-events.jsonl" }) } + boundary = $boundaryOutcome.BoundaryEvidence delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = [string]$descriptor.delegation.mechanism @@ -1086,8 +1301,10 @@ function Invoke-OpenCodeScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$boundaryOutcome.PairedArmVisible + grading_material_visible = [bool]$boundaryOutcome.GradingMaterialVisible + projection_escape_observed = [bool]$boundaryOutcome.ProjectionEscapeObserved nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1449,6 +1666,7 @@ function New-OpenCodeExecutionProjection { EvalId = $Inputs.Run.EvalId EvalName = $Inputs.Run.EvalName Mode = $Inputs.Run.Mode + ExecutionRole = $Inputs.Run.ExecutionRole PromptPath = $physicalPrompt PromptBytes = $Inputs.Run.PromptBytes PromptHash = $Inputs.Run.PromptHash @@ -1456,8 +1674,11 @@ function New-OpenCodeExecutionProjection { HomeDirectoryPath = $physicalHome SkillDirectoryPath = $physicalSkill CandidateSkillExposed = $Inputs.Run.CandidateSkillExposed + CandidateSkillName = $Inputs.Run.CandidateSkillName FixtureHash = $Inputs.Run.FixtureHash SkillHash = $Inputs.Run.SkillHash + CandidateInstructionHash = $Inputs.Run.CandidateInstructionHash + GitWorkspace = $Inputs.Run.GitWorkspace InteractionPath = $null InteractionHash = $Inputs.Run.InteractionHash Interaction = $Inputs.Run.Interaction @@ -2474,6 +2695,18 @@ function Invoke-OpenCodeExecute { } elseif ([string]::IsNullOrWhiteSpace($finalText)) { $reason = 'opencode_did_not_return_final_response'; $warnings.Add('OpenCode exited successfully without a text response.') } + $boundaryAssessment = if ($hardFilesystem) { + ConvertTo-OpenCodeBoundaryAssessment + } else { + Get-OpenCodeBoundaryAssessment -Events @($parsed.Events) -Projection $projection + } + $boundaryOutcome = Apply-OpenCodeBoundaryAssessment -Status $status -FailureCode ([string]$reason) -FailureMessage ([string]$failureMessage) -Assessment $boundaryAssessment + $status = [string]$boundaryOutcome.Status + $reason = if ([string]::IsNullOrWhiteSpace([string]$boundaryOutcome.FailureCode)) { $reason } else { [string]$boundaryOutcome.FailureCode } + $failureMessage = if ([string]::IsNullOrWhiteSpace([string]$boundaryOutcome.FailureMessage)) { $failureMessage } else { [string]$boundaryOutcome.FailureMessage } + if ([string]$reason -eq 'opencode_projection_escape_observed') { + $failure = New-ExecutionFailure -Code 'opencode_projection_escape_observed' -Message $failureMessage + } $telemetry = [ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $true }) tokens = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } @@ -2551,6 +2784,7 @@ function Invoke-OpenCodeExecute { artifact = $transcriptArtifactPath sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } } + boundary = $boundaryOutcome.BoundaryEvidence delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = [string]$descriptor.delegation.mechanism @@ -2565,8 +2799,10 @@ function Invoke-OpenCodeExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$boundaryOutcome.PairedArmVisible + grading_material_visible = [bool]$boundaryOutcome.GradingMaterialVisible + projection_escape_observed = [bool]$boundaryOutcome.ProjectionEscapeObserved nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index b16ec8d..02358fa 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -335,6 +335,155 @@ function Test-PathInside { return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) } +function Resolve-ObservedUnixPath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$RelativePath + ) + + $base = ($BasePath -replace '\\', '/') + if (-not $base.StartsWith('/', [System.StringComparison]::Ordinal)) { + throw "Unix observed path base '$BasePath' must be absolute." + } + $segments = [System.Collections.Generic.List[string]]::new() + foreach ($segment in @($base -split '/')) { + if (-not [string]::IsNullOrWhiteSpace([string]$segment)) { $segments.Add([string]$segment) } + } + foreach ($segment in @(($RelativePath -replace '\\', '/') -split '/')) { + switch ([string]$segment) { + '' { continue } + '.' { continue } + '..' { + if ($segments.Count -gt 0) { $segments.RemoveAt($segments.Count - 1) } + continue + } + default { + $segments.Add([string]$segment) + } + } + } + return '/' + ([string]::Join('/', @($segments.ToArray()))) +} + +function Get-ObservedPathInfo { + param( + [AllowEmptyString()][string]$Path, + [AllowEmptyString()][string]$BasePath = '' + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return $null } + $trimmed = [string]$Path + while ($trimmed.Length -ge 2 -and ( + ($trimmed.StartsWith('"', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('"', [System.StringComparison]::Ordinal)) -or + ($trimmed.StartsWith("'", [System.StringComparison]::Ordinal) -and $trimmed.EndsWith("'", [System.StringComparison]::Ordinal)))) { + $trimmed = $trimmed.Substring(1, $trimmed.Length - 2) + } + $trimmed = $trimmed.Trim() + if ([string]::IsNullOrWhiteSpace($trimmed)) { return $null } + + if ($trimmed -match '^[A-Za-z]:[\\/]') { + return [pscustomobject]@{ + Style = 'windows' + FullPath = ConvertTo-ComparablePath -Path ($trimmed -replace '/', '\') + Raw = $trimmed + Absolute = $true + } + } + if ($trimmed -match '^\\\\') { + return [pscustomobject]@{ + Style = 'unc' + FullPath = ConvertTo-ComparablePath -Path ($trimmed -replace '/', '\') + Raw = $trimmed + Absolute = $true + } + } + if ($trimmed -match '^/') { + $normalized = ($trimmed -replace '\\', '/') + if ($normalized.Length -gt 1) { $normalized = $normalized.TrimEnd('/') } + return [pscustomobject]@{ + Style = 'unix' + FullPath = $normalized + Raw = $trimmed + Absolute = $true + } + } + if ([string]::IsNullOrWhiteSpace($BasePath)) { return $null } + $baseInfo = Get-ObservedPathInfo -Path $BasePath + if ($null -eq $baseInfo) { return $null } + switch ([string]$baseInfo.Style) { + 'unix' { + return [pscustomobject]@{ + Style = 'unix' + FullPath = Resolve-ObservedUnixPath -BasePath ([string]$baseInfo.FullPath) -RelativePath $trimmed + Raw = $trimmed + Absolute = $false + } + } + default { + try { + $combined = [System.IO.Path]::GetFullPath((Join-Path $BasePath ($trimmed -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + } catch { + return $null + } + return [pscustomobject]@{ + Style = [string]$baseInfo.Style + FullPath = ConvertTo-ComparablePath -Path $combined + Raw = $trimmed + Absolute = $false + } + } + } +} + +function Test-ObservedPathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = Get-ObservedPathInfo -Path $BasePath + $candidate = Get-ObservedPathInfo -Path $CandidatePath -BasePath $BasePath + if ($null -eq $base -or $null -eq $candidate -or [string]$base.Style -ne [string]$candidate.Style) { return $false } + $separator = if ([string]$base.Style -eq 'unix') { '/' } else { '\' } + $comparison = if ([string]$base.Style -eq 'unix') { [System.StringComparison]::Ordinal } else { [System.StringComparison]::OrdinalIgnoreCase } + return [string]$candidate.FullPath -eq [string]$base.FullPath -or ([string]$candidate.FullPath).StartsWith(([string]$base.FullPath + $separator), $comparison) +} + +function Get-ObservedPathTokensFromCommandText { + param([AllowEmptyString()][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { return @() } + $tokens = [System.Collections.Generic.List[string]]::new() + $pattern = '(?:"(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^"]+)"|''(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^'']+)''|(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^ \t\r\n"''`|;&,]+))' + foreach ($match in [regex]::Matches($Text, $pattern)) { + $candidate = @($match.Groups['double'].Value, $match.Groups['single'].Value, $match.Groups['bare'].Value) | + Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | + Select-Object -First 1 + if (-not [string]::IsNullOrWhiteSpace([string]$candidate) -and $tokens -notcontains [string]$candidate) { + $tokens.Add([string]$candidate) + } + } + return @($tokens.ToArray()) +} + +function Test-FileSystemCommandText { + param([AllowEmptyString()][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { return $false } + return $Text -match '(?i)\b(Get-ChildItem|Get-Content|Set-Content|Add-Content|Out-File|Copy-Item|Move-Item|Remove-Item|Select-String|type|cat|ls|dir|find|grep|rg|read|write|view|edit|glob)\b' +} + +function Get-RunExecutionRole { + param([Parameter(Mandatory = $true)][object]$Run) + + $role = [string](Get-JsonProperty -Object $Run -Name 'executionRole' -Default 'eval_arm') + if ([string]::IsNullOrWhiteSpace($role)) { $role = 'eval_arm' } + if ($role -notin @('eval_arm', 'phase2_analyzer')) { + throw "run.json executionRole '$role' is unsupported." + } + return $role +} + function Assert-SafeRelativePath { param( [Parameter(Mandatory = $true)][string]$RelativePath, @@ -822,6 +971,20 @@ function Test-NativeWorkerTerminalEvidence { $failures.Add('delegation_terminal_evidence') return [pscustomobject]@{ Valid = $false; Failures = @($failures); Delegation = $null } } + $executionRole = if ($Run.PSObject.Properties.Name -contains 'ExecutionRole' -and -not [string]::IsNullOrWhiteSpace([string]$Run.ExecutionRole)) { + [string]$Run.ExecutionRole + } else { + 'eval_arm' + } + $pairedArmVisible = [bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) + $forbiddenGradingVisible = if ($executionRole -eq 'phase2_analyzer') { + [bool](Get-JsonProperty -Object $delegation -Name 'paired_or_package_grading_material_visible' -Default (Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) + } else { + [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true) + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'projection_escape_observed' -Default $false)) { + $failures.Add('projection_escape_observed') + } # Different evidence requirements depending on terminal status. A # completed scripted run must provide full same-session interaction proof. @@ -878,8 +1041,7 @@ function Test-NativeWorkerTerminalEvidence { if (-not [bool](Get-JsonProperty -Object $delegation -Name 'terminal_result_capture' -Default $false)) { $failures.Add('terminal_result_capture') } - if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or - [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + if ($pairedArmVisible -or $forbiddenGradingVisible) { $failures.Add('paired_arm_and_grading_exclusion') } if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) -or @@ -889,8 +1051,7 @@ function Test-NativeWorkerTerminalEvidence { } else { # For non-success terminals, skip strict prompt/terminal-capture checks # but still validate working/home path alignment when provided. - if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or - [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + if ($pairedArmVisible -or $forbiddenGradingVisible) { $failures.Add('paired_arm_and_grading_exclusion') } } @@ -1011,6 +1172,10 @@ function Resolve-RunContract { if ($mode -notin @('with_skill', 'without_skill')) { throw "run.json mode '$mode' is not with_skill or without_skill." } + $executionRole = Get-RunExecutionRole -Run $run + if ($executionRole -eq 'phase2_analyzer' -and $mode -ne 'without_skill') { + throw 'phase2_analyzer run.json must use transport mode without_skill.' + } $candidateSkillName = [string](Get-JsonProperty -Object $run -Name 'candidateSkillName' -Default '') if ([string]::IsNullOrWhiteSpace($candidateSkillName)) { throw 'run.json must declare candidateSkillName for both with_skill and without_skill arms.' @@ -1119,6 +1284,7 @@ function Resolve-RunContract { EvalId = [int]$run.evalId EvalName = [string]$run.evalName Mode = $mode + ExecutionRole = $executionRole PromptPath = $promptPath PromptBytes = $promptBytes PromptHash = Get-Sha256HexFromBytes -Bytes $promptBytes From 1ab3e56389a398f8afd147016ff422c7a3689336 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 13:49:26 +0200 Subject: [PATCH 35/36] =?UTF-8?q?=E2=9C=85=20add=20eval-runner=20conforman?= =?UTF-8?q?ce=20and=20isolation=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive test suite for eval-runner isolation and conformance validation. test-copilot-boundaries.ps1 validates GitHub Copilot CLI boundary enforcement. test-integrity-finalization.ps1 verifies execution freeze and result integrity. test-orchestration.ps1 tests runner coordination and phase workflows. test-runner-conformance.ps1 validates runner protocol compliance and worker isolation contracts. --- .../tests/test-copilot-boundaries.ps1 | 33 +++- .../tests/test-integrity-finalization.ps1 | 174 ++++++++++++++++++ .../eval-runners/tests/test-orchestration.ps1 | 62 +++++++ .../tests/test-runner-conformance.ps1 | 50 +++++ 4 files changed, 317 insertions(+), 2 deletions(-) diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 16571d5..27f5aed 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -189,11 +189,40 @@ if ($inputText -eq 'failure') { exit 7 } $forged.evidence.delegation.paired_arm_visible = $false $forged.evidence.delegation.grading_material_visible = $false Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $forged -RunData $singleInputs.Run } 'bridge rejects transcript contradiction despite false invisibility flags' - $proof = [pscustomobject]@{ Root = 'C:/temp/projection'; PackageRoot = 'C:/source/.bot/package'; SourceRepositoryRoot = 'C:/source' } - foreach ($path in @('../../eval-metadata.json', '../../with_skill/repo', '../../without_skill/repo', '../../results/arm.json', '../../grading.json', '../../execution-freeze.json', '../../orchestration-state.json', '../../report.html', 'C:\source\AGENTS.md')) { + $proof = [pscustomobject]@{ + PackageRoot = 'C:/source/.bot/package' + SourceRepositoryRoot = 'C:/source' + RunRoot = 'C:/source/.bot/package/eval-01/with_skill' + WorkingDirectoryRoot = 'C:/source/.bot/package/eval-01/with_skill/repo' + } + foreach ($path in @('../../eval-metadata.json', '../../without_skill/repo', '../../results/arm.json', '../../../grading.json', '../../../execution-freeze.json', '../../../orchestration-state.json', '../../../report.html', 'C:\source\AGENTS.md')) { Assert-True (@(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = $path } } -Projection $proof).Count -gt 0) "captured forbidden access rejected: $path" } Assert-Equal 0 @(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = 'src/Widget.cs' } } -Projection $proof).Count 'ordinary staged source is allowed' + $phase2AnalyzerBoundary = [pscustomobject]@{ + PackageRoot = 'C:\prepared\iteration-1' + SourceRepositoryRoot = 'C:\Source\GitHub\codebeltnet\agentic' + RunRoot = 'C:\prepared\iteration-1\phase2\work\arm-3-without_skill' + WorkingDirectoryRoot = 'C:\prepared\iteration-1\phase2\work\arm-3-without_skill\repo' + ExecutionRole = 'phase2_analyzer' + } + $phase2InputBundleAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'input-bundle.json' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2InputBundleAccess.Contradictions).Count 'phase2 analyzer may read its own staged input bundle' + Assert-True ([bool]$phase2InputBundleAccess.OwnArmGradingMaterialVisible) 'phase2 analyzer records own-arm bundle visibility separately from forbidden grading visibility' + $phase2GraderAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'grader.md' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2GraderAccess.Contradictions).Count 'phase2 analyzer may read its own staged grader contract' + Assert-True ([bool]$phase2GraderAccess.OwnArmGradingMaterialVisible) 'phase2 analyzer records own-arm grader visibility separately from forbidden grading visibility' + $phase2EvidenceAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'evidence/with_skill/evidence/opencode-events.jsonl' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2EvidenceAccess.Contradictions).Count 'phase2 analyzer may read its own staged one-arm evidence artifacts' + $phase2PairedArmAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\arm-4-with_skill\repo\input-bundle.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PairedArmAccess.Contradictions).Count -gt 0) 'phase2 analyzer paired-arm artifact access is incompatible' + Assert-True ([bool]$phase2PairedArmAccess.PairedArmVisible) 'phase2 analyzer paired-arm access is classified separately' + $phase2PackageGradingAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\..\..\grading.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PackageGradingAccess.Contradictions).Count -gt 0) 'phase2 analyzer package-root grading access is incompatible' + Assert-True ([bool]$phase2PackageGradingAccess.PairedOrPackageGradingMaterialVisible) 'phase2 analyzer package-root grading access is classified separately' + $phase2PackageOrchestrationAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\..\..\execution-freeze.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PackageOrchestrationAccess.Contradictions).Count -gt 0) 'phase2 analyzer package-root orchestration access is incompatible' + Assert-True ([bool]$phase2PackageOrchestrationAccess.PairedOrPackageGradingMaterialVisible) 'phase2 analyzer package-root orchestration access remains forbidden' # --- P0 native-skill isolation regressions (model-free) --- Assert-True (@(New-CopilotCliArguments -Inputs $singleInputs) -contains '--excluded-tools=skill') 'Copilot removes the native skill tool from the model tool set for both arms' diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index fa406f6..d594caa 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -27,6 +27,16 @@ $repositoryRoot = (Resolve-Path (Join-Path $runnerRoot '..')).Path . (Join-Path $runnerRoot 'phase2-grading.ps1') . (Join-Path $runnerRoot 'package-integrity.ps1') . (Join-Path $runnerRoot 'fanout-process.ps1') +$invokePhase2AnalyzerPath = Join-Path $runnerRoot 'invoke-phase2-analyzer.ps1' +$invokePhase2Tokens = $null +$invokePhase2Errors = $null +$invokePhase2Ast = [Management.Automation.Language.Parser]::ParseFile($invokePhase2AnalyzerPath, [ref]$invokePhase2Tokens, [ref]$invokePhase2Errors) +if ($invokePhase2Errors.Count -gt 0) { throw "Cannot parse invoke-phase2-analyzer.ps1: $invokePhase2Errors" } +foreach ($definition in $invokePhase2Ast.EndBlock.Statements) { + if ($definition -is [Management.Automation.Language.FunctionDefinitionAst]) { + Invoke-Expression $definition.Extent.Text + } +} function Assert-True { param([bool]$Condition, [string]$Message) @@ -485,6 +495,88 @@ function Copy-TestGradingDocument { return ConvertTo-RunnerJson -Value $Document -Depth 100 | ConvertFrom-Json -Depth 100 } +function New-AnalyzerSemanticFixture { + param([Parameter(Mandatory = $true)][string]$Root) + + $fixtureRoot = Join-Path $Root 'phase2-analyzer-direct' + $resultsDirectory = Join-Path $fixtureRoot 'results' + New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null + $canonicalPath = Join-Path $resultsDirectory 'with-skill.result.json' + $metadataPath = Join-Path $fixtureRoot 'eval-metadata.json' + $canonical = [ordered]@{ + output = "alpha line`nbeta line" + output_files = @('with_skill/evidence/opencode-events.jsonl', 'with_skill/evidence/opencode-stderr.txt') + } + Write-TestJson -Path $canonicalPath -Value $canonical + Write-TestJson -Path $metadataPath -Value ([ordered]@{ + expected_output = 'alpha line' + assertions = @( + [ordered]@{ assertion = 'alpha line is present'; evidence_domain = 'output' } + [ordered]@{ assertion = 'beta line is present'; evidence_domain = 'output' } + ) + }) + $record = [pscustomobject]@{ + EvalDirectory = $fixtureRoot + ResultPath = $canonicalPath + MetadataPath = $metadataPath + ResultRelative = 'with_skill.result.json' + } + $assertions = @( + [ordered]@{ eval_id = 1; eval_name = 'phase2-direct'; configuration = 'with_skill'; assertion_index = 0; assertion = 'alpha line is present'; evidence_domain = 'output'; validator = $null; record = $record } + [ordered]@{ eval_id = 1; eval_name = 'phase2-direct'; configuration = 'with_skill'; assertion_index = 1; assertion = 'beta line is present'; evidence_domain = 'output'; validator = $null; record = $record } + ) + return [pscustomobject]@{ + Root = $fixtureRoot + Canonical = Read-RunnerJson -Path $canonicalPath + Record = $record + Worker = [pscustomobject]@{ + worker_id = 'arm-1-with_skill' + eval_id = 1 + eval_name = 'phase2-direct' + configuration = 'with_skill' + record = $record + assertions = $assertions + } + AnalyzerProfile = [pscustomobject]@{ + Hash = ('f' * 64) + Runner = 'fixture' + Model = 'fixture-model' + ReasoningEffort = $null + } + } +} + +function Assert-AnalyzerResponseAccepted { + param( + [Parameter(Mandatory = $true)][string]$Text, + [Parameter(Mandatory = $true)][string]$ExpectedNormalization, + [Parameter(Mandatory = $true)][object]$Fixture, + [Parameter(Mandatory = $true)][string]$Description + ) + + $response = ConvertFrom-AnalyzerResponse -Text $Text + Assert-Equal $ExpectedNormalization ([string]$response.TransportNormalization) "$Description normalization" + $grades = @(Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $Fixture.Worker -Canonical $Fixture.Canonical) + Assert-Equal 2 $grades.Count "$Description semantic grading entry count" +} + +function Assert-ActionRejected { + param( + [Parameter(Mandatory = $true)][scriptblock]$Action, + [Parameter(Mandatory = $true)][string]$Pattern, + [Parameter(Mandatory = $true)][string]$Description + ) + + $failed = $false + try { + & $Action | Out-Null + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "$Description unexpected failure: $($_.Exception.Message)" + } + Assert-True $failed "$Description unexpectedly passed" +} + $testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-integrity-finalization-' + [Guid]::NewGuid().ToString('N')) $oldReportMode = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE') try { @@ -760,6 +852,88 @@ for ($index = 0; $index -lt $count; $index++) { $validationSnapshot = Get-TestFileHashSnapshot -Paths $validationSideEffectPaths if ($Suite -in @('All', 'Grading')) { + $directAnalyzerFixture = New-AnalyzerSemanticFixture -Root $testRoot + $validAnalyzerFragment = [ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-fragment/1' + eval_id = 1 + configuration = 'with_skill' + grading = @( + [ordered]@{ + assertion_index = 0 + passed = $true + reason = 'alpha line is present in the frozen output.' + evidence_refs = @([ordered]@{ + artifact = 'with_skill.result.json' + domain = 'output' + start_line = 1 + end_line = 1 + quote = 'alpha line' + }) + } + [ordered]@{ + assertion_index = 1 + passed = $true + reason = 'beta line is present in the frozen output.' + evidence_refs = @([ordered]@{ + artifact = 'with_skill.result.json' + domain = 'output' + start_line = 1 + end_line = 1 + quote = 'beta line' + }) + } + ) + } + $validAnalyzerJson = ConvertTo-RunnerJson -Value $validAnalyzerFragment -Depth 100 + $fencedAnalyzerJson = [string]::Join([Environment]::NewLine, @('```json', $validAnalyzerJson, '```')) + $fencedAnalyzerJsonWithProse = [string]::Join([Environment]::NewLine, @('The grading fragment follows.', '```json', $validAnalyzerJson, '```')) + $duplicateFencedAnalyzerJson = [string]::Join([Environment]::NewLine, @('```json', $validAnalyzerJson, '```', '', '```json', $validAnalyzerJson, '```')) + Assert-AnalyzerResponseAccepted -Text $validAnalyzerJson -ExpectedNormalization 'raw_json' -Fixture $directAnalyzerFixture -Description 'raw analyzer JSON' + Assert-AnalyzerResponseAccepted -Text $fencedAnalyzerJson -ExpectedNormalization 'fenced_json' -Fixture $directAnalyzerFixture -Description 'fenced analyzer JSON' + Assert-AnalyzerResponseAccepted -Text $fencedAnalyzerJsonWithProse -ExpectedNormalization 'fenced_json_with_surrounding_text' -Fixture $directAnalyzerFixture -Description 'fenced analyzer JSON with prose' + Assert-ActionRejected -Description 'empty analyzer response rejected' -Pattern 'empty response' -Action { ConvertFrom-AnalyzerResponse -Text ' ' | Out-Null } + Assert-ActionRejected -Description 'malformed analyzer JSON rejected' -Pattern 'malformed JSON' -Action { ConvertFrom-AnalyzerResponse -Text '```json`n{"schema":`n```' | Out-Null } + Assert-ActionRejected -Description 'multiple analyzer JSON candidates rejected' -Pattern 'multiple (fenced blocks|JSON candidates)' -Action { ConvertFrom-AnalyzerResponse -Text $duplicateFencedAnalyzerJson | Out-Null } + $wrongSchemaFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongSchemaFragment.schema = 'wrong/schema' + Assert-ActionRejected -Description 'wrong analyzer fragment schema rejected' -Pattern 'unsupported schema' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongSchemaFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $wrongEvalFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongEvalFragment.eval_id = 2 + Assert-ActionRejected -Description 'wrong analyzer eval id rejected' -Pattern 'does not match the requested eval arm' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongEvalFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $wrongConfigurationFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongConfigurationFragment.configuration = 'without_skill' + Assert-ActionRejected -Description 'wrong analyzer configuration rejected' -Pattern 'does not match the requested eval arm' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongConfigurationFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $missingAssertionFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $missingAssertionFragment.grading = @($missingAssertionFragment.grading[0]) + Assert-ActionRejected -Description 'missing analyzer assertion index rejected' -Pattern 'grade cardinality' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $missingAssertionFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $duplicateAssertionFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $duplicateAssertionFragment.grading[1].assertion_index = 0 + Assert-ActionRejected -Description 'duplicate analyzer assertion index rejected' -Pattern 'duplicates assertion_index' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $duplicateAssertionFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $phase2BundleRoot = Join-Path $testRoot 'phase2-bundle-direct' + New-Item -ItemType Directory -Path $phase2BundleRoot -Force | Out-Null + $directBundle = New-AnalyzerRunBundle -Phase2Root $phase2BundleRoot -Worker $directAnalyzerFixture.Worker -AnalyzerProfile $directAnalyzerFixture.AnalyzerProfile -AnalyzerExecutionProfilePath (Join-Path $phase2BundleRoot 'analyzer-execution-profile.json') -GraderContractText '# deterministic grader' + $directBundleDocument = Read-TestJson -Path $directBundle.BundlePath + Assert-Equal 'with_skill.result.json,with_skill/evidence/opencode-events.jsonl,with_skill/evidence/opencode-stderr.txt' ([string]::Join(',', @($directBundleDocument.allowed_artifacts))) 'allowed_artifacts preserves individual array entries' + $directBundleRun = Resolve-RunContract -RunPath $directBundle.RunPath + Assert-Equal 'phase2_analyzer' ([string]$directBundleRun.ExecutionRole) 'Phase 2 analyzer run declares its transport role explicitly' + Assert-Equal 'without_skill' ([string]$directBundleRun.Mode) 'Phase 2 analyzer transport stays without_skill' + Assert-Equal 'with_skill' ([string]$directBundleDocument.configuration) 'Phase 2 bundle preserves the graded subject configuration independently of transport mode' + $skeleton = Invoke-TestTool -Path $validationScript -Arguments @('-ShowSkeleton') Assert-ToolPasses -Invocation $skeleton -Description 'grading skeleton emission' $skeletonDocument = $skeleton.Text | ConvertFrom-Json -Depth 100 diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index c98b4f0..91eff73 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -424,6 +424,58 @@ try { $validTerminalEvidence = New-TestNativeTerminalEvidence -Arm $terminalArm -RunData $terminalRunData -WorkerSessionId 'native-terminal-session' Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'valid terminal native-worker evidence is accepted' Assert-True (Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session') 'valid terminal evidence passes the assert gate' + $phase2AnalyzerRunRoot = Join-Path $testRoot 'phase2-analyzer-run' + New-Item -ItemType Directory -Path (Join-Path $phase2AnalyzerRunRoot 'repo'), (Join-Path $phase2AnalyzerRunRoot 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $phase2AnalyzerRunRoot 'prompt.md'), 'deterministic phase2 analyzer prompt', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $phase2AnalyzerRunRoot 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 1 + evalName = 'phase2-analyzer' + candidateSkillName = 'phase2-analyzer' + skillName = $null + mode = 'without_skill' + executionRole = 'phase2_analyzer' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $null + candidateInstructionHash = $null + }) + $phase2AnalyzerRunData = Resolve-RunContract -RunPath (Join-Path $phase2AnalyzerRunRoot 'run.json') + $phase2AnalyzerEvidence = [ordered]@{ + status = 'completed' + session = [ordered]@{ id = 'phase2-analyzer-session'; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = 1; eval_name = 'phase2-analyzer'; configuration = 'without_skill' } + requested = [ordered]@{ model = 'fixture-model' } + input = [ordered]@{ prompt_sha256 = [string]$phase2AnalyzerRunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = 'phase2-analyzer-session' + observed_model = 'fixture-model' + observed_working_directory = [string]$phase2AnalyzerRunData.WorkingDirectoryPath + observed_home = [string]$phase2AnalyzerRunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$phase2AnalyzerRunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + paired_or_package_grading_material_visible = $false + own_arm_grading_material_visible = $true + nested_model_execution = $false + model_execution_count = 1 + } + } + } + Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $phase2AnalyzerEvidence -Run $phase2AnalyzerRunData -RequestedModel 'fixture-model' -ExpectedWorkerSessionId 'phase2-analyzer-session').Valid) 'phase2 analyzer may inspect its own staged grader bundle without violating the paired-arm/grading boundary' # Runner-specific checks and the portable validator must make one terminal # decision. Preserve the exact runner codes while the common validator @@ -491,6 +543,16 @@ try { Assert-True (([string]::Join(',', @($caseState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match [regex]::Escape($ExpectedFailure)) "$Name records $ExpectedFailure" } + $phase2AnalyzerForbidden = Copy-TestObject -Value $phase2AnalyzerEvidence + $phase2AnalyzerForbidden.evidence.delegation.paired_or_package_grading_material_visible = $true + $phase2AnalyzerValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $phase2AnalyzerForbidden -Run $phase2AnalyzerRunData -RequestedModel 'fixture-model' -ExpectedWorkerSessionId 'phase2-analyzer-session' + Assert-True (-not [bool]$phase2AnalyzerValidation.Valid) 'phase2 analyzer still rejects paired/package grading visibility' + Assert-True (([string]::Join(',', @($phase2AnalyzerValidation.Failures))) -match 'paired_arm_and_grading_exclusion') 'phase2 analyzer violation reports paired/package grading failure' + + $projectionEscapeObserved = Copy-TestObject -Value $validTerminalEvidence + Add-Member -InputObject $projectionEscapeObserved.evidence.delegation -MemberType NoteProperty -Name projection_escape_observed -Value $true -Force + Invoke-TerminalEvidenceCase -Name 'projection escape observed' -Evidence $projectionEscapeObserved -ExpectedFailure 'projection_escape_observed' + $modelMismatch = Copy-TestObject -Value $validTerminalEvidence $modelMismatch.evidence.delegation.observed_model = 'different-model' Invoke-TerminalEvidenceCase -Name 'model mismatch' -Evidence $modelMismatch -ExpectedFailure 'requested_model' diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index f463a49..23d44ef 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -2665,17 +2665,67 @@ try { $continuationParserAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Get-OpenCodeContinuationCapability' }, $true)) $continuationArgumentAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'New-OpenCodeContinuationArguments' }, $true)) $turnProcessAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeTurnProcess' }, $true)) + $openCodeBoundaryHelperNames = @( + 'Add-OpenCodeBoundaryValueCandidate', + 'Get-OpenCodeBoundaryValueCandidates', + 'Get-OpenCodeLogicalPackageRoot', + 'Test-OpenCodeBoundaryPairedArmPath', + 'Test-OpenCodeBoundaryForbiddenGradingPath', + 'Get-OpenCodeBoundaryAssessment', + 'ConvertTo-OpenCodeBoundaryAssessment', + 'Apply-OpenCodeBoundaryAssessment' + ) Assert-Equal 1 $scriptedFunctionAst.Count 'OpenCode has one selected scripted execution function' Assert-Equal 1 $executeFunctionAst.Count 'OpenCode has one execution dispatcher' Assert-Equal 0 $legacyFunctionAst.Count 'OpenCode removes the dead legacy scripted continuation function' Assert-Equal 1 $continuationParserAst.Count 'OpenCode keeps installed-help exact --session parsing' Assert-Equal 1 $continuationArgumentAst.Count 'OpenCode keeps explicit --session argument construction' Assert-Equal 1 $turnProcessAst.Count 'OpenCode uses a direct CLI turn-process helper' + foreach ($helperName in $openCodeBoundaryHelperNames) { + $helperAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $helperName }, $true) | Select-Object -First 1) + Assert-Equal 1 $helperAst.Count "OpenCode keeps boundary helper $helperName" + Invoke-Expression $helperAst[0].Extent.Text + } $scriptedFunctionText = [string]$scriptedFunctionAst[0].Extent.Text $executeFunctionText = [string]$executeFunctionAst[0].Extent.Text Assert-True ($scriptedFunctionText.Contains('Invoke-OpenCodeTurnProcess') -and $scriptedFunctionText.Contains('New-OpenCodeContinuationArguments') -and $scriptedFunctionText.Contains('Get-OpenCodeFutureTurnCanary')) 'selected OpenCode scripted transport uses direct CLI exact-session continuation and future-turn secrecy checks' Assert-True ($opencodeRunnerText -notmatch '(?i)Start-OpenCodeServer|Invoke-OpenCodeHttpRequest|SessionCreatePath|SessionMessagePath|/global/health|/doc|opencode\s+serve|serve_help_probe|opencode-server-synchronous-http') 'OpenCode runner has no eval server transport or loopback API probes' Assert-True ($executeFunctionText.Contains('Invoke-OpenCodeScriptedExecute') -and $executeFunctionText -notmatch '(?i)Invoke-OpenCodeScriptedExecuteLegacy\s+-Inputs') 'OpenCode dispatcher selects the direct CLI scripted transport for interaction runs' + $openCodeBoundaryProjection = [pscustomobject]@{ + Root = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture' + PhysicalWorkingDirectory = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo' + SourceRepositoryRoot = 'C:\Source\GitHub\codebeltnet\agentic' + LogicalRun = [pscustomobject]@{ + RunRoot = 'C:\prepared\iteration-1\eval-01\with_skill' + Mode = 'with_skill' + } + } + foreach ($event in @( + @{ Name = 'read inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\README.md' } } } + @{ Name = 'write inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'write'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\out.txt' } } } + @{ Name = 'repo-relative shell inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'shell'; command = 'Get-Content README.md' } } } + )) { + $assessment = Get-OpenCodeBoundaryAssessment -Events @($event.Event) -Projection $openCodeBoundaryProjection + Assert-Equal 0 @($assessment.Contradictions).Count "OpenCode boundary allows $($event.Name)" + } + foreach ($event in @( + @{ Name = 'read unrelated Windows repo'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\xunit\README.md' } } } + @{ Name = 'write unrelated Windows repo'; Event = @{ type = 'tool_use'; part = @{ tool = 'write'; path = 'C:\Source\GitHub\xunit\README.md' } } } + @{ Name = 'PowerShell filesystem shell escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'shell'; command = 'Get-ChildItem C:\Users\Administrator -Recurse' } } } + @{ Name = 'Linux absolute path escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = '/home/user/other-repo/file' } } } + @{ Name = 'macOS absolute path escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = '/Users/user/other-repo/file' } } } + @{ Name = 'source repository escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\codebeltnet\agentic\README.md' } } } + @{ Name = 'prepared package escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\prepared\iteration-1\eval-01\with_skill\repo\README.md' } } } + )) { + $assessment = Get-OpenCodeBoundaryAssessment -Events @($event.Event) -Projection $openCodeBoundaryProjection + Assert-True ([bool]$assessment.ProjectionEscapeObserved) "OpenCode boundary rejects $($event.Name)" + Assert-True (@($assessment.Contradictions).Count -gt 0) "OpenCode boundary records contradiction evidence for $($event.Name)" + } + $outsideProjectionAssessment = Get-OpenCodeBoundaryAssessment -Events @(@{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\xunit\README.md' } }) -Projection $openCodeBoundaryProjection + $outsideProjectionOutcome = Apply-OpenCodeBoundaryAssessment -Status 'completed' -Assessment $outsideProjectionAssessment + Assert-Equal 'incompatible' ([string]$outsideProjectionOutcome.Status) 'observed OpenCode projection escape makes a completed arm incompatible' + Assert-Equal 'opencode_projection_escape_observed' ([string]$outsideProjectionOutcome.FailureCode) 'observed OpenCode projection escape surfaces the dedicated failure code' + Assert-True (@($outsideProjectionOutcome.BoundaryEvidence.contradictions).Count -gt 0) 'OpenCode boundary outcome preserves structured contradiction evidence' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $withoutRawPath = Join-Path $iteration 'conformance\results\without-skill.execution-result.json' From df9f4051bacdb4ab0dc00b00a063ca1c73fda5b0 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 10 Sep 2026 19:30:37 +0200 Subject: [PATCH 36/36] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20extract=20path=20uti?= =?UTF-8?q?lities=20and=20add=20containment=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactor path handling across eval runners by extracting common utilities into runner-common.ps1. Adds Get-ObservedPathSegments, Resolve-ObservedPathSegments, Join-ObservedPath, New-ObservedPathInfo, Get-ObservedParentPath, and Get-ObservedRelativePath functions to centralize path logic and improve maintainability. Replace inline path resolution in github-copilot/isolation.ps1 and opencode/runner.ps1 to use these utilities. Add comprehensive test coverage in test-runner-conformance.ps1 for path containment across Windows, Unix, and UNC styles. --- .../eval-runners/github-copilot/isolation.ps1 | 4 +- scripts/eval-runners/opencode/runner.ps1 | 7 +- scripts/eval-runners/runner-common.ps1 | 166 ++++++++++++------ .../tests/test-runner-conformance.ps1 | 28 +++ 4 files changed, 143 insertions(+), 62 deletions(-) diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 index 1236b3d..6b56720 100644 --- a/scripts/eval-runners/github-copilot/isolation.ps1 +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -99,7 +99,7 @@ function Test-CopilotBoundaryForbiddenGradingPath { $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } $relative = if (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath) { - [System.IO.Path]::GetRelativePath($packageRoot, $ResolvedPath).Replace('\', '/') + Get-ObservedRelativePath -BasePath $packageRoot -CandidatePath $ResolvedPath } else { '' } @@ -143,7 +143,7 @@ function Get-CopilotBoundaryAssessment { $insideSource = -not [string]::IsNullOrWhiteSpace($sourceRepositoryRoot) -and (Test-ObservedPathInside -BasePath $sourceRepositoryRoot -CandidatePath $resolvedPath) if ($executionRole -eq 'phase2_analyzer') { if ($insideWorking) { - $relative = [System.IO.Path]::GetRelativePath($workingDirectoryRoot, $resolvedPath).Replace('\', '/') + $relative = Get-ObservedRelativePath -BasePath $workingDirectoryRoot -CandidatePath $resolvedPath if ($relative -in @('input-bundle.json', 'grader.md')) { $ownArmGradingVisible = $true } continue } diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 89a78b1..86c1312 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -680,7 +680,9 @@ function Get-OpenCodeLogicalPackageRoot { $logicalRun = Get-JsonProperty -Object $Projection -Name 'LogicalRun' -Default $null $logicalRunRoot = [string](Get-JsonProperty -Object $logicalRun -Name 'RunRoot' -Default '') if ([string]::IsNullOrWhiteSpace($logicalRunRoot)) { return '' } - return [System.IO.Path]::GetFullPath((Split-Path -Parent (Split-Path -Parent $logicalRunRoot))) + $evalRoot = Get-ObservedParentPath -Path $logicalRunRoot + if ([string]::IsNullOrWhiteSpace($evalRoot)) { return '' } + return [string](Get-ObservedParentPath -Path $evalRoot) } function Test-OpenCodeBoundaryPairedArmPath { @@ -705,7 +707,8 @@ function Test-OpenCodeBoundaryForbiddenGradingPath { $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } - $relative = [System.IO.Path]::GetRelativePath($packageRoot, $ResolvedPath).Replace('\', '/') + $relative = Get-ObservedRelativePath -BasePath $packageRoot -CandidatePath $ResolvedPath + if ([string]::IsNullOrWhiteSpace($relative)) { return $false } return $relative -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)(?:$|/)' -or $relative -match '^(?i)(results|tools|progress)(?:/|$)' } diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 02358fa..da942b5 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -335,34 +335,75 @@ function Test-PathInside { return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) } -function Resolve-ObservedUnixPath { +function Get-ObservedPathSegments { + param([AllowEmptyString()][string]$PathText) + + if ([string]::IsNullOrWhiteSpace($PathText)) { return @() } + $segments = [System.Collections.Generic.List[string]]::new() + foreach ($segment in @($PathText -split '[\\/]')) { + if ($null -ne $segment -and [string]$segment -ne '') { $segments.Add([string]$segment) } + } + return @($segments.ToArray()) +} + +function Resolve-ObservedPathSegments { param( - [Parameter(Mandatory = $true)][string]$BasePath, - [Parameter(Mandatory = $true)][string]$RelativePath + [AllowEmptyCollection()][string[]]$BaseSegments = @(), + [AllowEmptyCollection()][string[]]$PathSegments = @() ) - $base = ($BasePath -replace '\\', '/') - if (-not $base.StartsWith('/', [System.StringComparison]::Ordinal)) { - throw "Unix observed path base '$BasePath' must be absolute." + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($segment in @($BaseSegments)) { + if ($null -ne $segment -and [string]$segment -ne '') { $resolved.Add([string]$segment) } } - $segments = [System.Collections.Generic.List[string]]::new() - foreach ($segment in @($base -split '/')) { - if (-not [string]::IsNullOrWhiteSpace([string]$segment)) { $segments.Add([string]$segment) } - } - foreach ($segment in @(($RelativePath -replace '\\', '/') -split '/')) { + foreach ($segment in @($PathSegments)) { switch ([string]$segment) { '' { continue } '.' { continue } '..' { - if ($segments.Count -gt 0) { $segments.RemoveAt($segments.Count - 1) } + if ($resolved.Count -gt 0) { $resolved.RemoveAt($resolved.Count - 1) } continue } default { - $segments.Add([string]$segment) + $resolved.Add([string]$segment) } } } - return '/' + ([string]::Join('/', @($segments.ToArray()))) + return @($resolved.ToArray()) +} + +function Join-ObservedPath { + param( + [Parameter(Mandatory = $true)][string]$Style, + [Parameter(Mandatory = $true)][string]$Root, + [AllowEmptyCollection()][string[]]$Segments = @() + ) + + if (@($Segments).Count -eq 0) { return $Root } + $separator = if ([string]$Style -eq 'unix') { '/' } else { '\' } + if ($Root.EndsWith($separator, [System.StringComparison]::Ordinal)) { + return $Root + [string]::Join($separator, @($Segments)) + } + return $Root + $separator + [string]::Join($separator, @($Segments)) +} + +function New-ObservedPathInfo { + param( + [Parameter(Mandatory = $true)][string]$Style, + [Parameter(Mandatory = $true)][string]$Root, + [AllowEmptyCollection()][string[]]$Segments = @(), + [Parameter(Mandatory = $true)][string]$Raw, + [Parameter(Mandatory = $true)][bool]$Absolute + ) + + return [pscustomobject]@{ + Style = $Style + Root = $Root + Segments = @($Segments) + FullPath = Join-ObservedPath -Style $Style -Root $Root -Segments @($Segments) + Raw = $Raw + Absolute = $Absolute + } } function Get-ObservedPathInfo { @@ -382,57 +423,57 @@ function Get-ObservedPathInfo { if ([string]::IsNullOrWhiteSpace($trimmed)) { return $null } if ($trimmed -match '^[A-Za-z]:[\\/]') { - return [pscustomobject]@{ - Style = 'windows' - FullPath = ConvertTo-ComparablePath -Path ($trimmed -replace '/', '\') - Raw = $trimmed - Absolute = $true - } + $segments = Resolve-ObservedPathSegments -PathSegments (Get-ObservedPathSegments -PathText $trimmed.Substring(3)) + return New-ObservedPathInfo -Style 'windows' -Root ($trimmed.Substring(0, 1).ToUpperInvariant() + ':\') -Segments $segments -Raw $trimmed -Absolute $true } if ($trimmed -match '^\\\\') { - return [pscustomobject]@{ - Style = 'unc' - FullPath = ConvertTo-ComparablePath -Path ($trimmed -replace '/', '\') - Raw = $trimmed - Absolute = $true - } + $uncSegments = @(Get-ObservedPathSegments -PathText $trimmed.Substring(2)) + if ($uncSegments.Count -lt 2) { return $null } + $pathSegments = if ($uncSegments.Count -gt 2) { @($uncSegments[2..($uncSegments.Count - 1)]) } else { @() } + $segments = Resolve-ObservedPathSegments -PathSegments $pathSegments + return New-ObservedPathInfo -Style 'unc' -Root ('\\' + $uncSegments[0] + '\' + $uncSegments[1]) -Segments $segments -Raw $trimmed -Absolute $true } if ($trimmed -match '^/') { - $normalized = ($trimmed -replace '\\', '/') - if ($normalized.Length -gt 1) { $normalized = $normalized.TrimEnd('/') } - return [pscustomobject]@{ - Style = 'unix' - FullPath = $normalized - Raw = $trimmed - Absolute = $true - } + $segments = Resolve-ObservedPathSegments -PathSegments (Get-ObservedPathSegments -PathText $trimmed.Substring(1)) + return New-ObservedPathInfo -Style 'unix' -Root '/' -Segments $segments -Raw $trimmed -Absolute $true } if ([string]::IsNullOrWhiteSpace($BasePath)) { return $null } $baseInfo = Get-ObservedPathInfo -Path $BasePath if ($null -eq $baseInfo) { return $null } - switch ([string]$baseInfo.Style) { - 'unix' { - return [pscustomobject]@{ - Style = 'unix' - FullPath = Resolve-ObservedUnixPath -BasePath ([string]$baseInfo.FullPath) -RelativePath $trimmed - Raw = $trimmed - Absolute = $false - } - } - default { - try { - $combined = [System.IO.Path]::GetFullPath((Join-Path $BasePath ($trimmed -replace '/', [System.IO.Path]::DirectorySeparatorChar))) - } catch { - return $null - } - return [pscustomobject]@{ - Style = [string]$baseInfo.Style - FullPath = ConvertTo-ComparablePath -Path $combined - Raw = $trimmed - Absolute = $false - } - } + $segments = Resolve-ObservedPathSegments -BaseSegments @($baseInfo.Segments) -PathSegments (Get-ObservedPathSegments -PathText $trimmed) + return New-ObservedPathInfo -Style ([string]$baseInfo.Style) -Root ([string]$baseInfo.Root) -Segments $segments -Raw $trimmed -Absolute $false +} + +function Get-ObservedParentPath { + param([AllowEmptyString()][string]$Path) + + $info = Get-ObservedPathInfo -Path $Path + if ($null -eq $info) { return $null } + if (@($info.Segments).Count -eq 0) { return [string]$info.FullPath } + $parentSegments = if (@($info.Segments).Count -gt 1) { @($info.Segments[0..(@($info.Segments).Count - 2)]) } else { @() } + return Join-ObservedPath -Style ([string]$info.Style) -Root ([string]$info.Root) -Segments $parentSegments +} + +function Get-ObservedRelativePath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = Get-ObservedPathInfo -Path $BasePath + $candidate = Get-ObservedPathInfo -Path $CandidatePath -BasePath $BasePath + if ($null -eq $base -or $null -eq $candidate -or [string]$base.Style -ne [string]$candidate.Style) { return $null } + if (-not (Test-ObservedPathInside -BasePath ([string]$base.FullPath) -CandidatePath ([string]$candidate.FullPath))) { return $null } + + $comparison = if ([string]$base.Style -eq 'unix') { [System.StringComparison]::Ordinal } else { [System.StringComparison]::OrdinalIgnoreCase } + $baseSegments = @($base.Segments) + $candidateSegments = @($candidate.Segments) + if ($candidateSegments.Count -lt $baseSegments.Count) { return $null } + for ($index = 0; $index -lt $baseSegments.Count; $index++) { + if (-not [string]::Equals([string]$baseSegments[$index], [string]$candidateSegments[$index], $comparison)) { return $null } } + if ($candidateSegments.Count -eq $baseSegments.Count) { return '.' } + return [string]::Join('/', @($candidateSegments[$baseSegments.Count..($candidateSegments.Count - 1)])) } function Test-ObservedPathInside { @@ -444,9 +485,16 @@ function Test-ObservedPathInside { $base = Get-ObservedPathInfo -Path $BasePath $candidate = Get-ObservedPathInfo -Path $CandidatePath -BasePath $BasePath if ($null -eq $base -or $null -eq $candidate -or [string]$base.Style -ne [string]$candidate.Style) { return $false } - $separator = if ([string]$base.Style -eq 'unix') { '/' } else { '\' } $comparison = if ([string]$base.Style -eq 'unix') { [System.StringComparison]::Ordinal } else { [System.StringComparison]::OrdinalIgnoreCase } - return [string]$candidate.FullPath -eq [string]$base.FullPath -or ([string]$candidate.FullPath).StartsWith(([string]$base.FullPath + $separator), $comparison) + if (-not [string]::Equals([string]$base.Root, [string]$candidate.Root, $comparison)) { return $false } + + $baseSegments = @($base.Segments) + $candidateSegments = @($candidate.Segments) + if ($candidateSegments.Count -lt $baseSegments.Count) { return $false } + for ($index = 0; $index -lt $baseSegments.Count; $index++) { + if (-not [string]::Equals([string]$baseSegments[$index], [string]$candidateSegments[$index], $comparison)) { return $false } + } + return $true } function Get-ObservedPathTokensFromCommandText { @@ -851,6 +899,8 @@ function ConvertTo-ComparablePath { if ([string]::IsNullOrWhiteSpace($Path)) { return $null } try { + # Host filesystem normalization only. Observed/captured cross-platform + # evidence uses the lexical observed-path helpers above instead. $full = [System.IO.Path]::GetFullPath($Path) $full = Expand-WindowsShortPath -Path $full $root = [System.IO.Path]::GetPathRoot($full) diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 23d44ef..2483b35 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -2252,6 +2252,17 @@ function Assert-Throws { if (-not $thrown) { throw "ASSERT: $Message" } } +function Assert-ObservedPathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath, + [Parameter(Mandatory = $true)][bool]$Expected, + [Parameter(Mandatory = $true)][string]$Message + ) + + Assert-Equal $Expected ([bool](Test-ObservedPathInside -BasePath $BasePath -CandidatePath $CandidatePath)) $Message +} + function Write-TestJson { param([string]$Path, [object]$Value) New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null @@ -2657,6 +2668,20 @@ try { Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' Assert-True ($manifestBridgeText.Contains('Get-ManifestRunRecords') -and $manifestBridgeText.Contains('$record.ResultPath')) 'package-level bridge must resolve exact manifest records' Assert-True ($manifestBridgeText -notmatch 'with[-_]skill\.result\.json|without[-_]skill\.result\.json') 'package-level bridge must not encode arm-derived result filenames' + foreach ($case in @( + @{ Name = 'Windows inside'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection\repo\README.md'; Expected = $true } + @{ Name = 'Windows prefix collision'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection2\repo\README.md'; Expected = $false } + @{ Name = 'Windows dotdot escape'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection\..\outside\README.md'; Expected = $false } + @{ Name = 'Windows case-insensitive normalized separators'; Base = 'C:\TEMP\Projection'; Candidate = 'c:/temp/projection/repo/file.txt'; Expected = $true } + @{ Name = 'Unix inside'; Base = '/tmp/projection'; Candidate = '/tmp/projection/repo/README.md'; Expected = $true } + @{ Name = 'Unix prefix collision'; Base = '/tmp/projection'; Candidate = '/tmp/projection2/README.md'; Expected = $false } + @{ Name = 'Unix dotdot escape'; Base = '/tmp/projection'; Candidate = '/tmp/projection/../outside/README.md'; Expected = $false } + @{ Name = 'Unix remains case-sensitive'; Base = '/tmp/Projection'; Candidate = '/tmp/projection/file'; Expected = $false } + @{ Name = 'UNC inside'; Base = '\\server\share\projection'; Candidate = '\\server\share\projection\repo\file.txt'; Expected = $true } + @{ Name = 'UNC share boundary'; Base = '\\server\share\projection'; Candidate = '\\server\share-other\projection\file.txt'; Expected = $false } + )) { + Assert-ObservedPathInside -BasePath ([string]$case.Base) -CandidatePath ([string]$case.Candidate) -Expected ([bool]$case.Expected) -Message ("Observed path containment stays style-aware for {0}" -f [string]$case.Name) + } Assert-Equal 1 ([regex]::Matches($opencodeRunnerText, '\$directoryArgument = Get-SandboxVisiblePath').Count) 'OpenCode CLI argument construction assigns the sandbox directory once' $opencodeAst = Get-OpenCodeRunnerAst $scriptedFunctionAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeScriptedExecute' }, $true) | Select-Object -First 1) @@ -2700,6 +2725,9 @@ try { Mode = 'with_skill' } } + Assert-Equal 'C:\prepared\iteration-1' (Get-OpenCodeLogicalPackageRoot -Projection $openCodeBoundaryProjection) 'OpenCode derives the logical package root lexically from observed Windows paths' + Assert-True (Test-OpenCodeBoundaryForbiddenGradingPath -ResolvedPath 'C:\prepared\iteration-1\eval-01\eval-metadata.json' -Projection $openCodeBoundaryProjection) 'OpenCode grading-material detection stays lexical for observed Windows package paths' + Assert-True (Test-OpenCodeBoundaryPairedArmPath -ResolvedPath 'C:\prepared\iteration-1\eval-01\without_skill\repo\README.md' -Projection $openCodeBoundaryProjection) 'OpenCode paired-arm detection stays lexical for observed Windows package paths' foreach ($event in @( @{ Name = 'read inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\README.md' } } } @{ Name = 'write inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'write'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\out.txt' } } }
MetricWith skillWithout skillDelta