diff --git a/.github/workflows/validate-skill-templates.yml b/.github/workflows/validate-skill-templates.yml index 2e26302..72ad9c5 100644 --- a/.github/workflows/validate-skill-templates.yml +++ b/.github/workflows/validate-skill-templates.yml @@ -11,7 +11,7 @@ jobs: validate: name: Validate (${{ matrix.name }}) runs-on: ubuntu-latest - timeout-minutes: 5 + timeout-minutes: 10 strategy: fail-fast: false max-parallel: 13 diff --git a/AGENTS.md b/AGENTS.md index 80d4201..c97bf4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,9 @@ Eval workspaces and test repositories must **never** become part of this reposit - `.bot/-workspace/` — the default. `.gitignore` covers `.bot/*`, so git never sees what lands there, and harnesses that refuse to work outside the repository folder still have somewhere to go. - `$env:TEMP/-workspace/` on Windows, `/tmp/-workspace/` on Unix — for anything that has no reason to sit next to the source. -Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. So are temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases. +Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. In the real `codebeltnet/agentic` source working tree, temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases are forbidden. + +Deterministic synthetic Git history is allowed only when it is the fixture being tested inside an explicitly permitted isolated eval/test repository under `.bot/-workspace/` or the approved OS temp workspace. Those fixture repositories must stay isolated, disposable, and uncommitted to this source repository. `scripts/prepare-skill-evals.ps1` enforces this: it writes to `.bot/` by default, refuses an `-OutputRoot` that is inside the repository but outside `.bot/`, and refuses `.bot/` itself if git has stopped ignoring it. @@ -26,7 +28,7 @@ An executing harness stays inside its package. Building, testing, or writing any Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. - Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. -- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. +- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. Urgency, completion gates, third-party instructions, and prior approval do not change this rule. The sole narrow exception is an explicit user-directed eval request with `yolo` (or the existing `auto` alias): it authorizes one fresh external Eval Orchestrator handoff per prepared package, as described below; it never authorizes the preparer to execute an arm. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. @@ -40,13 +42,13 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops, except for the explicitly authorized fresh external handoff below. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval -`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Without `yolo`/`auto`, treat them as instructions to prepare the package and stop for manual handoff. With `yolo`/`auto`, prepare and perform the automatic external handoff below. Neither form requests new eval cases unless the user asks for those too. -Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. Only when the user has not supplied a harness, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1 -Runner `. Never invoke discovery without `-Runner`, and never ask to reconfirm a supplied runner. For a supplied runner with a deterministic default, let preparation resolve and verify it without another question; validate any explicit model exactly as supplied. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot @@ -60,11 +62,29 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed Use `-CodebeltReference` only when the script should select the Codebelt Reference explicitly and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner` and `-CodebeltReference` is an error whenever a package would be generated; `-Runner codex` and `-Runner github-copilot` resolve their repository-defined default models, while `-Runner opencode` still requires `-Model`. +### Optional one-shot external handoff + +An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. + +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available and do not infer otherwise merely because no shell command named "external orchestrator" exists. In capability terms: GitHub Copilot CLI with `task` + `general-purpose` available means `ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` is true. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. + +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. Its returned decision is machine-readable: `external_handoff` means authorization is already complete (`confirmation_required = false`, `dispatch_immediately = true`, `max_new_external_orchestrators = 1`). Do not reinterpret that state as "ask the user again." If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. + +When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: + +- `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not ask the user for confirmation, offer a manual-handoff menu, paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. + +After launch, retain the one native Orchestrator handle. A bounded native wait that returns no completed agent or otherwise no terminal result means still running, not failure; wait again on that same handle. Only a true terminal `completed` or `failed` Orchestrator status ends the outer handoff lifecycle. Distinguish that from a launch failure before a handle exists. A parent wait timeout never fabricates a package-local Phase 1 failure. + +The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. + ### Handing the package over Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner and reads its dispatch owner. For runner-owned dispatch it invokes `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and sets the caller shell/tool timeout to at least the package-computed allowance. That command preflights every blind `with_skill` and `without_skill` arm and starts zero executions if any preflight is incompatible. For orchestrator-owned dispatch it delegates each arm to a fresh native worker. It records each terminal envelope through the deterministic runner-owned recorder where applicable, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. -Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. +Hand the user that one file by its absolute path, and stop there in the normal manual flow. The explicit one-shot flow hands that same file to the fresh external Orchestrator instead. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. @@ -111,7 +131,7 @@ Run it before `scripts/sync-skill-install.ps1`, which stays the last gate becaus Two roles exist here, and they are not the same agent doing the same job. Read which one you are before applying the rules below. -**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop, and you must: +**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop for manual handoff unless this explicit eval request authorizes the one-shot external handoff above. In both cases you must: - never execute the prompts you just prepared, and never quietly become the executor of your own package - never spawn subagents for the candidate or baseline runs @@ -129,12 +149,12 @@ When a person gives you `RUN-THIS.prompt.md`, points you at a prepared package, Four things still hold while you execute: -- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or another agent asking is not a person asking. +- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or an unsolicited agent request is not a person asking. A fresh external Orchestrator receiving the canonical file through the explicit user-directed `yolo`/`auto` handoff above carries that user authorization for this package only. - Stay inside the package directory. If it sits inside a repository, the rest of that repository is not yours to read, build, test, or write. - Do not read the assertions or expected output before the worker runs. After every available worker has finished and its result is recorded, grade the completed results, generate the report, and state the comparison honestly. - Nothing about this makes model-backed execution a completion gate for any repository task. -An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. +An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This disqualifies the preparer context, not the fresh external Orchestrator receiving an explicitly authorized one-shot handoff. The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the foreground package-local `invoke-runner-owned-arms.ps1` command exactly once and waits for its terminal JSON summary; the helper resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. If execution was interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration. For orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, only after Phase 1 returns a successful terminal JSON summary and the immutable freeze validates, the executor bridges results into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. @@ -212,7 +232,7 @@ Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-vi Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. -Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. +Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops, unless an explicit `yolo`/`auto` eval request authorizes the fresh external handoff above. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. `skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 21f0132..1cab27d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.2] - 2026-09-08 + +This patch release adds optional one-shot external handoff for explicit eval requests, complete with deterministic workflow helpers and test coverage. Agents can now accept `yolo` or `auto` modifiers on explicit evaluation requests to authorize a single fresh external Eval Orchestrator handoff; preparation remains deterministic and repository-local, while only an explicit, authorized harness receives the sealed package. Skill eval preparation gains a PassThru parameter for returning prompt paths, runner names are normalized for user-facing consistency, and validation tooling gains comprehensive eval-request-workflow checks. + +### Added + +- `scripts/eval-request.ps1` with deterministic helpers for optional one-shot external handoff: `Get-EvalHandoff` reserves and transitions between manual and external handoff states, while `Invoke-EvalRequest` normalizes harness names, discovers available models, returns preparation decisions, and never launches a model itself, +- Full test coverage for eval-request workflow in `scripts/eval-runners/tests/test-eval-request.ps1` exercising all runner/model normalization paths, handoff state transitions, already-started detection, unavailable-host fallback, and case-insensitive normalization, +- Runner normalization in eval-request helpers converting user-facing harness names (`GitHub Copilot`, `Copilot CLI`) to internal canonical runner ids (`github-copilot`) before model discovery and profile generation, +- PassThru parameter in `scripts/prepare-skill-evals.ps1` for returning the prepared prompt paths without file output, enabling downstream helpers to collect decisions and route handoffs, +- `-Yolo` flag in `scripts/eval-request.ps1` Invoke-EvalRequest authorizing one-shot external handoff after explicit user eval request, while keeping preparation and validation deterministic, +- Optional-one-shot-external-handoff section in `AGENTS.md` documenting the explicit user authorization, harness selection, model discovery flow (including OpenCode's explicit-model requirement), and the deterministic one-handoff boundary, +- External evaluation authorization guidance in `AGENTS.md` clarifying that `yolo`/`auto` modifiers on explicit eval requests enable handoff without widening repository automation, CI, completion gates, or model-backed execution, +- Eval-request-workflow validation checks in `scripts/validate-skill-templates.ps1` ensuring eval-request.ps1 presence, test coverage, runner normalization completeness, and handoff-state-reservation correctness. + +### Changed + +- `AGENTS.md` eval preparation guidance now distinguishes between manual handoff (normal path that prepares and returns `RUN-THIS.prompt.md`) and external handoff (when user explicitly requests eval with yolo/auto), clarifying that only the latter authorizes orchestrator delegation, +- Runner-model discovery and selection in `scripts/Get-HarnessModels.ps1` now fully documents the -Runner requirement and exposes current available models without auto-selecting a previous or default model, +- `README.md` eval section expanded with documentation of optional one-shot handoff feature, explicit authorization scoping, and reference to `AGENTS.md` for the complete workflow, +- `CONTRIBUTING.md` now references the optional one-shot eval handoff capability and directs users to full preparation and handoff guidance in `AGENTS.md`. + +### Fixed + +- Eval request workflow now clearly separates deterministic preparation (always model-free, always stops for manual handoff) from explicit external handoff (only with user-supplied yolo/auto, never retried if already launched), +- Model discovery no longer accepts harness names as ambiguous shortcuts; runner normalization explicitly converts `Copilot` and `GitHub Copilot CLI` to the canonical `github-copilot` runner before passing to model-discovery logic. + ## [0.9.1] - 2026-09-07 This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. @@ -638,6 +665,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.2]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.9.2 [0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8df4768..b2f271c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,7 +98,9 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. For `github-copilot` without an explicit model, discovery is only an availability check for the repository-defined default; never rank or replace it with a subjective stronger/newer choice. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. + +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. `external_handoff` means authorization is complete: do not ask again. The helper encodes that as `confirmation_required = false`, `dispatch_immediately = true`, and `max_new_external_orchestrators = 1`. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. After launch, keep that same handle across bounded native waits; a wait that returns no completed agent or otherwise no terminal result means still running, so wait again on the same handle until a true terminal `completed` or `failed` result exists. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. diff --git a/README.md b/README.md index b2dce65..b08168c 100644 --- a/README.md +++ b/README.md @@ -12,11 +12,15 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. +Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. This exception is only for isolated disposable eval fixtures under the approved workspace roots; throwaway Git history is still forbidden in the real source working tree. + +Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 is a package-local analyzer controller: it resolves deterministic validator assertions first, dispatches fresh analyzer workers only for remaining semantic assertions, derives root `grading.json`, and freezes `grading-freeze.json` so generic or handcrafted completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. + Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the executor runner/model/configuration; `analyzer-profile.json` selects the independent grading runner/model; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning for execution. The analyzer default is also the Codebelt Reference profile, independent of the selected executor, and can be overridden with `-AnalyzerRunner`, `-AnalyzerModel`, and `-AnalyzerReasoningEffort`; preparation validates executor and analyzer models model-free before writing profiles. Offline deterministic tests may inject separate executor and analyzer catalogs with `-ModelCatalogPath` and `-AnalyzerModelCatalogPath`, but real preparation without catalogs still fails closed when discovery cannot verify the selected model. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, invokes the package-local Phase 2 analyzer controller, then finalizes reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: -Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. +Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, Phase 2 controller, grading application, and report adapter validate that ledger and never re-bless changed bytes. Phase 2 writes `phase2-state.json`, validator/analyzer evidence, `grading-freeze.json`, and the derived package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`) with structured `output`, `transcript`, and `validator` evidence refs. `apply-eval-grading.ps1` projects only frozen grading decisions onto canonical results, and `finalize-eval-package.ps1` owns the deterministic completion boundary: it succeeds only after validating Phase 1, the bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical grading, and all four report artifacts. A changed raw file or changed Phase 2 artifact requires a fresh authorized execution path, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. ```powershell pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot @@ -24,9 +28,11 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved to a runner; `-Runner github-copilot` uses the Codebelt Reference model, `-Runner codex` uses the Codex default, and `-Runner opencode` requires an explicit discovered `provider/model` selector. Direct noninteractive use without `-Runner` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness and fails immediately with the supported runner IDs when `-Runner` is omitted. Use `-CodebeltReference` only when you want the script to select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, normalized assertion domains, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the orchestrator invokes `invoke-phase2-analyzer.ps1`; it does not grade assertions itself, choose a grader model, retry failed analyzer workers, or write `grading.json`. The Phase 2 controller supplies each analyzer worker only that arm's expected output, unresolved semantic assertions, frozen output/transcript evidence, and packaged `tools/skill-creator/agents/grader.md`; paired arms, sibling evals, global skills, and candidate skill content are excluded unless a future assertion explicitly requires them. Missing telemetry is displayed as unavailable rather than estimated, and first-party reports separate execution usage/cost from analyzer usage/cost. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. -The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. `external_handoff` means authorization is complete: do not ask again. The helper encodes that as `confirmation_required = false`, `dispatch_immediately = true`, and `max_new_external_orchestrators = 1`. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. After launch, keep that same handle across bounded native waits; an empty wait or timeout means still running, so wait again on the same handle until a true terminal `completed` or `failed` result exists. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). + +The roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates package-local commands; the Phase 2 analyzer controller owns validator resolution, analyzer worker dispatch, and deterministic grading merge; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. @@ -131,7 +137,7 @@ Each `SKILL.md` description is lean activation metadata. The catalog below expla | Skill | Description | |-------|-------------| -| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | +| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping of tracked and non-ignored untracked files unless narrowed, exact path reconciliation before staging, and a final remaining-change check, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | | [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | @@ -278,6 +284,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith Commit messages are the most-read documentation in any codebase — yet they're usually an afterthought. "fix stuff", "wip", "address PR feedback" tells you nothing six months later. Writing good commits takes discipline, and when you're in flow, it's the first thing that slips. +**git-visual-commits** includes staged, unstaged, and non-ignored untracked files by default. It inventories individual files inside new directories, reconciles every path with the commit plan and staged changes, and checks for leftovers before reporting completion. Explicitly narrowed requests preserve excluded changes. + **git-visual-commits** handles the entire commit workflow— staging, diffing, crafting the message, choosing the right emoji — so every commit is consistent and meaningful without breaking your flow. Whether the agent authors the commit (`git bot commit`), you do (`git commit`), or you worked on it together (`git our commit`), the quality is the same. - **Gitmoji-first** — visual commit categories that are scannable at a glance diff --git a/scripts/eval-git-workspace.ps1 b/scripts/eval-git-workspace.ps1 new file mode 100644 index 0000000..0394705 --- /dev/null +++ b/scripts/eval-git-workspace.ps1 @@ -0,0 +1,52 @@ +# Declarative, local-only Git scenarios. No hooks, remotes, or model calls run. +function Assert-EvalGitScenario { + param([object]$Scenario) + if ($Scenario -is [bool]) { return } + if ($Scenario -isnot [pscustomobject]) { throw 'workspace.git must be a boolean or a declarative Git scenario.' } + $allowed = @('base_branch', 'feature_branch', 'commits') + foreach ($name in $Scenario.PSObject.Properties.Name) { if ($name -notin $allowed) { throw "Unknown workspace.git field '$name'." } } + foreach ($name in $allowed) { if ($name -notin $Scenario.PSObject.Properties.Name) { throw "workspace.git requires '$name'." } } + foreach ($branch in @($Scenario.base_branch, $Scenario.feature_branch)) { + if ($branch -isnot [string] -or $branch -notmatch '^[a-zA-Z0-9][a-zA-Z0-9/_-]*$' -or $branch.Contains('//') -or $branch.EndsWith('/')) { throw 'Invalid scenario branch name.' } + } + if ($Scenario.base_branch -eq $Scenario.feature_branch -or @($Scenario.commits).Count -lt 1) { throw 'Git scenario needs distinct base/feature branches and feature commits.' } + foreach ($commit in $Scenario.commits) { + if ([string]::IsNullOrWhiteSpace([string]$commit.message) -or $commit.files -isnot [pscustomobject]) { throw 'Each Git scenario commit requires message and files.' } + foreach ($name in $commit.PSObject.Properties.Name) { if ($name -notin @('message', 'files')) { throw "Unknown commit field '$name'." } } + foreach ($file in $commit.files.PSObject.Properties) { + if ($file.Name -notmatch '^[a-zA-Z0-9_][a-zA-Z0-9_./-]*$' -or $file.Name -match '(^|/)(\.\.?|\.git|bin|obj)(/|$)' -or $file.Name.EndsWith('/')) { throw "Unsafe Git scenario file '$($file.Name)'." } + if ($null -ne $file.Value -and $file.Value -isnot [string]) { throw 'Git scenario file content must be a string or null (delete).' } + } + } +} + +function Add-EvalGitScenario { + param([string]$RepoDirectory, [object]$Scenario) + Assert-EvalGitScenario -Scenario $Scenario + if ($Scenario -is [bool]) { return } + $identity = @('-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', '-c', 'core.autocrlf=false', '-c', 'core.hooksPath=') + function Invoke-ScenarioGit { param([string[]]$Arguments) + & git @identity -C $RepoDirectory @Arguments 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Git scenario command failed: $($Arguments -join ' ')" } + } + $oldAuthor = $env:GIT_AUTHOR_DATE; $oldCommitter = $env:GIT_COMMITTER_DATE + try { + $env:GIT_AUTHOR_DATE = '2020-01-02T00:00:00Z'; $env:GIT_COMMITTER_DATE = $env:GIT_AUTHOR_DATE + Invoke-ScenarioGit -Arguments @('branch', '-m', [string]$Scenario.base_branch) + # Local tracking refs and symbolic HEAD exercise default resolution + # without a network remote or paths back to the package. + Invoke-ScenarioGit -Arguments @('update-ref', "refs/remotes/origin/$($Scenario.base_branch)", 'HEAD') + Invoke-ScenarioGit -Arguments @('symbolic-ref', 'refs/remotes/origin/HEAD', "refs/remotes/origin/$($Scenario.base_branch)") + Invoke-ScenarioGit -Arguments @('checkout', '-b', [string]$Scenario.feature_branch, '--quiet') + foreach ($commit in $Scenario.commits) { + foreach ($file in $commit.files.PSObject.Properties) { + $path = [IO.Path]::GetFullPath((Join-Path $RepoDirectory $file.Name)) + if (-not $path.StartsWith([IO.Path]::GetFullPath($RepoDirectory) + [IO.Path]::DirectorySeparatorChar, [StringComparison]::OrdinalIgnoreCase)) { throw 'Git scenario path escaped repo.' } + if ($null -eq $file.Value) { if (Test-Path -LiteralPath $path) { Remove-Item -LiteralPath $path -Force } } + else { [void][IO.Directory]::CreateDirectory((Split-Path -Parent $path)); [IO.File]::WriteAllText($path, $file.Value, [Text.UTF8Encoding]::new($false)) } + } + Invoke-ScenarioGit -Arguments @('add', '-A') + Invoke-ScenarioGit -Arguments @('commit', '--quiet', '-m', [string]$commit.message) + } + } finally { $env:GIT_AUTHOR_DATE = $oldAuthor; $env:GIT_COMMITTER_DATE = $oldCommitter } +} diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html index 4881def..519a31a 100644 --- a/scripts/eval-report-template.html +++ b/scripts/eval-report-template.html @@ -266,6 +266,16 @@ function renderBenchmark() { const evals = DATA.evals || []; let html = '

Benchmark

Paired results are shown side by side. A missing metric means the harness did not record it; no value is estimated.

'; + const usage = DATA.metadata && DATA.metadata.usage ? DATA.metadata.usage : {}; + const executionUsage = usage.execution || {}; + const analyzerUsage = usage.analyzer || {}; + const display = (value, formatter = num) => value == null || value === '' ? 'unavailable' : formatter(value); + html += '
Execution cost / usage
'; + html += ''; + html += '
RunnerModelWorkersDurationTotal tokensInput tokensOutput tokensCost
' + esc(executionUsage.runner || 'unavailable') + '' + esc(executionUsage.model || 'unavailable') + '' + display(executionUsage.worker_count) + '' + display(executionUsage.duration_seconds, seconds) + '' + display(executionUsage.total_tokens) + '' + display(executionUsage.input_tokens) + '' + display(executionUsage.output_tokens) + '' + display(executionUsage.cost, money) + '
'; + html += '
Analyzer cost / usage

Analyzer usage is a separate Phase 2 stratum; it is not added to execution tokens or cost.

'; + html += ''; + html += '
RunnerModelProfile hashWorkersFailedDurationTotal tokensInput tokensOutput tokensCost
' + esc(analyzerUsage.runner || 'unavailable') + '' + esc(analyzerUsage.model || 'unavailable') + '' + esc(analyzerUsage.profile_sha256 || 'unavailable') + '' + display(analyzerUsage.worker_count) + '' + display(analyzerUsage.failed_worker_count) + '' + display(analyzerUsage.duration_seconds, seconds) + '' + display(analyzerUsage.total_tokens) + '' + display(analyzerUsage.input_tokens) + '' + display(analyzerUsage.output_tokens) + '' + display(analyzerUsage.cost, money) + '
'; html += '
Aggregate comparison
'; html += summaryRow('Pass rate', summaryMetric(evals, 'with_skill', passRate), summaryMetric(evals, 'without_skill', passRate), v => num(v, 2), true); html += summaryRow('Turns', summaryMetric(evals, 'with_skill', r => r && r.metrics.turns), summaryMetric(evals, 'without_skill', r => r && r.metrics.turns), v => num(v, 1)); diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 new file mode 100644 index 0000000..fc8025f --- /dev/null +++ b/scripts/eval-request.ps1 @@ -0,0 +1,331 @@ +<# +.SYNOPSIS + Deterministic helpers for the agent's explicit eval request workflow. Dot-source to use. +.DESCRIPTION + These helpers never launch a model. The interactive host consumes Get-EvalHandoff's + machine-readable decision and, when action=external_handoff, must treat authorization as + complete: confirmation_required=false, dispatch_immediately=true, and exactly one fresh + external Eval Orchestrator may be created. After launch, New-ExternalEvalOrchestratorState and + Update-ExternalEvalOrchestratorState model the outer host lifecycle: keep the same native + handle, treat a bounded wait with no terminal result as still running, and wait again on that + same handle until a true terminal completed/failed status exists. Do not wire this decision to + CI, hooks, preparation, validation, or completion-gate execution. For GitHub Copilot CLI, + task + general-purpose delegation is a valid external-orchestrator capability for this one-shot + handoff. +#> +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'eval-runners/runner-common.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/package-integrity.ps1') + +function Assert-PreparedEvalHandoffPackage { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath + ) + + $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path + if ([IO.Path]::GetFileName($path) -cne 'RUN-THIS.prompt.md') { + throw 'Handoff requires the prepared RUN-THIS.prompt.md file.' + } + + $package = Split-Path -Parent $path + try { + $manifestPath = Resolve-ManifestDeclaredPath -IterationDirectory $package -RelativePath 'manifest.json' -FieldName 'manifest.json' -Kind File -RequireExists + $manifest = Read-RunnerJson -Path $manifestPath + if ([string](Get-JsonProperty -Object $manifest -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-package/2') { + throw "manifest.json must declare 'codebeltnet/agentic/eval-package/2'." + } + if ([string](Get-JsonProperty -Object $manifest -Name 'execution' -Default '') -ne 'runner_handoff') { + throw "manifest.execution must be 'runner_handoff'." + } + $runnerPrompt = [string](Get-JsonProperty -Object $manifest -Name 'runner_prompt' -Default '') + if ([string]::IsNullOrWhiteSpace($runnerPrompt)) { + throw 'manifest.json must declare runner_prompt.' + } + $runnerPromptHash = [string](Get-JsonProperty -Object $manifest -Name 'runner_prompt_sha256' -Default '') + if ([string]::IsNullOrWhiteSpace($runnerPromptHash)) { + throw 'manifest.json must declare runner_prompt_sha256.' + } + if (-not (Test-Sha256 -Value $runnerPromptHash) -or $runnerPromptHash -cne $runnerPromptHash.ToLowerInvariant()) { + throw 'manifest.runner_prompt_sha256 must be a lowercase SHA-256.' + } + $resolvedPrompt = Resolve-ManifestDeclaredPath -IterationDirectory $package -RelativePath $runnerPrompt -FieldName 'runner_prompt' -Kind File -RequireExists + $comparison = if ($IsWindows) { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } + if (-not [string]::Equals([System.IO.Path]::GetFullPath($resolvedPrompt), [System.IO.Path]::GetFullPath($path), $comparison)) { + throw 'The supplied RUN-THIS.prompt.md is not the manifest-declared runner_prompt.' + } + $currentRunnerPromptHash = Get-Sha256HexFromFile -Path $resolvedPrompt + if ($currentRunnerPromptHash -cne $runnerPromptHash) { + throw 'manifest.runner_prompt_sha256 does not match the current RUN-THIS.prompt.md bytes. Requires a fresh package.' + } + [void](Get-ManifestRunRecords -IterationDirectory $package -Manifest $manifest) + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $package -Manifest $manifest) + [void](Assert-PackageRunnerIdentity -IterationDirectory $package -Manifest $manifest) + } catch { + throw "Handoff requires a valid prepared eval package produced by this repository: $($_.Exception.Message)" + } + + return [pscustomobject]@{ + PromptPath = $path + Package = $package + } +} + +function New-EvalHandoffDecision { + [CmdletBinding()] + param( + [Parameter(Mandatory)][ValidateSet('manual_handoff', 'external_handoff', 'already_started')][string]$Action, + [Parameter(Mandatory)][string]$PromptPath, + [Parameter(Mandatory)][string]$Reason, + [switch]$UserAuthorized, + [switch]$HostCanDelegateFreshOrchestrator, + [switch]$DispatchImmediately, + [int]$MaxNewExternalOrchestrators = 0, + [switch]$SameHandleRequired + ) + + $pendingWaitAction = if ($SameHandleRequired) { 'wait_same_handle_again' } else { 'none' } + $terminalStatuses = if ($SameHandleRequired) { @('completed', 'failed') } else { @() } + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-handoff-decision/1' + action = $Action + prompt_path = $PromptPath + reason = $Reason + user_authorized = $UserAuthorized.IsPresent + host_can_delegate_fresh_orchestrator = $HostCanDelegateFreshOrchestrator.IsPresent + confirmation_required = $false + dispatch_immediately = $DispatchImmediately.IsPresent + max_new_external_orchestrators = $MaxNewExternalOrchestrators + same_handle_required = $SameHandleRequired.IsPresent + pending_wait_action = $pendingWaitAction + terminal_statuses = @($terminalStatuses) + } +} + +function Assert-ExternalEvalHandoffDecision { + [CmdletBinding()] + param( + [Parameter(Mandatory)][object]$Decision + ) + + $schema = [string](Get-JsonProperty -Object $Decision -Name 'schema' -Default '') + if ($schema -ne 'codebeltnet/agentic/eval-handoff-decision/1') { + throw "External Eval handoff requires schema 'codebeltnet/agentic/eval-handoff-decision/1'." + } + if ([string](Get-JsonProperty -Object $Decision -Name 'action' -Default '') -ne 'external_handoff') { + throw "External Eval handoff requires action 'external_handoff'." + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'user_authorized' -Default $false)) { + throw 'External Eval handoff requires explicit user authorization.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'host_can_delegate_fresh_orchestrator' -Default $false)) { + throw 'External Eval handoff requires fresh-context host delegation capability.' + } + if ([bool](Get-JsonProperty -Object $Decision -Name 'confirmation_required' -Default $true)) { + throw 'External Eval handoff must not require another user confirmation.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'dispatch_immediately' -Default $false)) { + throw 'External Eval handoff must dispatch immediately.' + } + if ([int](Get-JsonProperty -Object $Decision -Name 'max_new_external_orchestrators' -Default 0) -ne 1) { + throw 'External Eval handoff permits exactly one fresh external Eval Orchestrator.' + } + if (-not [bool](Get-JsonProperty -Object $Decision -Name 'same_handle_required' -Default $false)) { + throw 'External Eval handoff must retain the same native Orchestrator handle across waits.' + } + if ([string](Get-JsonProperty -Object $Decision -Name 'pending_wait_action' -Default '') -ne 'wait_same_handle_again') { + throw "External Eval handoff must map a non-terminal bounded wait to 'wait_same_handle_again'." + } + $terminalStatuses = @((Get-JsonProperty -Object $Decision -Name 'terminal_statuses' -Default @())) + if ($terminalStatuses.Count -ne 2 -or $terminalStatuses[0] -ne 'completed' -or $terminalStatuses[1] -ne 'failed') { + throw "External Eval handoff must declare terminal statuses 'completed' and 'failed'." + } +} + +function New-ExternalEvalOrchestratorState { + [CmdletBinding(DefaultParameterSetName = 'Launched')] + param( + [Parameter(Mandatory)][object]$Decision, + [Parameter(Mandatory, ParameterSetName = 'Launched')][string]$NativeHandle, + [Parameter(Mandatory, ParameterSetName = 'LaunchFailed')][switch]$LaunchFailed, + [Parameter(ParameterSetName = 'LaunchFailed')][string]$FailureReason = 'The external Eval Orchestrator launch failed before a native handle existed.' + ) + + Assert-ExternalEvalHandoffDecision -Decision $Decision + $promptPath = [string](Get-JsonProperty -Object $Decision -Name 'prompt_path' -Default '') + if ($PSCmdlet.ParameterSetName -eq 'LaunchFailed') { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'launch_failed' + terminal = $true + native_handle = $null + wait_count = 0 + same_handle_required = $false + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = $FailureReason + } + } + + if ([string]::IsNullOrWhiteSpace($NativeHandle)) { + throw 'A launched external Eval Orchestrator requires a non-empty native handle.' + } + + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'running' + terminal = $false + native_handle = $NativeHandle + wait_count = 0 + same_handle_required = $true + pending_wait_action = 'wait_same_handle_again' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = 'The external Eval Orchestrator exists. Keep this same native handle and repeat bounded waits until a true terminal completed/failed result exists.' + } +} + +function Update-ExternalEvalOrchestratorState { + [CmdletBinding()] + param( + [Parameter(Mandatory)][object]$State, + [Parameter(Mandatory)][string]$NativeHandle, + [Parameter(Mandatory)][ValidateSet('pending', 'completed', 'failed')][string]$NativeWaitStatus, + [object]$TerminalResult = $null + ) + + $schema = [string](Get-JsonProperty -Object $State -Name 'schema' -Default '') + if ($schema -ne 'codebeltnet/agentic/eval-orchestrator-lifecycle/1') { + throw "External Eval Orchestrator state requires schema 'codebeltnet/agentic/eval-orchestrator-lifecycle/1'." + } + if ([bool](Get-JsonProperty -Object $State -Name 'terminal' -Default $false)) { + throw 'A terminal external Eval Orchestrator state cannot be waited again.' + } + $expectedHandle = [string](Get-JsonProperty -Object $State -Name 'native_handle' -Default '') + if ([string]::IsNullOrWhiteSpace($expectedHandle)) { + throw 'A running external Eval Orchestrator state must retain its native handle.' + } + if ($NativeHandle -cne $expectedHandle) { + throw "External Eval Orchestrator waits must retain the same native handle. Expected '$expectedHandle', received '$NativeHandle'." + } + + $waitCount = [int](Get-JsonProperty -Object $State -Name 'wait_count' -Default 0) + 1 + $promptPath = [string](Get-JsonProperty -Object $State -Name 'prompt_path' -Default '') + switch ($NativeWaitStatus) { + 'pending' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'running' + terminal = $false + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'wait_same_handle_again' + max_new_external_orchestrators = 0 + terminal_result = $null + reason = 'The bounded native wait returned no terminal Orchestrator result yet. Still running: wait again on this same handle and do not infer Phase 1 failure or create another Orchestrator.' + } + } + 'completed' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'completed' + terminal = $true + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $TerminalResult + reason = 'The same external Eval Orchestrator reached a true terminal completed state.' + } + } + 'failed' { + return [pscustomobject][ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestrator-lifecycle/1' + prompt_path = $promptPath + status = 'failed' + terminal = $true + native_handle = $expectedHandle + wait_count = $waitCount + same_handle_required = $true + pending_wait_action = 'none' + max_new_external_orchestrators = 0 + terminal_result = $TerminalResult + reason = 'The same external Eval Orchestrator reached a true terminal failed state.' + } + } + } +} + +function Get-EvalHandoff { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath, + [switch]$Yolo, + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + ) + + $validated = Assert-PreparedEvalHandoffPackage -PromptPath $PromptPath + $path = [string]$validated.PromptPath + $package = [string]$validated.Package + $decision = New-EvalHandoffDecision -Action 'manual_handoff' -PromptPath $path ` + -Reason 'Preparation complete; hand this file to an external Eval Orchestrator.' ` + -UserAuthorized:$Yolo.IsPresent -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent + if (-not $Yolo) { return [pscustomobject]$decision } + + # Reserve the handoff before the host delegates. An uncertain launch must never be retried. + $claim = Join-Path $package '.external-handoff-started' + if ((Test-Path -LiteralPath $claim) -or + (Test-Path -LiteralPath (Join-Path $package 'orchestration-state.json')) -or + (Test-Path -LiteralPath (Join-Path $package 'execution-freeze.json'))) { + return (New-EvalHandoffDecision -Action 'already_started' -PromptPath $path ` + -Reason 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent -SameHandleRequired) + } + if (-not $CanDelegateFreshOrchestrator) { + return (New-EvalHandoffDecision -Action 'manual_handoff' -PromptPath $path ` + -Reason 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$false) + } + try { + $stream = [IO.File]::Open($claim, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None) + $stream.Dispose() + } catch [IO.IOException] { + if (-not (Test-Path -LiteralPath $claim)) { throw } + return (New-EvalHandoffDecision -Action 'already_started' -PromptPath $path ` + -Reason 'A handoff was already reserved. Do not dispatch again.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator.IsPresent -SameHandleRequired) + } + return (New-EvalHandoffDecision -Action 'external_handoff' -PromptPath $path ` + -Reason 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; do not ask again, dispatch immediately, and keep the same native handle across bounded waits until its final result.' ` + -UserAuthorized:$true -HostCanDelegateFreshOrchestrator:$true -DispatchImmediately ` + -MaxNewExternalOrchestrators 1 -SameHandleRequired) +} + +function Invoke-EvalRequest { + [CmdletBinding()] + param( + [Parameter(Mandatory)][hashtable]$Preparation, + [switch]$Yolo, + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + ) + + $ErrorActionPreference = 'Stop' + # Splat existing preparation options unchanged; it alone resolves and verifies model policy. + # Collect every success before issuing any handoff, including multi-skill explicit requests. + if ($Preparation.ContainsKey('CollectResults')) { throw 'An eval request prepares packages; CollectResults is a separate forensic workflow.' } + $arguments = $Preparation.Clone() + $arguments.PassThru = $true + $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) + foreach ($path in $paths) { + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -CanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator + } +} diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index d3853fc..2b163c1 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -78,16 +78,7 @@ must validate that ledger; none of them can replace it or bless changed bytes. If a raw result or referenced artifact changes, the package is corrupted and requires a fresh Phase 1 execution. -The normal post-execution boundary is deterministic: the external Grader writes -only the package-root `grading.json` artifact (`codebeltnet/agentic/eval-grading/1`) -with exact assertion identities and `passed`/`evidence` decisions. The shared -`apply-eval-grading.ps1` helper projects those decisions onto canonical -`result.json` files and verifies that every non-grading field is unchanged. -`finalize-eval-package.ps1` then validates the freeze, bridge, canonical results, -and complete grading, invokes the existing report adapter, and fails unless -`report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` -are all non-empty. A prose success message cannot substitute for its JSON -success summary. +The normal post-execution boundary is deterministic: `invoke-phase2-analyzer.ps1` resolves validator-domain assertions first, dispatches one fresh analyzer worker per remaining semantic arm with the persisted `analyzer-profile.json`, validates the returned runner/model/session evidence, writes `phase2-state.json`, freezes `grading-freeze.json`, and derives package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`) with exact assertion identities, structured evidence domains and evidence refs. The shared `apply-eval-grading.ps1` helper projects those frozen decisions onto canonical `result.json` files and verifies that every non-grading field is unchanged. `finalize-eval-package.ps1` then validates Phase 1, the bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical results, and complete grading, invokes the existing report adapter, and fails unless `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` are all non-empty. A prose success message or handcrafted `grading.json` cannot substitute for its JSON success summary. The delegation contract has three distinct evidence levels: @@ -139,6 +130,14 @@ Operational permission, native skill isolation, and filesystem isolation are dis Native delegation mechanisms: +Copilot execution requires an allowlisted physical temporary projection outside the logical package and source checkout ancestry. Only the arm's staged repository, prepared home, prompt and candidate (with-skill only) are copied. Repository `AGENTS.md` and `.github/copilot-instructions.md` remain enabled and identical in both arms. Ambient ancestor instructions and filesystem links make projection preflight fail closed. Runtime repository changes are synchronized back after link validation; captures remain parent-owned in the logical arm. Execution evidence records both boundaries. On hosts without an outer sandbox this prevents package discovery through the working-directory ancestry, but does not deny arbitrary absolute host paths: projection is not OS confinement. Structured tool events are a conservative contradiction detector; forbidden package/source accesses make the arm incompatible, and the bridge independently rejects contradictory completed transcripts. Native events remain available for diagnosis. + +Copilot `session.usage_checkpoint` billing counters are cumulative: the last checkpoint wins, including across resumed turns. `promptCacheBreakState` entries are per-call cache snapshots, deduplicated by native `model_call_id`; exposed prompt/cache buckets are used only when `assistant.usage` is absent. Tool schema tokens and native snapshots remain in evidence, not generated-token buckets. Missing output tokens stay unavailable, and premium requests/nano-AI units are never converted to currency. See [GitHub's event contract](https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/streaming-events). + +Phase 2 workers receive only the exact packaged `tools/skill-creator/agents/grader.md`, the one arm's expected output, unresolved semantic assertions, frozen output lines and any explicitly required frozen transcript/artifact evidence. They do not receive the paired arm, sibling evals, benchmark summary, previous grades, user-global skills, candidate skill content, or an ambient repository checkout. Deterministic validation still checks provenance, source membership, verbatim quote presence, nonempty evidence, repeated/generic PASS reasons, analyzer profile hash, raw analyzer transcript hash, grading fragment hash, and worker/session uniqueness; it does not replace semantic analyzer judgement for unresolved semantic assertions. Execution evidence remains immutable and finalization remains exactly once. + +Historical integrity note: `dotnet-change-impact` Copilot iteration 4 is contaminated and must not be used as effectiveness evidence. Its missing `.external-handoff-started` is consistent with manual handoff, but the package records do not establish the original user request. The current helper reserves the marker before every external handoff and contains no removal path; no reservation-policy change is justified from absence alone. Codex iteration 2 and OpenCode iteration 3 have reservations. New protocol code applies only to newly prepared iterations. + - GitHub Copilot: runner-owned behavioral transport. The runner starts one fresh Copilot CLI session per eval execution (`copilot -C --model --output-format json`, prompt on stdin) and captures that @@ -300,12 +299,13 @@ in both paired arms. Personal Copilot configuration is excluded by run-local `COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; the runner does not copy the normal `.copilot` directory. Authentication prefers explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is -present, the trusted runner may resolve `gh auth token` outside the worker and -inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` -is never forwarded into the evaluated worker. `--secret-env-vars` removes every -listed token variable from shell and MCP child environments. Preflight does not -make a model request and therefore reports native keychain/service readiness as -conditional rather than claiming successful remote authentication. Codex's +present, the trusted runner resolves `gh auth token` outside the worker across +documented/default config candidates and injects only that token as a protected +environment variable. Host `GH_CONFIG_DIR` is never forwarded into the evaluated +worker. `--secret-env-vars` removes every listed token variable from shell and +MCP child environments. If neither explicit token variables nor trusted GitHub +CLI token fallback is resolvable, preflight fails closed before Phase 1 model +execution starts. Codex's compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox danger-full-access`; subscription eval arms use the runner-owned app-server path described above. It does not combine explicit sandbox selection with diff --git a/scripts/eval-runners/apply-eval-grading.ps1 b/scripts/eval-runners/apply-eval-grading.ps1 index c7edfbd..f819e41 100644 --- a/scripts/eval-runners/apply-eval-grading.ps1 +++ b/scripts/eval-runners/apply-eval-grading.ps1 @@ -20,6 +20,7 @@ Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot 'manifest-paths.ps1') . (Join-Path $PSScriptRoot 'execution-freeze.ps1') . (Join-Path $PSScriptRoot 'eval-grading-contract.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') function Write-GradingResultJson { param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) @@ -45,6 +46,7 @@ try { throw "Canonical result validation failed before grading application: $([string]::Join(' ', @($bridgeOutput | ForEach-Object { [string]$_ })))" } + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath) $gradingValidation = Assert-EvalGradingContract -IterationDirectory $iteration -GradingPath $GradingPath $records = @($gradingValidation.Records) $expected = $gradingValidation.Expected @@ -59,7 +61,15 @@ try { $assertions = @(Get-EvalMetadataAssertions -Record $record) for ($index = 0; $index -lt $assertions.Count; $index++) { $entry = $validated["$($record.EvalId)|$($record.Configuration)|$index"] - $newGrading.Add([ordered]@{ text = [string]$entry.assertion; passed = [bool]$entry.passed; evidence = [string]$entry.evidence }) + $newGrading.Add([ordered]@{ + text = [string]$entry.assertion + passed = [bool]$entry.passed + evidence = [string]$entry.evidence + evidence_domain = [string]$entry.evidence_domain + evidence_refs = @($entry.evidence_refs) + reason = [string]$entry.reason + source = [string](Get-JsonProperty -Object $entry -Name 'source' -Default '') + }) } $beforeNonGrading = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $canonical -PropertyName 'grading') # Keep the parsed canonical values as-is. ConvertFrom-Json reparses diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index d46e8de..c04cac3 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -259,6 +259,9 @@ try { $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) } + . (Join-Path $PSScriptRoot 'github-copilot/isolation.ps1') + Assert-CopilotCapturedBoundary -Raw $raw -RunData $runData + if ($RequireNativeDelegation) { if ([string]$raw.status -eq 'incompatible') { throw 'An incompatible native-worker arm is diagnostic only and cannot be bridged into a gradeable canonical result.' diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index 42843ee..4a87407 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -184,6 +184,108 @@ function Get-CodexBaseSkillSessionConfigValues { ) } +function Get-CodexSanitizedShellPath { + param( + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [AllowNull()][string]$GitDirectory = $null, + [AllowNull()][string]$WindowsRoot = $null + ) + + $separator = if ($Platform -eq 'windows') { ';' } else { ':' } + $entries = [System.Collections.Generic.List[string]]::new() + if ($Platform -eq 'windows') { + $root = if ([string]::IsNullOrWhiteSpace($WindowsRoot)) { [Environment]::GetEnvironmentVariable('SystemRoot') } else { $WindowsRoot } + if ([string]::IsNullOrWhiteSpace($root)) { $root = 'C:\Windows' } + $entries.Add((Join-CodexTargetPath -Root $root -Segments @('System32') -Platform windows)) + } else { + foreach ($path in @('/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin')) { + $entries.Add($path) + } + } + if (-not [string]::IsNullOrWhiteSpace($GitDirectory)) { $entries.Add([string]$GitDirectory) } + + $deduplicated = [System.Collections.Generic.List[string]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($entry in @($entries)) { + $value = [string]$entry + if ([string]::IsNullOrWhiteSpace($value)) { continue } + $normalized = $value.Trim().TrimEnd('\', '/') + if ([string]::IsNullOrWhiteSpace($normalized)) { continue } + if ($seen.Add($normalized)) { $deduplicated.Add($normalized) } + } + return [string]::Join($separator, [string[]]@($deduplicated.ToArray())) +} + +function Join-CodexTargetPath { + param( + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Root, + [Parameter(Mandatory = $true)][AllowEmptyString()][string[]]$Segments, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName) + ) + + $separator = if ($Platform -eq 'windows') { '\' } else { '/' } + $result = ([string]$Root).TrimEnd('\', '/') + foreach ($segment in @($Segments)) { + $value = ([string]$segment).Trim('\', '/') + if ([string]::IsNullOrWhiteSpace($value)) { continue } + if ([string]::IsNullOrWhiteSpace($result)) { + $result = $value + } else { + $result = $result + $separator + $value + } + } + return $result +} + +function Get-CodexShellEnvironmentPolicySet { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [AllowNull()][object]$GitCommandInfo = $null, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [AllowNull()][string]$WindowsRoot = $null + ) + + $gitDirectory = $null + if ($null -ne $GitCommandInfo) { + $gitSource = [string](Get-JsonProperty -Object $GitCommandInfo -Name 'Source' -Default '') + if (-not [string]::IsNullOrWhiteSpace($gitSource)) { $gitDirectory = Split-Path -Parent $gitSource } + } + $windowsRoot = if ([string]::IsNullOrWhiteSpace($WindowsRoot)) { [Environment]::GetEnvironmentVariable('SystemRoot') } else { $WindowsRoot } + $values = [ordered]@{ + PATH = Get-CodexSanitizedShellPath -Platform $Platform -GitDirectory $gitDirectory -WindowsRoot $windowsRoot + } + if ($Platform -eq 'windows') { + if ([string]::IsNullOrWhiteSpace($windowsRoot)) { $windowsRoot = 'C:\Windows' } + $values.SystemRoot = $windowsRoot + $values.ComSpec = Join-CodexTargetPath -Root $windowsRoot -Segments @('System32', 'cmd.exe') -Platform windows + $values.PATHEXT = '.COM;.EXE;.BAT;.CMD' + } + return $values +} + +function Test-CodexRunUsesGitWorkspace { + param([Parameter(Mandatory = $true)][object]$Run) + + return [bool](Get-JsonProperty -Object $Run -Name 'GitWorkspace' -Default $false) +} + +function Add-CodexShellEnvironmentPolicyConfigArguments { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Arguments, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$ShellEnvironmentSet, + [string]$SwitchName = '-c' + ) + + $Arguments.Add($SwitchName) + $Arguments.Add('shell_environment_policy.inherit=none') + foreach ($key in @($ShellEnvironmentSet.Keys)) { + $name = [string]$key + if ($name -notmatch '^[A-Za-z_][A-Za-z0-9_]*$') { throw "Unsupported Codex shell environment key '$name'." } + $Arguments.Add($SwitchName) + $Arguments.Add(('shell_environment_policy.set.{0}={1}' -f $name, (ConvertTo-CodexConfigStringLiteral -Value ([string]$ShellEnvironmentSet[$key])))) + } +} + function New-CodexSkillSuppressionSelector { param( [Parameter(Mandatory = $true)][object]$SkillEntry, @@ -268,12 +370,13 @@ function Add-CodexSessionConfigArguments { [AllowEmptyCollection()][object[]]$AmbientSkillEntries = @(), [string]$SwitchName = '-c', [switch]$IncludeShellEnvironmentPolicy, + [System.Collections.IDictionary]$ShellEnvironmentSet = $null, [bool]$IncludeNativeSkillSuppression = $true ) if ($IncludeShellEnvironmentPolicy) { - $Arguments.Add($SwitchName) - $Arguments.Add('shell_environment_policy.inherit=none') + if ($null -eq $ShellEnvironmentSet) { $ShellEnvironmentSet = [ordered]@{} } + Add-CodexShellEnvironmentPolicyConfigArguments -Arguments $Arguments -ShellEnvironmentSet $ShellEnvironmentSet -SwitchName $SwitchName } if (-not $IncludeNativeSkillSuppression) { return } foreach ($value in @(Get-CodexSkillSessionConfigValues -CandidateSkillName $CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries)) { @@ -587,6 +690,7 @@ function New-CodexNativeSkillProbeContext { ) $environmentKeys = @($Environment.Keys | ForEach-Object { [string]$_ } | Sort-Object) + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } return [ordered]@{ purpose = $Purpose codex_executable = [string]$CommandInfo.Source @@ -602,6 +706,7 @@ function New-CodexNativeSkillProbeContext { tmp = if ($Environment.Contains('TMP')) { [string]$Environment['TMP'] } else { $null } filtered_environment_keys = @($environmentKeys) shell_environment_policy = 'inherit=none' + shell_environment_policy_set_path = (Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo)['PATH'] app_server_arguments = @($Arguments) per_skill_suppression_selector_count = @($AmbientSkillEntries).Count native_skill_suppression_enabled = @($Arguments | Where-Object { [string]$_ -like 'skills.*' }).Count -gt 0 @@ -623,7 +728,9 @@ function Invoke-CodexAppServerSkillsListProbe { $deadline = $start.AddSeconds([Math]::Max(1, $TimeoutSeconds)) $arguments = [System.Collections.Generic.List[string]]::new() foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--strict-config', '--stdio')) { $arguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -IncludeShellEnvironmentPolicy -IncludeNativeSkillSuppression:$IncludeNativeSkillSuppression + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo + Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet -IncludeNativeSkillSuppression:$IncludeNativeSkillSuppression $psi = [System.Diagnostics.ProcessStartInfo]::new() $psi.FileName = $CommandInfo.FileName @@ -1113,7 +1220,9 @@ function Invoke-CodexAppServer { $suppressionSelectors = @(New-CodexNativeSkillSuppressionSelectors -AmbientSkillEntries $ambientSkillEntries) $appServerArguments = [System.Collections.Generic.List[string]]::new() foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--strict-config', '--stdio')) { $appServerArguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $appServerArguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $ambientSkillEntries -IncludeShellEnvironmentPolicy + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo + Add-CodexSessionConfigArguments -Arguments $appServerArguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $ambientSkillEntries -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet foreach ($argument in @($appServerArguments)) { [void]$psi.ArgumentList.Add([string]$argument) } # Shared progress context for the app-server protocol exchange. When the @@ -2087,11 +2196,13 @@ function New-CodexCliArguments { $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $outputArgument = Get-SandboxVisiblePath -HostPath $LastResponsePath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $gitCommandInfo = if (Test-CodexRunUsesGitWorkspace -Run $Inputs.Run) { Resolve-ExternalCommand -Name 'git' } else { $null } + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $gitCommandInfo -Platform $VisiblePlatform $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--strict-config', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'danger-full-access', '--config', 'shell_environment_policy.inherit=none')) { + foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--strict-config', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'danger-full-access')) { $arguments.Add([string]$argument) } - Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -SwitchName '--config' + Add-CodexSessionConfigArguments -Arguments $arguments -CandidateSkillName $Inputs.Run.CandidateSkillName -AmbientSkillEntries $AmbientSkillEntries -SwitchName '--config' -IncludeShellEnvironmentPolicy -ShellEnvironmentSet $shellEnvironmentSet foreach ($argument in @('--output-last-message', $outputArgument)) { $arguments.Add([string]$argument) } @@ -2207,6 +2318,40 @@ function Get-CodexCapabilityMap { return $capabilities } +function Invoke-CodexGitWorkspaceProbe { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$GitCommandInfo, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [int]$TimeoutSeconds = 30 + ) + + $shellEnvironmentSet = Get-CodexShellEnvironmentPolicySet -Inputs $Inputs -GitCommandInfo $GitCommandInfo -Platform $Platform + $sanitizedPath = [string]$shellEnvironmentSet['PATH'] + $environment = [ordered]@{} + foreach ($key in @($shellEnvironmentSet.Keys)) { $environment[$key] = [string]$shellEnvironmentSet[$key] } + $gitFileName = [IO.Path]::GetFileName([string]$GitCommandInfo.Source) + if ([string]::IsNullOrWhiteSpace($gitFileName)) { $gitFileName = 'git' } + try { + $process = Invoke-RunnerProcess -FileName $gitFileName -ArgumentList @('--version') -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $TimeoutSeconds + } catch { + return [pscustomobject]@{ + Available = $false + Detail = "git --version could not start with the sanitized Codex shell PATH: $($_.Exception.Message)" + SanitizedPath = $sanitizedPath + Process = $null + } + } + $stdout = [string]$process.Stdout + $available = -not $process.TimedOut -and $process.ExitCode -eq 0 -and $stdout -match '(?im)\bgit version\b' + return [pscustomobject]@{ + Available = $available + Detail = if ($available) { "git --version succeeded with the sanitized Codex shell PATH: $sanitizedPath" } else { "git --version failed with the sanitized Codex shell PATH (exit=$($process.ExitCode), timed_out=$($process.TimedOut))." } + SanitizedPath = $sanitizedPath + Process = $process + } +} + function Get-CodexPreflight { param([Parameter(Mandatory = $true)][object]$Inputs) @@ -2227,6 +2372,8 @@ function Get-CodexPreflight { $nativeWorkerObservation = $null $nativeSkillConfigObservation = $null $auth = Get-CodexAuthSource + $gitCommandInfo = $null + $gitWorkspaceProbe = $null if ($profile.Runner -ne 'codex') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") @@ -2244,6 +2391,21 @@ function Get-CodexPreflight { if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by codex.") } + if (Test-CodexRunUsesGitWorkspace -Run $run) { + $gitCommandInfo = Resolve-ExternalCommand -Name 'git' + if ($null -eq $gitCommandInfo) { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status failed -Detail 'gitWorkspace=true requires git, but Resolve-ExternalCommand could not locate git.')) + $reasons.Add('gitWorkspace=true requires git to be resolvable before Codex execution.') + } else { + $gitWorkspaceProbe = Invoke-CodexGitWorkspaceProbe -Inputs $Inputs -GitCommandInfo $gitCommandInfo -Platform $platform -TimeoutSeconds 30 + if ($gitWorkspaceProbe.Available) { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status passed -Detail $gitWorkspaceProbe.Detail)) + } else { + $checks.Add((New-PreflightCheck -Name 'git_workspace_tool_path' -Status failed -Detail $gitWorkspaceProbe.Detail)) + $reasons.Add('gitWorkspace=true requires git --version to succeed with the exact sanitized Codex child shell PATH.') + } + } + } if ($null -eq $commandInfo) { $reasons.Add('The Codex CLI executable is not available on PATH.') @@ -2365,11 +2527,11 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The selected transport starts an ephemeral thread and never supplies a resume, continue, or existing session identifier.')) if ($auth.Kind -eq 'subscription_file') { - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none. Native skill isolation is proven separately through skills/list discovery, ambient path suppression, behavioral skills/list verification, and runtime access evidence.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none and shell_environment_policy.set.PATH supplies only the sanitized runner tool path. Native skill isolation is proven separately through skills/list discovery, ambient path suppression, behavioral skills/list verification, and runtime access evidence.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "thread/start and turn/start set cwd to $($run.WorkingDirectoryPath); HOME and USERPROFILE remain staged under $($run.HomeDirectoryPath).")) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only auth.json is copied into a temporary auth-only CODEX_HOME and it is removed in finally; config.toml, skills, agents, sessions, memories, plugins, MCP configuration, and AGENTS.md are not copied. This does not claim hard filesystem confinement where none is available.')) } else { - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed. Child shell inheritance is disabled with shell_environment_policy.inherit=none and shell_environment_policy.set.PATH supplies only the sanitized runner tool path.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are not copied into the worker HOME.')) } @@ -2382,10 +2544,10 @@ function Get-CodexPreflight { $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction) { $mechanisms.Add('same-thread repeated turn/start for scripted interaction') } else { $mechanisms.Add('no session continuation') } } else { - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } $document = New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) @@ -2888,11 +3050,11 @@ function Invoke-CodexExecute { } $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + skills/list + thread/start + turn/start', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=dangerFullAccess', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } $continuationMechanism = if ($null -ne $Inputs.Run.Interaction) { 'same-thread repeated turn/start for scripted interaction' } else { 'no session continuation' } $mechanisms.Add($continuationMechanism) } else { - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--strict-config', '--ignore-user-config', '--ignore-rules', '--sandbox danger-full-access', 'shell_environment_policy.inherit=none', 'shell_environment_policy.set.PATH sanitized runner tool path', 'session skills.include_instructions=false', 'session skills.bundled.enabled=false', 'session skills.config ambient path disables', 'pre-turn debug prompt-input native-skill suppression proof', 'pre-turn skills/list ambient-state verification', 'runtime ambient skill access validation', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index f0bf233..0e05500 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -185,8 +185,8 @@ "prompt_fidelity": { "const": true }, "prompt_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, "terminal_result_capture": { "const": true }, - "paired_arm_visible": { "const": false }, - "grading_material_visible": { "const": false }, + "paired_arm_visible": { "type": "boolean" }, + "grading_material_visible": { "type": "boolean" }, "nested_model_execution": { "const": false }, "model_execution_count": { "const": 1 } } @@ -197,5 +197,12 @@ "warnings": { "type": "array", "items": { "type": "string" } }, "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, "attempt_count": { "const": 1 } - } + }, + "allOf": [{ + "if": { "properties": { "status": { "const": "completed" } } }, + "then": { "properties": { "evidence": { "properties": { "delegation": { "properties": { + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false } + } } } } } } + }] } diff --git a/scripts/eval-runners/contracts/grading.schema.json b/scripts/eval-runners/contracts/grading.schema.json index 23b4aac..bfb32b5 100644 --- a/scripts/eval-runners/contracts/grading.schema.json +++ b/scripts/eval-runners/contracts/grading.schema.json @@ -7,12 +7,26 @@ "required": ["schema", "grading"], "properties": { "schema": { "const": "codebeltnet/agentic/eval-grading/1" }, + "metadata": { + "type": "object", + "additionalProperties": true, + "properties": { + "schema": { "type": "string" }, + "generated_by": { "type": "string" }, + "analyzer_runner": { "type": "string" }, + "analyzer_model": { "type": "string" }, + "analyzer_reasoning_effort": { "type": ["string", "null"] }, + "analyzer_profile_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "phase2_state": { "type": "string" }, + "grading_freeze": { "type": "string" } + } + }, "grading": { "type": "array", "items": { "type": "object", "additionalProperties": false, - "required": ["eval_id", "eval_name", "configuration", "assertion_index", "assertion", "passed", "evidence"], + "required": ["eval_id", "eval_name", "configuration", "assertion_index", "assertion", "passed", "evidence", "evidence_domain", "evidence_refs", "reason"], "properties": { "eval_id": { "type": "integer", "minimum": 1 }, "eval_name": { "type": "string", "minLength": 1 }, @@ -20,7 +34,30 @@ "assertion_index": { "type": "integer", "minimum": 0 }, "assertion": { "type": "string", "minLength": 1 }, "passed": { "type": "boolean" }, - "evidence": { "type": "string" } + "evidence": { "type": "string", "minLength": 1, "pattern": "\\S" }, + "evidence_domain": { "enum": ["output", "transcript", "validator"] }, + "evidence_refs": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": true, + "required": ["artifact"], + "properties": { + "artifact": { "type": "string", "minLength": 1 }, + "domain": { "enum": ["output", "transcript", "validator"] }, + "start_line": { "type": "integer", "minimum": 1 }, + "end_line": { "type": "integer", "minimum": 1 }, + "quote": { "type": "string" }, + "rule": { "type": "string" }, + "version": { "type": "integer", "minimum": 1 }, + "event": { "type": ["string", "null"] }, + "passed": { "type": "boolean" } + } + } + }, + "reason": { "type": "string", "minLength": 1, "pattern": "\\S" }, + "source": { "enum": ["analyzer", "validator"] } } } } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index c1a7127..e55d80c 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -1,156 +1,212 @@ -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'runner-common.ps1') -} -if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'manifest-paths.ps1') -} -if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { - . (Join-Path $PSScriptRoot 'execution-freeze.ps1') -} - -function Get-EvalGradingSkeleton { - return [ordered]@{ - schema = (Get-RunnerSchemaNames).Grading - grading = @() - } -} - -function Get-EvalGradingEntryKey { - param([Parameter(Mandatory = $true)][object]$Entry) - - return "$(Get-JsonProperty -Object $Entry -Name 'eval_id' -Default 0)|$(Get-JsonProperty -Object $Entry -Name 'configuration' -Default '')|$(Get-JsonProperty -Object $Entry -Name 'assertion_index' -Default -1)" -} - -function Get-EvalMetadataAssertions { - param([Parameter(Mandatory = $true)][object]$Record) - - $metadata = Read-RunnerJson -Path $Record.MetadataPath - $assertions = @(Get-JsonProperty -Object $metadata -Name 'assertions' -Default @()) - if ($assertions.Count -eq 0) { throw "Metadata for '$($Record.EvalName)' declares no assertions." } - return @($assertions | ForEach-Object { [string]$_ }) -} - -function Assert-EvalGradingEntryShape { - param([Parameter(Mandatory = $true)][object]$Entry) - - $allowed = @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence') - foreach ($name in @(Get-JsonPropertyNames -Object $Entry)) { - if ($allowed -notcontains $name) { throw "grading.json entry contains unsupported field '$name'." } - } - foreach ($name in $allowed) { - if (-not (Test-JsonProperty -Object $Entry -Name $name)) { throw "grading.json entry is missing '$name'." } - } - $evalId = 0 - try { $evalId = [int]$Entry.eval_id } catch { throw 'grading.json eval_id must be an integer.' } - if ($evalId -lt 1) { throw 'grading.json eval_id must be positive.' } - $index = 0 - try { $index = [int]$Entry.assertion_index } catch { throw 'grading.json assertion_index must be an integer.' } - if ($index -lt 0 -or [double]$Entry.assertion_index -ne $index) { throw 'grading.json assertion_index must be a non-negative integer.' } - if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } - if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } - if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } - if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } -} - -function Assert-EvalGradingContract { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [string]$GradingPath = 'grading.json' - ) - - $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path - Assert-SafeRelativePath -RelativePath $GradingPath -FieldName 'grading path' - $gradingFullPath = Resolve-ContainedPath -BasePath $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File - if (-not (Test-Path -LiteralPath $gradingFullPath -PathType Leaf)) { - throw "Grading is incomplete: grading-only artifact '$GradingPath' is missing." - } - - $gradingDocument = Read-RunnerJson -Path $gradingFullPath - $schemas = Get-RunnerSchemaNames - if ([string](Get-JsonProperty -Object $gradingDocument -Name 'schema' -Default '') -ne $schemas.Grading) { - throw "grading.json must declare '$($schemas.Grading)'." - } - $topLevelAllowed = @('schema', 'grading') - foreach ($name in @(Get-JsonPropertyNames -Object $gradingDocument)) { - if ($topLevelAllowed -notcontains $name) { throw "grading.json contains unsupported field '$name'; the Grader may author only grading entries." } - } - if (-not (Test-JsonProperty -Object $gradingDocument -Name 'grading')) { - throw "grading.json is missing 'grading'." - } - $submittedValue = Get-JsonProperty -Object $gradingDocument -Name 'grading' -Default $null - if ($null -eq $submittedValue -or $submittedValue -is [string] -or -not ($submittedValue -is [System.Collections.IEnumerable])) { - throw 'grading.json grading must be an array.' - } - $submitted = @($submittedValue) - foreach ($entry in $submitted) { - Assert-EvalGradingEntryShape -Entry $entry - } - - $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState - $manifest = $freezeValidation.Manifest - $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') - if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { - throw "grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." - } - - $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) - $expected = @{} - $canonicalByKey = @{} - foreach ($record in $records) { - $assertions = @(Get-EvalMetadataAssertions -Record $record) - $canonical = Read-RunnerJson -Path $record.ResultPath - if ([int]$canonical.eval_id -ne [int]$record.EvalId -or [string]$canonical.eval_name -ne [string]$record.EvalName -or [string]$canonical.configuration -ne [string]$record.Configuration) { - throw "Canonical result '$($record.ResultRelative)' does not match its exact manifest identity." - } - if ([string]$canonical.execution_status -ne 'completed') { - throw "Grading is incomplete: '$($record.EvalName)/$($record.Configuration)' is not a completed execution." - } - $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] = $canonical - for ($index = 0; $index -lt $assertions.Count; $index++) { - $key = "$($record.EvalId)|$($record.Configuration)|$index" - $expected[$key] = [ordered]@{ - eval_id = [int]$record.EvalId - eval_name = [string]$record.EvalName - configuration = [string]$record.Configuration - assertion_index = $index - assertion = [string]$assertions[$index] - } - } - } - if ($submitted.Count -ne $expected.Count) { - throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." - } - - $validated = @{} - foreach ($entry in $submitted) { - Assert-EvalGradingEntryShape -Entry $entry - $key = Get-EvalGradingEntryKey -Entry $entry - if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } - if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } - $target = $expected[$key] - if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { - throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." - } - $validated[$key] = $entry - } - foreach ($key in $expected.Keys) { - if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } - } - - return [pscustomobject]@{ - IterationDirectory = $iteration - GradingPath = $GradingPath - GradingFullPath = $gradingFullPath - GradingDocument = $gradingDocument - FreezeValidation = $freezeValidation - Manifest = $manifest - Records = @($records) - Expected = $expected - Validated = $validated - CanonicalByKey = $canonicalByKey - GradedAssertions = $expected.Count - } -} +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'runner-common.ps1') +} +if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +} +if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'execution-freeze.ps1') +} +if (-not (Get-Command Normalize-EvalAssertion -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'phase2-grading.ps1') +} + +function Get-EvalGradingSkeleton { + return [ordered]@{ + schema = (Get-RunnerSchemaNames).Grading + grading = @() + } +} + +function Get-EvalGradingEntryKey { + param([Parameter(Mandatory = $true)][object]$Entry) + + return "$(Get-JsonProperty -Object $Entry -Name 'eval_id' -Default 0)|$(Get-JsonProperty -Object $Entry -Name 'configuration' -Default '')|$(Get-JsonProperty -Object $Entry -Name 'assertion_index' -Default -1)" +} + +function Get-EvalMetadataAssertions { + param([Parameter(Mandatory = $true)][object]$Record) + + return @(Get-EvalMetadataAssertionObjects -Record $Record) +} + +function Assert-EvalGradingEntryShape { + param([Parameter(Mandatory = $true)][object]$Entry) + + $allowed = @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason', 'source') + foreach ($name in @(Get-JsonPropertyNames -Object $Entry)) { + if ($allowed -notcontains $name) { throw "grading.json entry contains unsupported field '$name'." } + } + foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason')) { + if (-not (Test-JsonProperty -Object $Entry -Name $name)) { throw "grading.json entry is missing '$name'." } + } + $evalId = 0 + try { $evalId = [int]$Entry.eval_id } catch { throw 'grading.json eval_id must be an integer.' } + if ($evalId -lt 1) { throw 'grading.json eval_id must be positive.' } + $index = 0 + try { $index = [int]$Entry.assertion_index } catch { throw 'grading.json assertion_index must be an integer.' } + if ($index -lt 0 -or [double]$Entry.assertion_index -ne $index) { throw 'grading.json assertion_index must be a non-negative integer.' } + if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } + if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } + if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } + if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } + if ([string]::IsNullOrWhiteSpace($Entry.evidence)) { throw 'grading.json evidence must be non-empty.' } + if ([string]$Entry.evidence_domain -notin @('output', 'transcript', 'validator')) { throw "grading.json evidence_domain '$($Entry.evidence_domain)' is unsupported." } + $refs = @(Get-JsonProperty -Object $Entry -Name 'evidence_refs' -Default @()) + if ($refs.Count -eq 0) { throw 'grading.json evidence_refs must be a non-empty array.' } + if ([string]::IsNullOrWhiteSpace([string]$Entry.reason)) { throw 'grading.json reason must be non-empty.' } + if ((Test-JsonProperty -Object $Entry -Name 'source') -and [string]$Entry.source -notin @('analyzer', 'validator')) { + throw "grading.json source '$($Entry.source)' is unsupported." + } +} + +# Reject generic, templated, or tautological PASS reasons. A reason must explain HOW the cited observation establishes +# the specific assertion, not restate that the assertion passed or that the output was "evaluated". Iteration 9 passed +# 76 assertions with reasons equivalent to "Assertion evaluated against output"; that class must fail closed. +function Test-GenericGradingReason { + param([string]$Reason, [string]$Assertion) + $normalized = ([regex]::Replace([string]$Reason, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim() + if ([string]::IsNullOrWhiteSpace($normalized)) { return $true } + if ($normalized -eq (([regex]::Replace([string]$Assertion, '\s+', ' ')).Trim().TrimEnd('.', '!').Trim())) { return $true } + $genericPatterns = @( + '^(?i)eval(?:uation)? completed(?: with output)?$', + '^(?i)(?:the )?(?:assertion|requirement|expectation|condition|criteri(?:on|a))(?: is| was| has been)?(?: fully| clearly)? (?:met|satisfied|passed|verified|confirmed|evaluated|true|correct|valid|present|fulfilled|checked|held|holds|passes)$', + '(?i)evaluated against (?:the )?(?:output|transcript|response|result|evidence|assertion)', + '(?i)(?:output|response|transcript|result) (?:was |is )?(?:evaluated|matches|meets|satisfies|supports|confirms|contains) (?:the )?(?:assertion|requirement|expectation)', + '^(?i)(?:passed|verified|confirmed|as expected|done|looks good|correct|ok|success(?:ful)?|valid|complete)$', + '^(?i)(?:this )?(?:matches|meets|satisfies|establishes|proves|confirms)(?: the)?(?: assertion| requirement| expectation)?$' + ) + foreach ($pattern in $genericPatterns) { if ($normalized -match $pattern) { return $true } } + return $false +} + +function Assert-EvalPassEvidence { + param([object]$Entry, [object]$Canonical, [object]$Expected) + + $transcriptArtifacts = $null + if ([string](Get-JsonProperty -Object $Entry -Name 'evidence_domain' -Default '') -eq 'transcript') { + $transcriptArtifacts = @(Get-CanonicalTranscriptArtifacts -Record $Expected.record -Canonical $Canonical) + } + [void](Test-GradeEvidenceReference -Grade $Entry -Expected $Expected -Canonical $Canonical -TranscriptArtifacts $transcriptArtifacts) + if (-not $Entry.passed) { return } + $reason = [string](Get-JsonProperty -Object $Entry -Name 'reason' -Default '') + if (Test-GenericGradingReason -Reason $reason -Assertion ([string]$Entry.assertion)) { + throw 'PASS evidence must explain how the cited observation establishes this assertion, not restate that it passed or was evaluated.' + } +} + +function Assert-EvalGradingContract { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' + ) + + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + Assert-SafeRelativePath -RelativePath $GradingPath -FieldName 'grading path' + $gradingFullPath = Resolve-ContainedPath -BasePath $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File + if (-not (Test-Path -LiteralPath $gradingFullPath -PathType Leaf)) { + throw "Grading is incomplete: grading-only artifact '$GradingPath' is missing." + } + + $gradingDocument = Read-RunnerJson -Path $gradingFullPath + $schemas = Get-RunnerSchemaNames + if ([string](Get-JsonProperty -Object $gradingDocument -Name 'schema' -Default '') -ne $schemas.Grading) { + throw "grading.json must declare '$($schemas.Grading)'." + } + $topLevelAllowed = @('schema', 'grading', 'metadata') + foreach ($name in @(Get-JsonPropertyNames -Object $gradingDocument)) { + if ($topLevelAllowed -notcontains $name) { throw "grading.json contains unsupported field '$name'; the Grader may author only grading entries." } + } + if (-not (Test-JsonProperty -Object $gradingDocument -Name 'grading')) { + throw "grading.json is missing 'grading'." + } + $submittedValue = Get-JsonProperty -Object $gradingDocument -Name 'grading' -Default $null + if ($null -eq $submittedValue -or $submittedValue -is [string] -or -not ($submittedValue -is [System.Collections.IEnumerable])) { + throw 'grading.json grading must be an array.' + } + $submitted = @($submittedValue) + foreach ($entry in $submitted) { + Assert-EvalGradingEntryShape -Entry $entry + } + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + $manifest = $freezeValidation.Manifest + $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') + if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { + throw "grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." + } + + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) + $expected = @{} + $canonicalByKey = @{} + foreach ($record in $records) { + $assertions = @(Get-EvalMetadataAssertions -Record $record) + $canonical = Read-RunnerJson -Path $record.ResultPath + if ([int]$canonical.eval_id -ne [int]$record.EvalId -or [string]$canonical.eval_name -ne [string]$record.EvalName -or [string]$canonical.configuration -ne [string]$record.Configuration) { + throw "Canonical result '$($record.ResultRelative)' does not match its exact manifest identity." + } + if ([string]$canonical.execution_status -ne 'completed') { + throw "Grading is incomplete: '$($record.EvalName)/$($record.Configuration)' is not a completed execution." + } + $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] = $canonical + for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertion = $assertions[$index] + $key = "$($record.EvalId)|$($record.Configuration)|$index" + $expected[$key] = [ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + } + } + } + if ($submitted.Count -ne $expected.Count) { + throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." + } + + $validated = @{} + $passEvidence = @{} + foreach ($entry in $submitted) { + Assert-EvalGradingEntryShape -Entry $entry + $key = Get-EvalGradingEntryKey -Entry $entry + if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } + if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } + $target = $expected[$key] + if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion -or [string]$entry.evidence_domain -ne [string]$target.evidence_domain) { + throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." + } + $armKey = "$($entry.eval_id)|$($entry.configuration)" + $record = @($records | Where-Object { $_.EvalId -eq $entry.eval_id -and $_.Configuration -eq $entry.configuration })[0] + Assert-EvalPassEvidence -Entry $entry -Canonical $canonicalByKey[$armKey] -Expected $target + if ($entry.passed) { + $evidenceKey = $armKey + '|' + ([regex]::Replace($entry.evidence.Trim(), '\s+', ' ')).ToLowerInvariant() + if ($passEvidence.ContainsKey($evidenceKey)) { throw 'Repeated PASS evidence across assertions is not assertion-specific.' } + $passEvidence[$evidenceKey] = $true + } + $validated[$key] = $entry + } + foreach ($key in $expected.Keys) { + if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } + } + + return [pscustomobject]@{ + IterationDirectory = $iteration + GradingPath = $GradingPath + GradingFullPath = $gradingFullPath + GradingDocument = $gradingDocument + FreezeValidation = $freezeValidation + Manifest = $manifest + Records = @($records) + Expected = $expected + Validated = $validated + CanonicalByKey = $canonicalByKey + GradedAssertions = $expected.Count + } +} diff --git a/scripts/eval-runners/finalize-eval-package.ps1 b/scripts/eval-runners/finalize-eval-package.ps1 index e041f8e..2396517 100644 --- a/scripts/eval-runners/finalize-eval-package.ps1 +++ b/scripts/eval-runners/finalize-eval-package.ps1 @@ -23,6 +23,7 @@ Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot 'orchestration.ps1') . (Join-Path $PSScriptRoot 'execution-freeze.ps1') . (Join-Path $PSScriptRoot 'package-integrity.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') function Invoke-FinalizerCommand { param( @@ -132,9 +133,11 @@ try { } [void](Invoke-FinalizerCommand -ScriptPath $manifestBridge -Arguments $bridgeArgs -Description 'Manifest bridge') [void](Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState) + $gradingFreeze = Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath $gradingScript = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/apply-eval-grading.ps1') -FieldName 'grading application helper' -Kind File -RequireExists [void](Invoke-FinalizerCommand -ScriptPath $gradingScript -Arguments @('-IterationDirectory', $iteration, '-GradingPath', $GradingPath) -Description 'Grading application') + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $GradingPath) $gradedAssertions = Assert-FinalizerGrading -Records $records -IterationDirectory $iteration $reportRelative = [string](Get-JsonProperty -Object $manifest.report -Name 'tool' -Default 'tools/generate-eval-report.ps1') @@ -155,6 +158,9 @@ try { iteration = $iteration runner = $profile.Runner model = $profile.Model + analyzer_runner = [string]$gradingFreeze.Analyzer.Runner + analyzer_model = [string]$gradingFreeze.Analyzer.Model + analyzer_profile_sha256 = [string]$gradingFreeze.Analyzer.Hash expected_arms = $records.Count completed_arms = $records.Count graded_assertions = $gradedAssertions diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 new file mode 100644 index 0000000..6b56720 --- /dev/null +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -0,0 +1,260 @@ +function New-CopilotBoundaryContext { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [AllowEmptyString()][string]$PackageRoot = '', + [AllowEmptyString()][string]$SourceRepositoryRoot = '' + ) + + $executionRole = if ($RunData.PSObject.Properties.Name -contains 'ExecutionRole' -and -not [string]::IsNullOrWhiteSpace([string]$RunData.ExecutionRole)) { + [string]$RunData.ExecutionRole + } else { + 'eval_arm' + } + return [pscustomobject]@{ + PackageRoot = $PackageRoot + SourceRepositoryRoot = $SourceRepositoryRoot + RunRoot = [string]$RunData.RunRoot + WorkingDirectoryRoot = [string]$RunData.WorkingDirectoryPath + ExecutionRole = $executionRole + } +} + +function Add-CopilotBoundaryValueCandidate { + param( + [AllowNull()][object]$Value, + [Parameter(Mandatory = $true)][string]$PropertyPath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[object]]$Candidates + ) + + if ($null -eq $Value) { return } + if ($Value -is [string]) { + $leaf = [regex]::Replace($PropertyPath, '.*[.\[]', '').TrimEnd(']') + if ([string]::IsNullOrWhiteSpace($leaf)) { return } + $kind = if ($leaf -match '^(?i)(command|cmd)$') { + 'command' + } elseif ($leaf -match '^(?i)(path|paths|file|files|filepath|filepaths|directory|cwd|root|target|targets)$') { + 'path' + } else { + $null + } + if ($null -ne $kind) { + $Candidates.Add([pscustomobject]@{ + kind = $kind + source = $PropertyPath + value = [string]$Value + }) + } + return + } + if ($Value -is [System.Collections.IDictionary]) { + foreach ($key in @($Value.Keys)) { + Add-CopilotBoundaryValueCandidate -Value $Value[$key] -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$key)) -Candidates $Candidates + } + return + } + if ($Value -is [System.Collections.IEnumerable] -and $Value -isnot [string]) { + $index = 0 + foreach ($item in $Value) { + Add-CopilotBoundaryValueCandidate -Value $item -PropertyPath ([string]::Concat($PropertyPath, '[', $index, ']')) -Candidates $Candidates + $index++ + } + return + } + foreach ($property in @($Value.PSObject.Properties)) { + Add-CopilotBoundaryValueCandidate -Value $property.Value -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$property.Name)) -Candidates $Candidates + } +} + +function Get-CopilotBoundaryValueCandidates { + param([AllowNull()][object]$Data) + + $candidates = [System.Collections.Generic.List[object]]::new() + Add-CopilotBoundaryValueCandidate -Value $Data -PropertyPath 'data' -Candidates $candidates + return @($candidates.ToArray()) +} + +function Test-CopilotBoundaryPairedArmPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Boundary + ) + + $runRoot = [string](Get-JsonProperty -Object $Boundary -Name 'RunRoot' -Default '') + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + if (-not [string]::IsNullOrWhiteSpace($runRoot) -and (Test-ObservedPathInside -BasePath $runRoot -CandidatePath $ResolvedPath)) { return $false } + return ($ResolvedPath -replace '\\', '/') -match '(?i)(?:^|/)(arm-\d+-(?:with_skill|without_skill)|with_skill|without_skill)(?:/|$)' +} + +function Test-CopilotBoundaryForbiddenGradingPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Boundary + ) + + $normalizedPath = $ResolvedPath -replace '\\', '/' + if ($normalizedPath -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)$') { + return $true + } + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + $relative = if (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath) { + Get-ObservedRelativePath -BasePath $packageRoot -CandidatePath $ResolvedPath + } else { + '' + } + return $relative -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)(?:$|/)' -or + $relative -match '^(?i)(results|tools|progress)(?:/|$)' +} + +function Get-CopilotBoundaryAssessment { + param([object]$Data, [object]$Boundary) + + # Structured tool arguments/commands can contradict projection isolation; + # their absence never proves OS confinement. + $executionRole = [string](Get-JsonProperty -Object $Boundary -Name 'ExecutionRole' -Default 'eval_arm') + if ([string]::IsNullOrWhiteSpace($executionRole)) { $executionRole = 'eval_arm' } + $workingDirectoryRoot = [string](Get-JsonProperty -Object $Boundary -Name 'WorkingDirectoryRoot' -Default (Get-JsonProperty -Object $Boundary -Name 'Root' -Default '')) + $runRoot = [string](Get-JsonProperty -Object $Boundary -Name 'RunRoot' -Default $workingDirectoryRoot) + $packageRoot = [string](Get-JsonProperty -Object $Boundary -Name 'PackageRoot' -Default '') + $sourceRepositoryRoot = [string](Get-JsonProperty -Object $Boundary -Name 'SourceRepositoryRoot' -Default '') + $contradictions = [System.Collections.Generic.List[string]]::new() + $ownArmGradingVisible = $false + $pairedArmVisible = $false + $pairedOrPackageGradingVisible = $false + + foreach ($candidate in @(Get-CopilotBoundaryValueCandidates -Data $Data)) { + $pathValues = if ([string]$candidate.kind -eq 'command') { + if (Test-FileSystemCommandText -Text ([string]$candidate.value)) { + @(Get-ObservedPathTokensFromCommandText -Text ([string]$candidate.value)) + } else { + @() + } + } else { + @([string]$candidate.value) + } + foreach ($pathValue in $pathValues) { + $observed = Get-ObservedPathInfo -Path $pathValue -BasePath $workingDirectoryRoot + if ($null -eq $observed) { continue } + $resolvedPath = [string]$observed.FullPath + $insideWorking = -not [string]::IsNullOrWhiteSpace($workingDirectoryRoot) -and (Test-ObservedPathInside -BasePath $workingDirectoryRoot -CandidatePath $resolvedPath) + $insideRun = -not [string]::IsNullOrWhiteSpace($runRoot) -and (Test-ObservedPathInside -BasePath $runRoot -CandidatePath $resolvedPath) + $insidePackage = -not [string]::IsNullOrWhiteSpace($packageRoot) -and (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $resolvedPath) + $insideSource = -not [string]::IsNullOrWhiteSpace($sourceRepositoryRoot) -and (Test-ObservedPathInside -BasePath $sourceRepositoryRoot -CandidatePath $resolvedPath) + if ($executionRole -eq 'phase2_analyzer') { + if ($insideWorking) { + $relative = Get-ObservedRelativePath -BasePath $workingDirectoryRoot -CandidatePath $resolvedPath + if ($relative -in @('input-bundle.json', 'grader.md')) { $ownArmGradingVisible = $true } + continue + } + if ($insideRun) { + $contradictions.Add("Tool event references analyzer transport path '$pathValue' outside the staged repo bundle.") + } elseif ($insidePackage) { + $contradictions.Add("Tool event references package path '$pathValue' outside the staged analyzer bundle.") + } elseif ($insideSource) { + $contradictions.Add("Tool event references source repository path '$pathValue' outside the staged analyzer bundle.") + } else { + $contradictions.Add("Tool event references path '$pathValue' outside the staged analyzer bundle.") + } + } else { + if ($insideRun) { + continue + } + if ($insidePackage) { + $contradictions.Add("Tool event references forbidden package path '$pathValue'.") + } elseif ($insideSource) { + $contradictions.Add("Tool event references forbidden source repository path '$pathValue'.") + } else { + $contradictions.Add("Tool event references path '$pathValue' outside the physical projection.") + } + } + if (Test-CopilotBoundaryPairedArmPath -ResolvedPath $resolvedPath -Boundary ([pscustomobject]@{ PackageRoot = $packageRoot; RunRoot = $runRoot })) { $pairedArmVisible = $true } + if (Test-CopilotBoundaryForbiddenGradingPath -ResolvedPath $resolvedPath -Boundary ([pscustomobject]@{ PackageRoot = $packageRoot })) { $pairedOrPackageGradingVisible = $true } + } + } + + return [pscustomobject]@{ + Contradictions = @($contradictions | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -Unique) + OwnArmGradingMaterialVisible = $ownArmGradingVisible + PairedArmVisible = $pairedArmVisible + PairedOrPackageGradingMaterialVisible = $pairedOrPackageGradingVisible + } +} + +function Find-CopilotBoundaryContradictions { + param([object]$Data, [object]$Projection) + + return @((Get-CopilotBoundaryAssessment -Data $Data -Boundary $Projection).Contradictions) +} + +# Conservative detector for a native Copilot skill activation of the evaluated candidate. The native skill tool is +# excluded and skill discovery roots are isolated, so a candidate skill tool call or a native skill-resolution result +# naming the candidate is an isolation violation in BOTH arms. It fires only on a native 'skill' tool identity or a +# native skill-resolution result (a skillSource marker) that also names the candidate, so an ordinary file read of the +# staged with_skill candidate copy or a mere text mention never trips it. +function Find-CopilotNativeSkillActivation { + param([object]$Data, [string]$CandidateSkillName, [string]$EventType = '') + if ($null -eq $Data -or [string]::IsNullOrWhiteSpace($CandidateSkillName)) { return } + $json = ($Data | ConvertTo-Json -Depth 100 -Compress) + $toolIdentity = $false + foreach ($field in @('toolName', 'tool', 'name', 'tool_name')) { + $value = [string](Get-JsonProperty -Object $Data -Name $field -Default '') + if ($value -match '^(?i)skill$') { $toolIdentity = $true } + } + $skillResolution = $json -match '(?i)"skill[_]?[Ss]ource"' + if (-not ($toolIdentity -or $skillResolution)) { return } + if ($json -match ('(?i)(^|[^A-Za-z0-9._-])' + [regex]::Escape($CandidateSkillName) + '([^A-Za-z0-9._-]|$)')) { + "Native Copilot skill activation referenced the candidate '$CandidateSkillName'; the candidate must never be available through the native skill mechanism." + } +} + +# Independent, bridge-side re-verification of the candidate-instruction identity proof. with_skill must embed the exact +# frozen candidate instruction bytes (prompt prefix before the working-environment marker) whose hash equals the frozen +# candidateInstructionHash; without_skill must carry neither a hash nor an embedded candidate instruction section. This +# never trusts a runner boolean: it recomputes from the staged prompt bytes and run.json. +function Assert-CopilotCandidateInstructionBoundary { + param([object]$RunData) + $boundary = "`n`n# Working environment" + $promptText = ([Text.Encoding]::UTF8.GetString([byte[]]$RunData.PromptBytes)) -replace "`r`n", "`n" -replace "`r", "`n" + $markerIndex = $promptText.IndexOf($boundary, [StringComparison]::Ordinal) + if ($RunData.Mode -eq 'with_skill') { + $expected = [string]$RunData.CandidateInstructionHash + if ([string]::IsNullOrWhiteSpace($expected)) { throw 'with_skill run must carry candidateInstructionHash; candidate identity is unproven.' } + if ($markerIndex -lt 0) { throw 'with_skill prompt has no working-environment boundary; the injected candidate instructions cannot be isolated for hashing.' } + $instruction = $promptText.Substring(0, $markerIndex) + $actual = ([Convert]::ToHexString([Security.Cryptography.SHA256]::HashData([Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + if ($actual -ne $expected) { throw 'with_skill injected candidate instructions do not hash to the frozen candidateInstructionHash; candidate identity is unproven.' } + if ($instruction -notmatch '(?im)^##\s*Skill:') { throw 'with_skill prompt prefix does not contain the candidate instruction section.' } + } else { + if (-not [string]::IsNullOrWhiteSpace([string]$RunData.CandidateInstructionHash)) { throw 'without_skill must not declare candidateInstructionHash.' } + $prefix = if ($markerIndex -lt 0) { $promptText } else { $promptText.Substring(0, $markerIndex) } + if ($prefix -match '(?im)^##\s*Skill:') { throw 'without_skill prompt embeds a candidate instruction section; the baseline must receive no candidate instructions.' } + } +} + +function Assert-CopilotCapturedBoundary { + param([object]$Raw, [object]$RunData) + if ($Raw.runner.name -ne 'github-copilot' -or $Raw.status -ne 'completed') { return } + $paths = Get-JsonProperty -Object $Raw.evidence -Name execution_paths -Default $null + if (-not [bool](Get-JsonProperty -Object $paths -Name projection_proven -Default $false)) { throw 'Completed Copilot execution lacks a proven physical projection.' } + $package = Split-Path -Parent (Split-Path -Parent $RunData.RunRoot) + $source = [string](Get-JsonProperty -Object $paths -Name source_repository_root -Default '') + $physical = [string](Get-JsonProperty -Object $paths -Name physical_run_root -Default '') + if ([string]::IsNullOrWhiteSpace($physical) -or (Test-PathInside -BasePath $package -CandidatePath $physical) -or ($source -and (Test-PathInside -BasePath $source -CandidatePath $physical))) { throw 'Invalid Copilot physical projection boundary.' } + Assert-CopilotCandidateInstructionBoundary -RunData $RunData + $proof = New-CopilotBoundaryContext -RunData $RunData -PackageRoot $package -SourceRepositoryRoot $source + $candidateSkillName = [string]$RunData.CandidateSkillName + $transcript = @($Raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -eq 'evidence/copilot-events.jsonl' }) + if ($transcript.Count -ne 1) { throw 'Copilot native transcript is missing.' } + $path = Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $transcript[0].path -FieldName 'Copilot transcript' -Kind File + $parsed = ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($path)) + if (@($parsed.Errors).Count) { throw 'Copilot transcript contains unparseable events; boundary inspection is incomplete.' } + foreach ($event in $parsed.Events) { + $eventType = [string](Get-JsonProperty -Object $event -Name type -Default '') + $data = Get-JsonProperty -Object $event -Name data -Default $null + if ($eventType -match '^(tool\.|command\.)') { + if (@((Get-CopilotBoundaryAssessment -Data $data -Boundary $proof).Contradictions).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + if (@(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $candidateSkillName -EventType $eventType).Count) { throw 'Copilot transcript shows native candidate skill activation; grading is forbidden.' } + } + } +} diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index f63bf46..db27b88 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -27,12 +27,32 @@ param( $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot '..\runner-common.ps1') +. (Join-Path $PSScriptRoot 'isolation.ps1') # GitHub Copilot checks these token variables before its OS credential store and # GitHub CLI fallback. The values are forwarded only to the Copilot process; # --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +# The native skill tool ('skill') is the mechanism iteration 9 used to load an ambient candidate in the baseline. It is +# removed from the model's tool set in both arms; the candidate is delivered only through the frozen prompt instructions. +$copilotExcludedTools = @('skill') +# The single installed-help-proven boundary between the injected candidate instructions and the rest of the prompt +# wrapper. Preparation guarantees the candidate instruction section is the exact prompt prefix before this marker, so the +# runner and bridge can hash only the canonical candidate instruction bytes without re-deriving the skill body. +$copilotCandidateInstructionBoundary = "`n`n# Working environment" $script:copilotHomeCleanupSafe = $true +$script:copilotLogicalInputs = $null +$script:copilotProjection = $null +$script:copilotBoundaryContext = $null +$script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() +$script:copilotOwnArmGradingMaterialVisible = $false +$script:copilotPairedArmVisible = $false +$script:copilotPairedOrPackageGradingMaterialVisible = $false +# The evaluated candidate skill name (both arms) and any observed native-skill activation of it. A non-empty violation +# list at the end of execute is a fail-closed isolation breach: the candidate must never reach the worker natively. +$script:copilotCandidateSkillName = $null +$script:copilotNativeSkillViolations = [System.Collections.Generic.List[string]]::new() +$script:copilotNativeSkillCatalog = $null function Invoke-CopilotProcess { param( @@ -135,6 +155,9 @@ $descriptor = [ordered]@{ filesystem_confinement = 'conditional' ambient_candidate_skill_exclusion = 'supported' candidate_skill_exposure = 'supported' + native_skill_tool_exclusion = 'supported' + native_skill_catalog_probe = 'supported' + candidate_instruction_hash = 'supported' prompt_fidelity = 'supported' model_configuration_lock = 'supported' response_capture = 'supported' @@ -146,7 +169,7 @@ $descriptor = [ordered]@{ file_evidence = 'conditional' cost_telemetry = 'unsupported' credential_child_filtering = 'supported' - native_skill_activation_evidence = 'unsupported' + native_skill_activation_evidence = 'supported' # Behavioral evaluation transport is runner-owned: the runner starts one # fresh Copilot CLI session per eval execution and captures the session's # own terminal evidence. Copilot's native task/general-purpose subagent @@ -203,35 +226,70 @@ function Get-CopilotTokenVariable { return $null } -function Get-CopilotGhConfigDirectory { - # GH_CONFIG_DIR is an authentication-state exception to the isolated - # Copilot configuration roots. Resolve it from GitHub CLI's documented - # precedence without reading or logging any credential file. +function Add-CopilotGhConfigCandidate { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Candidates, + [string]$Path + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return } + try { + $normalized = [System.IO.Path]::GetFullPath($Path) + if (-not $Candidates.Contains($normalized)) { $Candidates.Add($normalized) } + } catch { } +} + +function Get-CopilotExplicitGhConfigDirectory { $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') - if ([string]::IsNullOrWhiteSpace($configured)) { - $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') - if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { - $configured = Join-Path $xdgConfig 'gh' - } elseif ((Get-PlatformName) -eq 'windows') { - $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) - if (-not [string]::IsNullOrWhiteSpace($applicationData)) { - $configured = Join-Path $applicationData 'GitHub CLI' - } - } else { - $userHome = [Environment]::GetEnvironmentVariable('HOME') - if ([string]::IsNullOrWhiteSpace($userHome)) { - $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) - } + if ([string]::IsNullOrWhiteSpace($configured)) { return $null } + return [string]$configured +} + +function Get-CopilotGhConfigDirectories { + # GH_CONFIG_DIR is an authentication-state exception to the isolated + # Copilot configuration roots. Build documented/default candidates and keep + # only existing directories; token resolution tries each candidate in order. + $candidates = [System.Collections.Generic.List[string]]::new() + + Add-CopilotGhConfigCandidate -Candidates $candidates -Path ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $xdgConfig 'gh') + } + + if ((Get-PlatformName) -eq 'windows') { + $applicationDataFromEnvironment = [Environment]::GetEnvironmentVariable('APPDATA') + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromEnvironment)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromEnvironment 'GitHub CLI') + } + + $applicationDataFromProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromProfile)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromProfile 'GitHub CLI') + } + } else { + foreach ($userHome in @( + [Environment]::GetEnvironmentVariable('HOME'), + [Environment]::GetEnvironmentVariable('USERPROFILE'), + [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + )) { if (-not [string]::IsNullOrWhiteSpace($userHome)) { - $configured = Join-Path (Join-Path $userHome '.config') 'gh' + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path (Join-Path $userHome '.config') 'gh') } } } - if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { - return $null + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($candidate in @($candidates)) { + if (Test-Path -LiteralPath $candidate -PathType Container) { $resolved.Add($candidate) } } - return [System.IO.Path]::GetFullPath($configured) + return @($resolved.ToArray()) +} + +function Get-CopilotGhConfigDirectory { + $directories = @(Get-CopilotGhConfigDirectories) + if ($directories.Count -eq 0) { return $null } + return [string]$directories[0] } function Get-CopilotGitHubCliToken { @@ -240,26 +298,66 @@ function Get-CopilotGitHubCliToken { return $null } - $environment = New-RunnerProbeEnvironment + $baseEnvironment = New-RunnerProbeEnvironment foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { $value = [Environment]::GetEnvironmentVariable($name) if (-not [string]::IsNullOrWhiteSpace($value)) { - $environment[$name] = $value + $baseEnvironment[$name] = $value + } + } + + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory + $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() + $useAmbientAttempt = $true + if (-not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory)) { + # An explicit GH_CONFIG_DIR selects the GitHub CLI identity. It is + # authoritative: do not search other config roots when it fails. + $candidateConfigDirectories.Add([string]$explicitGhConfigDirectory) + $useAmbientAttempt = $false + } else { + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) + } } } - $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + if ($useAmbientAttempt) { + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing + # host default can still resolve when no explicit config was supplied. + $candidateConfigDirectories.Add('') + } + + $probeRoot = Resolve-RunnerProbeTempRoot + $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null try { - $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 - if ($process.TimedOut -or $process.ExitCode -ne 0) { - return $null - } - $token = ([string]$process.Stdout).Trim() - if ([string]::IsNullOrWhiteSpace($token)) { - return $null + foreach ($candidateDirectory in @($candidateConfigDirectories.ToArray())) { + $environment = [ordered]@{} + foreach ($key in @($baseEnvironment.Keys)) { + $environment[[string]$key] = [string]$baseEnvironment[$key] + } + if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { + [void]$environment.Remove('GH_CONFIG_DIR') + } else { + $environment['GH_CONFIG_DIR'] = [string]$candidateDirectory + } + + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + continue + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + continue + } + return [pscustomobject]@{ + Token = $token + GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + GhConfigExplicit = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) + } } - return $token + return $null } finally { if (Test-Path -LiteralPath $probeDirectory) { Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue @@ -268,6 +366,8 @@ function Get-CopilotGitHubCliToken { } function Resolve-CopilotAuthentication { + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory + $explicitGhConfigProvided = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) $tokenVariable = Get-CopilotTokenVariable if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { return [pscustomobject]@{ @@ -275,24 +375,37 @@ function Resolve-CopilotAuthentication { TokenVariable = $tokenVariable TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided + NonInteractiveReady = $true } } $githubCliToken = Get-CopilotGitHubCliToken - if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + if ($null -ne $githubCliToken -and -not [string]::IsNullOrWhiteSpace([string]$githubCliToken.Token)) { return [pscustomobject]@{ Source = 'github_cli_token' TokenVariable = 'GH_TOKEN' - TokenValue = $githubCliToken + TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true + GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + GitHubCliConfigExplicit = [bool](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigExplicit' -Default $false) + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided + NonInteractiveReady = $true } } + $fallbackSource = if ($null -ne (Resolve-ExternalCommand -Name 'copilot')) { 'copilot_os_keychain_unverified' } else { 'none' } return [pscustomobject]@{ - Source = 'copilot_os_keychain_or_github_cli_unverified' + Source = $fallbackSource TokenVariable = $null TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided + NonInteractiveReady = $false } } @@ -457,6 +570,11 @@ function New-CopilotCliArguments { '--allow-all', '--no-ask-user', '--disable-builtin-mcps', + # The eval never needs native skill lookup: with_skill already embeds the exact frozen candidate + # instructions in the prompt, and without_skill intentionally has none. Removing the native skill tool from + # the model's tool set closes the native-skill activation path for BOTH arms (installed-version-supported + # --excluded-tools; the candidate 'skill' tool is what iteration 9 used to load an ambient candidate). + ('--excluded-tools=' + ($copilotExcludedTools -join ',')), '--no-color', '--log-level', 'none', '--no-auto-update', @@ -484,6 +602,10 @@ function Get-CopilotCapabilityMap { } $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + $capabilities['native_skill_tool_exclusion'] = 'supported' + $capabilities['native_skill_activation_evidence'] = 'supported' + $capabilities['candidate_instruction_hash'] = if ($Inputs.Run.Mode -eq 'with_skill') { 'supported' } else { 'excluded' } + $capabilities['native_skill_catalog_probe'] = if ($null -ne $script:copilotNativeSkillCatalog -and [bool]$script:copilotNativeSkillCatalog.available) { 'supported' } else { 'conditional' } $capabilities['scripted_multi_turn_same_session'] = if ($null -eq $Inputs.Run.Interaction) { 'conditional' } elseif ($null -ne $ContinuationCapability -and [bool]$ContinuationCapability.Available) { @@ -502,6 +624,13 @@ function Get-CopilotPreflight { $warnings = [System.Collections.Generic.List[string]]::new() $profile = $Inputs.Profile $run = $Inputs.Run + try { + if ($null -eq $script:copilotProjection) { [void](Get-CopilotProjectionPlan -Inputs $Inputs) } + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status passed -Detail 'Allowlisted physical workspace outside package/source ancestry is required.')) + } catch { + $reasons.Add($_.Exception.Message) + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status failed -Detail $_.Exception.Message)) + } $commandInfo = Resolve-ExternalCommand -Name 'copilot' $platform = Get-PlatformName $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } @@ -557,7 +686,7 @@ function Get-CopilotPreflight { } } else { $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--output-format', '--model', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + foreach ($flag in @('--output-format', '--model', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--excluded-tools', '--secret-env-vars')) { if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Copilot CLI does not advertise required flag '$flag'.") } @@ -567,13 +696,18 @@ function Get-CopilotPreflight { foreach ($forbidden in @('--resume', '-r', '--continue', '--session-id', '--connect', '--yolo')) { if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Copilot invocation must not use session-continuation or shortcut option '$forbidden'.") } } - foreach ($required in @('--output-format', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + foreach ($required in @('--output-format', '--allow-all', '--no-ask-user', '--disable-builtin-mcps', '--excluded-tools', '--secret-env-vars')) { $present = @($constructed) -contains $required - if ($required -eq '--secret-env-vars') { - $present = $present -or (@($constructed | Where-Object { $_ -like '--secret-env-vars=*' }).Count -gt 0) + if ($required -in @('--secret-env-vars', '--excluded-tools')) { + $present = $present -or (@($constructed | Where-Object { $_ -like "$required=*" }).Count -gt 0) } if (-not $present) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } } + $excludedToolsArgument = @($constructed | Where-Object { $_ -like '--excluded-tools=*' }) + if ($excludedToolsArgument.Count -eq 1) { + $excludedList = ($excludedToolsArgument[0] -replace '^--excluded-tools=', '') -split ',' + if ('skill' -notin @($excludedList)) { $reasons.Add('The constructed Copilot invocation must exclude the native skill tool for both arms.') } + } $promptOptionCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count if ($promptOptionCount -ne 0) { $reasons.Add('The constructed Copilot invocation must not place the prompt in argv; prompt delivery uses stdin.') } if ($reasons.Count -eq 0) { @@ -598,14 +732,43 @@ function Get-CopilotPreflight { } } + $nativeSkillCatalog = [ordered]@{ available = $false; candidate = [string]$run.CandidateSkillName; candidate_present = $false; candidate_enabled = $false; proven_absent = $false; entry_count = 0; entries = @(); error = 'not probed' } + if ($null -ne $commandInfo) { + try { + $nativeSkillCatalog = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $commandInfo -Environment (New-CopilotEnvironment -Inputs $Inputs -WithoutAuthentication) -WorkingDirectory $run.WorkingDirectoryPath -CandidateSkillName ([string]$run.CandidateSkillName) + } catch { $nativeSkillCatalog.error = $_.Exception.Message } + $script:copilotNativeSkillCatalog = $nativeSkillCatalog + if ($nativeSkillCatalog.available -and $nativeSkillCatalog.candidate_enabled) { + $reasons.Add("The candidate skill '$($run.CandidateSkillName)' is resolvable and enabled in the isolated native Copilot skill catalog; native discovery would contaminate the eval.") + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status failed -Detail "copilot skill list --json resolved the candidate '$($run.CandidateSkillName)' as an enabled native skill inside the isolated environment.")) + } elseif ($nativeSkillCatalog.available) { + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status passed -Detail "copilot skill list --json proves the candidate '$($run.CandidateSkillName)' is not an enabled native skill in the isolated environment ($($nativeSkillCatalog.entry_count) skills visible).")) + } else { + $checks.Add((New-PreflightCheck -Name 'native_skill_catalog' -Status unavailable -Detail "copilot skill list --json was not usable ($($nativeSkillCatalog.error)); the native skill tool exclusion and isolated discovery roots still enforce candidate exclusion.")) + $warnings.Add('The native-skill-catalog probe was unavailable; candidate native-skill exclusion still holds through --excluded-tools=skill, isolated discovery roots, and runtime/bridge detection.') + } + } + $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) + $configDetail = if ([bool]$authState.GitHubCliConfigExplicit) { + 'explicit GH_CONFIG_DIR' + } elseif ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { + 'ambient/default' + } else { + 'platform-discovered GH_CONFIG_DIR candidate' + } + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) - $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') + if ([bool]$authState.ExplicitGhConfigProvided) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or fix the selected GH_CONFIG_DIR authentication state.') + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') + } } if ($platform -notin @('linux', 'macos')) { @@ -646,7 +809,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction -and $continuationCapability.Available) { $mechanisms.Add(("explicit Copilot {0} continuation selected from installed help" -f $continuationCapability.Flag)) $mechanisms.Add('no implicit last-session continuation') @@ -668,6 +831,26 @@ function Get-CopilotPreflight { exact_session_required = $true implicit_continuation = $false } + authentication = [ordered]@{ + source = [string]$authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + github_cli_config_explicit = [bool]$authState.GitHubCliConfigExplicit + explicit_gh_config_dir_provided = [bool]$authState.ExplicitGhConfigProvided + token_value_observed = $false + } + native_skill_isolation = [ordered]@{ + excluded_tools = @($copilotExcludedTools) + native_skill_tool_disabled = $true + native_skill_dynamic_retrieval_disabled = $true + native_skill_catalog_probe_available = [bool]$nativeSkillCatalog.available + native_skill_catalog_proven_absent = [bool]$nativeSkillCatalog.proven_absent + candidate = [string]$run.CandidateSkillName + candidate_present = [bool]$nativeSkillCatalog.candidate_present + candidate_enabled = [bool]$nativeSkillCatalog.candidate_enabled + catalog = $nativeSkillCatalog + } } return $document } @@ -692,7 +875,6 @@ function New-CopilotEnvironment { } # Model-free help/version bootstrap must use the same cache roots too. if ($WithoutAuthentication) { return New-RunnerEnvironment -Run $Inputs.Run -Additional $additional } - $tokenVariable = Get-CopilotTokenVariable $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'github_cli_token') { $additional[$authState.TokenVariable] = $authState.TokenValue @@ -737,6 +919,7 @@ function Write-CopilotCapture { [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text ) + if ($null -ne $script:copilotLogicalInputs) { $RunData = $script:copilotLogicalInputs } $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) @@ -768,6 +951,8 @@ function Read-CopilotEvents { $usageNumToolCalls = 0 $usageSeen = $false $toolStarts = 0 + $lastCheckpoint = $null + $checkpointCalls = [ordered]@{} $sessionError = $null $eventCounts = @{} $sessionIds = [System.Collections.Generic.List[string]]::new() @@ -784,6 +969,18 @@ function Read-CopilotEvents { } if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + if ($eventType -match '^(tool\.|command\.)' -and $null -ne $script:copilotBoundaryContext) { + $assessment = Get-CopilotBoundaryAssessment -Data $data -Boundary $script:copilotBoundaryContext + foreach ($violation in @($assessment.Contradictions)) { $script:copilotBoundaryViolations.Add($violation) } + $script:copilotOwnArmGradingMaterialVisible = $script:copilotOwnArmGradingMaterialVisible -or [bool]$assessment.OwnArmGradingMaterialVisible + $script:copilotPairedArmVisible = $script:copilotPairedArmVisible -or [bool]$assessment.PairedArmVisible + $script:copilotPairedOrPackageGradingMaterialVisible = $script:copilotPairedOrPackageGradingMaterialVisible -or [bool]$assessment.PairedOrPackageGradingMaterialVisible + } + if ($eventType -match '^(tool\.|command\.)' -and -not [string]::IsNullOrWhiteSpace($script:copilotCandidateSkillName)) { + foreach ($activation in @(Find-CopilotNativeSkillActivation -Data $data -CandidateSkillName $script:copilotCandidateSkillName -EventType $eventType)) { + $script:copilotNativeSkillViolations.Add($activation) + } + } foreach ($eventSessionId in @(Get-CopilotEventSessionIds -Event $event)) { if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } } @@ -794,6 +991,19 @@ function Read-CopilotEvents { if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } } switch ($eventType) { + 'session.usage_checkpoint' { + # Billing is session-cumulative. Cache-break entries are last + # call snapshots, deduplicated by native call identity. + $lastCheckpoint = $data + foreach ($conversation in @(Get-JsonProperty -Object $data -Name 'promptCacheBreakState' -Default @())) { + $models = Get-JsonProperty -Object $conversation -Name 'models' -Default @{} + foreach ($name in @(Get-JsonPropertyNames -Object $models)) { + $call = Get-JsonProperty -Object $models -Name $name + $id = [string](Get-JsonProperty -Object $call -Name 'model_call_id' -Default '') + if ($id) { $checkpointCalls[$id] = $call } + } + } + } 'assistant.message' { $assistantMessageObserved = $true $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') @@ -836,6 +1046,14 @@ function Read-CopilotEvents { $finalText = [string]::Join("`n", $assistantContents) } $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + if (-not $usageSeen -and $checkpointCalls.Count -gt 0) { + foreach ($call in $checkpointCalls.Values) { + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $call -Name 'prompt_tokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $call -Name 'cache_read' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $call -Name 'cache_write' -Default $null) + } + $usageSeen = $true + } return [pscustomobject]@{ FinalText = $finalText @@ -845,6 +1063,8 @@ function Read-CopilotEvents { UsageOutput = $usageOutput UsageCacheRead = $usageCacheRead UsageCacheWrite = $usageCacheWrite + UsageCheckpoint = $lastCheckpoint + CheckpointCalls = @($checkpointCalls.Values) ToolCalls = $toolCalls SessionError = $sessionError EventCounts = $eventCounts @@ -891,13 +1111,105 @@ function Invoke-CopilotTurnProcess { return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds } +# Model-free native-skill-catalog probe. The installed Copilot CLI exposes `copilot skill list --json`, a machine-readable +# enumeration of every skill grouped by source with an `enabled` flag. Run inside the isolated eval environment, it proves +# the candidate is not resolvable from any native discovery source (inherited/personal ~/.agents|~/.copilot skills, +# project .github/skills, plugins, custom). It never consumes model tokens. A candidate that is present AND enabled is an +# isolation breach; probe unavailability is a warning only, because --excluded-tools=skill plus isolated discovery roots +# plus runtime/bridge detection still enforce the invariant. +function Invoke-CopilotNativeSkillCatalogProbe { + param( + [object]$CommandInfo, + [System.Collections.IDictionary]$Environment, + [string]$WorkingDirectory, + [Parameter(Mandatory = $true)][string]$CandidateSkillName + ) + + $probe = [ordered]@{ + available = $false + candidate = $CandidateSkillName + candidate_present = $false + candidate_enabled = $false + proven_absent = $false + entry_count = 0 + entries = @() + error = $null + } + if ($null -eq $CommandInfo) { $probe.error = 'copilot executable unavailable'; return $probe } + try { + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @('skill', 'list', '--json')) -WorkingDirectory $WorkingDirectory -Environment $Environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { $probe.error = "copilot skill list exited $($process.ExitCode)"; return $probe } + $parsed = $null + try { $parsed = [string]$process.Stdout | ConvertFrom-Json -ErrorAction Stop } catch { $probe.error = 'copilot skill list output was not JSON'; return $probe } + $entries = @($parsed) + $summary = [System.Collections.Generic.List[object]]::new() + $present = $false + $enabled = $false + foreach ($entry in $entries) { + $name = [string](Get-JsonProperty -Object $entry -Name 'name' -Default '') + $source = [string](Get-JsonProperty -Object $entry -Name 'source' -Default '') + $entryEnabled = [bool](Get-JsonProperty -Object $entry -Name 'enabled' -Default $true) + $summary.Add([ordered]@{ name = $name; source = $source; enabled = $entryEnabled }) + if ($name -eq $CandidateSkillName) { $present = $true; if ($entryEnabled) { $enabled = $true } } + } + $probe.available = $true + $probe.entry_count = $entries.Count + $probe.entries = @($summary) + $probe.candidate_present = $present + $probe.candidate_enabled = $enabled + # The candidate cannot activate natively when it is either absent or present-but-disabled. + $probe.proven_absent = -not $enabled + return $probe + } catch { + $probe.error = $_.Exception.Message + return $probe + } +} + +# Positive candidate-identity proof. The canonical candidate instruction bytes are the prompt prefix before the +# working-environment marker; preparation guarantees that prefix is exactly the frozen candidate instruction section. The +# hash of those bytes must equal the frozen candidateInstructionHash (with_skill); the baseline must carry neither a hash +# nor an embedded candidate section. Only the candidate instruction content is hashed, so unrelated wrapper edits do not +# invalidate the proof. +function Get-CopilotCandidateInstructionEvidence { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $promptText = ([System.Text.Encoding]::UTF8.GetString([byte[]]$Inputs.Run.PromptBytes)) -replace "`r`n", "`n" -replace "`r", "`n" + $markerIndex = $promptText.IndexOf($copilotCandidateInstructionBoundary, [System.StringComparison]::Ordinal) + $expected = [string]$Inputs.Run.CandidateInstructionHash + $result = [ordered]@{ expected = $null; injected = $null; verified = $false; violations = @() } + $violations = [System.Collections.Generic.List[string]]::new() + if ($Inputs.Run.Mode -eq 'with_skill') { + $result.expected = $expected + if ([string]::IsNullOrWhiteSpace($expected)) { + $violations.Add('with_skill candidateInstructionHash is required; candidate identity is unproven without it.') + $result.verified = $false + } elseif ($markerIndex -lt 0) { + $violations.Add('with_skill prompt has no working-environment boundary; candidate instruction bytes cannot be isolated for hashing.') + } else { + $instruction = $promptText.Substring(0, $markerIndex) + $injected = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + $result.injected = $injected + $result.verified = ($injected -eq $expected) + if (-not $result.verified) { $violations.Add('with_skill injected candidate instructions do not hash to the frozen candidateInstructionHash; candidate identity is unproven.') } + if ($instruction -notmatch '(?im)^##\s*Skill:') { $violations.Add('with_skill prompt prefix does not contain the candidate instruction section.') } + } + } else { + $result.verified = $true + $prefix = if ($markerIndex -lt 0) { $promptText } else { $promptText.Substring(0, $markerIndex) } + if ($prefix -match '(?im)^##\s*Skill:') { $violations.Add('without_skill prompt embeds a candidate instruction section; the baseline must receive no candidate instructions.') } + if (-not [string]::IsNullOrWhiteSpace($expected)) { $violations.Add('without_skill declares a candidateInstructionHash; the baseline must have none.') } + } + $result.violations = @($violations) + return $result +} + function Invoke-CopilotScriptedExecute { param( [Parameter(Mandatory = $true)][object]$Inputs, [Parameter(Mandatory = $true)][object]$Preflight, [Parameter(Mandatory = $true)][object]$ExecutionDescriptor ) - $started = [DateTime]::UtcNow $commandInfo = Resolve-ExternalCommand -Name 'copilot' $environment = New-CopilotEnvironment -Inputs $Inputs @@ -947,7 +1259,8 @@ function Invoke-CopilotScriptedExecute { $failureMessage = $null for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { - $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $Inputs.Run + $turnSourceRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $turnSourceRun $arguments = @($baseArguments) $targetSessionId = $null if ($turnIndex -gt 0) { @@ -1091,6 +1404,7 @@ function Invoke-CopilotScriptedExecute { $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') @@ -1167,8 +1481,11 @@ function Invoke-CopilotScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$script:copilotPairedArmVisible + grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + paired_or_package_grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + own_arm_grading_material_visible = [bool]$script:copilotOwnArmGradingMaterialVisible nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1189,13 +1506,198 @@ function Invoke-CopilotScriptedExecute { $resultFinalResponse = if ($status -eq 'completed') { $finalText } else { $null } $resultFinalResponseReason = if ($status -eq 'completed') { $null } else { 'native_interaction_incompatible' } $result = New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $resultFinalResponse -FinalResponseReason $resultFinalResponseReason -StartedUtc $firstProcess.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $capturedSessionId -IsolationCapabilities (Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -ContinuationCapability $continuationCapability) -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Copilot accepted the requested model alias and configuration; scripted turns retained the exact requested model on every invocation.'; observations = [ordered]@{ model = $Inputs.Profile.Model; observed_models = @($observedModels.ToArray()); continuation_flag = $continuationCapability.Flag } }) -Telemetry $telemetry -Artifacts @($artifacts.ToArray()) -Warnings @($warnings.ToArray()) -Evidence $evidence -AttemptCount 1 - if ($status -eq 'completed') { [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $Inputs.Run) } + if ($status -eq 'completed') { + $validationRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $validationRun) + } return $result } function Invoke-CopilotExecute { param([Parameter(Mandatory = $true)][object]$Inputs) + # Keep all orchestration data in the parent process. Only repo, prepared + # home, prompt bytes and this arm's candidate cross the projection boundary. + $plan = Get-CopilotProjectionPlan -Inputs $Inputs + $logicalInputs = $Inputs + $script:copilotLogicalInputs = $Inputs + $script:copilotProjection = $plan + $script:copilotBoundaryContext = $null + $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() + $script:copilotOwnArmGradingMaterialVisible = $false + $script:copilotPairedArmVisible = $false + $script:copilotPairedOrPackageGradingMaterialVisible = $false + $script:copilotCandidateSkillName = [string]$Inputs.Run.CandidateSkillName + $script:copilotNativeSkillViolations = [System.Collections.Generic.List[string]]::new() + $script:copilotNativeSkillCatalog = $null + [void](New-Item -ItemType Directory -Path $plan.Root -ErrorAction Stop) + try { + $physicalRun = $Inputs.Run.PSObject.Copy() + $physicalRun.RunRoot = $plan.Root + foreach ($field in @('WorkingDirectoryPath', 'HomeDirectoryPath', 'SkillDirectoryPath')) { + $source = [string]$Inputs.Run.$field + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $relative = [IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $source) + $destination = Join-Path $plan.Root $relative + Copy-CopilotProjectionTree -Source $source -Destination $destination + $physicalRun.$field = $destination + } + # Future scripted inputs stay parent-owned even when staged in repo/home. + if ($null -ne $Inputs.Run.Interaction) { + foreach ($turn in $Inputs.Run.Interaction.turns) { + $source = [string](Get-JsonProperty -Object $turn -Name source -Default '') + if ($source) { + Assert-SafeRelativePath -RelativePath $source -FieldName 'interaction source' + $path = Join-Path $plan.Root $source + if (Test-Path -LiteralPath $path -PathType Leaf) { Remove-Item -LiteralPath $path -Force } + } + } + } + $physicalRun.PromptPath = Join-Path $plan.Root 'prompt.md' + [IO.File]::WriteAllBytes($physicalRun.PromptPath, $Inputs.Run.PromptBytes) + $script:copilotBoundaryContext = New-CopilotBoundaryContext -RunData $physicalRun -PackageRoot $plan.PackageRoot -SourceRepositoryRoot $plan.SourceRepositoryRoot + $projectedFiles = @(Get-ChildItem -LiteralPath $physicalRun.WorkingDirectoryPath -Recurse -Force -File | ForEach-Object { [IO.Path]::GetRelativePath($physicalRun.WorkingDirectoryPath, $_.FullName) }) + $physicalInputs = [pscustomobject]@{ Run = $physicalRun; Profile = $Inputs.Profile } + $result = Invoke-CopilotWithPreparedHome -Inputs $physicalInputs -Action { Invoke-CopilotProjectedExecute -Inputs $physicalInputs } + $result.evidence.execution_paths = [ordered]@{ + projection = 'physical_temp_outside_logical_package' + logical_run_root = $logicalInputs.Run.RunRoot + logical_working_directory = $logicalInputs.Run.WorkingDirectoryPath + logical_home_directory = $logicalInputs.Run.HomeDirectoryPath + physical_run_root = $plan.Root + physical_working_directory = $physicalRun.WorkingDirectoryPath + physical_home_directory = $physicalRun.HomeDirectoryPath + source_repository_root = $plan.SourceRepositoryRoot + projection_proven = $true + hard_filesystem_confinement = [bool]$result.isolation.hard_filesystem_confinement + } + $result.evidence.boundary = [ordered]@{ proof = 'allowlisted_physical_projection'; contradictions = @($script:copilotBoundaryViolations.ToArray()); event_inspection = 'contradiction_detector_not_confinement' } + # Runtime native-skill activations observed during the model turn(s), captured before the candidate-instruction + # identity proof is folded in, so the resolution flag reflects only actual native activation. + $runtimeNativeSkillActivations = @($script:copilotNativeSkillViolations | Select-Object -Unique) + $candidateInstruction = Get-CopilotCandidateInstructionEvidence -Inputs $logicalInputs + $result.evidence.candidate_instruction_hash_expected = $candidateInstruction.expected + $result.evidence.candidate_instruction_hash_injected = $candidateInstruction.injected + $result.evidence.candidate_instruction_hash_verified = [bool]$candidateInstruction.verified + foreach ($violation in @($candidateInstruction.violations)) { $script:copilotNativeSkillViolations.Add($violation) } + $catalog = $script:copilotNativeSkillCatalog + $result.evidence.native_skill = [ordered]@{ + native_skill_tool_disabled = $true + native_skill_dynamic_retrieval_disabled = $true + excluded_tools = @($copilotExcludedTools) + native_skill_catalog_probe_available = [bool]($null -ne $catalog -and $catalog.available) + native_skill_catalog_proven_absent = [bool]($null -ne $catalog -and $catalog.proven_absent) + candidate_native_skill_resolution = ($runtimeNativeSkillActivations.Count -gt 0) + candidate_native_skill_activations = $runtimeNativeSkillActivations + catalog = $catalog + } + $capturePath = Join-Path $logicalInputs.Run.RunRoot 'evidence/copilot-events.jsonl' + if (Test-Path -LiteralPath $capturePath) { + $usageWarnings = [System.Collections.Generic.List[string]]::new() + # Reparse the combined native stream once, so resumed cumulative + # checkpoints and repeated call snapshots are never summed twice. + $usage = Read-CopilotEvents -Parsed (ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($capturePath))) -Warnings $usageWarnings + if ($usage.ParseErrorCount -gt 0) { $script:copilotBoundaryViolations.Add('Unparseable native events prevent complete boundary inspection.') } + $result.evidence.usage_checkpoint = $usage.UsageCheckpoint + $result.evidence.usage_checkpoint_calls = $usage.CheckpointCalls + if ($usage.UsageSeen) { + $buckets = [ordered]@{} + foreach ($mapping in @(@('input_tokens', 'UsageInput'), @('output_tokens', 'UsageOutput'), @('cache_read_tokens', 'UsageCacheRead'), @('cache_write_tokens', 'UsageCacheWrite'))) { + if ($null -ne $usage.($mapping[1])) { $buckets[$mapping[0]] = [int64]$usage.($mapping[1]) } + } + if ($buckets.Count) { $result.telemetry.tokens = New-AvailableMetric -Value $buckets } + } + } + $isolationViolations = @(@($script:copilotBoundaryViolations) + @($script:copilotNativeSkillViolations) | Where-Object { $_ } | Select-Object -Unique) + if ($isolationViolations.Count -gt 0) { + $result.status = 'incompatible' + $result.isolation.status = 'unverified' + $result.isolation.level = 'unsupported' + $result.isolation.hard_filesystem_confinement = $false + $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $isolationViolations)) + } + $result.evidence.boundary.contradictions = @($script:copilotBoundaryViolations | Select-Object -Unique) + if ($result.evidence.Contains('native_skill')) { + $result.evidence.native_skill.candidate_native_skill_resolution = ($result.evidence.native_skill.candidate_native_skill_resolution -or (@($script:copilotNativeSkillViolations).Count -gt 0)) + } + # Do not copy runtime links back into the logical package. + Assert-CopilotProjectionTree -Path $physicalRun.WorkingDirectoryPath + Assert-CopilotProjectionTree -Path $logicalInputs.Run.WorkingDirectoryPath + foreach ($file in @(Get-ChildItem -LiteralPath $logicalInputs.Run.WorkingDirectoryPath -Recurse -Force -File)) { + $relative = [IO.Path]::GetRelativePath($logicalInputs.Run.WorkingDirectoryPath, $file.FullName) + if ($projectedFiles -contains $relative -and -not (Test-Path -LiteralPath (Join-Path $physicalRun.WorkingDirectoryPath $relative))) { + if (-not (Test-PathInside -BasePath $logicalInputs.Run.WorkingDirectoryPath -CandidatePath $file.FullName)) { throw 'Unsafe projected output deletion.' } + Remove-Item -LiteralPath $file.FullName -Force + } + } + Copy-CopilotProjectionTree -Source $physicalRun.WorkingDirectoryPath -Destination $logicalInputs.Run.WorkingDirectoryPath + return $result + } finally { + if (-not $script:copilotHomeCleanupSafe) { throw 'Copilot process termination is unproven; projection retained and execution fails closed.' } + if (-not (Test-PathInside -BasePath $plan.Parent -CandidatePath $plan.Root) -or [IO.Path]::GetFileName($plan.Root) -notmatch '^agentic-copilot-projection-[0-9a-f]{32}$') { throw 'Unsafe Copilot projection cleanup path.' } + Remove-Item -LiteralPath $plan.Root -Recurse -Force + $script:copilotLogicalInputs = $null + $script:copilotProjection = $null + $script:copilotBoundaryContext = $null + $script:copilotOwnArmGradingMaterialVisible = $false + $script:copilotPairedArmVisible = $false + $script:copilotPairedOrPackageGradingMaterialVisible = $false + } +} + +function Assert-CopilotProjectionTree { + param([string]$Path) + $item = Get-Item -LiteralPath $Path -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw "Copilot projection refuses filesystem link '$Path'." } + if ($item.PSIsContainer) { + foreach ($child in Get-ChildItem -LiteralPath $Path -Force) { Assert-CopilotProjectionTree -Path $child.FullName } + } +} + +function Copy-CopilotProjectionTree { + param([string]$Source, [string]$Destination) + Assert-CopilotProjectionTree -Path $Source + [void][IO.Directory]::CreateDirectory($Destination) + foreach ($child in Get-ChildItem -LiteralPath $Source -Force) { Copy-Item -LiteralPath $child.FullName -Destination $Destination -Recurse -Force } +} + +function Get-CopilotProjectionPlan { + param([object]$Inputs) + $sourceRoot = $null + $packageRoot = Split-Path -Parent (Split-Path -Parent $Inputs.Run.RunRoot) + foreach ($start in @($Inputs.Run.RunRoot, $PSScriptRoot)) { + $cursor = $start + while ($cursor) { + if (Test-Path -LiteralPath (Join-Path $cursor '.git')) { $sourceRoot = $cursor; break } + $cursor = Split-Path -Parent $cursor + } + if ($sourceRoot) { break } + } + $parent = [IO.Path]::GetFullPath([IO.Path]::GetTempPath()) + foreach ($forbidden in @($packageRoot, $sourceRoot) | Where-Object { $_ }) { + if (Test-PathInside -BasePath $forbidden -CandidatePath $parent) { throw 'Copilot cannot establish a physical projection outside package/source ancestry.' } + } + # A linked temp parent or ambient instruction file would invalidate ancestry + # isolation. Fail closed rather than disable legitimate repo instructions. + $cursor = $parent + while ($cursor) { + $item = Get-Item -LiteralPath $cursor -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw 'Copilot projection parent contains a filesystem link.' } + foreach ($instruction in @('AGENTS.md', '.github/copilot-instructions.md', '.github/instructions', '.git')) { + if (Test-Path -LiteralPath (Join-Path $cursor $instruction)) { throw "Copilot projection would inherit ambient '$instruction'." } + } + $cursor = Split-Path -Parent $cursor + } + foreach ($path in @($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.HomeDirectoryPath, $Inputs.Run.SkillDirectoryPath) | Where-Object { $_ }) { Assert-CopilotProjectionTree -Path $path } + foreach ($indirection in @('.git', '.git/objects/info/alternates', '.git/commondir')) { + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.WorkingDirectoryPath $indirection) -PathType Leaf) { throw "Copilot projection refuses external Git indirection '$indirection'." } + } + return [pscustomobject]@{ Root = Join-Path $parent ('agentic-copilot-projection-' + [Guid]::NewGuid().ToString('N')); Parent = $parent; PackageRoot = $packageRoot; SourceRepositoryRoot = $sourceRoot } +} + +function Invoke-CopilotProjectedExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + $preflight = Get-CopilotPreflight -Inputs $Inputs $started = [DateTime]::UtcNow $sessionId = [Guid]::NewGuid().ToString('D') @@ -1286,13 +1788,14 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') @@ -1351,8 +1854,11 @@ function Invoke-CopilotExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$script:copilotPairedArmVisible + grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + paired_or_package_grading_material_visible = [bool]$script:copilotPairedOrPackageGradingMaterialVisible + own_arm_grading_material_visible = [bool]$script:copilotOwnArmGradingMaterialVisible nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/invoke-phase2-analyzer.ps1 b/scripts/eval-runners/invoke-phase2-analyzer.ps1 new file mode 100644 index 0000000..c27c4ce --- /dev/null +++ b/scripts/eval-runners/invoke-phase2-analyzer.ps1 @@ -0,0 +1,841 @@ +<#! +.SYNOPSIS + Runs package-local Phase 2 grading with explicit analyzer provenance. + +.DESCRIPTION + This is the normal Phase 2 boundary after Phase 1 execution is frozen and + bridged. It resolves deterministic validator-domain assertions before any + analyzer worker starts, dispatches one fresh analyzer session per remaining + semantic arm, persists analyzer evidence and progress, freezes Phase 2, and + deterministically writes package-root grading.json. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [ValidateRange(1, 128)][int]$Concurrency = 16, + [ValidateRange(1, 86400)][int]$TimeoutSeconds = 900 +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'package-integrity.ps1') +. (Join-Path $PSScriptRoot 'fanout-process.ps1') +. (Join-Path $PSScriptRoot 'phase2-grading.ps1') + +function Write-Phase2Summary { + param([Parameter(Mandatory = $true)][object]$Value, [int]$ExitCode = 0) + + Write-RunnerJson -Value $Value -Compress -AsOutput + if ($ExitCode -ne 0) { exit $ExitCode } +} + +function Save-Phase2State { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$State) + + $State.updated_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + Write-RunnerJsonFile -Path $Path -Value $State +} + +function Resolve-AnalyzerRunner { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile + ) + + $runnerToolsDirectory = Resolve-PackageRunnerToolsDirectory -IterationDirectory $IterationDirectory -Manifest $Manifest + $resolverPath = Join-Path $runnerToolsDirectory 'resolve-runner.ps1' + if (-not (Test-Path -LiteralPath $resolverPath -PathType Leaf)) { throw "Package-local Eval Runner resolver is missing at '$resolverPath'." } + $resolutionOutput = & pwsh -NoProfile -NonInteractive -File $resolverPath ([string]$AnalyzerProfile.Runner) 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Analyzer runner '$($AnalyzerProfile.Runner)' could not be resolved package-locally: $([string]::Join(' ', @($resolutionOutput)))" } + $resolution = ([string]::Join([Environment]::NewLine, @($resolutionOutput | ForEach-Object { [string]$_ }))) | ConvertFrom-Json + $runnerRelative = [string](Get-JsonProperty -Object $resolution -Name 'path' -Default '') + $runnerPath = Resolve-ContainedPath -BasePath $runnerToolsDirectory -RelativePath $runnerRelative -FieldName 'resolved analyzer runner path' -Kind File + $descriptor = Get-PackageRunnerDescriptorFromPath -RunnerName ([string]$AnalyzerProfile.Runner) -RunnerPath $runnerPath -RunnerToolsDirectory $runnerToolsDirectory + if ([string](Get-JsonProperty -Object $descriptor.harness -Name 'name' -Default '') -ne [string]$AnalyzerProfile.Harness) { + throw "Analyzer profile harness '$($AnalyzerProfile.Harness)' does not match package-local runner descriptor '$($descriptor.harness.name)'." + } + $delegation = Get-JsonProperty -Object $descriptor -Name 'delegation' -Default $null + if ([string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default '') -ne 'runner') { + throw "Analyzer runner '$($AnalyzerProfile.Runner)' must provide runner-owned fresh analyzer sessions for attributable Phase 2 grading." + } + return [pscustomobject]@{ RunnerToolsDirectory = $runnerToolsDirectory; RunnerPath = $runnerPath; Descriptor = $descriptor; Resolution = $resolution } +} + +function New-AnalyzerExecutionProfile { + param([Parameter(Mandatory = $true)][object]$AnalyzerProfile) + + return [ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + reasoning_effort = $AnalyzerProfile.ReasoningEffort + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = $TimeoutSeconds + concurrency = $Concurrency + } +} + +function New-AnalyzerPrompt { + param( + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][string]$GraderContract + ) + + $bundleJson = ConvertTo-RunnerJson -Value $Bundle -Depth 100 + return @" +# Phase 2 semantic analyzer + +You are grading exactly one evaluation arm. Use the grader contract below and the input bundle below. Return only JSON. Do not inspect any repository, paired arm, sibling eval, previous grade, native skill, plugin, MCP server, or global configuration. + +## Response schema + +Return this exact JSON shape: + +````json +{ + "schema": "codebeltnet/agentic/eval-analyzer-fragment/1", + "eval_id": $($Bundle.eval_id), + "configuration": "$($Bundle.configuration)", + "grading": [ + { + "assertion_index": 0, + "passed": true, + "reason": "Specific explanation tied to the cited frozen evidence.", + "evidence_refs": [ + { + "artifact": "$($Bundle.canonical_result)", + "domain": "output", + "start_line": 1, + "end_line": 1, + "quote": "verbatim text from those lines" + } + ] + } + ] +} +```` + +Rules: +- Grade only the assertion indexes listed in the input bundle. +- Use output-domain refs only for output assertions and transcript-domain refs only for transcript assertions. +- Cite only artifacts listed in this one-arm bundle. +- A quote, when present, must be verbatim from the cited lines or event. +- Do not cite the paired arm or any sibling eval. +- Do not retry, ask for another model, or broaden the task. + +## Packaged grader contract + +````markdown +$GraderContract +```` + +## Input bundle + +````json +$bundleJson +```` +"@ +} + +function Resolve-AnalyzerTranscriptStagedPath { + param( + [Parameter(Mandatory = $true)][string]$RepoRoot, + [Parameter(Mandatory = $true)][string]$Artifact + ) + + Assert-SafeRelativePath -RelativePath $Artifact -FieldName 'transcript evidence artifact' + $relative = ('evidence/{0}' -f $Artifact.Replace('\', '/')) + $candidate = [System.IO.Path]::GetFullPath((Join-Path $RepoRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $RepoRoot -CandidatePath $candidate)) { + throw "Transcript artifact '$Artifact' resolves outside the analyzer workspace." + } + return [pscustomobject]@{ + Relative = $relative + FullPath = $candidate + } +} + +function New-AnalyzerRunBundle { + param( + [Parameter(Mandatory = $true)][string]$Phase2Root, + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile, + [Parameter(Mandatory = $true)][string]$AnalyzerExecutionProfilePath, + [Parameter(Mandatory = $true)][string]$GraderContractText + ) + + $workerId = [string]$Worker.worker_id + $runRoot = Join-Path (Join-Path $Phase2Root 'work') $workerId + $repoRoot = Join-Path $runRoot 'repo' + $homeRoot = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null + $canonical = Read-RunnerJson -Path $Worker.record.ResultPath + $output = [string](Get-JsonProperty -Object $canonical -Name 'output' -Default '') + $outputLines = @(Get-ArmOutputLines -Output $output) + $lineRecords = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt $outputLines.Count; $index++) { + $lineRecords.Add([ordered]@{ line = $index + 1; text = [string]$outputLines[$index] }) + } + $metadata = Read-RunnerJson -Path $Worker.record.MetadataPath + + # Stage transcript artifacts when any assertion requires transcript-domain evidence. + $transcriptAssertions = @($Worker.assertions | Where-Object { [string]$_.evidence_domain -eq 'transcript' }) + $stagedTranscripts = [System.Collections.Generic.List[object]]::new() + if ($transcriptAssertions.Count -gt 0) { + $evidenceDir = Join-Path $repoRoot 'evidence' + New-Item -ItemType Directory -Path $evidenceDir -Force | Out-Null + $stagedPathMap = @{} + foreach ($artifact in @(Get-CanonicalTranscriptArtifactPaths -Canonical $canonical)) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $Worker.record -Artifact $artifact -AllowMissing + if ($null -eq $sourcePath) { continue } + $snapshot = Get-TranscriptArtifactSnapshot -Record $Worker.record -Artifact $artifact -SourcePath $sourcePath + $stagedLocation = Resolve-AnalyzerTranscriptStagedPath -RepoRoot $repoRoot -Artifact $artifact + $stagedKey = ConvertTo-ComparablePath -Path $stagedLocation.FullPath + if ($stagedPathMap.ContainsKey($stagedKey)) { + $existing = [string]$stagedPathMap[$stagedKey] + if (-not [string]::Equals($existing, [string]$snapshot.SourcePath, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Transcript artifacts '$existing' and '$($snapshot.SourcePath)' resolve to the same analyzer staging path '$($stagedLocation.Relative)'." + } + continue + } + $stagedPathMap[$stagedKey] = [string]$snapshot.SourcePath + $parent = Split-Path -Parent $stagedLocation.FullPath + if (-not [string]::IsNullOrWhiteSpace($parent)) { + New-Item -ItemType Directory -Path $parent -Force | Out-Null + } + if (Test-Path -LiteralPath $stagedLocation.FullPath -PathType Leaf) { + throw "Transcript staging path '$($stagedLocation.Relative)' already exists in the analyzer workspace." + } + [System.IO.File]::Copy($snapshot.SourcePath, $stagedLocation.FullPath, $false) + $entry = $snapshot.Entry + $entry['staged_artifact'] = [string]$stagedLocation.Relative + $entry['staged_sha256'] = Get-Sha256HexFromFile -Path $stagedLocation.FullPath + $stagedTranscripts.Add($entry) + } + } + + $allowedArtifacts = [System.Collections.Generic.List[string]]::new() + $allowedArtifacts.Add([string]$Worker.record.ResultRelative) + foreach ($artifact in @((Get-JsonProperty -Object $canonical -Name 'output_files' -Default @()) | ForEach-Object { [string]$_ })) { + if (-not [string]::IsNullOrWhiteSpace($artifact)) { $allowedArtifacts.Add($artifact) } + } + + $bundle = [ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-input/1' + worker_id = $workerId + eval_id = [int]$Worker.eval_id + eval_name = [string]$Worker.eval_name + configuration = [string]$Worker.configuration + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + expected_output = [string](Get-JsonProperty -Object $metadata -Name 'expected_output' -Default '') + canonical_result = [string]$Worker.record.ResultRelative + canonical_result_sha256 = Get-Sha256HexFromFile -Path $Worker.record.ResultPath + frozen_output = [ordered]@{ + artifact = [string]$Worker.record.ResultRelative + lines = @($lineRecords.ToArray()) + } + allowed_artifacts = @($allowedArtifacts.ToArray()) + assertions = @($Worker.assertions | ForEach-Object { + [ordered]@{ + assertion_index = [int]$_.assertion_index + assertion = [string]$_.assertion + evidence_domain = [string]$_.evidence_domain + } + }) + } + if ($stagedTranscripts.Count -gt 0) { + $bundle['frozen_transcripts'] = @($stagedTranscripts.ToArray()) + } + $bundlePath = Join-Path $repoRoot 'input-bundle.json' + Write-RunnerJsonFile -Path $bundlePath -Value $bundle + $graderPath = Join-Path $repoRoot 'grader.md' + [System.IO.File]::WriteAllText($graderPath, $GraderContractText, [System.Text.UTF8Encoding]::new($false)) + $bundleHash = Get-Sha256HexFromFile -Path $bundlePath + $prompt = New-AnalyzerPrompt -Bundle $bundle -GraderContract $GraderContractText + [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), $prompt, [System.Text.UTF8Encoding]::new($false)) + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = [int]$Worker.eval_id + evalName = ('phase2-{0}' -f [string]$Worker.eval_name) + candidateSkillName = 'phase2-analyzer' + skillName = $null + iteration = 1 + mode = 'without_skill' + executionRole = 'phase2_analyzer' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @('input-bundle.json', 'grader.md') + fixtureHash = Get-JsonFingerprint -Object $bundle + skillHash = $null + candidateInstructionHash = $null + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + $runPath = Join-Path $runRoot 'run.json' + Write-RunnerJsonFile -Path $runPath -Value $run + return [pscustomobject]@{ + WorkerId = $workerId + RunRoot = $runRoot + RunPath = $runPath + BundlePath = $bundlePath + BundleHash = $bundleHash + PromptPath = Join-Path $runRoot 'prompt.md' + AnalyzerExecutionProfilePath = $AnalyzerExecutionProfilePath + StagedTranscripts = @($stagedTranscripts.ToArray()) + } +} + +function Invoke-AnalyzerPreflight { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][string]$ProgressLogPath + ) + + $stdoutPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase2-preflight-' + [Guid]::NewGuid().ToString('N') + '.stdout') + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase2-preflight-' + [Guid]::NewGuid().ToString('N') + '.stderr') + $child = $null + try { + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $RunnerPath, 'preflight', '-Run', $Bundle.RunPath, '-Profile', $Bundle.AnalyzerExecutionProfilePath) -WorkingDirectory (Split-Path -Parent $Bundle.RunPath) -StdoutPath $stdoutPath -StderrPath $stderrPath -TimeoutSeconds (Get-RunnerPreflightTimeoutSeconds) -Runner 'phase2' -WorkerId $Bundle.WorkerId -EvalId $null -Configuration 'analyzer' -Phase 'preflight' -ProgressLogPath $ProgressLogPath + $exitCode = Complete-RunnerChildProcess -Child $child + $child = $null + $stdout = if (Test-Path -LiteralPath $stdoutPath -PathType Leaf) { [System.IO.File]::ReadAllText($stdoutPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } + return [pscustomobject]@{ ExitCode = $exitCode; Result = $result; Stdout = $stdout; Bundle = $Bundle } + } finally { + if ($null -ne $child) { try { [void](Complete-RunnerChildProcess -Child $child -TimeoutSeconds 1) } catch { } } + foreach ($path in @($stdoutPath, $stderrPath)) { Remove-Item -LiteralPath $path -Force -ErrorAction SilentlyContinue } + } +} + +function ConvertFrom-AnalyzerJsonDocument { + param( + [Parameter(Mandatory = $true)][string]$Json, + [Parameter(Mandatory = $true)][ValidateSet('raw_json', 'fenced_json', 'fenced_json_with_surrounding_text')][string]$TransportNormalization + ) + + $trimmed = $Json.Trim() + if (-not ($trimmed.StartsWith('{', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('}', [System.StringComparison]::Ordinal))) { + throw 'response contained no JSON object candidate.' + } + try { + return [pscustomobject]@{ + Fragment = $trimmed | ConvertFrom-Json -Depth 100 + TransportNormalization = $TransportNormalization + } + } catch { + throw $_.Exception + } +} + +function Get-AnalyzerFencedBlocks { + param([Parameter(Mandatory = $true)][string]$Text) + + $lines = [regex]::Split($Text, "\r?\n") + $blocks = [System.Collections.Generic.List[object]]::new() + $blockStart = $null + $blockInfo = '' + $bodyLines = $null + for ($index = 0; $index -lt $lines.Count; $index++) { + $line = [string]$lines[$index] + if ($null -eq $blockStart) { + $open = [regex]::Match($line, '^[ \t]*```(?[^\r\n`]*)[ \t]*$') + if ($open.Success) { + $blockStart = $index + $blockInfo = [string]$open.Groups['info'].Value + $bodyLines = [System.Collections.Generic.List[string]]::new() + } + continue + } + + if ($line -match '^[ \t]*```[ \t]*$') { + $leading = if ($blockStart -gt 0) { [string]::Join([Environment]::NewLine, @($lines[0..($blockStart - 1)])) } else { '' } + $trailing = if ($index + 1 -lt $lines.Count) { [string]::Join([Environment]::NewLine, @($lines[($index + 1)..($lines.Count - 1)])) } else { '' } + $blocks.Add([pscustomobject]@{ + Info = $blockInfo + Body = [string]::Join([Environment]::NewLine, @($bodyLines.ToArray())) + LeadingText = $leading + TrailingText = $trailing + }) + $blockStart = $null + $blockInfo = '' + $bodyLines = $null + continue + } + + $bodyLines.Add($line) + } + if ($null -ne $blockStart) { + throw 'Analyzer returned malformed JSON: the fenced block is not terminated.' + } + return @($blocks.ToArray()) +} + +function ConvertFrom-AnalyzerResponse { + param([Parameter(Mandatory = $true)][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { throw 'Analyzer returned an empty response.' } + $trimmed = $Text.Trim() + $candidates = [System.Collections.Generic.List[object]]::new() + $parseErrors = [System.Collections.Generic.List[string]]::new() + + if ($trimmed.StartsWith('{', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('}', [System.StringComparison]::Ordinal)) { + try { + $candidates.Add((ConvertFrom-AnalyzerJsonDocument -Json $trimmed -TransportNormalization 'raw_json')) + } catch { + $parseErrors.Add([string]$_.Exception.Message) + } + } + + $fenceBlocks = @(Get-AnalyzerFencedBlocks -Text $Text) + if ($fenceBlocks.Count -gt 1) { + throw 'Analyzer returned multiple fenced blocks; the response must contain exactly one analyzer JSON document.' + } + if ($fenceBlocks.Count -eq 1) { + $block = $fenceBlocks[0] + $info = [string]$block.Info + if (-not [string]::IsNullOrWhiteSpace($info) -and $info.Trim() -cne 'json') { + throw 'Analyzer returned malformed JSON: the fenced block must be a JSON fence.' + } + $leading = [string]$block.LeadingText + $trailing = [string]$block.TrailingText + $normalization = if ([string]::IsNullOrWhiteSpace($leading) -and [string]::IsNullOrWhiteSpace($trailing)) { + 'fenced_json' + } else { + 'fenced_json_with_surrounding_text' + } + try { + $candidates.Add((ConvertFrom-AnalyzerJsonDocument -Json ([string]$block.Body) -TransportNormalization $normalization)) + } catch { + $parseErrors.Add([string]$_.Exception.Message) + } + } + + if ($candidates.Count -gt 1) { + throw 'Analyzer returned multiple JSON candidates; the response must contain exactly one analyzer JSON document.' + } + if ($candidates.Count -eq 1) { + return $candidates[0] + } + if ($parseErrors.Count -gt 0) { + throw "Analyzer returned malformed JSON: $($parseErrors[0])" + } + throw 'Analyzer returned malformed JSON: response contained no unambiguous JSON object.' +} + +function Confirm-AnalyzerFragment { + param( + [Parameter(Mandatory = $true)][object]$Fragment, + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$Canonical, + [AllowNull()][object[]]$StagedTranscripts = $null + ) + + if ([string](Get-JsonProperty -Object $Fragment -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-analyzer-fragment/1') { throw 'Analyzer fragment has an unsupported schema.' } + if ([int](Get-JsonProperty -Object $Fragment -Name 'eval_id' -Default 0) -ne [int]$Worker.eval_id -or [string](Get-JsonProperty -Object $Fragment -Name 'configuration' -Default '') -ne [string]$Worker.configuration) { + throw 'Analyzer fragment does not match the requested eval arm.' + } + $grades = @(Get-JsonProperty -Object $Fragment -Name 'grading' -Default @()) + $expectedByIndex = @{} + foreach ($assertion in @($Worker.assertions)) { $expectedByIndex[[int]$assertion.assertion_index] = $assertion } + if ($grades.Count -ne @($Worker.assertions).Count) { throw 'Analyzer fragment grade cardinality does not match unresolved semantic assertions.' } + $seenIndexes = [System.Collections.Generic.HashSet[int]]::new() + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($grade in $grades) { + $index = [int](Get-JsonProperty -Object $grade -Name 'assertion_index' -Default -1) + if (-not $expectedByIndex.ContainsKey($index)) { throw "Analyzer fragment includes unexpected assertion_index '$index'." } + if (-not $seenIndexes.Add($index)) { throw "Analyzer fragment duplicates assertion_index '$index'." } + $expected = $expectedByIndex[$index] + if ((Get-JsonProperty -Object $grade -Name 'passed' -Default $null) -isnot [bool]) { throw 'Analyzer fragment passed must be boolean.' } + $reason = [string](Get-JsonProperty -Object $grade -Name 'reason' -Default '') + if ([string]::IsNullOrWhiteSpace($reason)) { throw 'Analyzer fragment reason must be non-empty.' } + $refs = @(Get-JsonProperty -Object $grade -Name 'evidence_refs' -Default @()) + $entry = ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$grade.passed) -Reason $reason -EvidenceRefs $refs -Source 'analyzer' -Evidence $reason + [void](Test-GradeEvidenceReference -Grade $entry -Expected $expected -Canonical $Canonical -TranscriptArtifacts $StagedTranscripts) + $entries.Add($entry) + } + if ($seenIndexes.Count -ne $expectedByIndex.Count) { + $missing = @($expectedByIndex.Keys | Where-Object { -not $seenIndexes.Contains([int]$_) } | Sort-Object) + throw "Analyzer fragment is missing assertion_index '$($missing[0])'." + } + return @($entries.ToArray()) +} + +function Get-ObservedAnalyzerModel { + param([Parameter(Mandatory = $true)][object]$Raw) + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + $observed = [string](Get-JsonProperty -Object $delegation -Name 'observed_model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($observed)) { return $observed } + $resolved = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'resolved' -Default $null) -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($resolved)) { return $resolved } + return '' +} + +function New-AnalyzerResult { + param( + [Parameter(Mandatory = $true)][object]$Worker, + [Parameter(Mandatory = $true)][object]$Bundle, + [Parameter(Mandatory = $true)][object]$Raw, + [Parameter(Mandatory = $true)][object]$AnalyzerProfile, + [Parameter(Mandatory = $true)][string]$RawRelative, + [Parameter(Mandatory = $true)][string]$FragmentRelative, + [Parameter(Mandatory = $true)][object[]]$Grades, + [string]$ResponseTransportNormalization = '', + [string]$Status = 'completed', + [string]$Failure = '' + ) + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + $tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'tokens' -Default $null + $toolCalls = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'tool_calls' -Default $null + $cost = Get-JsonProperty -Object (Get-JsonProperty -Object $Raw -Name 'telemetry' -Default $null) -Name 'cost' -Default $null + $rawPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $RawRelative -FieldName 'analyzer raw execution result' -Kind File -RequireExists + $fragmentPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $FragmentRelative -FieldName 'analyzer grading fragment' -Kind File -RequireExists + return [ordered]@{ + schema = $script:AnalyzerResultSchema + worker_id = [string]$Worker.worker_id + eval_id = [int]$Worker.eval_id + eval_name = [string]$Worker.eval_name + configuration = [string]$Worker.configuration + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + requested_runner = [string]$AnalyzerProfile.Runner + requested_model = [string]$AnalyzerProfile.Model + requested_reasoning_effort = $AnalyzerProfile.ReasoningEffort + resolved_runner = [string](Get-JsonProperty -Object $Raw.runner -Name 'name' -Default '') + resolved_model = [string](Get-JsonProperty -Object $Raw.requested -Name 'model' -Default '') + observed_model = Get-ObservedAnalyzerModel -Raw $Raw + harness_name = [string](Get-JsonProperty -Object $Raw.harness -Name 'name' -Default '') + harness_version = [string](Get-JsonProperty -Object $Raw.harness -Name 'version' -Default '') + session_id = [string](Get-JsonProperty -Object $delegation -Name 'worker_session_id' -Default (Get-JsonProperty -Object $Raw.session -Name 'id' -Default '')) + started_utc = [string](Get-JsonProperty -Object $Raw -Name 'started_utc' -Default '') + finished_utc = [string](Get-JsonProperty -Object $Raw -Name 'finished_utc' -Default '') + duration_ms = [int64]([double](Get-JsonProperty -Object $Raw -Name 'duration_seconds' -Default 0) * 1000) + terminal_status = $Status + failure = $Failure + input_bundle_sha256 = [string]$Bundle.BundleHash + grader_contract_sha256 = Get-Sha256HexFromFile -Path (Join-Path (Split-Path -Parent $Bundle.BundlePath) 'grader.md') + assertions_sha256 = Get-JsonFingerprint -Object @($Worker.assertions) + raw_execution_result = $RawRelative + raw_execution_result_sha256 = Get-Sha256HexFromFile -Path $rawPath + grading_fragment = $FragmentRelative + grading_fragment_sha256 = Get-Sha256HexFromFile -Path $fragmentPath + response_transport_normalization = $ResponseTransportNormalization + parsed_grading = @($Grades) + input_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'input_tokens' -Default $null + output_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'output_tokens' -Default $null + cache_read_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'cached_input_tokens' -Default $null + cache_write_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'cache_write_tokens' -Default $null + total_tokens = Get-JsonProperty -Object (Get-JsonProperty -Object $tokens -Name 'value' -Default $null) -Name 'total_tokens' -Default $null + cost = Get-JsonProperty -Object $cost -Name 'value' -Default $null + tool_calls = Get-JsonProperty -Object $toolCalls -Name 'value' -Default $null + attempt_count = 1 + } +} + +function Get-RootGradingMetadata { + param([Parameter(Mandatory = $true)][object]$AnalyzerProfile) + + return [ordered]@{ + schema = 'codebeltnet/agentic/eval-grading-metadata/1' + generated_by = 'phase2-controller' + analyzer_runner = [string]$AnalyzerProfile.Runner + analyzer_model = [string]$AnalyzerProfile.Model + analyzer_reasoning_effort = $AnalyzerProfile.ReasoningEffort + analyzer_profile_sha256 = [string]$AnalyzerProfile.Hash + phase2_state = 'phase2-state.json' + grading_freeze = 'grading-freeze.json' + } +} + +try { + $manifestPath = Join-Path $iteration 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { throw 'manifest.json is missing.' } + $manifest = Read-RunnerJson -Path $manifestPath + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iteration -Manifest $manifest) + $executionIdentity = Assert-PackageRunnerIdentity -IterationDirectory $iteration -Manifest $manifest + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + [void](Assert-FanoutPhase1Success -Aggregate $freezeValidation.Aggregate -MessagePrefix 'Phase 2 Phase 1') + $bridgePath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/bridge-manifest-results.ps1') -FieldName 'manifest bridge' -Kind File -RequireExists + $bridgeArgs = @('-IterationDirectory', $iteration, '-RequireComplete', '-RequireParallelDispatch') + if ([string](Get-JsonProperty -Object $executionIdentity.Descriptor.delegation -Name 'dispatch_owner' -Default '') -eq 'runner') { $bridgeArgs += '-RequireNativeDelegation' } + $bridgeOutput = & pwsh -NoProfile -NonInteractive -File $bridgePath @bridgeArgs 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Manifest bridge failed before Phase 2: $([string]::Join(' ', @($bridgeOutput)))" } + + $statePath = Join-Path $iteration 'phase2-state.json' + $freezePath = Join-Path $iteration 'grading-freeze.json' + if (Test-Path -LiteralPath $freezePath -PathType Leaf) { + $validatedFreeze = Assert-GradingFreeze -IterationDirectory $iteration + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'already_frozen' + analyzer_profile_sha256 = [string]$validatedFreeze.Analyzer.Hash + grading_freeze = 'grading-freeze.json' + grading = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default 'grading.json') + }) + exit 0 + } + if (Test-Path -LiteralPath $statePath -PathType Leaf) { + throw 'Phase 2 state already exists without a valid grading freeze; refusing to duplicate analyzer work. Use a fresh package iteration.' + } + + $analyzerProfile = Resolve-AnalyzerProfile -IterationDirectory $iteration -Manifest $manifest + $analyzerRunner = Resolve-AnalyzerRunner -IterationDirectory $iteration -Manifest $manifest -AnalyzerProfile $analyzerProfile + $phase2Root = Join-Path $iteration 'phase2' + $validatorRoot = Join-Path $phase2Root 'validators' + $resultRoot = Join-Path $phase2Root 'results' + $fragmentRoot = Join-Path $phase2Root 'fragments' + $progressRoot = Join-Path $iteration 'progress' + New-Item -ItemType Directory -Path $phase2Root, $validatorRoot, $resultRoot, $fragmentRoot, $progressRoot -Force | Out-Null + $progressLogPath = Join-Path $progressRoot 'phase2-progress.jsonl' + + $analyzerExecutionProfilePath = Join-Path $phase2Root 'analyzer-execution-profile.json' + Write-RunnerJsonFile -Path $analyzerExecutionProfilePath -Value (New-AnalyzerExecutionProfile -AnalyzerProfile $analyzerProfile) + $analyzerExecutionProfile = Resolve-ExecutionProfile -ProfilePath $analyzerExecutionProfilePath + $graderContractPath = Get-GraderContractPath -IterationDirectory $iteration + $graderContractText = [System.IO.File]::ReadAllText($graderContractPath, [System.Text.UTF8Encoding]::new($false)) + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) + $semanticWorkers = [System.Collections.Generic.List[object]]::new() + $validatorEntries = [System.Collections.Generic.List[object]]::new() + $grades = [System.Collections.Generic.List[object]]::new() + + foreach ($record in $records) { + $canonical = Read-RunnerJson -Path $record.ResultPath + $semanticAssertions = [System.Collections.Generic.List[object]]::new() + $assertionIndex = 0 + foreach ($assertion in @(Get-EvalMetadataAssertionObjects -Record $record)) { + $expected = [ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $assertionIndex + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + } + if ([string]$assertion.evidence_domain -eq 'validator') { + $validatorResult = Invoke-GradingValidatorRule -Rule ([string]$assertion.validator) -Canonical $canonical -Record $record + $validatorRelative = ('phase2/validators/{0}-{1}-{2}.json' -f [int]$record.EvalId, [string]$record.Configuration, $assertionIndex) + $validatorPath = Join-Path $iteration ($validatorRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $validatorPath -Value $validatorResult + $validatorEntries.Add([ordered]@{ + path = $validatorRelative + sha256 = Get-Sha256HexFromFile -Path $validatorPath + status = if ([bool]$validatorResult.passed) { 'passed' } else { 'failed' } + rule = [string]$assertion.validator + eval_id = [int]$record.EvalId + configuration = [string]$record.Configuration + assertion_index = $assertionIndex + }) + $grades.Add((ConvertTo-GradingEntry -Expected $expected -Passed ([bool]$validatorResult.passed) -Reason ([string]$validatorResult.reason) -EvidenceRefs @($validatorResult.evidence_refs) -Source 'validator' -Evidence ([string]$validatorResult.reason))) + } else { + $semanticAssertions.Add($expected) + } + $assertionIndex++ + } + if ($semanticAssertions.Count -gt 0) { + $semanticWorkers.Add([ordered]@{ + worker_id = Get-ArmKey -EvalId ([int]$record.EvalId) -Configuration ([string]$record.Configuration) + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + record = $record + assertions = @($semanticAssertions.ToArray()) + }) + } + } + + $state = [ordered]@{ + schema = $script:Phase2Schema + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + analyzer_runner = [string]$analyzerProfile.Runner + analyzer_model = [string]$analyzerProfile.Model + expected_worker_ids = @($semanticWorkers | ForEach-Object { [string]$_.worker_id }) + pending = @($semanticWorkers | ForEach-Object { [string]$_.worker_id }) + active = [ordered]@{} + completed = [ordered]@{} + started_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + updated_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + concurrency = $Concurrency + validator_results = @($validatorEntries.ToArray()) + analyzer_results = @() + grading_freeze = $null + status = 'running' + } + Save-Phase2State -Path $statePath -State $state + + $bundles = @{} + foreach ($worker in @($semanticWorkers)) { + $bundle = New-AnalyzerRunBundle -Phase2Root $phase2Root -Worker $worker -AnalyzerProfile $analyzerProfile -AnalyzerExecutionProfilePath $analyzerExecutionProfile.Path -GraderContractText $graderContractText + $bundles[$worker.worker_id] = $bundle + } + + $preflightFailures = [System.Collections.Generic.List[string]]::new() + foreach ($worker in @($semanticWorkers)) { + $preflight = Invoke-AnalyzerPreflight -RunnerPath $analyzerRunner.RunnerPath -Bundle $bundles[$worker.worker_id] -ProgressLogPath $progressLogPath + if ([int]$preflight.ExitCode -ne 0 -or $null -eq $preflight.Result -or [string](Get-JsonProperty -Object $preflight.Result -Name 'status' -Default '') -ne 'compatible') { + $preflightFailures.Add([string]$worker.worker_id) + } + } + if ($preflightFailures.Count -gt 0) { + $state.status = 'failed' + Save-Phase2State -Path $statePath -State $state + throw "Phase 2 analyzer preflight failed before execution for: $($preflightFailures -join ', ')." + } + + $executorSessions = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($entry in @($freezeValidation.Freeze.executions)) { + $session = [string](Get-JsonProperty -Object $entry -Name 'worker_session_id' -Default (Get-JsonProperty -Object $entry -Name 'thread_id' -Default '')) + if (-not [string]::IsNullOrWhiteSpace($session)) { [void]$executorSessions.Add($session) } + } + + $running = [System.Collections.Generic.List[object]]::new() + $completedAnalyzerEntries = [System.Collections.Generic.List[object]]::new() + $failed = [System.Collections.Generic.List[string]]::new() + $workersById = @{} + foreach ($worker in @($semanticWorkers)) { $workersById[[string]$worker.worker_id] = $worker } + + while (@($state.pending).Count -gt 0 -or $running.Count -gt 0) { + while (@($state.pending).Count -gt 0 -and $running.Count -lt $Concurrency) { + $workerId = [string]@($state.pending)[0] + $state.pending = @(@($state.pending) | Where-Object { [string]$_ -ne $workerId }) + $state.active[$workerId] = [ordered]@{ worker_id = $workerId; accepted_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); attempt_count = 1 } + Save-Phase2State -Path $statePath -State $state + $bundle = $bundles[$workerId] + $rawRelative = "phase2/results/$workerId.execution-result.json" + $stderrRelative = "phase2/results/$workerId.stderr.txt" + $rawPath = Join-Path $iteration ($rawRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $stderrPath = Join-Path $iteration ($stderrRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $analyzerRunner.RunnerPath, 'execute', '-Run', $bundle.RunPath, '-Profile', $analyzerExecutionProfile.Path) -WorkingDirectory (Split-Path -Parent $bundle.RunPath) -StdoutPath $rawPath -StderrPath $stderrPath -TimeoutSeconds $TimeoutSeconds -Runner ([string]$analyzerProfile.Runner) -WorkerId $workerId -EvalId $workersById[$workerId].eval_id -Configuration $workersById[$workerId].configuration -Phase 'phase2-analyzer' -ProgressLogPath $progressLogPath + $running.Add([pscustomobject]@{ worker_id = $workerId; child = $child; Process = $child.Process; raw_relative = $rawRelative; stderr_relative = $stderrRelative; bundle = $bundle }) + [Console]::Error.WriteLine(("Phase 2: {0}/{1} analyzer workers terminal; active: {2}; analyzer: {3}/{4}" -f @($state.completed.Keys).Count, @($state.expected_worker_ids).Count, $workerId, [string]$analyzerProfile.Runner, [string]$analyzerProfile.Model)) + } + if ($running.Count -eq 0) { break } + $completedIndex = Wait-AnyRunnerChild -Running $running + $item = $running[$completedIndex] + $running.RemoveAt($completedIndex) + $exitCode = Complete-RunnerChildProcess -Child $item.child + $workerId = [string]$item.worker_id + $worker = $workersById[$workerId] + $rawPath = Join-Path $iteration ([string]$item.raw_relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $terminalStatus = 'completed' + $terminalFailure = '' + $workerGrades = @() + try { + if ($null -eq $exitCode -or [int]$exitCode -ne 0) { throw "Analyzer runner exited with code '$exitCode'." } + $raw = Read-RunnerJson -Path $rawPath + [void](Assert-ExecutionResult -Result $raw) + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $raw -Run (Resolve-RunContract -RunPath $item.bundle.RunPath) -RequestedModel ([string]$analyzerProfile.Model) -ExpectedRunner ([string]$analyzerProfile.Runner) -ExpectedMechanism ([string]$analyzerRunner.Descriptor.delegation.mechanism)) + if ([string]$raw.status -ne 'completed') { throw "Analyzer runner terminal status '$($raw.status)' is not completed." } + $observedModel = Get-ObservedAnalyzerModel -Raw $raw + if ([string]::IsNullOrWhiteSpace($observedModel) -or $observedModel -ne [string]$analyzerProfile.Model) { throw "Analyzer observed model '$observedModel' does not match requested '$($analyzerProfile.Model)'." } + $sessionId = [string](Get-JsonProperty -Object $raw.session -Name 'id' -Default '') + if ([string]::IsNullOrWhiteSpace($sessionId)) { throw 'Analyzer result did not expose a session id.' } + if ($executorSessions.Contains($sessionId)) { throw 'Analyzer session reused an executor session identity.' } + if (@($completedAnalyzerEntries | Where-Object { [string]$_.session_id -eq $sessionId }).Count -gt 0) { throw 'Analyzer worker session was reused across arms.' } + $responseText = [string](Get-JsonProperty -Object $raw.final_response -Name 'text' -Default '') + $response = ConvertFrom-AnalyzerResponse -Text $responseText + $fragment = $response.Fragment + $canonical = Read-RunnerJson -Path $worker.record.ResultPath + $workerGrades = @(Confirm-AnalyzerFragment -Fragment $fragment -Worker $worker -Canonical $canonical -StagedTranscripts $item.bundle.StagedTranscripts) + $fragmentRelative = "phase2/fragments/$workerId.grading-fragment.json" + $fragmentPath = Join-Path $iteration ($fragmentRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $fragmentPath -Value $fragment + foreach ($grade in $workerGrades) { $grades.Add($grade) } + $analyzerResult = New-AnalyzerResult -Worker $worker -Bundle $item.bundle -Raw $raw -AnalyzerProfile $analyzerProfile -RawRelative ([string]$item.raw_relative) -FragmentRelative $fragmentRelative -Grades $workerGrades -ResponseTransportNormalization ([string]$response.TransportNormalization) + $resultRelative = "phase2/results/$workerId.analyzer-result.json" + $resultPath = Join-Path $iteration ($resultRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Write-RunnerJsonFile -Path $resultPath -Value $analyzerResult + $completedAnalyzerEntries.Add([ordered]@{ path = $resultRelative; sha256 = Get-Sha256HexFromFile -Path $resultPath; status = 'completed'; worker_id = $workerId; session_id = $sessionId }) + $state.completed[$workerId] = [ordered]@{ worker_id = $workerId; status = 'completed'; analyzer_result = $resultRelative; analyzer_result_sha256 = Get-Sha256HexFromFile -Path $resultPath; terminal_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); worker_session_id = $sessionId; attempt_count = 1 } + } catch { + $terminalStatus = 'failed' + $terminalFailure = $_.Exception.Message + $failed.Add("$workerId`: $terminalFailure") + $state.completed[$workerId] = [ordered]@{ worker_id = $workerId; status = 'failed'; terminal_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow); failure = $terminalFailure; attempt_count = 1 } + } finally { + if ($state.active.Contains($workerId)) { $state.active.Remove($workerId) } + $state.analyzer_results = @($completedAnalyzerEntries.ToArray()) + if ($terminalStatus -eq 'failed') { $state.status = 'failed' } + Save-Phase2State -Path $statePath -State $state + [Console]::Error.WriteLine(("Phase 2: {0}/{1} analyzer workers terminal; active: {2}; analyzer: {3}/{4}" -f @($state.completed.Keys).Count, @($state.expected_worker_ids).Count, ([string]::Join(',', @($state.active.Keys))), [string]$analyzerProfile.Runner, [string]$analyzerProfile.Model)) + } + } + + if ($failed.Count -gt 0) { + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'failed' + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + validator_results = @($validatorEntries).Count + analyzer_workers = @($semanticWorkers).Count + failed = @($failed.ToArray()) + phase2_state = 'phase2-state.json' + }) -ExitCode 2 + } + + $rootMetadata = Get-RootGradingMetadata -AnalyzerProfile $analyzerProfile + $rootGrading = New-RootGradingDocument -Grades @($grades.ToArray()) -Metadata $rootMetadata + $gradingRelative = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default 'grading.json') + $gradingPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $gradingRelative -FieldName 'grading path' -Kind File + Write-RunnerJsonFile -Path $gradingPath -Value $rootGrading + $state.analyzer_results = @($completedAnalyzerEntries.ToArray()) + $state.status = 'completed' + Save-Phase2State -Path $statePath -State $state + # Collect unique transcript artifact hashes across all semantic workers for the freeze. + $transcriptArtifacts = [System.Collections.Generic.List[object]]::new() + $seenTranscriptKeys = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($worker in @($semanticWorkers)) { + foreach ($st in @($bundles[$worker.worker_id].StagedTranscripts)) { + $artifact = [string]$st.artifact + $sourceHash = [string]$st.source_sha256 + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $worker.record -Artifact $artifact + $iterRelative = [System.IO.Path]::GetRelativePath($iteration, $sourcePath).Replace('\', '/') + $key = "$iterRelative`:$sourceHash" + if ($seenTranscriptKeys.Add($key)) { + $transcriptArtifacts.Add([ordered]@{ artifact = $iterRelative; sha256 = $sourceHash }) + } + } + } + $freeze = New-GradingFreezeDocument -IterationDirectory $iteration -Manifest $manifest -State $state -Grades @($grades.ToArray()) -Metadata $rootMetadata -TranscriptArtifacts @($transcriptArtifacts.ToArray()) + Write-RunnerJsonFile -Path $freezePath -Value $freeze + $state.grading_freeze = [ordered]@{ path = 'grading-freeze.json'; sha256 = Get-Sha256HexFromFile -Path $freezePath } + Write-RunnerJsonFile -Path $statePath -Value $state + [void](Assert-GradingFreeze -IterationDirectory $iteration -GradingPath $gradingRelative) + Write-Phase2Summary -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-phase2-summary/1' + status = 'completed' + analyzer_profile_sha256 = [string]$analyzerProfile.Hash + validator_results = @($validatorEntries).Count + analyzer_workers = @($semanticWorkers).Count + grading = $gradingRelative + grading_sha256 = Get-Sha256HexFromFile -Path $gradingPath + grading_freeze = 'grading-freeze.json' + phase2_state = 'phase2-state.json' + progress_log = 'progress/phase2-progress.jsonl' + }) +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 1dce3a5..86c1312 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -620,6 +620,210 @@ function Remove-OpenCodeAnsiSequences { return [regex]::Replace($Text, "`e\[[0-?]*[ -/]*[@-~]", '') } +function Add-OpenCodeBoundaryValueCandidate { + param( + [AllowNull()][object]$Value, + [Parameter(Mandatory = $true)][string]$PropertyPath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[object]]$Candidates + ) + + if ($null -eq $Value) { return } + if ($Value -is [string]) { + $leaf = [regex]::Replace($PropertyPath, '.*[.\[]', '').TrimEnd(']') + if ([string]::IsNullOrWhiteSpace($leaf)) { return } + $kind = if ($leaf -match '^(?i)(command|cmd)$') { + 'command' + } elseif ($leaf -match '^(?i)(path|paths|file|files|filepath|filepaths|directory|cwd|root|target|targets|location)$') { + 'path' + } else { + $null + } + if ($null -ne $kind) { + $Candidates.Add([pscustomobject]@{ + kind = $kind + source = $PropertyPath + value = [string]$Value + }) + } + return + } + if ($Value -is [System.Collections.IDictionary]) { + foreach ($key in @($Value.Keys)) { + Add-OpenCodeBoundaryValueCandidate -Value $Value[$key] -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$key)) -Candidates $Candidates + } + return + } + if ($Value -is [System.Collections.IEnumerable] -and $Value -isnot [string]) { + $index = 0 + foreach ($item in $Value) { + Add-OpenCodeBoundaryValueCandidate -Value $item -PropertyPath ([string]::Concat($PropertyPath, '[', $index, ']')) -Candidates $Candidates + $index++ + } + return + } + foreach ($property in @($Value.PSObject.Properties)) { + Add-OpenCodeBoundaryValueCandidate -Value $property.Value -PropertyPath ([string]::Concat($PropertyPath, '.', [string]$property.Name)) -Candidates $Candidates + } +} + +function Get-OpenCodeBoundaryValueCandidates { + param([AllowNull()][object]$Value) + + $candidates = [System.Collections.Generic.List[object]]::new() + Add-OpenCodeBoundaryValueCandidate -Value $Value -PropertyPath 'event' -Candidates $candidates + return @($candidates.ToArray()) +} + +function Get-OpenCodeLogicalPackageRoot { + param([Parameter(Mandatory = $true)][object]$Projection) + + $logicalRun = Get-JsonProperty -Object $Projection -Name 'LogicalRun' -Default $null + $logicalRunRoot = [string](Get-JsonProperty -Object $logicalRun -Name 'RunRoot' -Default '') + if ([string]::IsNullOrWhiteSpace($logicalRunRoot)) { return '' } + $evalRoot = Get-ObservedParentPath -Path $logicalRunRoot + if ([string]::IsNullOrWhiteSpace($evalRoot)) { return '' } + return [string](Get-ObservedParentPath -Path $evalRoot) +} + +function Test-OpenCodeBoundaryPairedArmPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Projection + ) + + $logicalRun = Get-JsonProperty -Object $Projection -Name 'LogicalRun' -Default $null + $logicalRunRoot = [string](Get-JsonProperty -Object $logicalRun -Name 'RunRoot' -Default '') + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + if (-not [string]::IsNullOrWhiteSpace($logicalRunRoot) -and (Test-ObservedPathInside -BasePath $logicalRunRoot -CandidatePath $ResolvedPath)) { return $false } + return ($ResolvedPath -replace '\\', '/') -match '(?i)(?:^|/)(arm-\d+-(?:with_skill|without_skill)|with_skill|without_skill)(?:/|$)' +} + +function Test-OpenCodeBoundaryForbiddenGradingPath { + param( + [Parameter(Mandatory = $true)][string]$ResolvedPath, + [Parameter(Mandatory = $true)][object]$Projection + ) + + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + if ([string]::IsNullOrWhiteSpace($packageRoot) -or -not (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $ResolvedPath)) { return $false } + $relative = Get-ObservedRelativePath -BasePath $packageRoot -CandidatePath $ResolvedPath + if ([string]::IsNullOrWhiteSpace($relative)) { return $false } + return $relative -match '(?i)(?:^|/)(eval-metadata\.json|grading\.json|execution-freeze\.json|orchestration-state\.json|benchmark\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)(?:$|/)' -or + $relative -match '^(?i)(results|tools|progress)(?:/|$)' +} + +function Get-OpenCodeBoundaryAssessment { + param( + [AllowEmptyCollection()][object[]]$Events = @(), + [Parameter(Mandatory = $true)][object]$Projection + ) + + $contradictions = [System.Collections.Generic.List[object]]::new() + $projectionRoot = [string](Get-JsonProperty -Object $Projection -Name 'Root' -Default '') + $physicalWorkingDirectory = [string](Get-JsonProperty -Object $Projection -Name 'PhysicalWorkingDirectory' -Default $projectionRoot) + $sourceRepositoryRoot = [string](Get-JsonProperty -Object $Projection -Name 'SourceRepositoryRoot' -Default '') + $packageRoot = Get-OpenCodeLogicalPackageRoot -Projection $Projection + + foreach ($event in @($Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ($eventType -ne 'tool_use') { continue } + $tool = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $event -Name 'part' -Default $null) -Name 'tool' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '')) + foreach ($candidate in @(Get-OpenCodeBoundaryValueCandidates -Value $event)) { + $pathValues = if ([string]$candidate.kind -eq 'command') { + if (Test-FileSystemCommandText -Text ([string]$candidate.value)) { + @(Get-ObservedPathTokensFromCommandText -Text ([string]$candidate.value)) + } else { + @() + } + } else { + @([string]$candidate.value) + } + foreach ($pathValue in $pathValues) { + $observed = Get-ObservedPathInfo -Path $pathValue -BasePath $physicalWorkingDirectory + if ($null -eq $observed) { continue } + $resolvedPath = [string]$observed.FullPath + if (-not [string]::IsNullOrWhiteSpace($projectionRoot) -and (Test-ObservedPathInside -BasePath $projectionRoot -CandidatePath $resolvedPath)) { continue } + + $category = if (-not [string]::IsNullOrWhiteSpace($packageRoot) -and (Test-ObservedPathInside -BasePath $packageRoot -CandidatePath $resolvedPath)) { + if (Test-OpenCodeBoundaryForbiddenGradingPath -ResolvedPath $resolvedPath -Projection $Projection) { 'package_grading_material' } + elseif (Test-OpenCodeBoundaryPairedArmPath -ResolvedPath $resolvedPath -Projection $Projection) { 'paired_arm' } + else { 'package_root_outside_projection' } + } elseif (-not [string]::IsNullOrWhiteSpace($sourceRepositoryRoot) -and (Test-ObservedPathInside -BasePath $sourceRepositoryRoot -CandidatePath $resolvedPath)) { + 'source_repository_outside_projection' + } else { + 'outside_projection' + } + $message = switch ($category) { + 'package_grading_material' { "Structured OpenCode tool evidence referenced package grading/orchestration material outside the physical projection: '$pathValue'." } + 'paired_arm' { "Structured OpenCode tool evidence referenced a paired-arm/package path outside the physical projection: '$pathValue'." } + 'package_root_outside_projection' { "Structured OpenCode tool evidence referenced a package path outside the physical projection: '$pathValue'." } + 'source_repository_outside_projection' { "Structured OpenCode tool evidence referenced a source-repository path outside the physical projection: '$pathValue'." } + default { "Structured OpenCode tool evidence referenced a path outside the physical projection: '$pathValue'." } + } + $contradictions.Add([ordered]@{ + category = $category + tool = $tool + source = [string]$candidate.source + observed_path = [string]$pathValue + resolved_path = $resolvedPath + event_type = $eventType + message = $message + }) + } + } + } + + return [pscustomobject]@{ + Contradictions = @($contradictions.ToArray()) + ProjectionEscapeObserved = $contradictions.Count -gt 0 + PairedArmVisible = @($contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'paired_arm' }).Count -gt 0 + GradingMaterialVisible = @($contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'package_grading_material' }).Count -gt 0 + } +} + +function ConvertTo-OpenCodeBoundaryAssessment { + param([AllowEmptyCollection()][object[]]$Contradictions = @()) + + return [pscustomobject]@{ + Contradictions = @($Contradictions) + ProjectionEscapeObserved = @($Contradictions).Count -gt 0 + PairedArmVisible = @($Contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'paired_arm' }).Count -gt 0 + GradingMaterialVisible = @($Contradictions | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'category' -Default '') -eq 'package_grading_material' }).Count -gt 0 + } +} + +function Apply-OpenCodeBoundaryAssessment { + param( + [Parameter(Mandatory = $true)][string]$Status, + [AllowEmptyString()][string]$FailureCode = '', + [AllowEmptyString()][string]$FailureMessage = '', + [Parameter(Mandatory = $true)][object]$Assessment + ) + + $appliedStatus = $Status + $appliedFailureCode = $FailureCode + $appliedFailureMessage = $FailureMessage + if ([bool](Get-JsonProperty -Object $Assessment -Name 'ProjectionEscapeObserved' -Default $false) -and $Status -eq 'completed') { + $appliedStatus = 'incompatible' + $appliedFailureCode = 'opencode_projection_escape_observed' + $appliedFailureMessage = 'OpenCode structured tool evidence proves access outside the physical run projection.' + } + return [pscustomobject]@{ + Status = $appliedStatus + FailureCode = $appliedFailureCode + FailureMessage = $appliedFailureMessage + ProjectionEscapeObserved = [bool](Get-JsonProperty -Object $Assessment -Name 'ProjectionEscapeObserved' -Default $false) + PairedArmVisible = [bool](Get-JsonProperty -Object $Assessment -Name 'PairedArmVisible' -Default $false) + GradingMaterialVisible = [bool](Get-JsonProperty -Object $Assessment -Name 'GradingMaterialVisible' -Default $false) + BoundaryEvidence = [ordered]@{ + proof = 'physical_run_projection' + contradictions = @(Get-JsonProperty -Object $Assessment -Name 'Contradictions' -Default @()) + event_inspection = 'structured_tool_contradiction_detector' + } + } +} + function Get-OpenCodeContinuationCapability { param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$HelpText) @@ -801,6 +1005,7 @@ function Invoke-OpenCodeScriptedExecute { $artifacts = [System.Collections.Generic.List[object]]::new() $warnings = [System.Collections.Generic.List[string]]::new() $nativeFailures = [System.Collections.Generic.List[string]]::new() + $boundaryContradictions = [System.Collections.Generic.List[object]]::new() $eventCounts = @{} $observedModels = [System.Collections.Generic.List[string]]::new() $usageBuckets = [ordered]@{} @@ -865,6 +1070,10 @@ function Invoke-OpenCodeScriptedExecute { $artifacts.Add($turnStderrArtifact) $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { [pscustomobject]@{ Events = @(); Errors = @() } } else { ConvertFrom-JsonLines -Text $process.Stdout } foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode turn $turnNumber event parse error: $parseError") } + if (-not $hardFilesystem) { + $turnBoundary = Get-OpenCodeBoundaryAssessment -Events @($parsed.Events) -Projection $projection + foreach ($contradiction in @($turnBoundary.Contradictions)) { $boundaryContradictions.Add($contradiction) } + } $parsedEvents = Read-OpenCodeScriptedTurn -Parsed $parsed -Warnings $warnings $turnTiming = [ordered]@{ turn = $turnNumber @@ -964,6 +1173,14 @@ function Invoke-OpenCodeScriptedExecute { $failureMessage = 'OpenCode scripted interaction did not complete every ordered user/assistant turn.' } if ($status -eq 'completed' -and $nativeFailures.Count -gt 0) { $status = 'incompatible' } + $boundaryAssessment = ConvertTo-OpenCodeBoundaryAssessment -Contradictions @($boundaryContradictions.ToArray()) + $boundaryOutcome = Apply-OpenCodeBoundaryAssessment -Status $status -FailureCode ([string]$failureCode) -FailureMessage ([string]$failureMessage) -Assessment $boundaryAssessment + $status = [string]$boundaryOutcome.Status + $failureCode = [string]$boundaryOutcome.FailureCode + $failureMessage = [string]$boundaryOutcome.FailureMessage + if ([bool]$boundaryOutcome.ProjectionEscapeObserved -and $nativeFailures -notcontains 'projection_escape_observed') { + $nativeFailures.Add('projection_escape_observed') + } if ([string]::IsNullOrWhiteSpace($capturedSessionId)) { $capturedSessionId = [Guid]::NewGuid().ToString('D') } $finished = $lastProcess.FinishedUtc $durationSeconds = [Math]::Round(($finished - $firstProcess.StartedUtc).TotalSeconds, 3) @@ -1072,6 +1289,7 @@ function Invoke-OpenCodeScriptedExecute { complete_structured_transcript = [bool]$terminalCapture turn_artifacts = @($nativeTurns.ToArray() | ForEach-Object { "evidence/opencode-turn-$(Get-JsonProperty -Object $_ -Name 'turn' -Default 0)-events.jsonl" }) } + boundary = $boundaryOutcome.BoundaryEvidence delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = [string]$descriptor.delegation.mechanism @@ -1086,8 +1304,10 @@ function Invoke-OpenCodeScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$boundaryOutcome.PairedArmVisible + grading_material_visible = [bool]$boundaryOutcome.GradingMaterialVisible + projection_escape_observed = [bool]$boundaryOutcome.ProjectionEscapeObserved nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1449,6 +1669,7 @@ function New-OpenCodeExecutionProjection { EvalId = $Inputs.Run.EvalId EvalName = $Inputs.Run.EvalName Mode = $Inputs.Run.Mode + ExecutionRole = $Inputs.Run.ExecutionRole PromptPath = $physicalPrompt PromptBytes = $Inputs.Run.PromptBytes PromptHash = $Inputs.Run.PromptHash @@ -1456,8 +1677,11 @@ function New-OpenCodeExecutionProjection { HomeDirectoryPath = $physicalHome SkillDirectoryPath = $physicalSkill CandidateSkillExposed = $Inputs.Run.CandidateSkillExposed + CandidateSkillName = $Inputs.Run.CandidateSkillName FixtureHash = $Inputs.Run.FixtureHash SkillHash = $Inputs.Run.SkillHash + CandidateInstructionHash = $Inputs.Run.CandidateInstructionHash + GitWorkspace = $Inputs.Run.GitWorkspace InteractionPath = $null InteractionHash = $Inputs.Run.InteractionHash Interaction = $Inputs.Run.Interaction @@ -2474,6 +2698,18 @@ function Invoke-OpenCodeExecute { } elseif ([string]::IsNullOrWhiteSpace($finalText)) { $reason = 'opencode_did_not_return_final_response'; $warnings.Add('OpenCode exited successfully without a text response.') } + $boundaryAssessment = if ($hardFilesystem) { + ConvertTo-OpenCodeBoundaryAssessment + } else { + Get-OpenCodeBoundaryAssessment -Events @($parsed.Events) -Projection $projection + } + $boundaryOutcome = Apply-OpenCodeBoundaryAssessment -Status $status -FailureCode ([string]$reason) -FailureMessage ([string]$failureMessage) -Assessment $boundaryAssessment + $status = [string]$boundaryOutcome.Status + $reason = if ([string]::IsNullOrWhiteSpace([string]$boundaryOutcome.FailureCode)) { $reason } else { [string]$boundaryOutcome.FailureCode } + $failureMessage = if ([string]::IsNullOrWhiteSpace([string]$boundaryOutcome.FailureMessage)) { $failureMessage } else { [string]$boundaryOutcome.FailureMessage } + if ([string]$reason -eq 'opencode_projection_escape_observed') { + $failure = New-ExecutionFailure -Code 'opencode_projection_escape_observed' -Message $failureMessage + } $telemetry = [ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $true }) tokens = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } @@ -2551,6 +2787,7 @@ function Invoke-OpenCodeExecute { artifact = $transcriptArtifactPath sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } } + boundary = $boundaryOutcome.BoundaryEvidence delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = [string]$descriptor.delegation.mechanism @@ -2565,8 +2802,10 @@ function Invoke-OpenCodeExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + execution_role = [string]$Inputs.Run.ExecutionRole + paired_arm_visible = [bool]$boundaryOutcome.PairedArmVisible + grading_material_visible = [bool]$boundaryOutcome.GradingMaterialVisible + projection_escape_observed = [bool]$boundaryOutcome.ProjectionEscapeObserved nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/phase2-grading.ps1 b/scripts/eval-runners/phase2-grading.ps1 new file mode 100644 index 0000000..d55bca0 --- /dev/null +++ b/scripts/eval-runners/phase2-grading.ps1 @@ -0,0 +1,704 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'runner-common.ps1') +} +if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +} +if (-not (Get-Command Assert-ExecutionFreeze -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'execution-freeze.ps1') +} + +$script:Phase2Schema = 'codebeltnet/agentic/eval-phase2-state/1' +$script:GradingFreezeSchema = 'codebeltnet/agentic/eval-grading-freeze/1' +$script:AnalyzerProfileSchema = 'codebeltnet/agentic/eval-analyzer-profile/1' +$script:AnalyzerResultSchema = 'codebeltnet/agentic/eval-analyzer-result/1' +$script:ValidatorResultSchema = 'codebeltnet/agentic/eval-validator-result/1' + +function Get-Phase2SchemaNames { + return [ordered]@{ + Phase2State = $script:Phase2Schema + GradingFreeze = $script:GradingFreezeSchema + AnalyzerProfile = $script:AnalyzerProfileSchema + AnalyzerResult = $script:AnalyzerResultSchema + ValidatorResult = $script:ValidatorResultSchema + RootGrading = (Get-RunnerSchemaNames).Grading + } +} + +function Get-AssertionText { + param([Parameter(Mandatory = $true)][object]$Assertion) + + if ($Assertion -is [string]) { return [string]$Assertion } + $text = [string](Get-JsonProperty -Object $Assertion -Name 'assertion' -Default (Get-JsonProperty -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { throw 'Assertion text must be non-empty.' } + return $text +} + +function Normalize-EvalAssertion { + param( + [Parameter(Mandatory = $true)][object]$Assertion, + [Parameter(Mandatory = $true)][int]$Index + ) + + if ($Assertion -is [string]) { + $text = [string]$Assertion + if ([string]::IsNullOrWhiteSpace($text)) { throw "Assertion $Index is empty." } + return [ordered]@{ + assertion = $text + evidence_domain = 'output' + validator = $null + } + } + + $text = [string](Get-JsonProperty -Object $Assertion -Name 'assertion' -Default (Get-JsonProperty -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { throw "Assertion $Index is missing assertion text." } + $domain = [string](Get-JsonProperty -Object $Assertion -Name 'evidence_domain' -Default 'output') + if ($domain -notin @('output', 'transcript', 'validator')) { throw "Assertion $Index evidence_domain '$domain' is unsupported." } + $validator = Get-JsonProperty -Object $Assertion -Name 'validator' -Default $null + if ($domain -eq 'validator' -and [string]::IsNullOrWhiteSpace([string]$validator)) { throw "Assertion $Index with evidence_domain=validator must declare validator." } + if ($domain -ne 'validator' -and -not [string]::IsNullOrWhiteSpace([string]$validator)) { throw "Assertion $Index declares validator for non-validator evidence_domain '$domain'." } + + return [ordered]@{ + assertion = $text + evidence_domain = $domain + validator = if ($null -eq $validator -or [string]::IsNullOrWhiteSpace([string]$validator)) { $null } else { [string]$validator } + } +} + +function Get-NormalizedAssertions { + param([Parameter(Mandatory = $true)][object[]]$Assertions) + + $normalized = [System.Collections.Generic.List[object]]::new() + for ($index = 0; $index -lt @($Assertions).Count; $index++) { + $normalized.Add((Normalize-EvalAssertion -Assertion $Assertions[$index] -Index $index)) + } + if ($normalized.Count -eq 0) { throw 'Each eval must declare at least one assertion.' } + return @($normalized.ToArray()) +} + +function Get-EvalMetadataAssertionObjects { + param([Parameter(Mandatory = $true)][object]$Record) + + $metadata = Read-RunnerJson -Path $Record.MetadataPath + $assertions = @(Get-JsonProperty -Object $metadata -Name 'assertions' -Default @()) + return @(Get-NormalizedAssertions -Assertions $assertions) +} + +function Get-ArmKey { + param([Parameter(Mandatory = $true)][int]$EvalId, [Parameter(Mandatory = $true)][string]$Configuration) + + return ('arm-{0}-{1}' -f $EvalId, $Configuration) +} + +function Get-ArmOutputLines { + param([Parameter(Mandatory = $true)][string]$Output) + + $lines = $Output -split "`r?`n", -1 + return @($lines) +} + +function Get-CanonicalTranscriptArtifactPaths { + param([Parameter(Mandatory = $true)][object]$Canonical) + + $artifacts = [System.Collections.Generic.List[string]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($value in @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @())) { + $artifact = [string]$value + if ([string]::IsNullOrWhiteSpace($artifact)) { continue } + if ($seen.Add($artifact)) { $artifacts.Add($artifact) } + } + $executionResultFile = [string](Get-JsonProperty -Object $Canonical -Name 'execution_result_file' -Default '') + if (-not [string]::IsNullOrWhiteSpace($executionResultFile) -and $seen.Add($executionResultFile)) { + $artifacts.Add($executionResultFile) + } + return @($artifacts.ToArray()) +} + +function Resolve-TranscriptArtifactSourcePath { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Artifact, + [switch]$AllowMissing + ) + + Assert-SafeRelativePath -RelativePath $Artifact -FieldName 'transcript evidence artifact' + $basePath = (Resolve-Path -LiteralPath $Record.EvalDirectory -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $basePath ($Artifact -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $basePath -CandidatePath $candidate)) { + throw "transcript evidence artifact '$Artifact' resolves outside the frozen eval directory." + } + if (-not (Test-Path -LiteralPath $candidate -PathType Leaf)) { + if ($AllowMissing) { return $null } + throw "transcript evidence artifact '$Artifact' does not exist in the frozen eval directory." + } + $resolved = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $basePath -CandidatePath $resolved)) { + throw "transcript evidence artifact '$Artifact' resolves through a link outside the frozen eval directory." + } + return $resolved +} + +function Get-TranscriptArtifactSnapshot { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Artifact, + [string]$SourcePath = '' + ) + + if ([string]::IsNullOrWhiteSpace($SourcePath)) { + $SourcePath = Resolve-TranscriptArtifactSourcePath -Record $Record -Artifact $Artifact + } + + $entry = [ordered]@{ + artifact = [string]$Artifact + source_sha256 = Get-Sha256HexFromFile -Path $SourcePath + } + if ($Artifact -match '\.jsonl$') { + $events = [System.Collections.Generic.List[object]]::new() + $fileLines = [System.IO.File]::ReadAllLines($SourcePath, [System.Text.UTF8Encoding]::new($false)) + foreach ($fileLine in $fileLines) { + if ([string]::IsNullOrWhiteSpace($fileLine)) { continue } + $sourceText = [string]$fileLine + try { + $parsed = $fileLine | ConvertFrom-Json -Depth 100 + $eventType = [string](Get-JsonProperty -Object $parsed -Name 'type' -Default '') + $content = Get-JsonProperty -Object $parsed -Name 'content' -Default $null + if ($null -eq $content) { $content = Get-JsonProperty -Object $parsed -Name 'data' -Default $null } + $contentStr = if ($null -ne $content) { [string]$content } else { ConvertTo-Json $parsed -Compress -Depth 10 } + $events.Add([ordered]@{ + event_index = $events.Count + type = $eventType + content = $contentStr + source_text = $sourceText + }) + } catch { + $events.Add([ordered]@{ + event_index = $events.Count + type = '' + content = $sourceText + source_text = $sourceText + }) + } + } + $entry['events'] = @($events.ToArray()) + } else { + $fileLines = [System.IO.File]::ReadAllLines($SourcePath, [System.Text.UTF8Encoding]::new($false)) + $textLines = [System.Collections.Generic.List[object]]::new() + for ($lineIndex = 0; $lineIndex -lt $fileLines.Count; $lineIndex++) { + $textLines.Add([ordered]@{ line = $lineIndex + 1; text = [string]$fileLines[$lineIndex] }) + } + $entry['lines'] = @($textLines.ToArray()) + } + + return [pscustomobject]@{ + Artifact = [string]$Artifact + SourcePath = $SourcePath + Entry = $entry + } +} + +function Get-CanonicalTranscriptArtifacts { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Canonical + ) + + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($artifact in @(Get-CanonicalTranscriptArtifactPaths -Canonical $Canonical)) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $Record -Artifact $artifact -AllowMissing + if ($null -eq $sourcePath) { continue } + $entries.Add((Get-TranscriptArtifactSnapshot -Record $Record -Artifact $artifact -SourcePath $sourcePath).Entry) + } + return @($entries.ToArray()) +} + +function New-OutputEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Output, + [Parameter(Mandatory = $true)][string]$Quote + ) + + $lines = @(Get-ArmOutputLines -Output $Output) + if ([string]::IsNullOrWhiteSpace($Quote)) { throw 'Output evidence quote must be non-empty.' } + for ($index = 0; $index -lt $lines.Count; $index++) { + $line = [string]$lines[$index] + if ($line.Contains($Quote, [StringComparison]::Ordinal)) { + return [ordered]@{ + artifact = [string]$Record.ResultRelative + domain = 'output' + start_line = $index + 1 + end_line = $index + 1 + quote = $Quote + } + } + } + throw 'Output evidence quote is absent from the frozen canonical output.' +} + +function Test-CommandText { + param([object]$Command) + + $text = [string](Get-JsonProperty -Object $Command -Name 'command' -Default (Get-JsonProperty -Object $Command -Name 'tool' -Default '')) + return $text +} + +function Get-NormalizedCommandEvidence { + param([Parameter(Mandatory = $true)][object]$Canonical) + + $commands = [System.Collections.Generic.List[object]]::new() + foreach ($command in @(Get-JsonProperty -Object $Canonical -Name 'shell_commands' -Default @())) { + $text = Test-CommandText -Command $command + if ([string]::IsNullOrWhiteSpace($text)) { continue } + $commands.Add([ordered]@{ + command = $text + status = Get-JsonProperty -Object $command -Name 'status' -Default $null + exit_code = Get-JsonProperty -Object $command -Name 'exit_code' -Default $null + }) + } + return @($commands.ToArray()) +} + +function Test-CommandMatches { + param( + [Parameter(Mandatory = $true)][object[]]$Commands, + [Parameter(Mandatory = $true)][string[]]$Patterns + ) + + foreach ($command in @($Commands)) { + $text = [string](Get-JsonProperty -Object $command -Name 'command' -Default '') + foreach ($pattern in @($Patterns)) { + if ($text -match $pattern) { return $command } + } + } + return $null +} + +function New-ValidatorEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][string]$Rule, + [Parameter(Mandatory = $true)][bool]$Passed, + [object]$Evidence = $null + ) + + return [ordered]@{ + artifact = [string]$Record.ResultRelative + domain = 'validator' + rule = $Rule + version = 1 + passed = $Passed + event = if ($null -eq $Evidence) { $null } else { Get-JsonProperty -Object $Evidence -Name 'command' -Default $null } + } +} + +function Invoke-GradingValidatorRule { + param( + [Parameter(Mandatory = $true)][string]$Rule, + [Parameter(Mandatory = $true)][object]$Canonical, + [Parameter(Mandatory = $true)][object]$Record + ) + + $commands = @(Get-NormalizedCommandEvidence -Canonical $Canonical) + $filesWritten = @(Get-JsonProperty -Object $Canonical -Name 'files_written' -Default @()) + $candidateSkill = 'dotnet-change-impact' + $matched = $null + $passed = $false + $reason = '' + switch ($Rule) { + 'git.current_branch_observed' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(branch\s+--show-current|rev-parse\s+--abbrev-ref\s+HEAD|status\b)') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event observed the current branch.' } else { 'No structured command event observed current-branch resolution.' } + } + 'git.default_branch_resolved' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(symbolic-ref\s+refs/remotes/origin/HEAD|remote\s+show\s+origin|merge-base\b|rev-parse\s+origin/(HEAD|main|master|trunk))') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event resolved a local base/default branch.' } else { 'No structured command event resolved a local base/default branch.' } + } + 'git.branch_commits_collected' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\b(log|rev-list)\b.*(\.\.|origin/HEAD|trunk|main|master)') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event collected branch-only commits.' } else { 'No structured command event collected branch-only commits.' } + } + 'git.three_dot_diff_observed' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\bgit\b.*\bdiff\b.*\.\.\.') + $passed = $null -ne $matched + $reason = if ($passed) { 'A structured command event collected a three-dot net diff.' } else { 'No structured command event collected a three-dot net diff.' } + } + 'skill.no_candidate_package_search' { + $matched = Test-CommandMatches -Commands $commands -Patterns @("(?i)\b(search|find|list)\b.*$([regex]::Escape($candidateSkill))", "(?i)\b(dotnet|npm|pip|winget|choco|brew)\b.*\b(search|list)\b.*$([regex]::Escape($candidateSkill))") + $passed = $null -eq $matched + $reason = if ($passed) { 'No structured command event searched for the candidate skill as a package or tool.' } else { 'A structured command event searched for the candidate skill as a package or tool.' } + } + 'skill.no_candidate_install_attempt' { + $matched = Test-CommandMatches -Commands $commands -Patterns @("(?i)\b(install|add)\b.*$([regex]::Escape($candidateSkill))", "(?i)\b(dotnet|npm|pip|winget|choco|brew)\b.*\b(install|add)\b.*$([regex]::Escape($candidateSkill))") + $passed = $null -eq $matched + $reason = if ($passed) { 'No structured command event attempted to install the candidate skill as a package or tool.' } else { 'A structured command event attempted to install the candidate skill as a package or tool.' } + } + 'workspace.no_unnecessary_mutation' { + $matched = Test-CommandMatches -Commands $commands -Patterns @('(?i)\b(dotnet\s+(add|new|tool\s+install|restore)|npm\s+install|pip\s+install|git\s+(commit|checkout|switch|merge|rebase|reset)|Set-Content|Out-File|New-Item|Remove-Item)\b') + $passed = ($null -eq $matched -and $filesWritten.Count -eq 0) + $reason = if ($passed) { 'No structured file-write evidence or mutation command was captured for this read-only classification task.' } else { 'Mutation evidence was captured for this read-only classification task.' } + } + default { throw "Unsupported validator rule '$Rule'." } + } + + return [ordered]@{ + schema = $script:ValidatorResultSchema + rule = $Rule + version = 1 + eval_id = [int]$Record.EvalId + eval_name = [string]$Record.EvalName + configuration = [string]$Record.Configuration + passed = [bool]$passed + reason = $reason + evidence_refs = @((New-ValidatorEvidenceRef -Record $Record -Rule $Rule -Passed ([bool]$passed) -Evidence $matched)) + } +} + +function Get-GraderContractPath { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $path = Join-Path $IterationDirectory 'tools/skill-creator/agents/grader.md' + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { throw "Packaged grader contract is missing at '$path'." } + return $path +} + +function Resolve-AnalyzerProfile { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $relative = [string](Get-JsonProperty -Object $Manifest -Name 'analyzer_profile' -Default 'analyzer-profile.json') + $path = Resolve-ManifestDeclaredPath -IterationDirectory $IterationDirectory -RelativePath $relative -FieldName 'analyzer_profile' -Kind File -RequireExists + $profile = Read-RunnerJson -Path $path + $schemas = Get-Phase2SchemaNames + if ([string](Get-JsonProperty -Object $profile -Name 'schema' -Default '') -ne $schemas.AnalyzerProfile) { throw 'analyzer-profile.json has an unsupported schema.' } + foreach ($name in @('contract_version', 'runner', 'harness', 'model', 'selection_source')) { + if ([string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $profile -Name $name -Default ''))) { throw "analyzer-profile.json must declare non-empty '$name'." } + } + if ([string](Get-JsonProperty -Object $profile -Name 'contract_version' -Default '') -ne $schemas.RootGrading) { throw 'analyzer-profile.json contract_version does not match the grading schema.' } + $source = [string](Get-JsonProperty -Object $profile -Name 'selection_source' -Default '') + if ($source -notin @('codebelt-reference', 'explicit')) { throw "analyzer-profile.json selection_source '$source' is unsupported." } + $runner = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') + if ($runner -notmatch '^[a-z0-9][a-z0-9-]*$') { throw 'analyzer-profile.json runner must be a safe lowercase runner id.' } + return [pscustomobject]@{ + Path = $path + RelativePath = $relative + Profile = $profile + Hash = Get-Sha256HexFromFile -Path $path + Runner = $runner + Model = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') + ReasoningEffort = Get-JsonProperty -Object $profile -Name 'reasoning_effort' -Default $null + Harness = [string](Get-JsonProperty -Object $profile -Name 'harness' -Default '') + SelectionSource = $source + } +} + +function Get-ExpectedGradeRecords { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest | Sort-Object EvalId, Configuration) + $expected = [System.Collections.Generic.List[object]]::new() + foreach ($record in $records) { + $assertions = @(Get-EvalMetadataAssertionObjects -Record $record) + for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertion = $assertions[$index] + $expected.Add([ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertion.assertion + evidence_domain = [string]$assertion.evidence_domain + validator = Get-JsonProperty -Object $assertion -Name 'validator' -Default $null + record = $record + }) + } + } + return @($expected.ToArray()) +} + +function ConvertTo-GradingEntry { + param( + [Parameter(Mandatory = $true)][object]$Expected, + [Parameter(Mandatory = $true)][bool]$Passed, + [Parameter(Mandatory = $true)][string]$Reason, + [Parameter(Mandatory = $true)][object[]]$EvidenceRefs, + [string]$Source = 'analyzer', + [string]$Evidence = '' + ) + + if ([string]::IsNullOrWhiteSpace($Evidence)) { + $Evidence = $Reason + } + return [ordered]@{ + eval_id = [int]$Expected.eval_id + eval_name = [string]$Expected.eval_name + configuration = [string]$Expected.configuration + assertion_index = [int]$Expected.assertion_index + assertion = [string]$Expected.assertion + passed = [bool]$Passed + evidence_domain = [string]$Expected.evidence_domain + evidence_refs = @($EvidenceRefs) + reason = $Reason + evidence = $Evidence + source = $Source + } +} + +function Get-GradeKey { + param([Parameter(Mandatory = $true)][object]$Grade) + + return ('{0}|{1}|{2}' -f [int](Get-JsonProperty -Object $Grade -Name 'eval_id' -Default 0), [string](Get-JsonProperty -Object $Grade -Name 'configuration' -Default ''), [int](Get-JsonProperty -Object $Grade -Name 'assertion_index' -Default -1)) +} + +function Test-GradeEvidenceReference { + param( + [Parameter(Mandatory = $true)][object]$Grade, + [Parameter(Mandatory = $true)][object]$Expected, + [Parameter(Mandatory = $true)][object]$Canonical, + [AllowNull()][object[]]$TranscriptArtifacts = $null + ) + + $domain = [string](Get-JsonProperty -Object $Grade -Name 'evidence_domain' -Default '') + if ($domain -ne [string]$Expected.evidence_domain) { throw "grade '$((Get-GradeKey -Grade $Grade))' evidence_domain does not match the assertion." } + $refs = @(Get-JsonProperty -Object $Grade -Name 'evidence_refs' -Default @()) + if ($refs.Count -eq 0) { throw "grade '$((Get-GradeKey -Grade $Grade))' must declare evidence_refs." } + foreach ($ref in $refs) { + $refDomain = [string](Get-JsonProperty -Object $ref -Name 'domain' -Default $domain) + if ($refDomain -ne $domain) { throw "grade '$((Get-GradeKey -Grade $Grade))' has a mismatched evidence ref domain." } + $artifact = [string](Get-JsonProperty -Object $ref -Name 'artifact' -Default '') + if ([string]::IsNullOrWhiteSpace($artifact)) { throw "grade '$((Get-GradeKey -Grade $Grade))' has an empty evidence artifact ref." } + if ($domain -eq 'output') { + if ($artifact -ne [string]$Expected.record.ResultRelative) { throw 'output evidence must cite this arm canonical result only.' } + $line = [int](Get-JsonProperty -Object $ref -Name 'start_line' -Default 0) + $endLine = [int](Get-JsonProperty -Object $ref -Name 'end_line' -Default $line) + $lines = @(Get-ArmOutputLines -Output ([string]$Canonical.output)) + if ($line -lt 1 -or $endLine -lt $line -or $endLine -gt $lines.Count) { throw 'output evidence line range is outside the frozen output.' } + $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $span = [string]::Join("`n", @($lines[($line - 1)..($endLine - 1)])) + if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { throw 'output evidence quote is absent from the referenced frozen output lines.' } + } + } elseif ($domain -eq 'validator') { + $rule = [string](Get-JsonProperty -Object $ref -Name 'rule' -Default '') + if ($rule -ne [string]$Expected.validator) { throw 'validator evidence must cite the assertion validator rule.' } + } elseif ($domain -eq 'transcript') { + if ($artifact -eq [string]$Expected.record.ResultRelative) { throw 'transcript evidence must not cite final output prose.' } + $allowed = @(Get-JsonProperty -Object $Canonical -Name 'output_files' -Default @()) + $execResultFile = [string](Get-JsonProperty -Object $Canonical -Name 'execution_result_file' -Default '') + if ($allowed -notcontains $artifact -and $artifact -ne $execResultFile) { throw 'transcript evidence must cite a frozen transcript/artifact for the same arm.' } + if ($null -eq $TranscriptArtifacts) { + throw 'transcript evidence requires the frozen transcript evidence set so the cited location can be grounded.' + } + $matchingArtifacts = @($TranscriptArtifacts | Where-Object { [string]$_.artifact -eq $artifact }) + if ($matchingArtifacts.Count -eq 0) { + throw 'transcript evidence artifact is not present in the frozen transcript evidence set; grounding cannot be verified.' + } + $transcriptEntry = $matchingArtifacts[0] + $hasEventLocator = Test-JsonProperty -Object $ref -Name 'event_index' + $hasStartLine = Test-JsonProperty -Object $ref -Name 'start_line' + $hasEndLine = Test-JsonProperty -Object $ref -Name 'end_line' + $hasLineLocator = $hasStartLine -or $hasEndLine + $quote = [string](Get-JsonProperty -Object $ref -Name 'quote' -Default '') + $hasEvents = Test-JsonProperty -Object $transcriptEntry -Name 'events' + $hasLines = Test-JsonProperty -Object $transcriptEntry -Name 'lines' + if ($hasEventLocator -and $hasLineLocator) { + throw 'transcript evidence must declare exactly one locator form.' + } + if (-not $hasEventLocator -and -not $hasLineLocator) { + throw 'transcript evidence must declare event_index or start_line/end_line to ground the cited location.' + } + if ($hasEventLocator) { + if ($hasLines) { + throw 'transcript evidence uses event_index but the frozen artifact has line-based content.' + } + if (-not $hasEvents) { + throw 'transcript evidence event_index cannot be applied because the frozen artifact has no event-based content.' + } + $events = @(Get-JsonProperty -Object $transcriptEntry -Name 'events' -Default @()) + $eventIndex = [int](Get-JsonProperty -Object $ref -Name 'event_index' -Default -1) + if ($eventIndex -lt 0 -or $eventIndex -ge $events.Count) { + throw "transcript evidence event_index $eventIndex is outside the frozen artifact ($($events.Count) events)." + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $eventText = [string](Get-JsonProperty -Object $events[$eventIndex] -Name 'source_text' -Default (Get-JsonProperty -Object $events[$eventIndex] -Name 'content' -Default '')) + if (-not $eventText.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen event.' + } + } + } else { + if (-not $hasStartLine -or -not $hasEndLine) { + throw 'transcript evidence line locator must declare both start_line and end_line.' + } + if ($hasEvents) { + throw 'transcript evidence uses start_line/end_line but the frozen artifact has event-based content.' + } + if (-not $hasLines) { + throw 'transcript evidence line locator cannot be applied because the frozen artifact has no line-based content.' + } + $startLine = [int](Get-JsonProperty -Object $ref -Name 'start_line' -Default 0) + $endLine = [int](Get-JsonProperty -Object $ref -Name 'end_line' -Default $startLine) + $lines = @(Get-JsonProperty -Object $transcriptEntry -Name 'lines' -Default @()) + if ($startLine -lt 1 -or $endLine -lt $startLine -or $endLine -gt $lines.Count) { + throw 'transcript evidence line range is outside the frozen artifact.' + } + if (-not [string]::IsNullOrWhiteSpace($quote)) { + $span = [string]::Join("`n", @($lines[($startLine - 1)..($endLine - 1)] | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + if (-not $span.Contains($quote, [StringComparison]::Ordinal)) { + throw 'transcript evidence quote is absent from the referenced frozen transcript lines.' + } + } + } + } + } + return $true +} + +function New-RootGradingDocument { + param( + [Parameter(Mandatory = $true)][object[]]$Grades, + [object]$Metadata = $null + ) + + $ordered = @($Grades | Sort-Object @{ Expression = { [int](Get-JsonProperty -Object $_ -Name 'eval_id' -Default 0) } }, @{ Expression = { [string](Get-JsonProperty -Object $_ -Name 'configuration' -Default '') } }, @{ Expression = { [int](Get-JsonProperty -Object $_ -Name 'assertion_index' -Default 0) } }) + $document = [ordered]@{ + schema = (Get-RunnerSchemaNames).Grading + grading = @($ordered) + } + if ($null -ne $Metadata) { $document.metadata = $Metadata } + return $document +} + +function Get-GradingMergeHash { + param([Parameter(Mandatory = $true)][object]$Document) + + return Get-JsonFingerprint -Object $Document +} + +function Get-Phase2StateHashForFreeze { + param([Parameter(Mandatory = $true)][object]$State) + + return Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $State -PropertyName 'grading_freeze') +} + +function New-GradingFreezeDocument { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][object[]]$Grades, + [object]$Metadata = $null, + [object[]]$TranscriptArtifacts = @() + ) + + $analyzer = Resolve-AnalyzerProfile -IterationDirectory $IterationDirectory -Manifest $Manifest + $graderPath = Get-GraderContractPath -IterationDirectory $IterationDirectory + $expected = @(Get-ExpectedGradeRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + $root = New-RootGradingDocument -Grades $Grades -Metadata $Metadata + $validatorPaths = @(Get-JsonProperty -Object $State -Name 'validator_results' -Default @()) + $analyzerResults = @(Get-JsonProperty -Object $State -Name 'analyzer_results' -Default @()) + + $freeze = [ordered]@{ + schema = $script:GradingFreezeSchema + analyzer_profile = [ordered]@{ path = [string]$analyzer.RelativePath; sha256 = [string]$analyzer.Hash } + analyzer_profile_sha256 = [string]$analyzer.Hash + grader_contract = [ordered]@{ path = 'tools/skill-creator/agents/grader.md'; sha256 = Get-Sha256HexFromFile -Path $graderPath } + assertions_sha256 = Get-JsonFingerprint -Object @($expected | ForEach-Object { + [ordered]@{ + eval_id = [int]$_.eval_id + eval_name = [string]$_.eval_name + configuration = [string]$_.configuration + assertion_index = [int]$_.assertion_index + assertion = [string]$_.assertion + evidence_domain = [string]$_.evidence_domain + validator = Get-JsonProperty -Object $_ -Name 'validator' -Default $null + } + }) + validator_results = @($validatorPaths) + analyzer_results = @($analyzerResults) + transcript_artifacts = @($TranscriptArtifacts) + phase2_state_sha256 = Get-Phase2StateHashForFreeze -State $State + expected_grade_count = $expected.Count + grading_sha256 = Get-GradingMergeHash -Document $root + generated_utc = (Format-UtcTimestamp -Value ([DateTime]::UtcNow)) + } + return $freeze +} + +function Assert-GradingFreeze { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' + ) + + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifest = Read-RunnerJson -Path (Join-Path $iteration 'manifest.json') + $statePath = Join-Path $iteration 'phase2-state.json' + $freezePath = Join-Path $iteration 'grading-freeze.json' + if (-not (Test-Path -LiteralPath $freezePath -PathType Leaf)) { throw 'grading-freeze.json is missing; handcrafted grading.json cannot finalize.' } + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { throw 'Phase 2 state is missing.' } + $state = Read-RunnerJson -Path $statePath + $freeze = Read-RunnerJson -Path $freezePath + if ([string](Get-JsonProperty -Object $state -Name 'schema' -Default '') -ne $script:Phase2Schema) { throw 'phase2-state.json has an unsupported schema.' } + if ([string](Get-JsonProperty -Object $freeze -Name 'schema' -Default '') -ne $script:GradingFreezeSchema) { throw 'grading-freeze.json has an unsupported schema.' } + if ([string](Get-JsonProperty -Object $state -Name 'status' -Default '') -ne 'completed') { throw 'phase2-state.json is not completed.' } + if (@(Get-JsonProperty -Object $state -Name 'pending' -Default @()).Count -ne 0) { throw 'phase2-state.json still has pending analyzer workers.' } + if (@(Get-JsonPropertyNames -Object (Get-JsonProperty -Object $state -Name 'active' -Default ([ordered]@{}))).Count -ne 0) { throw 'phase2-state.json still has active analyzer workers.' } + $expectedWorkers = @(Get-JsonProperty -Object $state -Name 'expected_worker_ids' -Default @()) + $completedWorkers = Get-JsonProperty -Object $state -Name 'completed' -Default ([ordered]@{}) + if (@(Get-JsonPropertyNames -Object $completedWorkers).Count -ne $expectedWorkers.Count) { throw 'phase2-state.json terminal analyzer worker count does not match expected_worker_ids.' } + $analyzerResultRefs = @(Get-JsonProperty -Object $state -Name 'analyzer_results' -Default @()) + if ($analyzerResultRefs.Count -ne $expectedWorkers.Count) { throw 'phase2-state.json analyzer result count does not match expected semantic worker count.' } + $analyzer = Resolve-AnalyzerProfile -IterationDirectory $iteration -Manifest $manifest + if ([string](Get-JsonProperty -Object $freeze -Name 'analyzer_profile_sha256' -Default '') -ne [string]$analyzer.Hash) { throw 'Analyzer profile changed after grading.' } + if ([string](Get-JsonProperty -Object (Get-JsonProperty -Object $freeze -Name 'analyzer_profile' -Default $null) -Name 'sha256' -Default '') -ne [string]$analyzer.Hash) { throw 'grading-freeze.json analyzer_profile hash mismatch.' } + $graderPath = Get-GraderContractPath -IterationDirectory $iteration + if ([string](Get-JsonProperty -Object (Get-JsonProperty -Object $freeze -Name 'grader_contract' -Default $null) -Name 'sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $graderPath)) { throw 'Packaged grader contract changed after Phase 2 freeze.' } + $stateHash = Get-Phase2StateHashForFreeze -State $state + if ([string](Get-JsonProperty -Object $freeze -Name 'phase2_state_sha256' -Default '') -ne $stateHash) { throw 'phase2-state.json changed after Phase 2 freeze.' } + + $gradingFullPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File -RequireExists + $grading = Read-RunnerJson -Path $gradingFullPath + if ([string](Get-JsonProperty -Object $freeze -Name 'grading_sha256' -Default '') -ne (Get-GradingMergeHash -Document $grading)) { throw 'Root grading.json was changed after deterministic Phase 2 merge.' } + + foreach ($validator in @(Get-JsonProperty -Object $freeze -Name 'validator_results' -Default @())) { + $path = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$validator.path) -FieldName 'validator result' -Kind File -RequireExists + if ([string]$validator.sha256 -ne (Get-Sha256HexFromFile -Path $path)) { throw "Validator result '$($validator.path)' changed after Phase 2 freeze." } + } + $workerSessions = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + foreach ($analyzerResult in @(Get-JsonProperty -Object $freeze -Name 'analyzer_results' -Default @())) { + $path = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$analyzerResult.path) -FieldName 'analyzer result' -Kind File -RequireExists + if ([string]$analyzerResult.sha256 -ne (Get-Sha256HexFromFile -Path $path)) { throw "Analyzer result '$($analyzerResult.path)' changed after Phase 2 freeze." } + $result = Read-RunnerJson -Path $path + if ([string](Get-JsonProperty -Object $result -Name 'analyzer_profile_sha256' -Default '') -ne [string]$analyzer.Hash) { throw "Analyzer result '$($analyzerResult.path)' uses a different analyzer profile." } + if ([string](Get-JsonProperty -Object $result -Name 'observed_model' -Default '') -ne [string]$analyzer.Model) { throw "Analyzer result '$($analyzerResult.path)' observed the wrong model." } + if ([int](Get-JsonProperty -Object $result -Name 'attempt_count' -Default 0) -ne 1) { throw "Analyzer worker '$($analyzerResult.path)' was retried." } + $sessionId = [string](Get-JsonProperty -Object $result -Name 'session_id' -Default '') + if ([string]::IsNullOrWhiteSpace($sessionId) -or -not $workerSessions.Add($sessionId)) { throw 'Analyzer worker/session identity was reused.' } + $rawPathValue = [string](Get-JsonProperty -Object $result -Name 'raw_execution_result' -Default '') + $rawPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $rawPathValue -FieldName 'analyzer raw execution result' -Kind File -RequireExists + if ([string](Get-JsonProperty -Object $result -Name 'raw_execution_result_sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $rawPath)) { throw "Analyzer raw transcript/evidence '$rawPathValue' changed after Phase 2 freeze." } + $fragmentPathValue = [string](Get-JsonProperty -Object $result -Name 'grading_fragment' -Default '') + $fragmentPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $fragmentPathValue -FieldName 'analyzer grading fragment' -Kind File -RequireExists + if ([string](Get-JsonProperty -Object $result -Name 'grading_fragment_sha256' -Default '') -ne (Get-Sha256HexFromFile -Path $fragmentPath)) { throw "Analyzer grading fragment '$fragmentPathValue' changed after Phase 2 freeze." } + } + foreach ($ta in @(Get-JsonProperty -Object $freeze -Name 'transcript_artifacts' -Default @())) { + $taPath = [string](Get-JsonProperty -Object $ta -Name 'artifact' -Default '') + $taHash = [string](Get-JsonProperty -Object $ta -Name 'sha256' -Default '') + if ([string]::IsNullOrWhiteSpace($taPath) -or [string]::IsNullOrWhiteSpace($taHash)) { throw 'grading-freeze.json transcript_artifacts entry is missing artifact or sha256.' } + $taFullPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $taPath -FieldName 'transcript artifact' -Kind File -RequireExists + if ((Get-Sha256HexFromFile -Path $taFullPath) -ne $taHash) { throw "Transcript artifact '$taPath' changed after Phase 2 freeze." } + } + + return [pscustomobject]@{ Path = $freezePath; Freeze = $freeze; State = $state; Manifest = $manifest; Analyzer = $analyzer; Grading = $grading } +} diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 89b26be..da942b5 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -24,6 +24,11 @@ function Get-RunnerSchemaNames { Interaction = 'codebeltnet/agentic/eval-interaction/1' ExecutionFreeze = 'codebeltnet/agentic/eval-execution-freeze/1' Grading = 'codebeltnet/agentic/eval-grading/1' + AnalyzerProfile = 'codebeltnet/agentic/eval-analyzer-profile/1' + Phase2State = 'codebeltnet/agentic/eval-phase2-state/1' + GradingFreeze = 'codebeltnet/agentic/eval-grading-freeze/1' + AnalyzerResult = 'codebeltnet/agentic/eval-analyzer-result/1' + ValidatorResult = 'codebeltnet/agentic/eval-validator-result/1' } } @@ -330,6 +335,203 @@ function Test-PathInside { return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) } +function Get-ObservedPathSegments { + param([AllowEmptyString()][string]$PathText) + + if ([string]::IsNullOrWhiteSpace($PathText)) { return @() } + $segments = [System.Collections.Generic.List[string]]::new() + foreach ($segment in @($PathText -split '[\\/]')) { + if ($null -ne $segment -and [string]$segment -ne '') { $segments.Add([string]$segment) } + } + return @($segments.ToArray()) +} + +function Resolve-ObservedPathSegments { + param( + [AllowEmptyCollection()][string[]]$BaseSegments = @(), + [AllowEmptyCollection()][string[]]$PathSegments = @() + ) + + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($segment in @($BaseSegments)) { + if ($null -ne $segment -and [string]$segment -ne '') { $resolved.Add([string]$segment) } + } + foreach ($segment in @($PathSegments)) { + switch ([string]$segment) { + '' { continue } + '.' { continue } + '..' { + if ($resolved.Count -gt 0) { $resolved.RemoveAt($resolved.Count - 1) } + continue + } + default { + $resolved.Add([string]$segment) + } + } + } + return @($resolved.ToArray()) +} + +function Join-ObservedPath { + param( + [Parameter(Mandatory = $true)][string]$Style, + [Parameter(Mandatory = $true)][string]$Root, + [AllowEmptyCollection()][string[]]$Segments = @() + ) + + if (@($Segments).Count -eq 0) { return $Root } + $separator = if ([string]$Style -eq 'unix') { '/' } else { '\' } + if ($Root.EndsWith($separator, [System.StringComparison]::Ordinal)) { + return $Root + [string]::Join($separator, @($Segments)) + } + return $Root + $separator + [string]::Join($separator, @($Segments)) +} + +function New-ObservedPathInfo { + param( + [Parameter(Mandatory = $true)][string]$Style, + [Parameter(Mandatory = $true)][string]$Root, + [AllowEmptyCollection()][string[]]$Segments = @(), + [Parameter(Mandatory = $true)][string]$Raw, + [Parameter(Mandatory = $true)][bool]$Absolute + ) + + return [pscustomobject]@{ + Style = $Style + Root = $Root + Segments = @($Segments) + FullPath = Join-ObservedPath -Style $Style -Root $Root -Segments @($Segments) + Raw = $Raw + Absolute = $Absolute + } +} + +function Get-ObservedPathInfo { + param( + [AllowEmptyString()][string]$Path, + [AllowEmptyString()][string]$BasePath = '' + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return $null } + $trimmed = [string]$Path + while ($trimmed.Length -ge 2 -and ( + ($trimmed.StartsWith('"', [System.StringComparison]::Ordinal) -and $trimmed.EndsWith('"', [System.StringComparison]::Ordinal)) -or + ($trimmed.StartsWith("'", [System.StringComparison]::Ordinal) -and $trimmed.EndsWith("'", [System.StringComparison]::Ordinal)))) { + $trimmed = $trimmed.Substring(1, $trimmed.Length - 2) + } + $trimmed = $trimmed.Trim() + if ([string]::IsNullOrWhiteSpace($trimmed)) { return $null } + + if ($trimmed -match '^[A-Za-z]:[\\/]') { + $segments = Resolve-ObservedPathSegments -PathSegments (Get-ObservedPathSegments -PathText $trimmed.Substring(3)) + return New-ObservedPathInfo -Style 'windows' -Root ($trimmed.Substring(0, 1).ToUpperInvariant() + ':\') -Segments $segments -Raw $trimmed -Absolute $true + } + if ($trimmed -match '^\\\\') { + $uncSegments = @(Get-ObservedPathSegments -PathText $trimmed.Substring(2)) + if ($uncSegments.Count -lt 2) { return $null } + $pathSegments = if ($uncSegments.Count -gt 2) { @($uncSegments[2..($uncSegments.Count - 1)]) } else { @() } + $segments = Resolve-ObservedPathSegments -PathSegments $pathSegments + return New-ObservedPathInfo -Style 'unc' -Root ('\\' + $uncSegments[0] + '\' + $uncSegments[1]) -Segments $segments -Raw $trimmed -Absolute $true + } + if ($trimmed -match '^/') { + $segments = Resolve-ObservedPathSegments -PathSegments (Get-ObservedPathSegments -PathText $trimmed.Substring(1)) + return New-ObservedPathInfo -Style 'unix' -Root '/' -Segments $segments -Raw $trimmed -Absolute $true + } + if ([string]::IsNullOrWhiteSpace($BasePath)) { return $null } + $baseInfo = Get-ObservedPathInfo -Path $BasePath + if ($null -eq $baseInfo) { return $null } + $segments = Resolve-ObservedPathSegments -BaseSegments @($baseInfo.Segments) -PathSegments (Get-ObservedPathSegments -PathText $trimmed) + return New-ObservedPathInfo -Style ([string]$baseInfo.Style) -Root ([string]$baseInfo.Root) -Segments $segments -Raw $trimmed -Absolute $false +} + +function Get-ObservedParentPath { + param([AllowEmptyString()][string]$Path) + + $info = Get-ObservedPathInfo -Path $Path + if ($null -eq $info) { return $null } + if (@($info.Segments).Count -eq 0) { return [string]$info.FullPath } + $parentSegments = if (@($info.Segments).Count -gt 1) { @($info.Segments[0..(@($info.Segments).Count - 2)]) } else { @() } + return Join-ObservedPath -Style ([string]$info.Style) -Root ([string]$info.Root) -Segments $parentSegments +} + +function Get-ObservedRelativePath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = Get-ObservedPathInfo -Path $BasePath + $candidate = Get-ObservedPathInfo -Path $CandidatePath -BasePath $BasePath + if ($null -eq $base -or $null -eq $candidate -or [string]$base.Style -ne [string]$candidate.Style) { return $null } + if (-not (Test-ObservedPathInside -BasePath ([string]$base.FullPath) -CandidatePath ([string]$candidate.FullPath))) { return $null } + + $comparison = if ([string]$base.Style -eq 'unix') { [System.StringComparison]::Ordinal } else { [System.StringComparison]::OrdinalIgnoreCase } + $baseSegments = @($base.Segments) + $candidateSegments = @($candidate.Segments) + if ($candidateSegments.Count -lt $baseSegments.Count) { return $null } + for ($index = 0; $index -lt $baseSegments.Count; $index++) { + if (-not [string]::Equals([string]$baseSegments[$index], [string]$candidateSegments[$index], $comparison)) { return $null } + } + if ($candidateSegments.Count -eq $baseSegments.Count) { return '.' } + return [string]::Join('/', @($candidateSegments[$baseSegments.Count..($candidateSegments.Count - 1)])) +} + +function Test-ObservedPathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = Get-ObservedPathInfo -Path $BasePath + $candidate = Get-ObservedPathInfo -Path $CandidatePath -BasePath $BasePath + if ($null -eq $base -or $null -eq $candidate -or [string]$base.Style -ne [string]$candidate.Style) { return $false } + $comparison = if ([string]$base.Style -eq 'unix') { [System.StringComparison]::Ordinal } else { [System.StringComparison]::OrdinalIgnoreCase } + if (-not [string]::Equals([string]$base.Root, [string]$candidate.Root, $comparison)) { return $false } + + $baseSegments = @($base.Segments) + $candidateSegments = @($candidate.Segments) + if ($candidateSegments.Count -lt $baseSegments.Count) { return $false } + for ($index = 0; $index -lt $baseSegments.Count; $index++) { + if (-not [string]::Equals([string]$baseSegments[$index], [string]$candidateSegments[$index], $comparison)) { return $false } + } + return $true +} + +function Get-ObservedPathTokensFromCommandText { + param([AllowEmptyString()][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { return @() } + $tokens = [System.Collections.Generic.List[string]]::new() + $pattern = '(?:"(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^"]+)"|''(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^'']+)''|(?(?:[A-Za-z]:[\\/]|/|\.\.?[\\/])[^ \t\r\n"''`|;&,]+))' + foreach ($match in [regex]::Matches($Text, $pattern)) { + $candidate = @($match.Groups['double'].Value, $match.Groups['single'].Value, $match.Groups['bare'].Value) | + Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | + Select-Object -First 1 + if (-not [string]::IsNullOrWhiteSpace([string]$candidate) -and $tokens -notcontains [string]$candidate) { + $tokens.Add([string]$candidate) + } + } + return @($tokens.ToArray()) +} + +function Test-FileSystemCommandText { + param([AllowEmptyString()][string]$Text) + + if ([string]::IsNullOrWhiteSpace($Text)) { return $false } + return $Text -match '(?i)\b(Get-ChildItem|Get-Content|Set-Content|Add-Content|Out-File|Copy-Item|Move-Item|Remove-Item|Select-String|type|cat|ls|dir|find|grep|rg|read|write|view|edit|glob)\b' +} + +function Get-RunExecutionRole { + param([Parameter(Mandatory = $true)][object]$Run) + + $role = [string](Get-JsonProperty -Object $Run -Name 'executionRole' -Default 'eval_arm') + if ([string]::IsNullOrWhiteSpace($role)) { $role = 'eval_arm' } + if ($role -notin @('eval_arm', 'phase2_analyzer')) { + throw "run.json executionRole '$role' is unsupported." + } + return $role +} + function Assert-SafeRelativePath { param( [Parameter(Mandatory = $true)][string]$RelativePath, @@ -697,6 +899,8 @@ function ConvertTo-ComparablePath { if ([string]::IsNullOrWhiteSpace($Path)) { return $null } try { + # Host filesystem normalization only. Observed/captured cross-platform + # evidence uses the lexical observed-path helpers above instead. $full = [System.IO.Path]::GetFullPath($Path) $full = Expand-WindowsShortPath -Path $full $root = [System.IO.Path]::GetPathRoot($full) @@ -817,6 +1021,20 @@ function Test-NativeWorkerTerminalEvidence { $failures.Add('delegation_terminal_evidence') return [pscustomobject]@{ Valid = $false; Failures = @($failures); Delegation = $null } } + $executionRole = if ($Run.PSObject.Properties.Name -contains 'ExecutionRole' -and -not [string]::IsNullOrWhiteSpace([string]$Run.ExecutionRole)) { + [string]$Run.ExecutionRole + } else { + 'eval_arm' + } + $pairedArmVisible = [bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) + $forbiddenGradingVisible = if ($executionRole -eq 'phase2_analyzer') { + [bool](Get-JsonProperty -Object $delegation -Name 'paired_or_package_grading_material_visible' -Default (Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) + } else { + [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true) + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'projection_escape_observed' -Default $false)) { + $failures.Add('projection_escape_observed') + } # Different evidence requirements depending on terminal status. A # completed scripted run must provide full same-session interaction proof. @@ -873,8 +1091,7 @@ function Test-NativeWorkerTerminalEvidence { if (-not [bool](Get-JsonProperty -Object $delegation -Name 'terminal_result_capture' -Default $false)) { $failures.Add('terminal_result_capture') } - if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or - [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + if ($pairedArmVisible -or $forbiddenGradingVisible) { $failures.Add('paired_arm_and_grading_exclusion') } if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) -or @@ -884,8 +1101,7 @@ function Test-NativeWorkerTerminalEvidence { } else { # For non-success terminals, skip strict prompt/terminal-capture checks # but still validate working/home path alignment when provided. - if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or - [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + if ($pairedArmVisible -or $forbiddenGradingVisible) { $failures.Add('paired_arm_and_grading_exclusion') } } @@ -1006,6 +1222,10 @@ function Resolve-RunContract { if ($mode -notin @('with_skill', 'without_skill')) { throw "run.json mode '$mode' is not with_skill or without_skill." } + $executionRole = Get-RunExecutionRole -Run $run + if ($executionRole -eq 'phase2_analyzer' -and $mode -ne 'without_skill') { + throw 'phase2_analyzer run.json must use transport mode without_skill.' + } $candidateSkillName = [string](Get-JsonProperty -Object $run -Name 'candidateSkillName' -Default '') if ([string]::IsNullOrWhiteSpace($candidateSkillName)) { throw 'run.json must declare candidateSkillName for both with_skill and without_skill arms.' @@ -1092,6 +1312,20 @@ function Resolve-RunContract { if ($mode -eq 'with_skill' -and -not (Test-Sha256 -Value ([string]$run.skillHash))) { throw 'with_skill run.json skillHash must be a SHA-256 value.' } + # The candidate-instruction hash is the positive identity proof for the injected candidate. It covers only the exact + # canonical candidate instruction bytes the with_skill prompt embeds, not the whole prompt wrapper, so unrelated + # wrapper edits never invalidate the candidate identity. The baseline must not carry it: no candidate is injected. + $candidateInstructionHash = [string](Get-JsonProperty -Object $run -Name 'candidateInstructionHash' -Default '') + if ($mode -eq 'with_skill') { + if ([string]::IsNullOrWhiteSpace($candidateInstructionHash)) { + throw 'with_skill run.json candidateInstructionHash is required; it must be a SHA-256 hash covering the exact frozen candidate instruction bytes injected into the prompt.' + } + if (-not (Test-Sha256 -Value $candidateInstructionHash)) { + throw 'with_skill run.json candidateInstructionHash must be a valid SHA-256 value covering the exact frozen candidate instruction bytes injected into the prompt.' + } + } elseif (-not [string]::IsNullOrWhiteSpace($candidateInstructionHash)) { + throw 'without_skill run.json must not declare candidateInstructionHash; the baseline receives no candidate instructions.' + } return [pscustomobject]@{ RunPath = $resolvedRunPath @@ -1100,6 +1334,7 @@ function Resolve-RunContract { EvalId = [int]$run.evalId EvalName = [string]$run.evalName Mode = $mode + ExecutionRole = $executionRole PromptPath = $promptPath PromptBytes = $promptBytes PromptHash = Get-Sha256HexFromBytes -Bytes $promptBytes @@ -1110,6 +1345,8 @@ function Resolve-RunContract { CandidateSkillName = $candidateSkillName FixtureHash = $fixtureHash SkillHash = if ($mode -eq 'with_skill') { [string]$run.skillHash } else { $null } + CandidateInstructionHash = if ($mode -eq 'with_skill') { $candidateInstructionHash } else { $null } + GitWorkspace = [bool](Get-JsonProperty -Object $run -Name 'gitWorkspace' -Default $false) InteractionPath = $interactionPath InteractionHash = $interactionHash Interaction = $interaction @@ -1470,8 +1707,19 @@ function Assert-PhaseOneEvidenceWritable { # package-level freeze exists, refusing to build another result prevents a # direct runner invocation (or the orchestrator-owned recorder) from # truncating or replacing frozen raw evidence. - $runRoot = [System.IO.Path]::GetFullPath([string]$Run.RunRoot) - $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runRoot) + # Projected runner inputs may replace RunRoot with a physical directory + # directly under /tmp. Derive the package location from the original + # manifest path so freeze checks remain valid on every platform. + $runPath = [string](Get-JsonProperty -Object $Run -Name 'RunPath' -Default '') + $runRoot = if ([string]::IsNullOrWhiteSpace($runPath)) { + [System.IO.Path]::GetFullPath([string]$Run.RunRoot) + } else { + [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetFullPath($runPath)) + } + $iterationDirectory = [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetDirectoryName($runRoot)) + if ([string]::IsNullOrWhiteSpace($iterationDirectory)) { + throw 'Cannot derive the eval package directory from the run contract.' + } $freezeRelativePath = 'execution-freeze.json' $manifestPath = Join-Path $iterationDirectory 'manifest.json' if (Test-Path -LiteralPath $manifestPath -PathType Leaf) { diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt index 24903ba..578c020 100644 --- a/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt +++ b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt @@ -6,6 +6,7 @@ Usage: copilot [options] --allow-all-tools Allow tools --no-ask-user Disable prompts --disable-builtin-mcps Disable built-in MCP servers + --excluded-tools Tools that will not be available to the model --secret-env-vars Secret environment variables --no-auto-update Disable updates --resume Resume a previous Copilot session by its exact session id diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt index c6dfd13..ac99a10 100644 --- a/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt +++ b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt @@ -6,6 +6,7 @@ Usage: copilot [options] --allow-all-tools Allow tools --no-ask-user Disable prompts --disable-builtin-mcps Disable built-in MCP servers + --excluded-tools Tools that will not be available to the model --secret-env-vars Secret environment variables --no-auto-update Disable updates --resume Resume the most recent Copilot session by id diff --git a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 index 5edb3b1..888f8c3 100644 --- a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 +++ b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 @@ -95,6 +95,30 @@ try { return (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'evidence-validation-failed') -PathType Leaf) } + function Get-ExtraTranscriptArtifacts { + $markerPath = Join-Path $inputs.Run.HomeDirectoryPath 'extra-transcript-artifacts' + if (-not (Test-Path -LiteralPath $markerPath -PathType Leaf)) { + return @() + } + return @( + [ordered]@{ + Path = 'evidence/a/events.jsonl' + Content = @('{"type":"assistant.terminal","content":"artifact-a exact frozen quote"}') + MediaType = 'application/x-ndjson; charset=utf-8' + } + [ordered]@{ + Path = 'evidence/b/events.jsonl' + Content = @('{"type":"assistant.terminal","content":"artifact-b exact frozen quote"}') + MediaType = 'application/x-ndjson; charset=utf-8' + } + [ordered]@{ + Path = 'evidence/logs/transcript.txt' + Content = @('alpha frozen transcript line', 'beta frozen transcript line') + MediaType = 'text/plain; charset=utf-8' + } + ) + } + if ($Command -eq 'preflight') { Write-FixtureEvent -Kind 'preflight' if (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'preflight-incompatible') -PathType Leaf) { @@ -159,6 +183,67 @@ try { } } } + $phase2BundlePath = Join-Path $inputs.Run.WorkingDirectoryPath 'input-bundle.json' + if (Test-Path -LiteralPath $phase2BundlePath -PathType Leaf) { + $bundle = Read-RunnerJson -Path $phase2BundlePath + $fragmentGrades = [System.Collections.Generic.List[object]]::new() + foreach ($assertion in @(Get-JsonProperty -Object $bundle -Name 'assertions' -Default @())) { + $assertionIndex = [int](Get-JsonProperty -Object $assertion -Name 'assertion_index' -Default 0) + $domain = [string](Get-JsonProperty -Object $assertion -Name 'evidence_domain' -Default 'output') + $artifact = [string](Get-JsonProperty -Object $bundle -Name 'canonical_result' -Default '') + $evidenceRef = $null + if ($domain -eq 'transcript') { + $transcripts = @(Get-JsonProperty -Object $bundle -Name 'frozen_transcripts' -Default @()) + $eventTranscript = @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'events' } | Select-Object -First 1) + if ($eventTranscript.Count -eq 1) { + $event = @(Get-JsonProperty -Object $eventTranscript[0] -Name 'events' -Default @() | Select-Object -First 1) + if ($event.Count -ne 1) { throw 'Phase 2 fixture expected one frozen transcript event.' } + $evidenceRef = [ordered]@{ + artifact = [string](Get-JsonProperty -Object $eventTranscript[0] -Name 'artifact' -Default '') + domain = 'transcript' + event_index = [int](Get-JsonProperty -Object $event[0] -Name 'event_index' -Default 0) + quote = [string](Get-JsonProperty -Object $event[0] -Name 'source_text' -Default (Get-JsonProperty -Object $event[0] -Name 'content' -Default '')) + } + } else { + $lineTranscript = @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'lines' } | Select-Object -First 1) + if ($lineTranscript.Count -ne 1) { throw 'Phase 2 fixture expected frozen transcript evidence for transcript-domain grading.' } + $lines = @(Get-JsonProperty -Object $lineTranscript[0] -Name 'lines' -Default @() | Select-Object -First 2) + if ($lines.Count -lt 1) { throw 'Phase 2 fixture expected at least one frozen transcript line.' } + $quote = [string]::Join("`n", @($lines | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + $evidenceRef = [ordered]@{ + artifact = [string](Get-JsonProperty -Object $lineTranscript[0] -Name 'artifact' -Default '') + domain = 'transcript' + start_line = [int](Get-JsonProperty -Object $lines[0] -Name 'line' -Default 1) + end_line = [int](Get-JsonProperty -Object $lines[$lines.Count - 1] -Name 'line' -Default 1) + quote = $quote + } + } + } else { + $line = @(Get-JsonProperty -Object (Get-JsonProperty -Object $bundle -Name 'frozen_output' -Default $null) -Name 'lines' -Default @() | Select-Object -First 1) + $lineNumber = if ($line.Count -eq 1) { [int](Get-JsonProperty -Object $line[0] -Name 'line' -Default 1) } else { 1 } + $quote = if ($line.Count -eq 1) { [string](Get-JsonProperty -Object $line[0] -Name 'text' -Default 'deterministic fixture response') } else { 'deterministic fixture response' } + $evidenceRef = [ordered]@{ + artifact = $artifact + domain = $domain + start_line = $lineNumber + end_line = $lineNumber + quote = $quote + } + } + $fragmentGrades.Add([ordered]@{ + assertion_index = $assertionIndex + passed = $true + reason = "The deterministic analyzer fixture cites frozen one-arm output for assertion $assertionIndex." + evidence_refs = @($evidenceRef) + }) + } + $fixtureFinalResponse = ConvertTo-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-fragment/1' + eval_id = [int](Get-JsonProperty -Object $bundle -Name 'eval_id' -Default 0) + configuration = [string](Get-JsonProperty -Object $bundle -Name 'configuration' -Default '') + grading = @($fragmentGrades.ToArray()) + }) -Compress + } $durationSeconds = [Math]::Round(($executeFinishUtc - $executeStartUtc).TotalSeconds, 3) $durationOverride = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_DURATION_SECONDS') $parsedDuration = 0.0 @@ -211,13 +296,28 @@ try { } [IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $eventLines) + "`n"), [Text.UTF8Encoding]::new($false)) $eventsArtifact = New-ArtifactReference -Run $inputs.Run -Path 'evidence/fixture-events.jsonl' -Scope run -MediaType 'application/x-ndjson; charset=utf-8' + $artifactReferences = [System.Collections.Generic.List[object]]::new() + $artifactReferences.Add($eventsArtifact) + foreach ($extra in @(Get-ExtraTranscriptArtifacts)) { + $artifactPath = Join-Path $inputs.Run.RunRoot (([string]$extra.Path) -replace '/', [IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $artifactPath) -Force | Out-Null + [IO.File]::WriteAllText($artifactPath, ([string]::Join("`n", @($extra.Content)) + "`n"), [Text.UTF8Encoding]::new($false)) + $artifactReferences.Add((New-ArtifactReference -Run $inputs.Run -Path ([string]$extra.Path) -Scope run -MediaType ([string]$extra.MediaType))) + } $evidence = [ordered]@{ capture = [ordered]@{ source = 'harness_native_transport'; terminal = $true; worker_authored = $false } + commands = @( + [ordered]@{ type = 'command_execution'; command = 'git branch --show-current'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git symbolic-ref refs/remotes/origin/HEAD --short'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git log origin/HEAD..HEAD --oneline'; status = 'completed'; exit_code = 0 } + [ordered]@{ type = 'command_execution'; command = 'git diff origin/HEAD...HEAD'; status = 'completed'; exit_code = 0 } + ) + files_written = @() delegation = [ordered]@{ dispatch_owner = 'runner' mechanism = 'deterministic-runner-owned-fixture' worker_session_id = $sessionId - observed_model = [string]$inputs.Profile.Model + observed_model = if ([string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_OBSERVED_MODEL'))) { [string]$inputs.Profile.Model } else { [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_OBSERVED_MODEL') } observed_working_directory = [string]$inputs.Run.WorkingDirectoryPath observed_home = [string]$inputs.Run.HomeDirectoryPath fresh_worker = $true @@ -257,7 +357,7 @@ try { 'cancelled' { New-ExecutionFailure -Code 'cancelled' -Message 'The deterministic runner-owned fixture was cancelled before completion.' } default { $null } } - $result = New-ExecutionResult -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Status $terminalStatus -FinalResponse $finalResponse -FinalResponseReason $finalResponseReason -StartedUtc ($executeStartUtc.ToString('o')) -FinishedUtc ($executeFinishUtc.ToString('o')) -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @('deterministic runner-owned fixture') -Telemetry ([ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fixture-events.jsonl'; complete = $true }); tokens = $fixtureTelemetryTokens; tool_calls = $fixtureTelemetryToolCalls; cost = New-UnavailableMetric -Reason 'fixture' }) -Artifacts @($eventsArtifact) -Evidence $evidence -AttemptCount 1 + $result = New-ExecutionResult -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Status $terminalStatus -FinalResponse $finalResponse -FinalResponseReason $finalResponseReason -StartedUtc ($executeStartUtc.ToString('o')) -FinishedUtc ($executeFinishUtc.ToString('o')) -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @('deterministic runner-owned fixture') -Telemetry ([ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fixture-events.jsonl'; complete = $true }); tokens = $fixtureTelemetryTokens; tool_calls = $fixtureTelemetryToolCalls; cost = New-UnavailableMetric -Reason 'fixture' }) -Artifacts @($artifactReferences.ToArray()) -Evidence $evidence -AttemptCount 1 [void](Assert-ExecutionResult -Result $result) Write-RunnerJson -Value $result -AsOutput } catch { diff --git a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 new file mode 100644 index 0000000..5f9a789 --- /dev/null +++ b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 @@ -0,0 +1,85 @@ +# Real package preparation with a fake model catalog; never launches a provider. +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scriptsRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scriptsRoot 'eval-runners\runner-common.ps1') +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('change-impact-workspace-' + [guid]::NewGuid().ToString('N')) +[void][IO.Directory]::CreateDirectory($workspace) +function Assert-True($Condition, $Message) { if (-not $Condition) { throw $Message } } +$tokens = $null +$parseErrors = $null +$codexAst = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $scriptsRoot 'eval-runners\codex\runner.ps1'), [ref]$tokens, [ref]$parseErrors) +if ($parseErrors.Count) { throw 'Codex runner did not parse for sanitized PATH projection.' } +foreach ($functionName in @('Join-CodexTargetPath', 'Get-CodexSanitizedShellPath')) { + $definition = @($codexAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $functionName }, $true)) + if ($definition.Count -ne 1) { throw "Codex function '$functionName' was not found." } + Invoke-Expression $definition[0].Extent.Text +} +function New-SanitizedGitEnvironment { + param([Parameter(Mandatory = $true)][object]$GitCommand) + $gitDirectory = Split-Path -Parent ([string]$GitCommand.Source) + $platform = Get-PlatformName + $environment = [ordered]@{ PATH = Get-CodexSanitizedShellPath -Platform $platform -GitDirectory $gitDirectory } + if ($platform -eq 'windows') { + $windowsRoot = [Environment]::GetEnvironmentVariable('SystemRoot') + if ([string]::IsNullOrWhiteSpace($windowsRoot)) { $windowsRoot = 'C:\Windows' } + $environment.SystemRoot = $windowsRoot + $environment.ComSpec = Join-Path (Join-Path $windowsRoot 'System32') 'cmd.exe' + $environment.PATHEXT = '.COM;.EXE;.BAT;.CMD' + } + return $environment +} +function Invoke-SanitizedGit { + param( + [Parameter(Mandatory = $true)][object]$GitCommand, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [Parameter(Mandatory = $true)][string]$Repo, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + $process = Invoke-RunnerProcess -FileName ([IO.Path]::GetFileName([string]$GitCommand.Source)) -ArgumentList (@('-C', $Repo) + $Arguments) -WorkingDirectory $Repo -Environment $Environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { throw "sanitized git $([string]::Join(' ', $Arguments)) failed: $($process.Stderr)" } + return @($process.Stdout -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) }) +} +try { + $catalog = Join-Path $workspace 'models.json' + [IO.File]::WriteAllText($catalog, '{"models":[{"id":"fixture-model"}]}') + $analyzerCatalog = Join-Path $workspace 'analyzer-models.json' + [IO.File]::WriteAllText($analyzerCatalog, '{"models":[{"id":"claude-haiku-4.5"}]}') + $prompt = & (Join-Path $scriptsRoot 'prepare-skill-evals.ps1') -Skill dotnet-change-impact -Eval 9 -Runner github-copilot -Model fixture-model -ModelCatalogPath $catalog -AnalyzerModelCatalogPath $analyzerCatalog -OutputRoot $workspace -PassThru + $package = Split-Path -Parent $prompt + $handoff = [IO.File]::ReadAllText($prompt) + Assert-True ($handoff.Contains('invoke-phase2-analyzer.ps1') -and $handoff.Contains('The Phase 2 controller, not this outer orchestrator')) 'Handoff must delegate grading to the package-local Phase 2 controller.' + $manifest = Get-Content (Join-Path $package 'manifest.json') -Raw | ConvertFrom-Json + $heads = @(); $diffs = @(); $refs = @() + $gitCommand = Resolve-ExternalCommand -Name 'git' + Assert-True ($null -ne $gitCommand) 'git must resolve for the sanitized Codex git-workspace projection.' + $sanitizedGitEnvironment = New-SanitizedGitEnvironment -GitCommand $gitCommand + foreach ($arm in @('with_skill', 'without_skill')) { + $repo = Join-Path (Split-Path -Parent (Join-Path $package $manifest.evals[0].runs.$arm.run_manifest)) 'repo' + Assert-True (Test-Path (Join-Path $repo '.git') -PathType Container) 'Eval 9 must stage a real .git repository.' + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('branch', '--show-current')) -eq 'feature/remove-legacy-api') 'Feature branch missing under sanitized PATH.' + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('symbolic-ref', 'refs/remotes/origin/HEAD', '--short')) -eq 'origin/trunk') 'Non-main default branch fallback missing under sanitized PATH.' + Assert-True ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('rev-list', '--count', 'origin/HEAD..HEAD')) -eq '1') 'Feature commit history missing under sanitized PATH.' + $diff = (Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('diff', 'origin/HEAD...HEAD')) -join "`n" + Assert-True ($diff.Contains('- public static Widget Parse(string value)')) 'Meaningful public API removal missing.' + Assert-True ([string]::IsNullOrWhiteSpace((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('status', '--porcelain')) -join '')) 'Staged repository must be clean under sanitized PATH.' + $heads += Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('rev-parse', 'HEAD') + $diffs += $diff + $refs += ((Invoke-SanitizedGit -GitCommand $gitCommand -Environment $sanitizedGitEnvironment -Repo $repo -Arguments @('show-ref')) -join "`n") + } + Assert-True ($heads[0] -ceq $heads[1] -and $diffs[0] -ceq $diffs[1] -and $refs[0] -ceq $refs[1]) 'Paired Git history, refs and diff must be identical.' + . (Join-Path $scriptsRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario $true + Assert-EvalGitScenario $false + foreach ($invalid in @('{"base_branch":"main","feature_branch":"main","commits":[]}', '{"base_branch":"main","feature_branch":"feature","commits":[{"message":"unsafe","files":{"../escape":"bad"}}]}')) { + $rejected = $false + try { Assert-EvalGitScenario ($invalid | ConvertFrom-Json) } catch { $rejected = $true } + Assert-True $rejected 'Unsafe declarative scenario must be rejected.' + } + 'PASS: eval 9 deterministic package Git history, pairing, grader handoff, boolean compatibility and unsafe-path rejection; no models.' +} finally { + if (-not ([IO.Path]::GetFileName($workspace) -match '^change-impact-workspace-[0-9a-f]{32}$')) { throw 'Unsafe test cleanup.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} diff --git a/scripts/eval-runners/tests/test-codex-paths.ps1 b/scripts/eval-runners/tests/test-codex-paths.ps1 index 0f31a1b..80e6ad7 100644 --- a/scripts/eval-runners/tests/test-codex-paths.ps1 +++ b/scripts/eval-runners/tests/test-codex-paths.ps1 @@ -10,7 +10,7 @@ $tokens = $null $parseErrors = $null $ast = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $PSScriptRoot '../codex/runner.ps1'), [ref]$tokens, [ref]$parseErrors) if ($parseErrors.Count) { throw 'Codex runner did not parse.' } -$names = @('ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') +$names = @('ConvertTo-CodexConfigStringLiteral', 'Get-CodexSanitizedShellPath', 'Join-CodexTargetPath', 'Get-CodexShellEnvironmentPolicySet', 'Add-CodexShellEnvironmentPolicyConfigArguments', 'ConvertTo-CodexComparablePath', 'ConvertTo-CodexComparableText', 'Get-CodexAmbientSkillRoot', 'Test-CodexTextReferencesRoot', 'Test-CodexPathInsideComparableRoot', 'Update-CodexNativeSkillRuntimeAccessEvidence') foreach ($name in $names) { $definition = @($ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $name }, $true)) if ($definition.Count -ne 1) { throw "Missing unique function '$name'." } @@ -24,6 +24,23 @@ if ((ConvertTo-CodexComparablePath $comparable) -cne $comparable) { throw 'Path if (-not (Test-CodexPathInsideComparableRoot $staged (Join-Path $staged 'FORMS.md'))) { throw 'Staged descendants must remain inside their root.' } if (Test-CodexPathInsideComparableRoot $staged ($staged + '-other/FORMS.md')) { throw 'A sibling prefix is not a descendant.' } +$windowsGit = 'C:\Program Files\Git\cmd' +$windowsPath = Get-CodexSanitizedShellPath -Platform windows -GitDirectory $windowsGit -WindowsRoot 'C:\Windows' +if ($windowsPath -cne "C:\Windows\System32;$windowsGit") { throw "Windows sanitized PATH was '$windowsPath'." } +if ((Join-CodexTargetPath -Root 'C:\Program Files' -Segments @('Git', 'cmd') -Platform windows) -cne 'C:\Program Files\Git\cmd') { throw 'Windows target path construction must not require a host Windows drive.' } +if ($windowsPath -match [regex]::Escape('C:\host-only\bin')) { throw 'Windows sanitized PATH exposed an arbitrary host-only directory.' } +$linuxPath = Get-CodexSanitizedShellPath -Platform linux -GitDirectory '/opt/git/bin' +if ($linuxPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/git/bin') { throw "Linux sanitized PATH was '$linuxPath'." } +if ((Join-CodexTargetPath -Root '/usr/local' -Segments @('bin') -Platform linux) -cne '/usr/local/bin') { throw 'POSIX target path construction must not depend on the host filesystem provider.' } +$macPath = Get-CodexSanitizedShellPath -Platform macos -GitDirectory '/usr/bin' +if ($macPath -cne '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin') { throw "macOS sanitized PATH should deduplicate /usr/bin but was '$macPath'." } +$shellEnvironment = Get-CodexShellEnvironmentPolicySet -Inputs ([ordered]@{}) -Platform windows -WindowsRoot 'C:\Windows' +if ([string]$shellEnvironment.ComSpec -cne 'C:\Windows\System32\cmd.exe') { throw "Windows ComSpec was '$($shellEnvironment.ComSpec)'." } +$configArgs = [System.Collections.Generic.List[string]]::new() +Add-CodexShellEnvironmentPolicyConfigArguments -Arguments $configArgs -ShellEnvironmentSet ([ordered]@{ PATH = $windowsPath; SystemRoot = 'C:\Windows'; ComSpec = 'C:\Windows\System32\cmd.exe'; PATHEXT = '.COM;.EXE;.BAT;.CMD' }) -SwitchName '--config' +if (@($configArgs | Where-Object { $_ -eq 'shell_environment_policy.inherit=none' }).Count -ne 1) { throw 'Codex config args must keep shell environment inheritance disabled.' } +if (@($configArgs | Where-Object { $_ -eq ('shell_environment_policy.set.PATH="' + $windowsPath.Replace('\', '\\') + '"') }).Count -ne 1) { throw 'Codex config args must set the sanitized child PATH.' } + foreach ($access in @('command', 'file')) { $isolation = [ordered]@{ ambient_skill_paths_observed = @($ambient); failures = @() } $parameters = @{ NativeSkillIsolation = $isolation; AllowedStagedSkillRoot = $staged } diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 9c1f6e9..27f5aed 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -9,6 +9,7 @@ Set-StrictMode -Version Latest $runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path . (Join-Path $runnerRoot 'runner-common.ps1') . (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'github-copilot/isolation.ps1') # Import definitions only: neither adapter dispatch nor a real CLI is invoked. foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', 'tests/test-runner-conformance.ps1', '../generate-eval-report.ps1')) { @@ -20,11 +21,11 @@ foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', Invoke-Expression $definition.Extent.Text } if ($file -eq 'github-copilot/runner.ps1' -and $definition -is [Management.Automation.Language.AssignmentStatementAst] -and - $definition.Left.Extent.Text -eq '$descriptor') { Invoke-Expression $definition.Extent.Text } + $definition.Left.Extent.Text -in @('$descriptor', '$copilotExcludedTools', '$copilotCandidateInstructionBoundary')) { Invoke-Expression $definition.Extent.Text } } } $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') -function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false } } +function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false; GitHubCliConfigDirectory = $null; NonInteractiveReady = $true } } function Resolve-SandboxCommand { param($Name) return $null } function Resolve-ExternalCommand { param($Name) if ($Name -ne 'copilot') { throw "Unexpected executable lookup: $Name" }; return $fakeCommand } function Get-CopilotPreflight { @@ -73,6 +74,14 @@ $gitExit = $LASTEXITCODE $record = [ordered]@{ turn = $turn; arguments = $arguments; cwd = (Get-Location).Path; ceiling = $env:GIT_CEILING_DIRECTORIES; cache = $env:COPILOT_CACHE_HOME; xdg = $env:XDG_CACHE_HOME; localappdata = $env:LOCALAPPDATA; appdata = $env:APPDATA; git_exit = $gitExit; git_root = [string]$gitResult; old_cache_exists = Test-Path -LiteralPath (Join-Path $runHome '.copilot-cache'); double_suffix_exists = Test-Path -LiteralPath (Join-Path $env:COPILOT_CACHE_HOME 'copilot') } [IO.File]::AppendAllText((Join-Path (Get-Location).Path 'fake-log.jsonl'), (($record | ConvertTo-Json -Compress) + "`n")) [IO.File]::WriteAllText((Join-Path (Get-Location).Path 'task-output.txt'), 'keep repo output') +$probe = [ordered]@{ metadata = Test-Path ../../eval-metadata.json; paired = (Test-Path ../../with_skill) -or (Test-Path ../../without_skill); candidate = Test-Path ../skill; staged_agents = Get-Content AGENTS.md -Raw; staged_copilot = Get-Content .github/copilot-instructions.md -Raw; ambient = @() } +$cursor = Split-Path -Parent (Get-Location).Path +while ($cursor) { + foreach ($name in @('AGENTS.md', '.github/copilot-instructions.md')) { if (Test-Path -LiteralPath (Join-Path $cursor $name)) { $probe.ambient += [IO.File]::ReadAllText((Join-Path $cursor $name)) } } + $cursor = Split-Path -Parent $cursor +} +[IO.File]::WriteAllText((Join-Path (Get-Location).Path 'projection-probe.json'), ($probe | ConvertTo-Json)) +if ($inputText -eq 'violation') { '{"type":"tool.execution_start","data":{"toolName":"view","arguments":{"path":"../../eval-metadata.json"}}}' } if ($inputText -eq 'timeout') { [Console]::Out.WriteLine('{"type":"session.start","data":{"sessionId":"fixture-session"}}'); [Console]::Out.Flush(); Start-Sleep -Seconds 30 } $text = if ($turn -eq 1) { 'Confirm before generation. Δ' } else { "Generated successfully.`nExact terminal text." } @{ type = 'session.start'; data = @{ sessionId = 'fixture-session' } } | ConvertTo-Json -Compress @@ -90,12 +99,19 @@ if ($inputText -eq 'failure') { exit 7 } $profilePath = Join-Path $iteration 'execution-profile.json' Write-TestJson -Path $profilePath -Value @{ schema = (Get-RunnerSchemaNames).Profile; runner = 'github-copilot'; model = 'fixture-model'; reasoning_effort = $null; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 15; concurrency = 1 } foreach ($run in @($with, $without)) { + [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'repo/.github')) + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/AGENTS.md'), 'STAGED_INSTRUCTIONS_CANARY') + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/.github/copilot-instructions.md'), 'STAGED_COPILOT_CANARY') [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'home/baseline/empty')) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/nested.bin'), [byte[]]@(0, 255, 10, 13, 42)) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/empty.bin'), [byte[]]@()) [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'evidence')) [IO.File]::WriteAllText((Join-Path $run.Root 'evidence/prepared.txt'), 'keep evidence') } + [IO.File]::WriteAllText((Join-Path $testRoot 'AGENTS.md'), 'FORBIDDEN_SOURCE_INSTRUCTIONS_CANARY') + [void][IO.Directory]::CreateDirectory((Join-Path $testRoot '.github')) + [IO.File]::WriteAllText((Join-Path $testRoot '.github/copilot-instructions.md'), 'FORBIDDEN_COPILOT_CANARY') + [IO.File]::WriteAllText((Join-Path (Split-Path -Parent $with.Root) 'eval-metadata.json'), 'FORBIDDEN_GRADING_CANARY') $inputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $with.Path; Profile = Resolve-ExecutionProfile -ProfilePath $profilePath } $baseline = Get-CopilotHomeBaseline -HomePath $inputs.Run.HomeDirectoryPath $baselineHash = Get-TestTreeHash -Root $inputs.Run.HomeDirectoryPath @@ -117,18 +133,19 @@ if ($inputText -eq 'failure') { exit 7 } $records = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) Assert-Equal 2 $records.Count 'both turns execute with shared runtime state' foreach ($record in $records) { - Assert-Equal $with.Root $record.ceiling 'exact child environment ceiling is the manifest RunRoot' + Assert-Equal $result.evidence.execution_paths.physical_run_root $record.ceiling 'child ceiling is the physical run root' + Assert-True (-not (Test-PathInside -BasePath $testRoot -CandidatePath $record.cwd)) 'physical cwd excludes source and package ancestry' Assert-True ($record.git_exit -ne 0) 'child git cannot discover outer/.git from non-Git staged repo' - Assert-Equal $inputs.Run.WorkingDirectoryPath $record.cwd 'child working directory retained' + Assert-Equal $result.evidence.execution_paths.physical_working_directory $record.cwd 'child uses projected working directory' $argsList = [string[]]$record.arguments - Assert-Equal $inputs.Run.WorkingDirectoryPath $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C retained' + Assert-Equal $record.cwd $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C agrees with process cwd' Assert-Equal 'fixture-model' $argsList[[Array]::IndexOf($argsList, '--model') + 1] 'model lock retained on every turn' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache') $record.xdg 'XDG cache root' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache') $record.xdg 'XDG cache root' Assert-True (-not $record.old_cache_exists -and -not $record.double_suffix_exists) 'no old or double-suffixed cache is created during execution' if ($IsWindows) { Assert-Equal $record.xdg $record.localappdata 'Windows cache fallback converges' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.config') $record.appdata 'Windows config root isolated' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.config') $record.appdata 'Windows config root isolated' } } Assert-True (@($records[1].arguments) -contains '--resume=fixture-session') 'exact session continuation retained' @@ -148,12 +165,13 @@ if ($inputText -eq 'failure') { exit 7 } if ($LASTEXITCODE -ne 0) { throw 'Could not initialize staged fixture Git repository.' } $singleInputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $without.Path; Profile = $inputs.Profile } $singleBaselineHash = Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath - foreach ($scenario in @('completed', 'failure', 'timeout')) { + foreach ($scenario in @('completed', 'failure', 'timeout', 'violation')) { $singleInputs.Run.PromptBytes = [Text.Encoding]::UTF8.GetBytes($scenario) $singleInputs.Profile.TimeoutSeconds = if ($scenario -eq 'timeout') { 2 } else { 15 } $singleResult = Invoke-CopilotWithPreparedHome -Inputs $singleInputs -Action { Invoke-CopilotExecute -Inputs $singleInputs } - $expectedStatus = if ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } + $expectedStatus = if ($scenario -eq 'violation') { 'incompatible' } elseif ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } Assert-Equal $expectedStatus $singleResult.status "$scenario terminal status" + [void](Assert-ExecutionResult -Result $singleResult) Assert-Equal $singleBaselineHash (Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath) "$scenario restores prepared baseline" Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'home/.cache'))) "$scenario removes generated cache" Assert-Equal 'keep repo output' ([IO.File]::ReadAllText((Join-Path $without.Root 'repo/task-output.txt'))) "$scenario preserves repo output" @@ -162,9 +180,133 @@ if ($inputText -eq 'failure') { exit 7 } $singleRecords = @(Get-Content -LiteralPath (Join-Path $without.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) foreach ($record in $singleRecords) { Assert-Equal 0 $record.git_exit 'Git still discovers staged repo/.git' - Assert-Equal ((Join-Path $without.Root 'repo').Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns staged root' + Assert-Equal ($record.cwd.Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns projected staged root' Assert-True ($record.cache -ne $records[0].cache) 'paired arms never share cache' } + Assert-True $singleResult.evidence.delegation.grading_material_visible 'contradiction prevents false invisibility claim' + $forged = $singleResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $forged.status = 'completed' + $forged.evidence.delegation.paired_arm_visible = $false + $forged.evidence.delegation.grading_material_visible = $false + Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $forged -RunData $singleInputs.Run } 'bridge rejects transcript contradiction despite false invisibility flags' + $proof = [pscustomobject]@{ + PackageRoot = 'C:/source/.bot/package' + SourceRepositoryRoot = 'C:/source' + RunRoot = 'C:/source/.bot/package/eval-01/with_skill' + WorkingDirectoryRoot = 'C:/source/.bot/package/eval-01/with_skill/repo' + } + foreach ($path in @('../../eval-metadata.json', '../../without_skill/repo', '../../results/arm.json', '../../../grading.json', '../../../execution-freeze.json', '../../../orchestration-state.json', '../../../report.html', 'C:\source\AGENTS.md')) { + Assert-True (@(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = $path } } -Projection $proof).Count -gt 0) "captured forbidden access rejected: $path" + } + Assert-Equal 0 @(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = 'src/Widget.cs' } } -Projection $proof).Count 'ordinary staged source is allowed' + $phase2AnalyzerBoundary = [pscustomobject]@{ + PackageRoot = 'C:\prepared\iteration-1' + SourceRepositoryRoot = 'C:\Source\GitHub\codebeltnet\agentic' + RunRoot = 'C:\prepared\iteration-1\phase2\work\arm-3-without_skill' + WorkingDirectoryRoot = 'C:\prepared\iteration-1\phase2\work\arm-3-without_skill\repo' + ExecutionRole = 'phase2_analyzer' + } + $phase2InputBundleAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'input-bundle.json' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2InputBundleAccess.Contradictions).Count 'phase2 analyzer may read its own staged input bundle' + Assert-True ([bool]$phase2InputBundleAccess.OwnArmGradingMaterialVisible) 'phase2 analyzer records own-arm bundle visibility separately from forbidden grading visibility' + $phase2GraderAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'grader.md' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2GraderAccess.Contradictions).Count 'phase2 analyzer may read its own staged grader contract' + Assert-True ([bool]$phase2GraderAccess.OwnArmGradingMaterialVisible) 'phase2 analyzer records own-arm grader visibility separately from forbidden grading visibility' + $phase2EvidenceAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = 'evidence/with_skill/evidence/opencode-events.jsonl' } } -Boundary $phase2AnalyzerBoundary + Assert-Equal 0 @($phase2EvidenceAccess.Contradictions).Count 'phase2 analyzer may read its own staged one-arm evidence artifacts' + $phase2PairedArmAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\arm-4-with_skill\repo\input-bundle.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PairedArmAccess.Contradictions).Count -gt 0) 'phase2 analyzer paired-arm artifact access is incompatible' + Assert-True ([bool]$phase2PairedArmAccess.PairedArmVisible) 'phase2 analyzer paired-arm access is classified separately' + $phase2PackageGradingAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\..\..\grading.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PackageGradingAccess.Contradictions).Count -gt 0) 'phase2 analyzer package-root grading access is incompatible' + Assert-True ([bool]$phase2PackageGradingAccess.PairedOrPackageGradingMaterialVisible) 'phase2 analyzer package-root grading access is classified separately' + $phase2PackageOrchestrationAccess = Get-CopilotBoundaryAssessment -Data @{ arguments = @{ path = '..\..\..\..\execution-freeze.json' } } -Boundary $phase2AnalyzerBoundary + Assert-True (@($phase2PackageOrchestrationAccess.Contradictions).Count -gt 0) 'phase2 analyzer package-root orchestration access is incompatible' + Assert-True ([bool]$phase2PackageOrchestrationAccess.PairedOrPackageGradingMaterialVisible) 'phase2 analyzer package-root orchestration access remains forbidden' + + # --- P0 native-skill isolation regressions (model-free) --- + Assert-True (@(New-CopilotCliArguments -Inputs $singleInputs) -contains '--excluded-tools=skill') 'Copilot removes the native skill tool from the model tool set for both arms' + # Native-skill activation detector: a candidate `skill` tool call or an inherited skill-resolution result is a breach. + Assert-True (@(Find-CopilotNativeSkillActivation -Data @{ toolName = 'skill'; arguments = @{ skill = 'demo-skill' } } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_start').Count -gt 0) 'candidate skill tool call is flagged' + Assert-True (@(Find-CopilotNativeSkillActivation -Data @{ skill = 'demo-skill'; found = $true; skillSource = 'inherited' } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_completed').Count -gt 0) 'inherited candidate skill resolution is flagged' + Assert-Equal 0 @(Find-CopilotNativeSkillActivation -Data @{ toolName = 'view'; arguments = @{ path = '../skill/demo-skill/SKILL.md' } } -CandidateSkillName 'demo-skill' -EventType 'tool.execution_start').Count 'ordinary read of the staged candidate copy is not native activation' + Assert-Equal 0 @(Find-CopilotNativeSkillActivation -Data @{ toolName = 'shell'; arguments = @{ command = 'echo demo-skill' } } -CandidateSkillName 'demo-skill' -EventType 'command.execute').Count 'a mere text mention of the candidate is not native activation' + + # Candidate-instruction identity proof (only the frozen instruction bytes are hashed, not the wrapper). + $demoInstruction = "# Operating instructions`n`n## Skill: demo-skill`n`nDo the demo work." + $demoInstructionHash = ([Convert]::ToHexString([Security.Cryptography.SHA256]::HashData([Text.Encoding]::UTF8.GetBytes($demoInstruction)))).ToLowerInvariant() + $withPrompt = "$demoInstruction`n`n# Working environment`n`nStay in the run.`n`n# task`n`nClassify." + $noSkillPrompt = "# Operating instructions`n`nNo special instructions.`n`n# Working environment`n`nStay in the run.`n`n# task`n`nClassify." + $withEvidence = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = $demoInstructionHash } }) + Assert-True $withEvidence.verified 'with_skill injected instructions hash exactly to the frozen candidate hash' + Assert-Equal $demoInstructionHash $withEvidence.injected 'injected candidate-instruction hash equals the frozen hash' + Assert-Equal 0 @($withEvidence.violations).Count 'a matching candidate-instruction hash yields no violation' + $wrapperChanged = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes("$demoInstruction`n`n# Working environment`n`nDifferent wrapper text entirely.`n`n# task`n`nOther."); CandidateInstructionHash = $demoInstructionHash } }) + Assert-True $wrapperChanged.verified 'unrelated prompt-wrapper changes do not invalidate the candidate identity proof' + $mismatch = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'with_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = ('0' * 64) } }) + Assert-True ((-not $mismatch.verified) -and @($mismatch.violations).Count -gt 0) 'a mismatched candidate-instruction hash is a violation' + $baselineClean = Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($noSkillPrompt); CandidateInstructionHash = $null } }) + Assert-True ($baselineClean.verified -and @($baselineClean.violations).Count -eq 0) 'clean baseline has no candidate injection and no hash' + Assert-True (@((Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($withPrompt); CandidateInstructionHash = $null } })).violations).Count -gt 0) 'baseline that embeds a candidate instruction section is a violation' + Assert-True (@((Get-CopilotCandidateInstructionEvidence -Inputs ([pscustomobject]@{ Run = [pscustomobject]@{ Mode = 'without_skill'; PromptBytes = [Text.Encoding]::UTF8.GetBytes($noSkillPrompt); CandidateInstructionHash = ('a' * 64) } })).violations).Count -gt 0) 'baseline that declares a candidate hash is a violation' + + # Native-skill-catalog probe: model-free, proves candidate absence/disablement or fails closed on an enabled candidate. + $catalogFake = Join-Path $testRoot 'catalog-fake.ps1' + [IO.File]::WriteAllText($catalogFake, 'param([Parameter(ValueFromRemainingArguments=$true)][string[]]$a); if ($env:FAKE_CATALOG) { Write-Output $env:FAKE_CATALOG } else { Write-Output "[]" }') + $catalogCommand = [pscustomobject]@{ FileName = (Get-Command pwsh).Source; Prefix = @('-NoProfile', '-NonInteractive', '-File', $catalogFake) } + function New-CatalogEnv { param([string]$Json) $env = New-RunnerProbeEnvironment; $env['FAKE_CATALOG'] = $Json; return $env } + $absentProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"customize-cloud-agent","source":"builtin","enabled":true}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($absentProbe.available -and $absentProbe.proven_absent -and -not $absentProbe.candidate_present) 'catalog probe proves an absent candidate is not natively resolvable' + $enabledProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"demo-skill","source":"personal","enabled":true}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($enabledProbe.available -and $enabledProbe.candidate_present -and $enabledProbe.candidate_enabled -and -not $enabledProbe.proven_absent) 'catalog probe flags an enabled ambient candidate' + $disabledProbe = Invoke-CopilotNativeSkillCatalogProbe -CommandInfo $catalogCommand -Environment (New-CatalogEnv '[{"name":"demo-skill","source":"personal","enabled":false}]') -WorkingDirectory $testRoot -CandidateSkillName 'demo-skill' + Assert-True ($disabledProbe.candidate_present -and -not $disabledProbe.candidate_enabled -and $disabledProbe.proven_absent) 'a disabled ambient candidate is present but cannot activate' + + # Bridge independently rejects a captured candidate native-skill activation even when runner booleans look clean. + $activationRun = New-TestRun -IterationDirectory (Join-Path $testRoot 'native-skill-activation') -Configuration without_skill + New-Item -ItemType Directory -Path (Join-Path $activationRun.Root 'evidence') -Force | Out-Null + [IO.File]::WriteAllText((Join-Path $activationRun.Root 'evidence/copilot-events.jsonl'), '{"type":"tool.execution_start","data":{"toolName":"skill","arguments":{"skill":"candidate"}}}' + "`n") + $activationInputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $activationRun.Path; Profile = Resolve-ExecutionProfile -ProfilePath $profilePath } + $activationRaw = @{ runner = @{ name = 'github-copilot' }; status = 'completed'; evidence = @{ execution_paths = @{ projection_proven = $true; physical_run_root = (Join-Path $testRoot 'phys-activation'); source_repository_root = '' } }; artifacts = @(@{ scope = 'run'; path = 'evidence/copilot-events.jsonl' }) } | ConvertTo-Json -Depth 20 | ConvertFrom-Json + Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $activationRaw -RunData $activationInputs.Run } 'bridge rejects a captured candidate native-skill activation' + $warnings = [Collections.Generic.List[string]]::new() + $checkpoint = @{ type = 'session.usage_checkpoint'; data = @{ totalPremiumRequests = 0.33; totalNanoAiu = 10; promptCacheBreakState = @(@{ models = @{ model = @{ model_call_id = 'call-1'; prompt_tokens = 100; cache_read = 70; cache_write = 20; tool_tokens = 15 } } }) } } + $last = $checkpoint | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $last.data.totalPremiumRequests = 0.66 + $last.data.totalNanoAiu = 20 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $last); Errors = @() } -Warnings $warnings + Assert-Equal 100 $usage.UsageInput 'repeated call snapshots are not summed' + Assert-Equal 70 $usage.UsageCacheRead 'cache read bucket retained without duplication' + Assert-Equal 20 $usage.UsageCacheWrite 'cache write bucket retained without duplication' + Assert-Equal 0.66 $usage.UsageCheckpoint.totalPremiumRequests 'final cumulative premium request counter is authoritative' + Assert-Equal 20 $usage.UsageCheckpoint.totalNanoAiu 'final cumulative nano AI units retained without currency conversion' + Assert-True ($null -eq $usage.UsageOutput) 'unexposed output tokens remain unavailable' + Assert-Equal 15 $usage.CheckpointCalls[0].tool_tokens 'tool schema tokens retained in native evidence only' + $secondCall = $last | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $secondCall.data.promptCacheBreakState[0].models.model.model_call_id = 'call-2' + $secondCall.data.promptCacheBreakState[0].models.model.prompt_tokens = 120 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $secondCall, $secondCall); Errors = @() } -Warnings $warnings + Assert-Equal 220 $usage.UsageInput 'distinct native calls count once each' + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, @{ type = 'assistant.usage'; data = @{ inputTokens = 7; outputTokens = 3 } }); Errors = @() } -Warnings $warnings + Assert-Equal 7 $usage.UsageInput 'native assistant usage takes precedence over cache snapshots' + Assert-Equal 3 $usage.UsageOutput 'actual exposed output count retained' + $savedTemp = $env:TEMP; $savedTmp = $env:TMP; $savedTmpDir = $env:TMPDIR + try { + $env:TEMP = $testRoot; $env:TMP = $testRoot; $env:TMPDIR = $testRoot + Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'temp inside source ancestry fails closed' + } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp; $env:TMPDIR = $savedTmpDir } + $runtimeLink = Join-Path $singleInputs.Run.WorkingDirectoryPath 'forbidden-link' + New-Item -ItemType $linkType -Path $runtimeLink -Target $testRoot | Out-Null + try { Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'linked projection input fails closed' } + finally { (Get-Item -LiteralPath $runtimeLink -Force).Delete() } + foreach ($arm in @($with, $without)) { + $probe = Get-Content (Join-Path $arm.Root 'repo/projection-probe.json') -Raw | ConvertFrom-Json + Assert-True (-not $probe.metadata -and -not $probe.paired) 'parent metadata and sibling arms are unreachable through projected ancestry' + Assert-Equal 0 @($probe.ambient).Count 'source AGENTS and Copilot instructions are excluded' + Assert-Equal 'STAGED_INSTRUCTIONS_CANARY' $probe.staged_agents 'staged AGENTS preserved identically' + Assert-Equal 'STAGED_COPILOT_CANARY' $probe.staged_copilot 'staged Copilot instructions preserved identically' + Assert-Equal ($arm.Root -eq $with.Root) $probe.candidate 'candidate material is only projected for with_skill' + } $transcript = Get-PortableTranscript -Raw $result -RunData $inputs.Run foreach ($text in @((Get-InteractionTurnText -Turn $inputs.Run.Interaction.turns[0] -RunData $inputs.Run), 'Confirm before generation. Δ', 'Yes, generate the key.', "Generated successfully.`nExact terminal text.", "Session: fixture-session`nSame session: true")) { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 new file mode 100644 index 0000000..fb42de0 --- /dev/null +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -0,0 +1,330 @@ +<# Deterministic request-to-handoff coverage: real preparation, fixture catalogs, fake host. No models. #> +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scripts 'eval-request.ps1') +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) +[void](New-Item -ItemType Directory -Path $workspace -Force) +$script:dispatches = [Collections.Generic.List[string]]::new() +$script:launches = [Collections.Generic.List[string]]::new() +$script:waits = [Collections.Generic.List[string]]::new() +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Assert-True([bool]$Condition, [string]$Message) { + if (-not $Condition) { throw $Message } +} +function Read-PackageManifest([string]$PromptPath) { + $manifestPath = Join-Path (Split-Path -Parent $PromptPath) 'manifest.json' + return [System.IO.File]::ReadAllText($manifestPath, $utf8NoBom) | ConvertFrom-Json +} +function Write-PackageManifest([string]$PromptPath, [object]$Manifest) { + $manifestPath = Join-Path (Split-Path -Parent $PromptPath) 'manifest.json' + [System.IO.File]::WriteAllText($manifestPath, (($Manifest | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} +function Assert-PreparedPromptBinding([string]$PromptPath, [string]$Description) { + $manifest = Read-PackageManifest -PromptPath $PromptPath + Assert-True ([string]$manifest.runner_prompt -ceq 'RUN-THIS.prompt.md') "$Description manifest.runner_prompt changed unexpectedly." + $declaredHash = [string]$manifest.runner_prompt_sha256 + Assert-True ($declaredHash -match '^[0-9a-f]{64}$') "$Description manifest.runner_prompt_sha256 must be a lowercase SHA-256." + Assert-True ($declaredHash -ceq (Get-Sha256HexFromFile -Path $PromptPath)) "$Description manifest.runner_prompt_sha256 must match RUN-THIS.prompt.md byte-for-byte." +} +function Assert-ExternalHandoffDecision([object]$Decision, [string]$Description) { + Assert-True ([string]$Decision.schema -ceq 'codebeltnet/agentic/eval-handoff-decision/1') "$Description schema changed unexpectedly." + Assert-True ([string]$Decision.action -ceq 'external_handoff') "$Description did not return external_handoff." + Assert-True ([bool]$Decision.user_authorized) "$Description lost explicit user authorization." + Assert-True ([bool]$Decision.host_can_delegate_fresh_orchestrator) "$Description lost fresh-context delegation capability." + Assert-True (-not [bool]$Decision.confirmation_required) "$Description reintroduced a confirmation state after external_handoff." + Assert-True ([bool]$Decision.dispatch_immediately) "$Description did not require immediate dispatch." + Assert-True ([int]$Decision.max_new_external_orchestrators -eq 1) "$Description allowed more or fewer than one new external Orchestrator." + Assert-True ([bool]$Decision.same_handle_required) "$Description no longer requires the same Orchestrator handle across waits." + Assert-True ([string]$Decision.pending_wait_action -ceq 'wait_same_handle_again') "$Description changed pending wait semantics." + Assert-True ((@($Decision.terminal_statuses) -join ',') -ceq 'completed,failed') "$Description changed terminal wait statuses." +} +function Assert-AlreadyStartedDecision([object]$Decision, [string]$Description) { + Assert-True ([string]$Decision.schema -ceq 'codebeltnet/agentic/eval-handoff-decision/1') "$Description schema changed unexpectedly." + Assert-True ([string]$Decision.action -ceq 'already_started') "$Description did not stay already_started." + Assert-True (-not [bool]$Decision.confirmation_required) "$Description requested confirmation while resuming an existing Orchestrator." + Assert-True (-not [bool]$Decision.dispatch_immediately) "$Description attempted a replacement Orchestrator dispatch." + Assert-True ([int]$Decision.max_new_external_orchestrators -eq 0) "$Description permitted a second Orchestrator." + Assert-True ([bool]$Decision.same_handle_required) "$Description lost the same-handle requirement." + Assert-True ([string]$Decision.pending_wait_action -ceq 'wait_same_handle_again') "$Description changed resume wait semantics." +} +function Invoke-FakeHost($Decision) { + if ($Decision.action -eq 'external_handoff') { + # The host's only input is the actual canonical handoff file, never an arm prompt. + Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' + Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + Assert-True (Test-Path -LiteralPath (Join-Path (Split-Path -Parent $Decision.prompt_path) '.external-handoff-started')) 'External handoff must be reserved before the host launch, for every runner.' + $script:dispatches.Add($Decision.prompt_path) + } +} +function Invoke-FakeExternalOrchestratorDispatch([object]$Decision, [string]$NativeHandle) { + Assert-ExternalHandoffDecision -Decision $Decision -Description 'Fake host dispatch' + Invoke-FakeHost $Decision + $script:launches.Add($NativeHandle) + return New-ExternalEvalOrchestratorState -Decision $Decision -NativeHandle $NativeHandle +} +function Invoke-FakeExternalOrchestratorWait([object]$State, [string]$NativeWaitStatus, [object]$TerminalResult = $null) { + $handle = [string]$State.native_handle + $script:waits.Add($handle) + return Update-ExternalEvalOrchestratorState -State $State -NativeHandle $handle -NativeWaitStatus $NativeWaitStatus -TerminalResult $TerminalResult +} +function New-Preparation([string]$Runner, [string]$Name) { + return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner + OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog; AnalyzerModelCatalogPath = $catalog } +} +function Assert-Failure([hashtable]$Options, [string]$Pattern) { + # The request must fail closed even in an ordinary interactive PowerShell caller. + $ErrorActionPreference = 'Continue' + $before = $script:dispatches.Count + $failed = $false + try { + Invoke-EvalRequest -Preparation $Options -Yolo -CanDelegateFreshOrchestrator | + ForEach-Object { Invoke-FakeHost $_ } + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected failure: $_" + } + Assert-True $failed 'Expected preparation failure.' + Assert-True ($script:dispatches.Count -eq $before) 'Failed preparation dispatched an Orchestrator.' +} + +function Assert-HandoffFailure([string]$PromptPath, [string]$Pattern, [string]$Description) { + $before = $script:dispatches.Count + $failed = $false + $claim = Join-Path (Split-Path -Parent $PromptPath) '.external-handoff-started' + $decision = $null + try { + $decision = Get-EvalHandoff -PromptPath $PromptPath -Yolo -CanDelegateFreshOrchestrator + Invoke-FakeHost $decision + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected handoff failure for ${Description}: $($_.Exception.Message)" + } + Assert-True $failed "$Description unexpectedly passed handoff validation." + Assert-True ($null -eq $decision -or $decision.action -ne 'external_handoff') "$Description returned external_handoff before failing." + Assert-True ($script:dispatches.Count -eq $before) "$Description dispatched an Orchestrator." + Assert-True (-not (Test-Path -LiteralPath $claim)) "$Description created .external-handoff-started before rejection." +} + +try { + $catalog = Join-Path $workspace 'models.json' + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'claude-opus-4.7' }, @{ id = 'provider/Exact.Model' }) } | + ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 + + $normalOptions = New-Preparation 'Codex' 'normal' + $normal = Invoke-EvalRequest -Preparation $normalOptions -CanDelegateFreshOrchestrator + Assert-PreparedPromptBinding -PromptPath $normal.prompt_path -Description 'Normal preparation' + Invoke-FakeHost $normal + Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' + Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + + # Force may replace an unstarted package, but must preserve every started package. + $normalOptions.Iteration = 1 + $normalOptions.Force = $true + $replacement = Invoke-EvalRequest -Preparation $normalOptions + Assert-True ($replacement.prompt_path -eq $normal.prompt_path) 'Force could not replace an unstarted package.' + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + $options = New-Preparation 'Codex' ([guid]::NewGuid().ToString('N')) + $options.Iteration = 1 + $options.Force = $true + if ($marker -eq '.external-handoff-started') { + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + Invoke-FakeHost $reserved + } else { + $reserved = Invoke-EvalRequest -Preparation $options + '{}' | Set-Content -LiteralPath (Join-Path (Split-Path $reserved.prompt_path) $marker) + } + $package = Split-Path $reserved.prompt_path + $beforeFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-Failure $options 'handoff or execution has already started' + $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." + } + + foreach ($case in @( + @{ Name = 'codex-default'; Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Name = 'copilot-default'; Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5'; StrongerModel = 'claude-opus-4.7'; UseLegacyAvailabilityAlias = $true }, + @{ Name = 'copilot-explicit-model'; Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Name = 'opencode-explicit-model'; Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + )) { + $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) + if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } + $decision = if ($case.ContainsKey('UseLegacyAvailabilityAlias')) { + Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + } else { + Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + } + Assert-PreparedPromptBinding -PromptPath $decision.prompt_path -Description $case.Name + $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.ContainsKey('StrongerModel')) { + Assert-True ($profile.model -cne $case.StrongerModel) 'Copilot replaced repository model policy with a subjective stronger-model choice.' + } + if ($case.Expected -eq 'codex') { + Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' + } + Assert-ExternalHandoffDecision -Decision $decision -Description $case.Name + $before = $script:dispatches.Count + $state = Invoke-FakeExternalOrchestratorDispatch -Decision $decision -NativeHandle ('dispatch-' + $case.Name) + Assert-True ($script:dispatches[$before] -ceq $decision.prompt_path) 'Host did not receive the exact generated RUN-THIS.prompt.md path.' + Assert-True ($state.status -eq 'running' -and -not $state.terminal -and $state.native_handle -ceq ('dispatch-' + $case.Name)) "$($case.Name) did not dispatch immediately into a running Orchestrator state." + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-AlreadyStartedDecision -Decision $again -Description ($case.Name + ' duplicate handoff') + Invoke-FakeHost $again + Assert-True ($script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' + $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo + Assert-AlreadyStartedDecision -Decision $unavailableAfterStart -Description ($case.Name + ' resume after dispatch') + } + + $copilotLaunchFailed = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'copilot-launch-failed') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $copilotLaunchFailed -Description 'Copilot launch-failed contract' + $failedLaunchState = New-ExternalEvalOrchestratorState -Decision $copilotLaunchFailed -LaunchFailed -FailureReason 'launch failed before a native handle existed' + Assert-True ($failedLaunchState.status -eq 'launch_failed' -and $failedLaunchState.terminal) 'Launch-failed Orchestrator state was not terminal.' + Assert-True ($null -eq $failedLaunchState.native_handle -and $failedLaunchState.wait_count -eq 0) 'Launch-failed Orchestrator state incorrectly retained a handle or wait history.' + Assert-True ($failedLaunchState.max_new_external_orchestrators -eq 0) 'Launch failure permitted a replacement Orchestrator.' + + $codexSuccess = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'codex-lifecycle-success') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $codexSuccess -Description 'Codex lifecycle success contract' + $launchBefore = $script:launches.Count + $waitBefore = $script:waits.Count + $codexState = Invoke-FakeExternalOrchestratorDispatch -Decision $codexSuccess -NativeHandle 'O1' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 0) 'Codex lifecycle did not start in a running state.' + $mismatchFailed = $false + try { + Update-ExternalEvalOrchestratorState -State $codexState -NativeHandle 'O2' -NativeWaitStatus 'pending' | Out-Null + } catch { + $mismatchFailed = $true + Assert-True ($_.Exception.Message -match 'same native handle') "Unexpected handle-mismatch failure: $($_.Exception.Message)" + } + Assert-True $mismatchFailed 'Codex lifecycle accepted a replacement Orchestrator handle.' + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'pending' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 1) 'First pending wait incorrectly terminated Codex lifecycle.' + Assert-True ($codexState.pending_wait_action -eq 'wait_same_handle_again') 'First pending wait did not require another bounded wait on O1.' + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'pending' + Assert-True ($codexState.status -eq 'running' -and -not $codexState.terminal -and $codexState.wait_count -eq 2) 'Second pending wait incorrectly terminated Codex lifecycle.' + $terminalResult = [pscustomobject]@{ native_handle = 'O1'; terminal_status = 'completed'; report_path = 'C:\fake\report.html' } + $codexState = Invoke-FakeExternalOrchestratorWait -State $codexState -NativeWaitStatus 'completed' -TerminalResult $terminalResult + $successLaunches = @($script:launches | Select-Object -Skip $launchBefore) + $successWaits = @($script:waits | Select-Object -Skip $waitBefore) + Assert-True ($successLaunches.Count -eq 1 -and $successLaunches[0] -ceq 'O1') 'Codex success lifecycle created more than one Orchestrator.' + Assert-True (($successWaits -join ',') -ceq 'O1,O1,O1') 'Codex success lifecycle stopped waiting on O1.' + Assert-True ($codexState.status -eq 'completed' -and $codexState.terminal -and $codexState.wait_count -eq 3) 'Codex success lifecycle did not end at terminal completion.' + Assert-True ($codexState.terminal_result.native_handle -ceq 'O1' -and $codexState.terminal_result.terminal_status -ceq 'completed') 'Codex success terminal result did not belong to O1.' + + $codexFailure = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'codex-lifecycle-failure') -Yolo -CanDelegateFreshOrchestrator + Assert-ExternalHandoffDecision -Decision $codexFailure -Description 'Codex lifecycle failure contract' + $launchBefore = $script:launches.Count + $waitBefore = $script:waits.Count + $codexFailedState = Invoke-FakeExternalOrchestratorDispatch -Decision $codexFailure -NativeHandle 'O1' + $codexFailedState = Invoke-FakeExternalOrchestratorWait -State $codexFailedState -NativeWaitStatus 'pending' + Assert-True ($codexFailedState.status -eq 'running' -and -not $codexFailedState.terminal -and $codexFailedState.wait_count -eq 1) 'Codex failure lifecycle ended during a pending wait.' + $failureResult = [pscustomobject]@{ native_handle = 'O1'; terminal_status = 'failed'; error = 'native orchestrator failed' } + $codexFailedState = Invoke-FakeExternalOrchestratorWait -State $codexFailedState -NativeWaitStatus 'failed' -TerminalResult $failureResult + $failureLaunches = @($script:launches | Select-Object -Skip $launchBefore) + $failureWaits = @($script:waits | Select-Object -Skip $waitBefore) + Assert-True ($failureLaunches.Count -eq 1 -and $failureLaunches[0] -ceq 'O1') 'Codex failure lifecycle created more than one Orchestrator.' + Assert-True (($failureWaits -join ',') -ceq 'O1,O1') 'Codex failure lifecycle stopped waiting on O1.' + Assert-True ($codexFailedState.status -eq 'failed' -and $codexFailedState.terminal -and $codexFailedState.wait_count -eq 2) 'Codex failure lifecycle did not return the real terminal failure.' + Assert-True ($codexFailedState.terminal_result.native_handle -ceq 'O1' -and $codexFailedState.terminal_result.terminal_status -ceq 'failed') 'Codex failure terminal result did not belong to O1.' + + $reference = New-Preparation 'unused' 'reference' + $reference.Remove('Runner') + $reference.CodebeltReference = $true + $reference.ReasoningEffort = 'high' + $reference.ConfigurationProfile = 'fixture-profile' + $reference.ToolProfile = 'fixture-tools' + $reference.TimeoutSeconds = 123 + $reference.Concurrency = 3 + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -CanDelegateFreshOrchestrator + $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' + Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' + + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'unavailable') -Yolo + $before = $script:dispatches.Count + Invoke-FakeHost $unavailable + Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' + Assert-True ([bool]$unavailable.user_authorized -and -not [bool]$unavailable.host_can_delegate_fresh_orchestrator) 'Unavailable host decision lost authorization/capability semantics.' + Assert-True (-not [bool]$unavailable.confirmation_required -and -not [bool]$unavailable.dispatch_immediately) 'Unavailable host decision reintroduced a confirmation or dispatch state.' + Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' + # Existing execution state also blocks automatic handoff, even without a handoff receipt. + '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-AlreadyStartedDecision -Decision $started -Description 'Existing Phase 1' + + Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' + $invalid = New-Preparation 'GitHub Copilot' 'invalid-model' + $invalid.Model = 'not-in-catalog' + Assert-Failure $invalid "Runner 'github-copilot' model 'not-in-catalog' could not be verified" + $invalid = New-Preparation 'Codex' 'invalid-default' + $emptyCatalog = Join-Path $workspace 'empty-models.json' + '{"models":[{"id":"unrelated-model"}]}' | Set-Content $emptyCatalog + $invalid.ModelCatalogPath = $emptyCatalog + Assert-Failure $invalid "Runner 'codex' model 'gpt-5.6-luna' could not be verified" + $invalid = New-Preparation 'Codex' 'missing-skill' + $invalid.Skill = 'no-such-skill' + Assert-Failure $invalid 'skill|directory|path' + + $standaloneRoot = Join-Path $workspace 'standalone' + [void](New-Item -ItemType Directory -Path $standaloneRoot -Force) + $standalonePrompt = Join-Path $standaloneRoot 'RUN-THIS.prompt.md' + [System.IO.File]::WriteAllText($standalonePrompt, '# forged handoff', [System.Text.UTF8Encoding]::new($false)) + Assert-HandoffFailure -PromptPath $standalonePrompt -Pattern 'valid prepared eval package' -Description 'standalone RUN-THIS.prompt.md' + + $pathMismatch = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'path-mismatch') + $pathMismatchManifest = Read-PackageManifest -PromptPath $pathMismatch.prompt_path + $pathMismatchManifest.runner_prompt = 'README.md' + $pathMismatchManifest.runner_prompt_sha256 = Get-Sha256HexFromFile -Path (Join-Path (Split-Path -Parent $pathMismatch.prompt_path) 'README.md') + Write-PackageManifest -PromptPath $pathMismatch.prompt_path -Manifest $pathMismatchManifest + Assert-HandoffFailure -PromptPath $pathMismatch.prompt_path -Pattern 'manifest-declared runner_prompt' -Description 'manifest runner_prompt path mismatch' + + $tamperedPrompt = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'tampered-prompt') + Assert-PreparedPromptBinding -PromptPath $tamperedPrompt.prompt_path -Description 'Prompt tamper fixture' + [System.IO.File]::AppendAllText($tamperedPrompt.prompt_path, "`n# tampered prompt`n", $utf8NoBom) + Assert-HandoffFailure -PromptPath $tamperedPrompt.prompt_path -Pattern 'runner_prompt_sha256|RUN-THIS\.prompt\.md bytes|Requires a fresh package' -Description 'tampered RUN-THIS.prompt.md' + + $missingPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'missing-prompt-hash') + $missingPromptHashManifest = Read-PackageManifest -PromptPath $missingPromptHash.prompt_path + [void]$missingPromptHashManifest.PSObject.Properties.Remove('runner_prompt_sha256') + Write-PackageManifest -PromptPath $missingPromptHash.prompt_path -Manifest $missingPromptHashManifest + Assert-HandoffFailure -PromptPath $missingPromptHash.prompt_path -Pattern 'runner_prompt_sha256' -Description 'missing runner_prompt_sha256' + + $malformedPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'malformed-prompt-hash') + $malformedPromptHashManifest = Read-PackageManifest -PromptPath $malformedPromptHash.prompt_path + $malformedPromptHashManifest.runner_prompt_sha256 = 'not-a-lowercase-sha256' + Write-PackageManifest -PromptPath $malformedPromptHash.prompt_path -Manifest $malformedPromptHashManifest + Assert-HandoffFailure -PromptPath $malformedPromptHash.prompt_path -Pattern 'runner_prompt_sha256' -Description 'malformed runner_prompt_sha256' + + $mismatchedPromptHash = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'mismatched-prompt-hash') + $mismatchedPromptHashManifest = Read-PackageManifest -PromptPath $mismatchedPromptHash.prompt_path + $mismatchedPromptHashManifest.runner_prompt_sha256 = ('1' * 64) + Write-PackageManifest -PromptPath $mismatchedPromptHash.prompt_path -Manifest $mismatchedPromptHashManifest + Assert-HandoffFailure -PromptPath $mismatchedPromptHash.prompt_path -Pattern 'runner_prompt_sha256|RUN-THIS\.prompt\.md bytes|Requires a fresh package' -Description 'mismatched runner_prompt_sha256' + + $tampered = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'tampered') + $tamperedPackage = Split-Path -Parent $tampered.prompt_path + [System.IO.File]::AppendAllText((Join-Path $tamperedPackage 'tools/eval-runners/resolve-runner.ps1'), "`n# tampered`n", [System.Text.UTF8Encoding]::new($false)) + Assert-HandoffFailure -PromptPath $tampered.prompt_path -Pattern 'valid prepared eval package|Requires a fresh package|changed after preparation' -Description 'tampered prepared package' + + # Check the real discovery call contract, not a mocked model resolver. + $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw + Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' + $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper.Contains("[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator")) 'External orchestrator capability alias changed unexpectedly.' + Assert-True ($helper.Contains('dispatch_immediately') -and $helper.Contains('wait_same_handle_again') -and $helper.Contains('New-ExternalEvalOrchestratorState') -and $helper.Contains('Update-ExternalEvalOrchestratorState')) 'Eval request helper lost the explicit handoff lifecycle contract.' + Assert-True (-not $helper.Contains('claude-haiku-4.5') -and -not $helper.Contains('gpt-5.6-luna') -and -not $helper.Contains('claude-opus-4.7')) 'Eval request helper must not embed model-selection policy.' + Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' + Write-Host 'PASS: normal/yolo requests, explicit no-confirmation handoff semantics, same-handle wait lifecycle, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' +} finally { + # The absolute target is the unique child allocated under the external test workspace above. + $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar + if (-not [IO.Path]::GetFullPath($workspace).StartsWith($allowed, [StringComparison]::OrdinalIgnoreCase)) { throw 'Unsafe test cleanup path.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index a32bb4d..d594caa 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -24,8 +24,19 @@ $repositoryRoot = (Resolve-Path (Join-Path $runnerRoot '..')).Path . (Join-Path $runnerRoot 'manifest-paths.ps1') . (Join-Path $runnerRoot 'orchestration.ps1') . (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'phase2-grading.ps1') . (Join-Path $runnerRoot 'package-integrity.ps1') . (Join-Path $runnerRoot 'fanout-process.ps1') +$invokePhase2AnalyzerPath = Join-Path $runnerRoot 'invoke-phase2-analyzer.ps1' +$invokePhase2Tokens = $null +$invokePhase2Errors = $null +$invokePhase2Ast = [Management.Automation.Language.Parser]::ParseFile($invokePhase2AnalyzerPath, [ref]$invokePhase2Tokens, [ref]$invokePhase2Errors) +if ($invokePhase2Errors.Count -gt 0) { throw "Cannot parse invoke-phase2-analyzer.ps1: $invokePhase2Errors" } +foreach ($definition in $invokePhase2Ast.EndBlock.Statements) { + if ($definition -is [Management.Automation.Language.FunctionDefinitionAst]) { + Invoke-Expression $definition.Extent.Text + } +} function Assert-True { param([bool]$Condition, [string]$Message) @@ -57,6 +68,19 @@ function Read-TestJson { return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 } +function Remove-TestProperty { + param( + [Parameter(Mandatory = $true)][object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + if ($Object -is [System.Collections.IDictionary]) { + [void]$Object.Remove($Name) + } else { + [void]$Object.PSObject.Properties.Remove($Name) + } +} + function Invoke-TestTool { param( [Parameter(Mandatory = $true)][string]$Path, @@ -136,6 +160,12 @@ function New-TestRun { $interactionHash = Get-Sha256HexFromFile -Path $interactionFile } + $candidateInstructionHash = $null + if ($Configuration -eq 'with_skill') { + $promptContent = "deterministic fixture prompt for $EvalName/$Configuration`n" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + $run = [ordered]@{ schema = (Get-RunnerSchemaNames).Run evalId = $EvalId @@ -155,6 +185,7 @@ function New-TestRun { inputFiles = @() fixtureHash = ('a' * 64) skillHash = if ($Configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = $candidateInstructionHash contract = [ordered]@{ sandboxRoot = '.' workingDirectory = 'repo' @@ -169,6 +200,10 @@ function New-TestRun { } Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + if ($EvalId -eq 2) { + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'extra-transcript-artifacts'), "1`n", [System.Text.UTF8Encoding]::new($false)) + } + return [pscustomobject]@{ Directory = $runDirectory RunPath = Join-Path $runDirectory 'run.json' @@ -176,6 +211,77 @@ function New-TestRun { } } +function Get-GradingEntry { + param( + [Parameter(Mandatory = $true)][object]$Document, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$Configuration, + [Parameter(Mandatory = $true)][int]$AssertionIndex + ) + + return @($Document.grading | Where-Object { + [int]$_.eval_id -eq $EvalId -and + [string]$_.configuration -eq $Configuration -and + [int]$_.assertion_index -eq $AssertionIndex + } | Select-Object -First 1)[0] +} + +function New-TestTranscriptEvidenceRef { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Canonical, + [ValidateSet('Events', 'Lines')][string]$Kind = 'Events', + [string]$Artifact = '' + ) + + $transcripts = @(Get-CanonicalTranscriptArtifacts -Record $Record -Canonical $Canonical) + if ([string]::IsNullOrWhiteSpace($Artifact)) { + $entry = if ($Kind -eq 'Events') { + @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'events' } | Select-Object -First 1) + } else { + @($transcripts | Where-Object { Test-JsonProperty -Object $_ -Name 'lines' } | Select-Object -First 1) + } + } else { + $entry = @($transcripts | Where-Object { [string]$_.artifact -eq $Artifact } | Select-Object -First 1) + } + if ($entry.Count -ne 1 -and [string]::IsNullOrWhiteSpace($Artifact) -and $Kind -eq 'Events') { + $entry = @((Get-TranscriptArtifactSnapshot -Record $Record -Artifact ("$([string]$Record.Configuration)/evidence/fixture-events.jsonl")).Entry) + } + if ($entry.Count -ne 1) { + throw "Missing $Kind transcript artifact '$Artifact' for deterministic grading coverage." + } + if ($Kind -eq 'Events') { + if (-not (Test-JsonProperty -Object $entry[0] -Name 'events')) { + throw "Transcript artifact '$([string]$entry[0].artifact)' does not expose events." + } + $event = @(Get-JsonProperty -Object $entry[0] -Name 'events' -Default @() | Select-Object -First 1) + if ($event.Count -ne 1) { + throw "Transcript artifact '$([string]$entry[0].artifact)' has no event content." + } + return [ordered]@{ + artifact = [string](Get-JsonProperty -Object $entry[0] -Name 'artifact' -Default '') + domain = 'transcript' + event_index = [int](Get-JsonProperty -Object $event[0] -Name 'event_index' -Default 0) + quote = [string](Get-JsonProperty -Object $event[0] -Name 'source_text' -Default (Get-JsonProperty -Object $event[0] -Name 'content' -Default '')) + } + } + + if (-not (Test-JsonProperty -Object $entry[0] -Name 'lines')) { + throw "Transcript artifact '$([string]$entry[0].artifact)' does not expose line content." + } + $lines = @(Get-JsonProperty -Object $entry[0] -Name 'lines' -Default @() | Select-Object -First 2) + if ($lines.Count -lt 1) { + throw "Transcript artifact '$([string]$entry[0].artifact)' has no line content." + } + return [ordered]@{ + artifact = [string](Get-JsonProperty -Object $entry[0] -Name 'artifact' -Default '') + domain = 'transcript' + start_line = [int](Get-JsonProperty -Object $lines[0] -Name 'line' -Default 1) + end_line = [int](Get-JsonProperty -Object $lines[$lines.Count - 1] -Name 'line' -Default 1) + quote = [string]::Join("`n", @($lines | ForEach-Object { [string](Get-JsonProperty -Object $_ -Name 'text' -Default '') })) + } +} + function New-TestGradingDocument { param([Parameter(Mandatory = $true)][object[]]$Records) @@ -183,15 +289,44 @@ function New-TestGradingDocument { foreach ($record in @($Records | Sort-Object EvalId, Configuration)) { $metadata = Read-TestJson -Path $record.MetadataPath $assertions = @($metadata.assertions) + $canonical = Read-TestJson -Path $record.ResultPath for ($index = 0; $index -lt $assertions.Count; $index++) { + $assertionText = if ($assertions[$index] -is [string]) { [string]$assertions[$index] } elseif ($assertions[$index].PSObject.Properties.Name -contains 'assertion') { [string]$assertions[$index].assertion } else { [string]$assertions[$index] } + $domain = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'evidence_domain') { [string]$assertions[$index].evidence_domain } else { 'output' } + $validator = if ($assertions[$index] -isnot [string] -and $assertions[$index].PSObject.Properties.Name -contains 'validator') { [string]$assertions[$index].validator } else { $null } + $output = [string]$canonical.output + $refs = if ($domain -eq 'validator') { + @([ordered]@{ + artifact = [string]$record.ResultRelative + domain = 'validator' + rule = $validator + version = 1 + passed = $true + event = 'deterministic fixture validator evidence' + }) + } elseif ($domain -eq 'transcript') { + @((New-TestTranscriptEvidenceRef -Record $record -Canonical $canonical -Kind Events)) + } else { + @([ordered]@{ + artifact = [string]$record.ResultRelative + domain = 'output' + start_line = 1 + end_line = 1 + quote = $output + }) + } $entries.Add([ordered]@{ eval_id = [int]$record.EvalId eval_name = [string]$record.EvalName configuration = [string]$record.Configuration assertion_index = $index - assertion = [string]$assertions[$index] + assertion = $assertionText passed = $true - evidence = 'deterministic grading-isolation fixture evidence' + evidence_domain = $domain + evidence_refs = @($refs) + reason = "The captured fixture response establishes assertion $index for this deterministic transport case." + evidence = "The captured fixture response establishes assertion $index for this deterministic transport case." + source = 'analyzer' }) } } @@ -360,6 +495,88 @@ function Copy-TestGradingDocument { return ConvertTo-RunnerJson -Value $Document -Depth 100 | ConvertFrom-Json -Depth 100 } +function New-AnalyzerSemanticFixture { + param([Parameter(Mandatory = $true)][string]$Root) + + $fixtureRoot = Join-Path $Root 'phase2-analyzer-direct' + $resultsDirectory = Join-Path $fixtureRoot 'results' + New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null + $canonicalPath = Join-Path $resultsDirectory 'with-skill.result.json' + $metadataPath = Join-Path $fixtureRoot 'eval-metadata.json' + $canonical = [ordered]@{ + output = "alpha line`nbeta line" + output_files = @('with_skill/evidence/opencode-events.jsonl', 'with_skill/evidence/opencode-stderr.txt') + } + Write-TestJson -Path $canonicalPath -Value $canonical + Write-TestJson -Path $metadataPath -Value ([ordered]@{ + expected_output = 'alpha line' + assertions = @( + [ordered]@{ assertion = 'alpha line is present'; evidence_domain = 'output' } + [ordered]@{ assertion = 'beta line is present'; evidence_domain = 'output' } + ) + }) + $record = [pscustomobject]@{ + EvalDirectory = $fixtureRoot + ResultPath = $canonicalPath + MetadataPath = $metadataPath + ResultRelative = 'with_skill.result.json' + } + $assertions = @( + [ordered]@{ eval_id = 1; eval_name = 'phase2-direct'; configuration = 'with_skill'; assertion_index = 0; assertion = 'alpha line is present'; evidence_domain = 'output'; validator = $null; record = $record } + [ordered]@{ eval_id = 1; eval_name = 'phase2-direct'; configuration = 'with_skill'; assertion_index = 1; assertion = 'beta line is present'; evidence_domain = 'output'; validator = $null; record = $record } + ) + return [pscustomobject]@{ + Root = $fixtureRoot + Canonical = Read-RunnerJson -Path $canonicalPath + Record = $record + Worker = [pscustomobject]@{ + worker_id = 'arm-1-with_skill' + eval_id = 1 + eval_name = 'phase2-direct' + configuration = 'with_skill' + record = $record + assertions = $assertions + } + AnalyzerProfile = [pscustomobject]@{ + Hash = ('f' * 64) + Runner = 'fixture' + Model = 'fixture-model' + ReasoningEffort = $null + } + } +} + +function Assert-AnalyzerResponseAccepted { + param( + [Parameter(Mandatory = $true)][string]$Text, + [Parameter(Mandatory = $true)][string]$ExpectedNormalization, + [Parameter(Mandatory = $true)][object]$Fixture, + [Parameter(Mandatory = $true)][string]$Description + ) + + $response = ConvertFrom-AnalyzerResponse -Text $Text + Assert-Equal $ExpectedNormalization ([string]$response.TransportNormalization) "$Description normalization" + $grades = @(Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $Fixture.Worker -Canonical $Fixture.Canonical) + Assert-Equal 2 $grades.Count "$Description semantic grading entry count" +} + +function Assert-ActionRejected { + param( + [Parameter(Mandatory = $true)][scriptblock]$Action, + [Parameter(Mandatory = $true)][string]$Pattern, + [Parameter(Mandatory = $true)][string]$Description + ) + + $failed = $false + try { + & $Action | Out-Null + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "$Description unexpected failure: $($_.Exception.Message)" + } + Assert-True $failed "$Description unexpectedly passed" +} + $testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-integrity-finalization-' + [Guid]::NewGuid().ToString('N')) $oldReportMode = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE') try { @@ -378,6 +595,9 @@ try { $fixtureDirectory = Join-Path $packageTools 'fixture' New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + $graderContractDirectory = Join-Path $iteration 'tools/skill-creator/agents' + New-Item -ItemType Directory -Path $graderContractDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $graderContractDirectory 'grader.md'), "# Deterministic grader contract`nGrade against frozen evidence only.`n", [System.Text.UTF8Encoding]::new($false)) $reportScript = Join-Path $iteration 'tools/test-report.ps1' $reportScriptText = @' @@ -410,14 +630,34 @@ for ($index = 0; $index -lt $count; $index++) { $evalName = if ($evalId -eq 2) { 'dotnet-strong-name-signing-confirmation' } else { 'integrity-eval-{0:d2}' -f $evalId } $evalDirectory = Join-Path $iteration $evalName New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - $assertion = if ($evalId -eq 2) { 'the protected operation is absent before confirmation and occurs only after the same-session confirmation turn' } else { 'the deterministic terminal response is captured' } + $assertions = if ($evalId -eq 2) { + @( + [ordered]@{ + assertion = 'the protected operation is absent before confirmation and occurs only after the same-session confirmation turn' + evidence_domain = 'transcript' + } + 'the response contains the fixture completion content' + ) + } elseif ($evalId -eq 3) { + @( + [ordered]@{ assertion = 'current Git branch was resolved'; evidence_domain = 'validator'; validator = 'git.current_branch_observed' } + [ordered]@{ assertion = 'base/default branch was resolved using local Git state'; evidence_domain = 'validator'; validator = 'git.default_branch_resolved' } + [ordered]@{ assertion = 'branch-only commits were collected'; evidence_domain = 'validator'; validator = 'git.branch_commits_collected' } + [ordered]@{ assertion = 'three-dot net diff was collected'; evidence_domain = 'validator'; validator = 'git.three_dot_diff_observed' } + [ordered]@{ assertion = 'no candidate package search was performed'; evidence_domain = 'validator'; validator = 'skill.no_candidate_package_search' } + [ordered]@{ assertion = 'no candidate install attempt was performed'; evidence_domain = 'validator'; validator = 'skill.no_candidate_install_attempt' } + [ordered]@{ assertion = 'no unnecessary project package or tool mutation was performed'; evidence_domain = 'validator'; validator = 'workspace.no_unnecessary_mutation' } + ) + } else { + @('the deterministic terminal response is captured', 'the response contains the fixture completion content') + } Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-metadata/1' eval_id = $evalId eval_name = $evalName prompt = "fixture prompt $evalId" expected_output = 'fixture output' - assertions = @($assertion) + assertions = @($assertions) }) $runs = [ordered]@{} foreach ($configuration in @('with_skill', 'without_skill')) { @@ -425,13 +665,23 @@ for ($index = 0; $index -lt $count; $index++) { $run = New-TestRun -IterationDirectory $iteration -EvalId $evalId -EvalName $evalName -Configuration $configuration -Interaction $interactionForRun $resultName = "$configuration.result.json" $executionName = "$configuration.execution-result.json" + $stubGrading = @($assertions | ForEach-Object { + [ordered]@{ + text = if ($_ -is [string]) { [string]$_ } elseif ($_.PSObject.Properties.Name -contains 'assertion') { [string]$_.assertion } else { [string]$_ } + passed = $null + evidence = '' + evidence_domain = if ($_ -isnot [string] -and $_.PSObject.Properties.Name -contains 'evidence_domain') { [string]$_.evidence_domain } else { 'output' } + evidence_refs = @() + reason = '' + } + }) Write-TestJson -Path (Join-Path $evalDirectory "results/$resultName") -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).PortableResult eval_id = $evalId eval_name = $evalName configuration = $configuration execution_status = 'unrun' - grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }) + grading = @($stubGrading) }) $runs[$configuration] = [ordered]@{ mode = $configuration @@ -472,8 +722,33 @@ for ($index = 0; $index -lt $count; $index++) { timeout_seconds = 60 concurrency = 3 } + $analyzerProfile = [ordered]@{ + schema = (Get-RunnerSchemaNames).AnalyzerProfile + contract_version = (Get-RunnerSchemaNames).Grading + runner = 'fixture' + harness = 'deterministic runner-owned fixture' + model = 'fixture-model' + reasoning_effort = $null + selection_source = 'explicit' + } Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $manifest Write-TestJson -Path (Join-Path $iteration 'execution-profile.json') -Value $profile + Write-TestJson -Path (Join-Path $iteration 'analyzer-profile.json') -Value $analyzerProfile + $manifest.analyzer_profile = 'analyzer-profile.json' + $manifest.analyzer_profile_sha256 = Get-Sha256HexFromFile -Path (Join-Path $iteration 'analyzer-profile.json') + $manifest.analyzer_selection = [ordered]@{ + runner = 'fixture' + harness = 'deterministic runner-owned fixture' + model = 'fixture-model' + reasoning_effort = $null + selection_source = 'explicit' + contract_version = (Get-RunnerSchemaNames).Grading + analyzer_profile_sha256 = [string]$manifest.analyzer_profile_sha256 + } + $manifest.phase2_controller = 'tools/eval-runners/invoke-phase2-analyzer.ps1' + $manifest.phase2_state = 'phase2-state.json' + $manifest.grading_freeze = 'grading-freeze.json' + Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $manifest $fanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' $fanout = Invoke-ForegroundPhaseOne -Path $fanoutScript -IterationDirectory $iteration @@ -562,12 +837,103 @@ for ($index = 0; $index -lt $count; $index++) { } $gradingPath = Join-Path $iteration 'grading.json' $validGrading = New-TestGradingDocument -Records $records + $transcriptEvalId = 2 + $transcriptAssertionIndex = 0 + $transcriptRecord = @($records | Where-Object { $_.EvalId -eq $transcriptEvalId -and $_.Configuration -eq 'with_skill' } | Select-Object -First 1)[0] + $transcriptCanonical = Read-TestJson -Path $transcriptRecord.ResultPath + $transcriptArtifacts = @(Get-CanonicalTranscriptArtifacts -Record $transcriptRecord -Canonical $transcriptCanonical) + $collisionEventArtifacts = @($transcriptArtifacts | Where-Object { [string]$_.artifact -like 'with_skill/evidence/*/events.jsonl' } | Sort-Object artifact) + $lineTranscriptArtifact = @($transcriptArtifacts | Where-Object { [string]$_.artifact -eq 'with_skill/evidence/logs/transcript.txt' } | Select-Object -First 1)[0] + Assert-Equal 2 $collisionEventArtifacts.Count 'fixture exposes two same-basename transcript artifacts' + Assert-True ($null -ne $lineTranscriptArtifact) 'fixture exposes a line-based transcript artifact' $validationScript = Join-Path $packageTools 'validate-eval-grading.ps1' $validationArguments = @('-IterationDirectory', $iteration, '-GradingPath', 'grading.json') $validationSideEffectPaths = Get-GradingValidationSideEffectPaths -IterationDirectory $iteration -Records $records $validationSnapshot = Get-TestFileHashSnapshot -Paths $validationSideEffectPaths if ($Suite -in @('All', 'Grading')) { + $directAnalyzerFixture = New-AnalyzerSemanticFixture -Root $testRoot + $validAnalyzerFragment = [ordered]@{ + schema = 'codebeltnet/agentic/eval-analyzer-fragment/1' + eval_id = 1 + configuration = 'with_skill' + grading = @( + [ordered]@{ + assertion_index = 0 + passed = $true + reason = 'alpha line is present in the frozen output.' + evidence_refs = @([ordered]@{ + artifact = 'with_skill.result.json' + domain = 'output' + start_line = 1 + end_line = 1 + quote = 'alpha line' + }) + } + [ordered]@{ + assertion_index = 1 + passed = $true + reason = 'beta line is present in the frozen output.' + evidence_refs = @([ordered]@{ + artifact = 'with_skill.result.json' + domain = 'output' + start_line = 1 + end_line = 1 + quote = 'beta line' + }) + } + ) + } + $validAnalyzerJson = ConvertTo-RunnerJson -Value $validAnalyzerFragment -Depth 100 + $fencedAnalyzerJson = [string]::Join([Environment]::NewLine, @('```json', $validAnalyzerJson, '```')) + $fencedAnalyzerJsonWithProse = [string]::Join([Environment]::NewLine, @('The grading fragment follows.', '```json', $validAnalyzerJson, '```')) + $duplicateFencedAnalyzerJson = [string]::Join([Environment]::NewLine, @('```json', $validAnalyzerJson, '```', '', '```json', $validAnalyzerJson, '```')) + Assert-AnalyzerResponseAccepted -Text $validAnalyzerJson -ExpectedNormalization 'raw_json' -Fixture $directAnalyzerFixture -Description 'raw analyzer JSON' + Assert-AnalyzerResponseAccepted -Text $fencedAnalyzerJson -ExpectedNormalization 'fenced_json' -Fixture $directAnalyzerFixture -Description 'fenced analyzer JSON' + Assert-AnalyzerResponseAccepted -Text $fencedAnalyzerJsonWithProse -ExpectedNormalization 'fenced_json_with_surrounding_text' -Fixture $directAnalyzerFixture -Description 'fenced analyzer JSON with prose' + Assert-ActionRejected -Description 'empty analyzer response rejected' -Pattern 'empty response' -Action { ConvertFrom-AnalyzerResponse -Text ' ' | Out-Null } + Assert-ActionRejected -Description 'malformed analyzer JSON rejected' -Pattern 'malformed JSON' -Action { ConvertFrom-AnalyzerResponse -Text '```json`n{"schema":`n```' | Out-Null } + Assert-ActionRejected -Description 'multiple analyzer JSON candidates rejected' -Pattern 'multiple (fenced blocks|JSON candidates)' -Action { ConvertFrom-AnalyzerResponse -Text $duplicateFencedAnalyzerJson | Out-Null } + $wrongSchemaFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongSchemaFragment.schema = 'wrong/schema' + Assert-ActionRejected -Description 'wrong analyzer fragment schema rejected' -Pattern 'unsupported schema' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongSchemaFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $wrongEvalFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongEvalFragment.eval_id = 2 + Assert-ActionRejected -Description 'wrong analyzer eval id rejected' -Pattern 'does not match the requested eval arm' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongEvalFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $wrongConfigurationFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $wrongConfigurationFragment.configuration = 'without_skill' + Assert-ActionRejected -Description 'wrong analyzer configuration rejected' -Pattern 'does not match the requested eval arm' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $wrongConfigurationFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $missingAssertionFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $missingAssertionFragment.grading = @($missingAssertionFragment.grading[0]) + Assert-ActionRejected -Description 'missing analyzer assertion index rejected' -Pattern 'grade cardinality' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $missingAssertionFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $duplicateAssertionFragment = Copy-TestGradingDocument -Document $validAnalyzerFragment + $duplicateAssertionFragment.grading[1].assertion_index = 0 + Assert-ActionRejected -Description 'duplicate analyzer assertion index rejected' -Pattern 'duplicates assertion_index' -Action { + $response = ConvertFrom-AnalyzerResponse -Text (ConvertTo-RunnerJson -Value $duplicateAssertionFragment -Depth 100) + Confirm-AnalyzerFragment -Fragment $response.Fragment -Worker $directAnalyzerFixture.Worker -Canonical $directAnalyzerFixture.Canonical | Out-Null + } + $phase2BundleRoot = Join-Path $testRoot 'phase2-bundle-direct' + New-Item -ItemType Directory -Path $phase2BundleRoot -Force | Out-Null + $directBundle = New-AnalyzerRunBundle -Phase2Root $phase2BundleRoot -Worker $directAnalyzerFixture.Worker -AnalyzerProfile $directAnalyzerFixture.AnalyzerProfile -AnalyzerExecutionProfilePath (Join-Path $phase2BundleRoot 'analyzer-execution-profile.json') -GraderContractText '# deterministic grader' + $directBundleDocument = Read-TestJson -Path $directBundle.BundlePath + Assert-Equal 'with_skill.result.json,with_skill/evidence/opencode-events.jsonl,with_skill/evidence/opencode-stderr.txt' ([string]::Join(',', @($directBundleDocument.allowed_artifacts))) 'allowed_artifacts preserves individual array entries' + $directBundleRun = Resolve-RunContract -RunPath $directBundle.RunPath + Assert-Equal 'phase2_analyzer' ([string]$directBundleRun.ExecutionRole) 'Phase 2 analyzer run declares its transport role explicitly' + Assert-Equal 'without_skill' ([string]$directBundleRun.Mode) 'Phase 2 analyzer transport stays without_skill' + Assert-Equal 'with_skill' ([string]$directBundleDocument.configuration) 'Phase 2 bundle preserves the graded subject configuration independently of transport mode' + $skeleton = Invoke-TestTool -Path $validationScript -Arguments @('-ShowSkeleton') Assert-ToolPasses -Invocation $skeleton -Description 'grading skeleton emission' $skeletonDocument = $skeleton.Text | ConvertFrom-Json -Depth 100 @@ -596,6 +962,139 @@ for ($index = 0; $index -lt $count; $index++) { Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'invalid grading entry validation' -ExpectedText 'passed must be a boolean' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'invalid grading validation' + foreach ($badEvidence in @('', " `t`n")) { + $bad = Copy-TestGradingDocument -Document $validGrading + $bad.grading[0].evidence = $badEvidence + Write-TestJson -Path $gradingPath -Value $bad + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'non-evidentiary PASS rejected' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'evidence rejection preserves frozen execution' + } + foreach ($badReason in @('Evaluation completed with output', 'Assertion evaluated against output', 'The assertion is met', 'Output matches the assertion')) { + $bad = Copy-TestGradingDocument -Document $validGrading + $bad.grading[0].reason = $badReason + $bad.grading[0].evidence = $badReason + Write-TestJson -Path $gradingPath -Value $bad + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'generic PASS reason rejected' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'reason rejection preserves frozen execution' + } + $badQuote = Copy-TestGradingDocument -Document $validGrading + $badQuote.grading[0].evidence_refs[0].quote = 'fabricated unavailable observation' + Write-TestJson -Path $gradingPath -Value $badQuote + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'missing output quote rejected' -ExpectedText 'quote is absent' + + $badDomain = Copy-TestGradingDocument -Document $validGrading + $badDomain.grading[0].evidence_domain = 'transcript' + Write-TestJson -Path $gradingPath -Value $badDomain + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'wrong evidence domain rejected' -ExpectedText 'assertion identity' + + $validTranscriptEvent = Copy-TestGradingDocument -Document $validGrading + $validTranscriptEventEntry = Get-GradingEntry -Document $validTranscriptEvent -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $validTranscriptEventEntry.evidence_refs = @((New-TestTranscriptEvidenceRef -Record $transcriptRecord -Canonical $transcriptCanonical -Kind Events -Artifact ([string]$collisionEventArtifacts[0].artifact))) + Write-TestJson -Path $gradingPath -Value $validTranscriptEvent + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'valid transcript event_index grading validation' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'valid transcript event-index grading validation' + + $missingTranscriptLocator = Copy-TestGradingDocument -Document $validGrading + $missingTranscriptLocatorEntry = Get-GradingEntry -Document $missingTranscriptLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + Remove-TestProperty -Object $missingTranscriptLocatorEntry.evidence_refs[0] -Name 'event_index' + Remove-TestProperty -Object $missingTranscriptLocatorEntry.evidence_refs[0] -Name 'quote' + Write-TestJson -Path $gradingPath -Value $missingTranscriptLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'artifact-only transcript evidence rejected' -ExpectedText 'must declare event_index or start_line/end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'artifact-only transcript evidence rejection' + + $quotedWithoutTranscriptLocator = Copy-TestGradingDocument -Document $validGrading + $quotedWithoutTranscriptLocatorEntry = Get-GradingEntry -Document $quotedWithoutTranscriptLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + Remove-TestProperty -Object $quotedWithoutTranscriptLocatorEntry.evidence_refs[0] -Name 'event_index' + $quotedWithoutTranscriptLocatorEntry.evidence_refs[0].quote = 'some text' + Write-TestJson -Path $gradingPath -Value $quotedWithoutTranscriptLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'quoted transcript evidence without locator rejected' -ExpectedText 'must declare event_index or start_line/end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'quoted transcript evidence without locator rejection' + + $invalidEventIndex = Copy-TestGradingDocument -Document $validGrading + $invalidEventIndexEntry = Get-GradingEntry -Document $invalidEventIndex -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $invalidEventIndexEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + event_index = 9 + }) + Write-TestJson -Path $gradingPath -Value $invalidEventIndex + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'out-of-range transcript event_index rejected' -ExpectedText 'event_index 9 is outside' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'transcript event-index bounds rejection' + + $validTranscriptLines = Copy-TestGradingDocument -Document $validGrading + $validTranscriptLinesEntry = Get-GradingEntry -Document $validTranscriptLines -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $validTranscriptLinesEntry.evidence_refs = @((New-TestTranscriptEvidenceRef -Record $transcriptRecord -Canonical $transcriptCanonical -Kind Lines -Artifact ([string]$lineTranscriptArtifact.artifact))) + Write-TestJson -Path $gradingPath -Value $validTranscriptLines + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'valid transcript line-range grading validation' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'valid transcript line-range grading validation' + + $invalidTranscriptLines = Copy-TestGradingDocument -Document $validGrading + $invalidTranscriptLinesEntry = Get-GradingEntry -Document $invalidTranscriptLines -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $invalidTranscriptLinesEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + start_line = 9 + end_line = 9 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $invalidTranscriptLines + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'out-of-range transcript line range rejected' -ExpectedText 'line range is outside' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'transcript line-range bounds rejection' + + $missingTranscriptEndLine = Copy-TestGradingDocument -Document $validGrading + $missingTranscriptEndLineEntry = Get-GradingEntry -Document $missingTranscriptEndLine -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $missingTranscriptEndLineEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + start_line = 1 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $missingTranscriptEndLine + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'partial transcript line locator rejected' -ExpectedText 'line locator must declare both start_line and end_line' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'partial transcript line locator rejection' + + $lineArtifactEventLocator = Copy-TestGradingDocument -Document $validGrading + $lineArtifactEventLocatorEntry = Get-GradingEntry -Document $lineArtifactEventLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $lineArtifactEventLocatorEntry.evidence_refs = @([ordered]@{ + artifact = [string]$lineTranscriptArtifact.artifact + domain = 'transcript' + event_index = 0 + quote = 'alpha frozen transcript line' + }) + Write-TestJson -Path $gradingPath -Value $lineArtifactEventLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'event locator on line-based transcript rejected' -ExpectedText 'uses event_index but the frozen artifact has line-based content' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'line-based transcript event locator rejection' + + $eventArtifactLineLocator = Copy-TestGradingDocument -Document $validGrading + $eventArtifactLineLocatorEntry = Get-GradingEntry -Document $eventArtifactLineLocator -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $eventArtifactLineLocatorEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + start_line = 1 + end_line = 1 + quote = 'artifact-a exact frozen quote' + }) + Write-TestJson -Path $gradingPath -Value $eventArtifactLineLocator + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'line locator on event-based transcript rejected' -ExpectedText 'uses start_line/end_line but the frozen artifact has event-based content' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'event-based transcript line locator rejection' + + $crossArtifactQuote = Copy-TestGradingDocument -Document $validGrading + $crossArtifactQuoteEntry = Get-GradingEntry -Document $crossArtifactQuote -EvalId $transcriptEvalId -Configuration 'with_skill' -AssertionIndex $transcriptAssertionIndex + $crossArtifactQuoteEntry.evidence_refs = @([ordered]@{ + artifact = [string]$collisionEventArtifacts[0].artifact + domain = 'transcript' + event_index = 0 + quote = 'artifact-b exact frozen quote' + }) + Write-TestJson -Path $gradingPath -Value $crossArtifactQuote + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'cross-artifact transcript quote rejected' -ExpectedText 'quote is absent from the referenced frozen event' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'cross-artifact transcript quote rejection' + + $repeated = Copy-TestGradingDocument -Document $validGrading + $repeated.grading[1].evidence = $repeated.grading[0].evidence + Write-TestJson -Path $gradingPath -Value $repeated + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'repeated PASS evidence rejected' -ExpectedText 'Repeated PASS evidence' + Write-TestJson -Path $gradingPath -Value $validGrading $validValidation = Invoke-TestTool -Path $validationScript -Arguments $validationArguments Assert-ToolPasses -Invocation $validValidation -Description 'valid grading validation' @@ -605,12 +1104,81 @@ for ($index = 0; $index -lt $count; $index++) { } Write-TestJson -Path $gradingPath -Value $validGrading + if ($Suite -in @('All', 'Application', 'Finalization')) { + $manualNoFreezeFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $manualNoFreezeFinalizer -Description 'finalizer rejects handcrafted grading without Phase 2 freeze' -ExpectedText 'grading-freeze.json is missing' + $phase2Script = Join-Path $packageTools 'invoke-phase2-analyzer.ps1' + $phase2 = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolPasses -Invocation $phase2 -Description 'fixture Phase 2 analyzer controller' + Assert-True (Test-Path -LiteralPath (Join-Path $iteration 'phase2-state.json') -PathType Leaf) 'Phase 2 writes phase2-state.json' + Assert-True (Test-Path -LiteralPath (Join-Path $iteration 'grading-freeze.json') -PathType Leaf) 'Phase 2 writes grading-freeze.json' + $phase2State = Read-TestJson -Path (Join-Path $iteration 'phase2-state.json') + Assert-True (@($phase2State.expected_worker_ids | Where-Object { [string]$_ -like 'arm-3-*' }).Count -eq 0) 'validator-only eval arms must require zero analyzer workers' + Assert-Equal 14 @($phase2State.validator_results).Count 'validator-only paired arms resolve process assertions deterministically' + Assert-Equal 4 @($phase2State.expected_worker_ids).Count 'semantic worker cardinality derives from unresolved assertions, not manifest arm count' + $phase2BundlePath = Join-Path $iteration 'phase2\work\arm-2-with_skill\repo\input-bundle.json' + $phase2Bundle = Read-TestJson -Path $phase2BundlePath + $stagedCollisionEntries = @($phase2Bundle.frozen_transcripts | Where-Object { [string]$_.artifact -like 'with_skill/evidence/*/events.jsonl' } | Sort-Object artifact) + Assert-Equal 2 $stagedCollisionEntries.Count 'Phase 2 stages both same-basename transcript artifacts' + Assert-True ([string]$stagedCollisionEntries[0].staged_artifact -cne [string]$stagedCollisionEntries[1].staged_artifact) 'same-basename transcript artifacts must stage to unique paths' + $phase2BundleRepo = Join-Path $iteration 'phase2\work\arm-2-with_skill\repo' + $stagedCollisionContents = [System.Collections.Generic.List[string]]::new() + foreach ($entry in $stagedCollisionEntries) { + $sourcePath = Resolve-TranscriptArtifactSourcePath -Record $transcriptRecord -Artifact ([string]$entry.artifact) + $stagedPath = Resolve-ContainedPath -BasePath $phase2BundleRepo -RelativePath ([string]$entry.staged_artifact) -FieldName 'staged transcript artifact' -Kind File + Assert-Equal ([string]$entry.source_sha256) (Get-Sha256HexFromFile -Path $sourcePath) "bundle preserves the source hash for $([string]$entry.artifact)" + Assert-Equal ([string]$entry.staged_sha256) (Get-Sha256HexFromFile -Path $stagedPath) "bundle preserves the staged hash for $([string]$entry.artifact)" + $sourceContent = [System.IO.File]::ReadAllText($sourcePath, [System.Text.UTF8Encoding]::new($false)) + $stagedContent = [System.IO.File]::ReadAllText($stagedPath, [System.Text.UTF8Encoding]::new($false)) + Assert-Equal $sourceContent $stagedContent "staged transcript preserves the frozen content for $([string]$entry.artifact)" + $stagedCollisionContents.Add($stagedContent) + } + Assert-True ([string]$stagedCollisionContents[0] -cne [string]$stagedCollisionContents[1]) 'same-basename transcript artifacts keep distinct staged content' + $rootGradingAfterPhase2 = Read-TestJson -Path $gradingPath + Assert-Equal 22 @($rootGradingAfterPhase2.grading).Count 'root grading cardinality derives from normalized assertions' + Assert-True (@($rootGradingAfterPhase2.grading | Where-Object { [string]$_.evidence_domain -eq 'validator' -and [string]$_.source -eq 'validator' }).Count -eq 14) 'validator assertions are resolved by deterministic validator results, not analyzer prose' + $validGrading = Read-TestJson -Path $gradingPath + # Fix 2: second invocation must return already_frozen, exit 0, perform zero new analyzer work. + $freezeBytes = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'grading-freeze.json')) + $stateBytes = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'phase2-state.json')) + $gradingBytes = [System.IO.File]::ReadAllBytes($gradingPath) + $phase2AnalyzerWorkerDirCountBefore = @(Get-ChildItem -LiteralPath (Join-Path $iteration 'phase2\work') -Directory -ErrorAction SilentlyContinue).Count + $phase2Again = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolPasses -Invocation $phase2Again -Description 'second Phase 2 invocation succeeds (already_frozen)' + $phase2AgainOutput = $phase2Again.Text | ConvertFrom-Json -ErrorAction SilentlyContinue + Assert-Equal 'already_frozen' ([string]$phase2AgainOutput.status) 'second Phase 2 invocation must report already_frozen status' + $freezeBytesAfter = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'grading-freeze.json')) + $stateBytesAfter = [System.IO.File]::ReadAllBytes((Join-Path $iteration 'phase2-state.json')) + $gradingBytesAfter = [System.IO.File]::ReadAllBytes($gradingPath) + Assert-True ([System.Linq.Enumerable]::SequenceEqual($freezeBytes, $freezeBytesAfter)) 'grading-freeze.json must be byte-identical after second Phase 2 invocation' + Assert-True ([System.Linq.Enumerable]::SequenceEqual($stateBytesAfter, $stateBytes)) 'phase2-state.json must be byte-identical after second Phase 2 invocation' + Assert-True ([System.Linq.Enumerable]::SequenceEqual($gradingBytesAfter, $gradingBytes)) 'grading.json must be byte-identical after second Phase 2 invocation' + $phase2AnalyzerWorkerDirCountAfter = @(Get-ChildItem -LiteralPath (Join-Path $iteration 'phase2\work') -Directory -ErrorAction SilentlyContinue).Count + Assert-Equal $phase2AnalyzerWorkerDirCountBefore $phase2AnalyzerWorkerDirCountAfter 'second Phase 2 invocation must create zero new analyzer worker directories' + # Tampered grading-freeze.json must fail rather than rerun. + $freezePath = Join-Path $iteration 'grading-freeze.json' + $originalFreezeBytes = [System.IO.File]::ReadAllBytes($freezePath) + $tamperedFreeze = Read-TestJson -Path $freezePath + $tamperedFreeze.grading_sha256 = ('0' * 64) + Write-TestJson -Path $freezePath -Value $tamperedFreeze + $phase2Tampered = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolFails -Invocation $phase2Tampered -Description 'tampered grading-freeze.json fails Phase 2 instead of rerunning' + [System.IO.File]::WriteAllBytes($freezePath, $originalFreezeBytes) + # Orphaned phase2-state.json without a valid freeze must fail closed. + $statePath = Join-Path $iteration 'phase2-state.json' + $originalStateBytes = [System.IO.File]::ReadAllBytes($statePath) + Remove-Item -LiteralPath $freezePath -Force + $phase2Orphaned = Invoke-TestTool -Path $phase2Script -Arguments @('-IterationDirectory', $iteration, '-Concurrency', '3', '-TimeoutSeconds', '60') + Assert-ToolFails -Invocation $phase2Orphaned -Description 'orphaned phase2-state.json without freeze fails closed' + Assert-True ([string]$phase2Orphaned.Text -match 'duplicate|fresh package|grading freeze') 'orphaned state must explain the retry is forbidden' + [System.IO.File]::WriteAllBytes($freezePath, $originalFreezeBytes) + } if ($Suite -in @('All', 'Application')) { $invalidDirectFinalizer = Copy-TestGradingDocument -Document $validGrading $invalidDirectFinalizer.grading[0].evidence = 123 Write-TestJson -Path $gradingPath -Value $invalidDirectFinalizer $directInvalidFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments - Assert-ToolFails -Invocation $directInvalidFinalizer -Description 'finalizer performs its own grading validation' -ExpectedText 'evidence must be a string' + Assert-ToolFails -Invocation $directInvalidFinalizer -Description 'finalizer rejects post-freeze grading mutation' -ExpectedText 'Root grading.json was changed' Assert-True (-not (Test-Path -LiteralPath (Join-Path $iteration 'report.html') -PathType Leaf)) 'invalid direct finalizer produces no report' Write-TestJson -Path $gradingPath -Value $validGrading @@ -626,13 +1194,13 @@ for ($index = 0; $index -lt $count; $index++) { $invalidGrading = [ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = @($validGrading.grading); output = 'raw output is forbidden here' } Write-TestJson -Path $gradingPath -Value $invalidGrading $invalidApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments - Assert-ToolFails -Invocation $invalidApply -Description 'grading artifact with raw output is rejected' -ExpectedText 'unsupported field' + Assert-ToolFails -Invocation $invalidApply -Description 'grading artifact with raw output is rejected' -ExpectedText 'Root grading.json was changed' $forbiddenGradingFields = @('model', 'harness', 'execution_result_sha256', 'session_id', 'telemetry') foreach ($forbiddenField in $forbiddenGradingFields) { $forbiddenEntries = @($validGrading.grading | ForEach-Object { $copy = [ordered]@{} - foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence')) { + foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence', 'evidence_domain', 'evidence_refs', 'reason', 'source')) { $copy[$name] = Get-JsonProperty -Object $_ -Name $name } $copy[$forbiddenField] = 'forbidden' @@ -640,7 +1208,7 @@ for ($index = 0; $index -lt $count; $index++) { }) Write-TestJson -Path $gradingPath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = $forbiddenEntries }) $forbiddenApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments - Assert-ToolFails -Invocation $forbiddenApply -Description "grading artifact with $forbiddenField is rejected" -ExpectedText 'unsupported field' + Assert-ToolFails -Invocation $forbiddenApply -Description "grading artifact with $forbiddenField is rejected" -ExpectedText 'Root grading.json was changed' } Write-TestJson -Path $gradingPath -Value $validGrading } diff --git a/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 index 9f79922..2b4f042 100644 --- a/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 +++ b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 @@ -1,184 +1,186 @@ -<# -.SYNOPSIS - Regression: mixed-terminal fan-out preserves raw statuses and records evidence_validation separately. -.DESCRIPTION - MODEL-FREE deterministic check. Does not execute any model. -#> - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'orchestration.ps1') - -function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } -function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } - -function New-TestNativeTerminalEvidence { - param( - [Parameter(Mandatory = $true)][object]$Arm, - [Parameter(Mandatory = $true)][object]$RunData, - [Parameter(Mandatory = $true)][string]$WorkerSessionId - ) - - return [ordered]@{ - status = 'completed' - session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } - requested = [ordered]@{ model = [string]$Arm.worker.model } - input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } - evidence = [ordered]@{ - delegation = [ordered]@{ - mechanism = 'deterministic-fake-native-worker' - worker_session_id = $WorkerSessionId - observed_model = [string]$Arm.worker.model - observed_working_directory = [string]$RunData.WorkingDirectoryPath - observed_home = [string]$RunData.HomeDirectoryPath - fresh_worker = $true - home_config_isolated = $true - prompt_fidelity = $true - prompt_sha256 = [string]$RunData.PromptHash - terminal_result_capture = $true - paired_arm_visible = $false - grading_material_visible = $false - nested_model_execution = $false - model_execution_count = 1 - } - } - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-mixed-terminal-' + [Guid]::NewGuid().ToString('N')) -$iteration = Join-Path $testRoot 'iteration-1' -New-Item -ItemType Directory -Path $iteration -Force | Out-Null - -# Create 4 eval arms -$manifestEvals = [System.Collections.Generic.List[object]]::new() -for ($evalId = 1; $evalId -le 4; $evalId++) { - $evalName = 'eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $iteration $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'without_skill') -Force | Out-Null - Write-Output "prepared $evalName" - $runs = [ordered]@{} - foreach ($configuration in @('with_skill','without_skill')) { - $runPath = Join-Path $evalDirectory $configuration - New-Item -ItemType Directory -Path $runPath -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null - New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null - if ($configuration -eq 'with_skill') { - New-Item -ItemType Directory -Path (Join-Path $runPath 'skill') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path (Join-Path $runPath 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) - } - [System.IO.File]::WriteAllText((Join-Path $runPath 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - $runJson = [ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName +<# +.SYNOPSIS + Regression: mixed-terminal fan-out preserves raw statuses and records evidence_validation separately. +.DESCRIPTION + MODEL-FREE deterministic check. Does not execute any model. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +function New-TestNativeTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$Arm, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + return [ordered]@{ + status = 'completed' + session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } + requested = [ordered]@{ model = [string]$Arm.worker.model } + input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $WorkerSessionId + observed_model = [string]$Arm.worker.model + observed_working_directory = [string]$RunData.WorkingDirectoryPath + observed_home = [string]$RunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$RunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-mixed-terminal-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# Create 4 eval arms +$manifestEvals = [System.Collections.Generic.List[object]]::new() +for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'without_skill') -Force | Out-Null + Write-Output "prepared $evalName" + $runs = [ordered]@{} + foreach ($configuration in @('with_skill','without_skill')) { + $runPath = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runPath -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null + if ($configuration -eq 'with_skill') { + New-Item -ItemType Directory -Path (Join-Path $runPath 'skill') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path (Join-Path $runPath 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $runJson = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - mustNotReadOutsideSandbox = $true - fixtureHash = ('a' * 64) - skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } - } - [System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) - $resultsDir = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null - $executionResultRel = "$evalName/results/$configuration.execution-result.json" - $resultRel = "$evalName/results/$configuration.result.json" - # create a canonical (empty) result.json so manifest validation is satisfied - [System.IO.File]::WriteAllText((Join-Path $resultsDir "$configuration.result.json"), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - $runs[$configuration] = [ordered]@{ mode = $configuration; run_manifest = "$evalName/$configuration/run.json"; execution_result = $executionResultRel; result = $resultRel } - } - [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) -} - -$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @($manifestEvals) } -$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'fake'; model = 'fixture-model'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 60; concurrency = 4 } - -$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile -$state = New-OrchestrationState -Plan $plan - -# Accept all workers -$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) -foreach ($d in $dispatches) { [void](Register-DelegationAccepted -State $state -WorkerId $d.worker_id -WorkerSessionId ('sess-' + $d.worker_id)) } -Assert-Equal 4 (Get-OrchestrationActiveCount -State $state) 'all workers active' - -# Prepare synthetic execution evidences -$arms = @($plan.arms) -$evidences = @{} -# arm1 -> timed_out -$arm1 = $arms[0] -$evidences[$arm1.worker_id] = [ordered]@{ - status = 'timed_out' - session = [ordered]@{ id = ('sess-' + $arm1.worker_id); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$arm1.eval_id; eval_name = [string]$arm1.eval_name; configuration = [string]$arm1.configuration } - requested = [ordered]@{ model = [string]$arm1.worker.model } - evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm1.worker_id); observed_model = [string]$arm1.worker.model; terminal_result_capture = $false } } -} -# arm2 -> completed (valid) -$arm2 = $arms[1] -$runData2 = Resolve-RunContract -RunPath ([string]$arm2.worker.run_manifest_path) -$evidences[$arm2.worker_id] = New-TestNativeTerminalEvidence -Arm $arm2 -RunData $runData2 -WorkerSessionId ('sess-' + $arm2.worker_id) -# arm3 -> failed -$arm3 = $arms[2] -$evidences[$arm3.worker_id] = [ordered]@{ - status = 'failed' - session = [ordered]@{ id = ('sess-' + $arm3.worker_id); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = [int]$arm3.eval_id; eval_name = [string]$arm3.eval_name; configuration = [string]$arm3.configuration } - requested = [ordered]@{ model = [string]$arm3.worker.model } - evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm3.worker_id); observed_model = [string]$arm3.worker.model; terminal_result_capture = $false } } -} -# arm4 -> completed -$arm4 = $arms[3] -$runData4 = Resolve-RunContract -RunPath ([string]$arm4.worker.run_manifest_path) -$evidences[$arm4.worker_id] = New-TestNativeTerminalEvidence -Arm $arm4 -RunData $runData4 -WorkerSessionId ('sess-' + $arm4.worker_id) - -# Sanity: ensure evidences exist for each arm -Write-Output "ARM KEYS:" -foreach ($arm in $arms) { Write-Output " - $($arm.worker_id)" } -Write-Output "EVIDENCE KEYS:" -foreach ($k in $evidences.Keys) { Write-Output " - $k" } - -# Register terminals -foreach ($arm in $arms) { - $w = [string]$arm.worker_id - if (-not $evidences.ContainsKey($w)) { throw "Missing synthetic evidence for $w" } - $exec = $evidences[$w] - [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $w -ExecutionEvidence $exec) -} - -# Verify ledger preserves raw statuses and records evidence_validation -foreach ($arm in $arms) { - $w = $arm.worker_id - $ledger = $state.completed[$w] - $raw = $evidences[$w] - Assert-Equal $raw.status $ledger.status "ledger.status should equal raw for $w" - $ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null - Write-Output "ledger[$w].native_worker_evidence_failures = $([string]::Join(', ', @($ledger.native_worker_evidence_failures | Where-Object {$_} )))" - Write-Output "ledger[$w].evidence_validation.status = $($ev.status)" - if ($raw.status -eq 'completed') { Assert-Equal 'passed' $ev.status "evidence_validation should pass for $w" } else { Assert-Equal 'failed' $ev.status "evidence_validation should fail for $w" } -} - -# Negative integrity: ledger mismatch must be rejected by Assert-FreezeTerminalLedgerEntry -# craft a fake record and raw object -$fakeRecord = [ordered]@{ EvalId = 999; Configuration = 'with_skill'; } -$fakeRaw = [ordered]@{ status = 'timed_out'; session = [ordered]@{ id = 'sess-fake' } } -# craft a state with a mismatched ledger entry -$badState = [ordered]@{ completed = [ordered]@{ 'arm-999-with_skill' = [ordered]@{ worker_id = 'arm-999-with_skill'; eval_id = 999; configuration = 'with_skill'; status = 'incompatible'; worker_session_id = 'sess-fake' } } } -$threw = $false -try { [void](Assert-FreezeTerminalLedgerEntry -Record $fakeRecord -Raw $fakeRaw -State $badState) } catch { $threw = $true } -Assert-True $threw 'Assert-FreezeTerminalLedgerEntry must reject ledger/raw status mismatch' - -Write-Output 'MIXED-TERMINAL REGRESSION: PASS' \ No newline at end of file + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + mustNotReadOutsideSandbox = $true + fixtureHash = ('a' * 64) + skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = if ($configuration -eq 'with_skill') { ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("terminal test prompt for $evalName/$configuration")))).ToLowerInvariant() } else { $null } + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) + $resultsDir = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null + $executionResultRel = "$evalName/results/$configuration.execution-result.json" + $resultRel = "$evalName/results/$configuration.result.json" + # create a canonical (empty) result.json so manifest validation is satisfied + [System.IO.File]::WriteAllText((Join-Path $resultsDir "$configuration.result.json"), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $runs[$configuration] = [ordered]@{ mode = $configuration; run_manifest = "$evalName/$configuration/run.json"; execution_result = $executionResultRel; result = $resultRel } + } + [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) +} + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @($manifestEvals) } +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'fake'; model = 'fixture-model'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 60; concurrency = 4 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# Accept all workers +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +foreach ($d in $dispatches) { [void](Register-DelegationAccepted -State $state -WorkerId $d.worker_id -WorkerSessionId ('sess-' + $d.worker_id)) } +Assert-Equal 4 (Get-OrchestrationActiveCount -State $state) 'all workers active' + +# Prepare synthetic execution evidences +$arms = @($plan.arms) +$evidences = @{} +# arm1 -> timed_out +$arm1 = $arms[0] +$evidences[$arm1.worker_id] = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('sess-' + $arm1.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm1.eval_id; eval_name = [string]$arm1.eval_name; configuration = [string]$arm1.configuration } + requested = [ordered]@{ model = [string]$arm1.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm1.worker_id); observed_model = [string]$arm1.worker.model; terminal_result_capture = $false } } +} +# arm2 -> completed (valid) +$arm2 = $arms[1] +$runData2 = Resolve-RunContract -RunPath ([string]$arm2.worker.run_manifest_path) +$evidences[$arm2.worker_id] = New-TestNativeTerminalEvidence -Arm $arm2 -RunData $runData2 -WorkerSessionId ('sess-' + $arm2.worker_id) +# arm3 -> failed +$arm3 = $arms[2] +$evidences[$arm3.worker_id] = [ordered]@{ + status = 'failed' + session = [ordered]@{ id = ('sess-' + $arm3.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm3.eval_id; eval_name = [string]$arm3.eval_name; configuration = [string]$arm3.configuration } + requested = [ordered]@{ model = [string]$arm3.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm3.worker_id); observed_model = [string]$arm3.worker.model; terminal_result_capture = $false } } +} +# arm4 -> completed +$arm4 = $arms[3] +$runData4 = Resolve-RunContract -RunPath ([string]$arm4.worker.run_manifest_path) +$evidences[$arm4.worker_id] = New-TestNativeTerminalEvidence -Arm $arm4 -RunData $runData4 -WorkerSessionId ('sess-' + $arm4.worker_id) + +# Sanity: ensure evidences exist for each arm +Write-Output "ARM KEYS:" +foreach ($arm in $arms) { Write-Output " - $($arm.worker_id)" } +Write-Output "EVIDENCE KEYS:" +foreach ($k in $evidences.Keys) { Write-Output " - $k" } + +# Register terminals +foreach ($arm in $arms) { + $w = [string]$arm.worker_id + if (-not $evidences.ContainsKey($w)) { throw "Missing synthetic evidence for $w" } + $exec = $evidences[$w] + [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $w -ExecutionEvidence $exec) +} + +# Verify ledger preserves raw statuses and records evidence_validation +foreach ($arm in $arms) { + $w = $arm.worker_id + $ledger = $state.completed[$w] + $raw = $evidences[$w] + Assert-Equal $raw.status $ledger.status "ledger.status should equal raw for $w" + $ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null + Write-Output "ledger[$w].native_worker_evidence_failures = $([string]::Join(', ', @($ledger.native_worker_evidence_failures | Where-Object {$_} )))" + Write-Output "ledger[$w].evidence_validation.status = $($ev.status)" + if ($raw.status -eq 'completed') { Assert-Equal 'passed' $ev.status "evidence_validation should pass for $w" } else { Assert-Equal 'failed' $ev.status "evidence_validation should fail for $w" } +} + +# Negative integrity: ledger mismatch must be rejected by Assert-FreezeTerminalLedgerEntry +# craft a fake record and raw object +$fakeRecord = [ordered]@{ EvalId = 999; Configuration = 'with_skill'; } +$fakeRaw = [ordered]@{ status = 'timed_out'; session = [ordered]@{ id = 'sess-fake' } } +# craft a state with a mismatched ledger entry +$badState = [ordered]@{ completed = [ordered]@{ 'arm-999-with_skill' = [ordered]@{ worker_id = 'arm-999-with_skill'; eval_id = 999; configuration = 'with_skill'; status = 'incompatible'; worker_session_id = 'sess-fake' } } } +$threw = $false +try { [void](Assert-FreezeTerminalLedgerEntry -Record $fakeRecord -Raw $fakeRaw -State $badState) } catch { $threw = $true } +Assert-True $threw 'Assert-FreezeTerminalLedgerEntry must reject ledger/raw status mismatch' + +Write-Output 'MIXED-TERMINAL REGRESSION: PASS' diff --git a/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 index d019935..f3a5d0c 100644 --- a/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 +++ b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 @@ -1,86 +1,86 @@ -<# -.SYNOPSIS - Regression: OpenCode timed-out interaction evidence accepted as honest terminal result. -.DESCRIPTION - MODEL-FREE deterministic check using synthetic opencode-style execution evidence. -#> - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'orchestration.ps1') - -function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } -function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-opencode-tmo-' + [Guid]::NewGuid().ToString('N')) -$iteration = Join-Path $testRoot 'iteration-1' -New-Item -ItemType Directory -Path $iteration -Force | Out-Null - -# single eval/arm -$evalName = 'eval-01' -$evalDirectory = Join-Path $iteration $evalName -New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null -# run manifest -$runPath = Join-Path $evalDirectory 'with_skill' +<# +.SYNOPSIS + Regression: OpenCode timed-out interaction evidence accepted as honest terminal result. +.DESCRIPTION + MODEL-FREE deterministic check using synthetic opencode-style execution evidence. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-opencode-tmo-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# single eval/arm +$evalName = 'eval-01' +$evalDirectory = Join-Path $iteration $evalName +New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null +# run manifest +$runPath = Join-Path $evalDirectory 'with_skill' $skillPath = Join-Path $runPath 'skill\test-skill' New-Item -ItemType Directory -Path $skillPath -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $skillPath 'SKILL.md'), '# deterministic fixture skill', [System.Text.UTF8Encoding]::new($false)) -$runJson = [ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = $evalName; candidateSkillName = 'test-skill'; skillName = 'test-skill'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/test-skill'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; fixtureHash = ('a' * 64); skillHash = ('b' * 64) } -[System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) -New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null -New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null -# results and metadata -$resultsDir = Join-Path $evalDirectory 'results' -New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null -[System.IO.File]::WriteAllText((Join-Path $resultsDir 'with_skill.result.json'), (([ordered]@{ eval_id = 1; configuration = 'with_skill'; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) -[System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = 1; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) - -$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @([ordered]@{ eval_id = 1; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = [ordered]@{ with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = "$evalName/with_skill/run.json"; execution_result = "$evalName/results/with_skill.execution-result.json"; result = "$evalName/results/with_skill.result.json" } } }) } - -$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'opencode'; model = 'opencode/muse-spark-1.2-contributor-free'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900; concurrency = 1 } - -$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile -$state = New-OrchestrationState -Plan $plan - -# accept the single worker -$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) -[void](Register-DelegationAccepted -State $state -WorkerId $dispatches[0].worker_id -WorkerSessionId ('sess-' + $dispatches[0].worker_id)) - -# craft opencode-style timed_out execution evidence -$workerId = $dispatches[0].worker_id -$execEvidence = [ordered]@{ - status = 'timed_out' - session = [ordered]@{ id = ('opencode-session-' + $workerId); fresh = $true; resumed = $false } - run = [ordered]@{ eval_id = 1; eval_name = $evalName; configuration = 'with_skill' } - requested = [ordered]@{ model = $profile.model } - input = [ordered]@{ prompt_sha256 = ('a' * 64) } - evidence = [ordered]@{ - delegation = [ordered]@{ - mechanism = 'opencode-native' - worker_session_id = ('opencode-session-' + $workerId) - observed_model = $profile.model - observed_working_directory = (Join-Path $runPath 'repo') - observed_home = (Join-Path $runPath 'home') - fresh_worker = $true - home_config_isolated = $true - # No terminal_result_capture because timed out before assistant response - terminal_result_capture = $false - # include HTTP timeout metadata - http = [ordered]@{ request_start_utc = (Get-Date).ToUniversalTime().ToString('o'); timeout_utc = (Get-Date).AddSeconds(30).ToUniversalTime().ToString('o'); classification = 'request_timeout' } - terminal_event = [ordered]@{ type = 'timeout'; reason = 'request_timeout' } - } - execution_paths = [ordered]@{ logical_working_directory = (Join-Path $runPath 'repo'); logical_home_directory = (Join-Path $runPath 'home') } - } -} - -# Register terminal; orchestration should preserve raw 'timed_out' and record evidence_validation failed -[void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $workerId -ExecutionEvidence $execEvidence) -$ledger = $state.completed[$workerId] -Assert-Equal 'timed_out' $ledger.status 'ledger must preserve raw timed_out status' -$ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null -Assert-Equal 'failed' $ev.status 'evidence_validation should be recorded as failed for a timed_out (no terminal capture)' - -Write-Output 'OPENCODE TIMED-OUT REGRESSION: PASS' \ No newline at end of file +$runJson = [ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = $evalName; candidateSkillName = 'test-skill'; skillName = 'test-skill'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/test-skill'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64); candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("opencode timedout fixture prompt")))).ToLowerInvariant() } +[System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) +New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null +New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null +# results and metadata +$resultsDir = Join-Path $evalDirectory 'results' +New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null +[System.IO.File]::WriteAllText((Join-Path $resultsDir 'with_skill.result.json'), (([ordered]@{ eval_id = 1; configuration = 'with_skill'; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) +[System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = 1; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @([ordered]@{ eval_id = 1; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = [ordered]@{ with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = "$evalName/with_skill/run.json"; execution_result = "$evalName/results/with_skill.execution-result.json"; result = "$evalName/results/with_skill.result.json" } } }) } + +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'opencode'; model = 'opencode/muse-spark-1.2-contributor-free'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900; concurrency = 1 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# accept the single worker +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +[void](Register-DelegationAccepted -State $state -WorkerId $dispatches[0].worker_id -WorkerSessionId ('sess-' + $dispatches[0].worker_id)) + +# craft opencode-style timed_out execution evidence +$workerId = $dispatches[0].worker_id +$execEvidence = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('opencode-session-' + $workerId); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = 1; eval_name = $evalName; configuration = 'with_skill' } + requested = [ordered]@{ model = $profile.model } + input = [ordered]@{ prompt_sha256 = ('a' * 64) } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'opencode-native' + worker_session_id = ('opencode-session-' + $workerId) + observed_model = $profile.model + observed_working_directory = (Join-Path $runPath 'repo') + observed_home = (Join-Path $runPath 'home') + fresh_worker = $true + home_config_isolated = $true + # No terminal_result_capture because timed out before assistant response + terminal_result_capture = $false + # include HTTP timeout metadata + http = [ordered]@{ request_start_utc = (Get-Date).ToUniversalTime().ToString('o'); timeout_utc = (Get-Date).AddSeconds(30).ToUniversalTime().ToString('o'); classification = 'request_timeout' } + terminal_event = [ordered]@{ type = 'timeout'; reason = 'request_timeout' } + } + execution_paths = [ordered]@{ logical_working_directory = (Join-Path $runPath 'repo'); logical_home_directory = (Join-Path $runPath 'home') } + } +} + +# Register terminal; orchestration should preserve raw 'timed_out' and record evidence_validation failed +[void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $workerId -ExecutionEvidence $execEvidence) +$ledger = $state.completed[$workerId] +Assert-Equal 'timed_out' $ledger.status 'ledger must preserve raw timed_out status' +$ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null +Assert-Equal 'failed' $ev.status 'evidence_validation should be recorded as failed for a timed_out (no terminal capture)' + +Write-Output 'OPENCODE TIMED-OUT REGRESSION: PASS' diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index 9b48850..91eff73 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -146,9 +146,11 @@ try { freshContextRequired = $true filesystemIsolationRequired = $true isolatedHomeRequired = $true + gitWorkspace = $false mustNotReadOutsideSandbox = $true fixtureHash = ('a' * 64) skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + candidateInstructionHash = if ($configuration -eq 'with_skill') { ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes("terminal test prompt for $evalName/$configuration")))).ToLowerInvariant() } else { $null } }) $resultFileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } Write-TestJson -Path (Join-Path $evalDirectory (Join-Path 'results' $resultFileName)) -Value ([ordered]@{ @@ -422,6 +424,58 @@ try { $validTerminalEvidence = New-TestNativeTerminalEvidence -Arm $terminalArm -RunData $terminalRunData -WorkerSessionId 'native-terminal-session' Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'valid terminal native-worker evidence is accepted' Assert-True (Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session') 'valid terminal evidence passes the assert gate' + $phase2AnalyzerRunRoot = Join-Path $testRoot 'phase2-analyzer-run' + New-Item -ItemType Directory -Path (Join-Path $phase2AnalyzerRunRoot 'repo'), (Join-Path $phase2AnalyzerRunRoot 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $phase2AnalyzerRunRoot 'prompt.md'), 'deterministic phase2 analyzer prompt', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $phase2AnalyzerRunRoot 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 1 + evalName = 'phase2-analyzer' + candidateSkillName = 'phase2-analyzer' + skillName = $null + mode = 'without_skill' + executionRole = 'phase2_analyzer' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $null + candidateInstructionHash = $null + }) + $phase2AnalyzerRunData = Resolve-RunContract -RunPath (Join-Path $phase2AnalyzerRunRoot 'run.json') + $phase2AnalyzerEvidence = [ordered]@{ + status = 'completed' + session = [ordered]@{ id = 'phase2-analyzer-session'; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = 1; eval_name = 'phase2-analyzer'; configuration = 'without_skill' } + requested = [ordered]@{ model = 'fixture-model' } + input = [ordered]@{ prompt_sha256 = [string]$phase2AnalyzerRunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = 'phase2-analyzer-session' + observed_model = 'fixture-model' + observed_working_directory = [string]$phase2AnalyzerRunData.WorkingDirectoryPath + observed_home = [string]$phase2AnalyzerRunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$phase2AnalyzerRunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + paired_or_package_grading_material_visible = $false + own_arm_grading_material_visible = $true + nested_model_execution = $false + model_execution_count = 1 + } + } + } + Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $phase2AnalyzerEvidence -Run $phase2AnalyzerRunData -RequestedModel 'fixture-model' -ExpectedWorkerSessionId 'phase2-analyzer-session').Valid) 'phase2 analyzer may inspect its own staged grader bundle without violating the paired-arm/grading boundary' # Runner-specific checks and the portable validator must make one terminal # decision. Preserve the exact runner codes while the common validator @@ -489,6 +543,16 @@ try { Assert-True (([string]::Join(',', @($caseState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match [regex]::Escape($ExpectedFailure)) "$Name records $ExpectedFailure" } + $phase2AnalyzerForbidden = Copy-TestObject -Value $phase2AnalyzerEvidence + $phase2AnalyzerForbidden.evidence.delegation.paired_or_package_grading_material_visible = $true + $phase2AnalyzerValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $phase2AnalyzerForbidden -Run $phase2AnalyzerRunData -RequestedModel 'fixture-model' -ExpectedWorkerSessionId 'phase2-analyzer-session' + Assert-True (-not [bool]$phase2AnalyzerValidation.Valid) 'phase2 analyzer still rejects paired/package grading visibility' + Assert-True (([string]::Join(',', @($phase2AnalyzerValidation.Failures))) -match 'paired_arm_and_grading_exclusion') 'phase2 analyzer violation reports paired/package grading failure' + + $projectionEscapeObserved = Copy-TestObject -Value $validTerminalEvidence + Add-Member -InputObject $projectionEscapeObserved.evidence.delegation -MemberType NoteProperty -Name projection_escape_observed -Value $true -Force + Invoke-TerminalEvidenceCase -Name 'projection escape observed' -Evidence $projectionEscapeObserved -ExpectedFailure 'projection_escape_observed' + $modelMismatch = Copy-TestObject -Value $validTerminalEvidence $modelMismatch.evidence.delegation.observed_model = 'different-model' Invoke-TerminalEvidenceCase -Name 'model mismatch' -Evidence $modelMismatch -ExpectedFailure 'requested_model' @@ -586,11 +650,14 @@ try { [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "fixture prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) $skillDirectory = $null $skillHash = $null + $candidateInstructionHash = $null if ($configuration -eq 'with_skill') { $skillDirectory = 'skill/candidate' New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) $skillHash = ('b' * 64) + $promptContent = "fixture prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() } Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run @@ -607,8 +674,10 @@ try { freshContextRequired = $true filesystemIsolationRequired = $true isolatedHomeRequired = $true + gitWorkspace = $false fixtureHash = ('a' * 64) skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash }) $resultDirectory = Join-Path $runDirectory 'results' New-Item -ItemType Directory -Path $resultDirectory -Force | Out-Null diff --git a/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 index 6eb5612..42fa78c 100644 --- a/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 +++ b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 @@ -1,452 +1,456 @@ -<#! -.SYNOPSIS - Deterministic Phase 1 aggregate fail-closed regressions. - -.DESCRIPTION - Exercises the real runner-owned Phase 1 path with mixed terminal outcomes - and with completed raw results whose evidence validation fails. MODEL-FREE. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -cne [string]$Actual) { - throw "ASSERT: $Message (expected '$Expected', got '$Actual')" - } -} - -function Assert-Contains { - param([string]$Text, [string]$Expected, [string]$Message) - if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { - throw "ASSERT: $Message (missing '$Expected')" - } -} - -function Write-TestJson { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) - - New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null - [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Read-TestJson { - param([Parameter(Mandatory = $true)][string]$Path) - - return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 -} - -function Invoke-TestTool { - param( - [Parameter(Mandatory = $true)][string]$Path, - [Parameter(Mandatory = $true)][string[]]$Arguments - ) - +<#! +.SYNOPSIS + Deterministic Phase 1 aggregate fail-closed regressions. + +.DESCRIPTION + Exercises the real runner-owned Phase 1 path with mixed terminal outcomes + and with completed raw results whose evidence validation fails. MODEL-FREE. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -cne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Assert-Contains { + param([string]$Text, [string]$Expected, [string]$Message) + if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { + throw "ASSERT: $Message (missing '$Expected')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-TestTool { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + $output = & pwsh -NoProfile -NonInteractive -File $Path @Arguments 2>&1 - return [pscustomobject]@{ - ExitCode = $LASTEXITCODE - Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - } -} - -function Invoke-ForegroundPhaseOne { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) - - # STDOUT is the machine protocol; STDERR is live observability. Capture them - # separately so relayed heartbeats never contaminate the terminal JSON. - $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') - try { + return [pscustomobject]@{ + ExitCode = $LASTEXITCODE + Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + } +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) + + # STDOUT is the machine protocol; STDERR is live observability. Capture them + # separately so relayed heartbeats never contaminate the terminal JSON. + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') + try { $output = & pwsh -NoProfile -NonInteractive -File $Path -IterationDirectory $IterationDirectory 2>$stderrPath - $exitCode = $LASTEXITCODE - $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - } finally { - Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue - } - $document = $text | ConvertFrom-Json -Depth 100 - return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } -} - -function Assert-ToolFails { - param( - [Parameter(Mandatory = $true)][object]$Invocation, - [Parameter(Mandatory = $true)][string]$Description, - [string]$ExpectedText = '' - ) - - if ([int]$Invocation.ExitCode -eq 0) { - throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" - } - if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { - Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description - } -} - -function New-TestRun { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [Parameter(Mandatory = $true)][int]$EvalId, - [Parameter(Mandatory = $true)][string]$EvalName - ) - - $runDirectory = Join-Path (Join-Path $IterationDirectory $EvalName) 'with_skill' - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $skillDirectory = Join-Path $runDirectory 'skill/test-skill' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $skillDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "phase1 aggregate prompt for $EvalName/with_skill`n", [System.Text.UTF8Encoding]::new($false)) - - $run = [ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $EvalId - evalName = $EvalName + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + } finally { + Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue + } + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } +} + +function Assert-ToolFails { + param( + [Parameter(Mandatory = $true)][object]$Invocation, + [Parameter(Mandatory = $true)][string]$Description, + [string]$ExpectedText = '' + ) + + if ([int]$Invocation.ExitCode -eq 0) { + throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" + } + if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { + Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description + } +} + +function New-TestRun { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$EvalName + ) + + $runDirectory = Join-Path (Join-Path $IterationDirectory $EvalName) 'with_skill' + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $skillDirectory = Join-Path $runDirectory 'skill/test-skill' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $skillDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "phase1 aggregate prompt for $EvalName/with_skill`n", [System.Text.UTF8Encoding]::new($false)) + + $promptBytes = [System.Text.Encoding]::UTF8.GetBytes("phase1 aggregate prompt for $EvalName/with_skill`n") + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData($promptBytes))).ToLowerInvariant() + + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $EvalId + evalName = $EvalName candidateSkillName = 'test-skill' - skillName = 'test-skill' - iteration = 1 - mode = 'with_skill' - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = 'skill/test-skill' - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - gitWorkspace = $false - inputFiles = @() - fixtureHash = ('a' * 64) - skillHash = ('b' * 64) - contract = [ordered]@{ - sandboxRoot = '.' - workingDirectory = 'repo' - homeDirectory = 'home' - mustNotReadOutsideSandbox = $true - mustNotExposeGlobalSkillsOrConfig = $true - } - } - Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run - - return [pscustomobject]@{ - RunDirectory = $runDirectory - HomeDirectory = $homeDirectory - } -} - -function New-ReportFixtureScript { - param([Parameter(Mandatory = $true)][string]$Path) - - $scriptText = @' -[CmdletBinding()] -param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) -$ErrorActionPreference = 'Stop' -foreach ($file in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory $file), "unexpected report artifact: $file`n", [System.Text.UTF8Encoding]::new($false)) -} -'@ - [System.IO.File]::WriteAllText($Path, $scriptText, [System.Text.UTF8Encoding]::new($false)) -} - -function Initialize-PhaseOneFailurePackage { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, - [int[]]$EvidenceFailureEvalIds = @() - ) - - $packageTools = Join-Path $IterationDirectory 'tools/eval-runners' - New-Item -ItemType Directory -Path $packageTools -Force | Out-Null - foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { - Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force - } - $fixtureDirectory = Join-Path $packageTools 'fixture' - New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force - New-ReportFixtureScript -Path (Join-Path $IterationDirectory 'tools/test-report.ps1') - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le 4; $evalId++) { - $evalName = 'phase1-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-metadata/1' - eval_id = $evalId - eval_name = $evalName - prompt = "fixture prompt $evalId" - expected_output = 'fixture output' - assertions = @('deterministic fixture assertion') - }) - - $run = New-TestRun -IterationDirectory $IterationDirectory -EvalId $evalId -EvalName $evalName - $status = if ($StatusesByEvalId.ContainsKey($evalId)) { [string]$StatusesByEvalId[$evalId] } else { 'completed' } - if ($status -ne 'completed') { - [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'terminal-status'), $status, [System.Text.UTF8Encoding]::new($false)) - } - if ($EvidenceFailureEvalIds -contains $evalId) { - [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'evidence-validation-failed'), 'fixture', [System.Text.UTF8Encoding]::new($false)) - } - - $resultsDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $resultsDirectory 'with_skill.result.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).PortableResult - eval_id = $evalId - eval_name = $evalName - configuration = 'with_skill' - execution_status = 'unrun' - grading = @([ordered]@{ text = 'deterministic fixture assertion'; passed = $null; evidence = '' }) - }) - - $manifestEvals.Add([ordered]@{ - eval_id = $evalId - eval_name = $evalName - directory = $evalName - metadata = "$evalName/eval-metadata.json" - runs = [ordered]@{ - with_skill = [ordered]@{ - mode = 'with_skill' - run_manifest = "$evalName/with_skill/run.json" - execution_result = "$evalName/results/with_skill.execution-result.json" - result = "$evalName/results/with_skill.result.json" - } - } - }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools - Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - skill_name = 'phase1-aggregate-fixture' - iteration = 1 - configurations = @('with_skill') + skillName = 'test-skill' + iteration = 1 + mode = 'with_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = 'skill/test-skill' + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = ('b' * 64) + candidateInstructionHash = $candidateInstructionHash + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + + return [pscustomobject]@{ + RunDirectory = $runDirectory + HomeDirectory = $homeDirectory + } +} + +function New-ReportFixtureScript { + param([Parameter(Mandatory = $true)][string]$Path) + + $scriptText = @' +[CmdletBinding()] +param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) +$ErrorActionPreference = 'Stop' +foreach ($file in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory $file), "unexpected report artifact: $file`n", [System.Text.UTF8Encoding]::new($false)) +} +'@ + [System.IO.File]::WriteAllText($Path, $scriptText, [System.Text.UTF8Encoding]::new($false)) +} + +function Initialize-PhaseOneFailurePackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @() + ) + + $packageTools = Join-Path $IterationDirectory 'tools/eval-runners' + New-Item -ItemType Directory -Path $packageTools -Force | Out-Null + foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { + Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force + } + $fixtureDirectory = Join-Path $packageTools 'fixture' + New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + New-ReportFixtureScript -Path (Join-Path $IterationDirectory 'tools/test-report.ps1') + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'phase1-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/1' + eval_id = $evalId + eval_name = $evalName + prompt = "fixture prompt $evalId" + expected_output = 'fixture output' + assertions = @('deterministic fixture assertion') + }) + + $run = New-TestRun -IterationDirectory $IterationDirectory -EvalId $evalId -EvalName $evalName + $status = if ($StatusesByEvalId.ContainsKey($evalId)) { [string]$StatusesByEvalId[$evalId] } else { 'completed' } + if ($status -ne 'completed') { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'terminal-status'), $status, [System.Text.UTF8Encoding]::new($false)) + } + if ($EvidenceFailureEvalIds -contains $evalId) { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'evidence-validation-failed'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + } + + $resultsDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $resultsDirectory 'with_skill.result.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = $evalId + eval_name = $evalName + configuration = 'with_skill' + execution_status = 'unrun' + grading = @([ordered]@{ text = 'deterministic fixture assertion'; passed = $null; evidence = '' }) + }) + + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + run_manifest = "$evalName/with_skill/run.json" + execution_result = "$evalName/results/with_skill.execution-result.json" + result = "$evalName/results/with_skill.result.json" + } + } + }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + skill_name = 'phase1-aggregate-fixture' + iteration = 1 + configurations = @('with_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Phase 1 aggregate fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - grading = 'grading.json' - report = [ordered]@{ tool = 'tools/test-report.ps1' } - evals = @($manifestEvals.ToArray()) - }) - Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 60 - concurrency = 4 - }) - - return [pscustomobject]@{ - IterationDirectory = $IterationDirectory - FanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' - BridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' - FinalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' - LogPath = Join-Path $IterationDirectory 'runner-events.jsonl' - Records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest (Read-TestJson -Path (Join-Path $IterationDirectory 'manifest.json')) | Sort-Object EvalId, Configuration) - } -} - -function Assert-Counts { - param( - [Parameter(Mandatory = $true)][object]$Source, - [Parameter(Mandatory = $true)][hashtable]$Expected, - [Parameter(Mandatory = $true)][string]$MessagePrefix - ) - - foreach ($name in @( - 'expected_count', - 'terminal_count', - 'completed_count', - 'failed_count', - 'timed_out_count', - 'cancelled_count', - 'incompatible_count', - 'evidence_validation_failed_count' - )) { - Assert-Equal $Expected[$name] (Get-JsonProperty -Object $Source -Name $name -Default $null) "$MessagePrefix $name" - } -} - -function Assert-ArmSummaryShape { - param([Parameter(Mandatory = $true)][object]$Summary, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($arm in @($Summary.arms)) { - foreach ($field in @('worker_id', 'eval_id', 'configuration', 'status', 'worker_session_id', 'evidence_validation')) { - Assert-True (Test-JsonProperty -Object $arm -Name $field) "$ScenarioName arm summary contains $field" - } - $evidenceValidation = Get-JsonProperty -Object $arm -Name 'evidence_validation' -Default $null - Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'status') "$ScenarioName arm summary contains evidence_validation.status" - Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'reasons') "$ScenarioName arm summary contains evidence_validation.reasons" - } -} - -function Assert-CanonicalResultsRemainUnrun { - param([Parameter(Mandatory = $true)][object[]]$Records, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($record in $Records) { - $result = Read-TestJson -Path $record.ResultPath - Assert-Equal 'unrun' ([string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '')) "$ScenarioName keeps $($record.ResultRelative) unbridged" - } -} - -function Assert-NoPhaseTwoArtifacts { - param([Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($relative in @('grading.json', 'report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { - Assert-True (-not (Test-Path -LiteralPath (Join-Path $IterationDirectory $relative) -PathType Leaf)) "$ScenarioName does not produce $relative" - } -} - -function Assert-NoRetries { - param([Parameter(Mandatory = $true)][string]$LogPath, [Parameter(Mandatory = $true)][int]$ExpectedArmCount, [Parameter(Mandatory = $true)][string]$ScenarioName) - - $events = @(Get-Content -LiteralPath $LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal ($ExpectedArmCount * 2) $events.Count "$ScenarioName records one preflight and one execute event per arm" - Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'preflight' }).Count "$ScenarioName records one preflight per arm" - Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'execute' }).Count "$ScenarioName records one execute per arm" - - foreach ($evalId in 1..$ExpectedArmCount) { - $executeEvents = @($events | Where-Object { $_.kind -eq 'execute' -and [int]$_.eval_id -eq $evalId -and [string]$_.configuration -eq 'with_skill' }) - Assert-Equal 1 $executeEvents.Count "$ScenarioName does not retry eval $evalId" - } -} - -function Assert-LedgerMatchesFrozenStatuses { - param([Parameter(Mandatory = $true)][object]$FreezeValidation, [Parameter(Mandatory = $true)][string]$ScenarioName) - - foreach ($entry in @($FreezeValidation.Freeze.executions)) { - $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name ([string]$entry.worker_id) -Default $null - Assert-Equal ([string]$entry.terminal_status) ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '')) "$ScenarioName preserves frozen ledger status for $($entry.worker_id)" - } -} - -function Invoke-PhaseOneFailureScenario { - param( - [Parameter(Mandatory = $true)][string]$ScenarioName, - [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, - [int[]]$EvidenceFailureEvalIds = @(), - [Parameter(Mandatory = $true)][hashtable]$ExpectedCounts, - [Parameter(Mandatory = $true)][string[]]$ExpectedFrozenStatuses, - [scriptblock]$AdditionalAssertions = $null - ) - - $iterationDirectory = Join-Path $testRoot $ScenarioName - $package = Initialize-PhaseOneFailurePackage -IterationDirectory $iterationDirectory -StatusesByEvalId $StatusesByEvalId -EvidenceFailureEvalIds $EvidenceFailureEvalIds - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $package.LogPath) - - $fanout = Invoke-ForegroundPhaseOne -Path $package.FanoutScript -IterationDirectory $iterationDirectory - Assert-Equal 2 $fanout.ExitCode "$ScenarioName Phase 1 exits non-zero" - $summary = $fanout.Document - Assert-Equal 'phase1' ([string](Get-JsonProperty -Object $summary -Name 'phase' -Default '')) "$ScenarioName summary identifies Phase 1" - Assert-Equal 'failed' ([string](Get-JsonProperty -Object $summary -Name 'status' -Default '')) "$ScenarioName summary is non-success" - Assert-Counts -Source $summary -Expected $ExpectedCounts -MessagePrefix "$ScenarioName summary" - Assert-ArmSummaryShape -Summary $summary -ScenarioName $ScenarioName - Assert-True (Test-Path -LiteralPath (Join-Path $iterationDirectory 'execution-freeze.json') -PathType Leaf) "$ScenarioName writes execution-freeze.json before failing" - $summaryFreeze = Get-JsonProperty -Object $summary -Name 'execution_freeze' -Default $null - Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'path') "$ScenarioName summary reports execution_freeze.path" - Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'sha256') "$ScenarioName summary reports execution_freeze.sha256" - - $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState - Assert-True (-not [bool]$freezeValidation.PhaseOneSuccess) "$ScenarioName frozen aggregate remains non-success" - Assert-Counts -Source $freezeValidation.Aggregate -Expected $ExpectedCounts -MessagePrefix "$ScenarioName frozen aggregate" - Assert-Equal ([string]::Join(',', $ExpectedFrozenStatuses)) ([string]::Join(',', @($freezeValidation.Freeze.executions | ForEach-Object { [string]$_.terminal_status }))) "$ScenarioName freeze preserves exact raw terminal statuses" - Assert-LedgerMatchesFrozenStatuses -FreezeValidation $freezeValidation -ScenarioName $ScenarioName - Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName - Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName - - $bridge = Invoke-TestTool -Path $package.BridgeScript -Arguments @('-IterationDirectory', $iterationDirectory, '-RequireComplete', '-RequireParallelDispatch', '-RequireNativeDelegation') - Assert-ToolFails -Invocation $bridge -Description "$ScenarioName complete bridge is blocked" -ExpectedText 'completion gate failed' - Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName - - $finalizer = Invoke-TestTool -Path $package.FinalizerScript -Arguments @('-IterationDirectory', $iterationDirectory) - Assert-ToolFails -Invocation $finalizer -Description "$ScenarioName finalizer is blocked" -ExpectedText 'Manifest bridge failed' - Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName - - if ($null -ne $AdditionalAssertions) { - & $AdditionalAssertions $summary $freezeValidation $package - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase1-aggregate-' + [Guid]::NewGuid().ToString('N')) -$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') -try { - Invoke-PhaseOneFailureScenario ` - -ScenarioName 'mixed-terminal' ` - -StatusesByEvalId @{ 1 = 'completed'; 2 = 'timed_out'; 3 = 'failed'; 4 = 'completed' } ` - -ExpectedCounts @{ - expected_count = 4 - terminal_count = 4 - completed_count = 2 - failed_count = 1 - timed_out_count = 1 - cancelled_count = 0 - incompatible_count = 0 - evidence_validation_failed_count = 0 - } ` - -ExpectedFrozenStatuses @('completed', 'timed_out', 'failed', 'completed') ` - -AdditionalAssertions { - param($Summary, $FreezeValidation, $Package) - - Assert-NoRetries -LogPath $Package.LogPath -ExpectedArmCount 4 -ScenarioName 'mixed-terminal' - foreach ($workerId in @('arm-1-with_skill', 'arm-2-with_skill', 'arm-3-with_skill', 'arm-4-with_skill')) { - $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name $workerId -Default $null - $evidenceValidation = Get-JsonProperty -Object $terminal -Name 'evidence_validation' -Default $null - Assert-Equal 'passed' ([string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '')) "mixed-terminal keeps honest evidence_validation for $workerId" - } - } - - Invoke-PhaseOneFailureScenario ` - -ScenarioName 'completed-with-evidence-failure' ` - -StatusesByEvalId @{ 1 = 'completed'; 2 = 'completed'; 3 = 'completed'; 4 = 'completed' } ` - -EvidenceFailureEvalIds @(3) ` - -ExpectedCounts @{ - expected_count = 4 - terminal_count = 4 - completed_count = 4 - failed_count = 0 - timed_out_count = 0 - cancelled_count = 0 - incompatible_count = 0 - evidence_validation_failed_count = 1 - } ` - -ExpectedFrozenStatuses @('completed', 'completed', 'completed', 'completed') ` - -AdditionalAssertions { - param($Summary, $FreezeValidation, $Package) - - $failedTerminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name 'arm-3-with_skill' -Default $null - $failedEvidence = Get-JsonProperty -Object $failedTerminal -Name 'evidence_validation' -Default $null - Assert-Equal 'completed' ([string](Get-JsonProperty -Object $failedTerminal -Name 'status' -Default '')) 'evidence-failure scenario keeps the raw completed status' - Assert-Equal 'failed' ([string](Get-JsonProperty -Object $failedEvidence -Name 'status' -Default '')) 'evidence-failure scenario records failed evidence validation' - Assert-Contains -Text ([string]::Join(', ', @((Get-JsonProperty -Object $failedEvidence -Name 'reasons' -Default @()) | ForEach-Object { [string]$_ }))) -Expected 'prompt_fidelity' -Message 'evidence-failure scenario preserves the validation reason' - } - - Write-Output 'Phase 1 aggregate regressions: PASS' -} finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + grading = 'grading.json' + report = [ordered]@{ tool = 'tools/test-report.ps1' } + evals = @($manifestEvals.ToArray()) + }) + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 4 + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + FanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' + BridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' + FinalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' + LogPath = Join-Path $IterationDirectory 'runner-events.jsonl' + Records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest (Read-TestJson -Path (Join-Path $IterationDirectory 'manifest.json')) | Sort-Object EvalId, Configuration) + } +} + +function Assert-Counts { + param( + [Parameter(Mandatory = $true)][object]$Source, + [Parameter(Mandatory = $true)][hashtable]$Expected, + [Parameter(Mandatory = $true)][string]$MessagePrefix + ) + + foreach ($name in @( + 'expected_count', + 'terminal_count', + 'completed_count', + 'failed_count', + 'timed_out_count', + 'cancelled_count', + 'incompatible_count', + 'evidence_validation_failed_count' + )) { + Assert-Equal $Expected[$name] (Get-JsonProperty -Object $Source -Name $name -Default $null) "$MessagePrefix $name" + } +} + +function Assert-ArmSummaryShape { + param([Parameter(Mandatory = $true)][object]$Summary, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($arm in @($Summary.arms)) { + foreach ($field in @('worker_id', 'eval_id', 'configuration', 'status', 'worker_session_id', 'evidence_validation')) { + Assert-True (Test-JsonProperty -Object $arm -Name $field) "$ScenarioName arm summary contains $field" + } + $evidenceValidation = Get-JsonProperty -Object $arm -Name 'evidence_validation' -Default $null + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'status') "$ScenarioName arm summary contains evidence_validation.status" + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'reasons') "$ScenarioName arm summary contains evidence_validation.reasons" + } +} + +function Assert-CanonicalResultsRemainUnrun { + param([Parameter(Mandatory = $true)][object[]]$Records, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($record in $Records) { + $result = Read-TestJson -Path $record.ResultPath + Assert-Equal 'unrun' ([string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '')) "$ScenarioName keeps $($record.ResultRelative) unbridged" + } +} + +function Assert-NoPhaseTwoArtifacts { + param([Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($relative in @('grading.json', 'report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $IterationDirectory $relative) -PathType Leaf)) "$ScenarioName does not produce $relative" + } +} + +function Assert-NoRetries { + param([Parameter(Mandatory = $true)][string]$LogPath, [Parameter(Mandatory = $true)][int]$ExpectedArmCount, [Parameter(Mandatory = $true)][string]$ScenarioName) + + $events = @(Get-Content -LiteralPath $LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal ($ExpectedArmCount * 2) $events.Count "$ScenarioName records one preflight and one execute event per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'preflight' }).Count "$ScenarioName records one preflight per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'execute' }).Count "$ScenarioName records one execute per arm" + + foreach ($evalId in 1..$ExpectedArmCount) { + $executeEvents = @($events | Where-Object { $_.kind -eq 'execute' -and [int]$_.eval_id -eq $evalId -and [string]$_.configuration -eq 'with_skill' }) + Assert-Equal 1 $executeEvents.Count "$ScenarioName does not retry eval $evalId" + } +} + +function Assert-LedgerMatchesFrozenStatuses { + param([Parameter(Mandatory = $true)][object]$FreezeValidation, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($entry in @($FreezeValidation.Freeze.executions)) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name ([string]$entry.worker_id) -Default $null + Assert-Equal ([string]$entry.terminal_status) ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '')) "$ScenarioName preserves frozen ledger status for $($entry.worker_id)" + } +} + +function Invoke-PhaseOneFailureScenario { + param( + [Parameter(Mandatory = $true)][string]$ScenarioName, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @(), + [Parameter(Mandatory = $true)][hashtable]$ExpectedCounts, + [Parameter(Mandatory = $true)][string[]]$ExpectedFrozenStatuses, + [scriptblock]$AdditionalAssertions = $null + ) + + $iterationDirectory = Join-Path $testRoot $ScenarioName + $package = Initialize-PhaseOneFailurePackage -IterationDirectory $iterationDirectory -StatusesByEvalId $StatusesByEvalId -EvidenceFailureEvalIds $EvidenceFailureEvalIds + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $package.LogPath) + + $fanout = Invoke-ForegroundPhaseOne -Path $package.FanoutScript -IterationDirectory $iterationDirectory + Assert-Equal 2 $fanout.ExitCode "$ScenarioName Phase 1 exits non-zero" + $summary = $fanout.Document + Assert-Equal 'phase1' ([string](Get-JsonProperty -Object $summary -Name 'phase' -Default '')) "$ScenarioName summary identifies Phase 1" + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $summary -Name 'status' -Default '')) "$ScenarioName summary is non-success" + Assert-Counts -Source $summary -Expected $ExpectedCounts -MessagePrefix "$ScenarioName summary" + Assert-ArmSummaryShape -Summary $summary -ScenarioName $ScenarioName + Assert-True (Test-Path -LiteralPath (Join-Path $iterationDirectory 'execution-freeze.json') -PathType Leaf) "$ScenarioName writes execution-freeze.json before failing" + $summaryFreeze = Get-JsonProperty -Object $summary -Name 'execution_freeze' -Default $null + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'path') "$ScenarioName summary reports execution_freeze.path" + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'sha256') "$ScenarioName summary reports execution_freeze.sha256" + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState + Assert-True (-not [bool]$freezeValidation.PhaseOneSuccess) "$ScenarioName frozen aggregate remains non-success" + Assert-Counts -Source $freezeValidation.Aggregate -Expected $ExpectedCounts -MessagePrefix "$ScenarioName frozen aggregate" + Assert-Equal ([string]::Join(',', $ExpectedFrozenStatuses)) ([string]::Join(',', @($freezeValidation.Freeze.executions | ForEach-Object { [string]$_.terminal_status }))) "$ScenarioName freeze preserves exact raw terminal statuses" + Assert-LedgerMatchesFrozenStatuses -FreezeValidation $freezeValidation -ScenarioName $ScenarioName + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + $bridge = Invoke-TestTool -Path $package.BridgeScript -Arguments @('-IterationDirectory', $iterationDirectory, '-RequireComplete', '-RequireParallelDispatch', '-RequireNativeDelegation') + Assert-ToolFails -Invocation $bridge -Description "$ScenarioName complete bridge is blocked" -ExpectedText 'completion gate failed' + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + + $finalizer = Invoke-TestTool -Path $package.FinalizerScript -Arguments @('-IterationDirectory', $iterationDirectory) + Assert-ToolFails -Invocation $finalizer -Description "$ScenarioName finalizer is blocked" -ExpectedText 'Manifest bridge failed' + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + if ($null -ne $AdditionalAssertions) { + & $AdditionalAssertions $summary $freezeValidation $package + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase1-aggregate-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'mixed-terminal' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'timed_out'; 3 = 'failed'; 4 = 'completed' } ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 2 + failed_count = 1 + timed_out_count = 1 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 0 + } ` + -ExpectedFrozenStatuses @('completed', 'timed_out', 'failed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + Assert-NoRetries -LogPath $Package.LogPath -ExpectedArmCount 4 -ScenarioName 'mixed-terminal' + foreach ($workerId in @('arm-1-with_skill', 'arm-2-with_skill', 'arm-3-with_skill', 'arm-4-with_skill')) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name $workerId -Default $null + $evidenceValidation = Get-JsonProperty -Object $terminal -Name 'evidence_validation' -Default $null + Assert-Equal 'passed' ([string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '')) "mixed-terminal keeps honest evidence_validation for $workerId" + } + } + + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'completed-with-evidence-failure' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'completed'; 3 = 'completed'; 4 = 'completed' } ` + -EvidenceFailureEvalIds @(3) ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 4 + failed_count = 0 + timed_out_count = 0 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 1 + } ` + -ExpectedFrozenStatuses @('completed', 'completed', 'completed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + $failedTerminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name 'arm-3-with_skill' -Default $null + $failedEvidence = Get-JsonProperty -Object $failedTerminal -Name 'evidence_validation' -Default $null + Assert-Equal 'completed' ([string](Get-JsonProperty -Object $failedTerminal -Name 'status' -Default '')) 'evidence-failure scenario keeps the raw completed status' + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $failedEvidence -Name 'status' -Default '')) 'evidence-failure scenario records failed evidence validation' + Assert-Contains -Text ([string]::Join(', ', @((Get-JsonProperty -Object $failedEvidence -Name 'reasons' -Default @()) | ForEach-Object { [string]$_ }))) -Expected 'prompt_fidelity' -Message 'evidence-failure scenario preserves the validation reason' + } + + Write-Output 'Phase 1 aggregate regressions: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 index adcb44a..60cc404 100644 --- a/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 +++ b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 @@ -1,244 +1,249 @@ -<#! -.SYNOPSIS - Deterministic foreground runner-owned Phase 1 lifecycle tests. - -.DESCRIPTION - Exercises the restored runner-owned topology: one foreground - invoke-runner-owned-arms.ps1 invocation per iteration. The fixture runner - is model-free and never calls an AI CLI. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -ne [string]$Actual) { - throw "ASSERT: $Message (expected '$Expected', got '$Actual')" - } -} - -function Write-TestJson { - param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) - - New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null - [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Read-TestJson { - param([Parameter(Mandatory = $true)][string]$Path) - - return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 -} - -function Invoke-ForegroundPhaseOne { - param([Parameter(Mandatory = $true)][string]$IterationDirectory) - - $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' - # STDOUT carries the machine protocol; STDERR carries live observability. - # Keep them separate so heartbeats never corrupt the terminal JSON. - $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') - try { - $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>$stderrPath - $exitCode = $LASTEXITCODE - $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - } finally { - Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue - } - $document = $text | ConvertFrom-Json -Depth 100 - return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } -} - -function New-ForegroundPackage { - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [int]$EvalCount = 2, - [int]$Concurrency = 2 - ) - - $tools = Join-Path $IterationDirectory 'tools\eval-runners' - New-Item -ItemType Directory -Path $tools -Force | Out-Null - foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { - Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force - } - $fixtureRunnerDirectory = Join-Path $tools 'fixture' - New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force - Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 30 - concurrency = $Concurrency - }) - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { - $evalName = 'foreground-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) - $runs = [ordered]@{} - foreach ($configuration in @('with_skill', 'without_skill')) { - $runDirectory = Join-Path $evalDirectory $configuration - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $resultDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "foreground prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - $skillDirectory = $null - $skillHash = $null - if ($configuration -eq 'with_skill') { - $skillDirectory = 'skill/candidate' - New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) - $skillHash = ('b' * 64) - } - Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName +<#! +.SYNOPSIS + Deterministic foreground runner-owned Phase 1 lifecycle tests. + +.DESCRIPTION + Exercises the restored runner-owned topology: one foreground + invoke-runner-owned-arms.ps1 invocation per iteration. The fixture runner + is model-free and never calls an AI CLI. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' + # STDOUT carries the machine protocol; STDERR carries live observability. + # Keep them separate so heartbeats never corrupt the terminal JSON. + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('phase1-stderr-' + [Guid]::NewGuid().ToString('N') + '.log') + try { + $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>$stderrPath + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + } finally { + Remove-Item -LiteralPath $stderrPath -Force -ErrorAction SilentlyContinue + } + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Stderr = $stderr; Document = $document } +} + +function New-ForegroundPackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [int]$EvalCount = 2, + [int]$Concurrency = 2 + ) + + $tools = Join-Path $IterationDirectory 'tools\eval-runners' + New-Item -ItemType Directory -Path $tools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $tools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = $Concurrency + }) + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { + $evalName = 'foreground-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $resultDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "foreground prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + $candidateInstructionHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + $promptContent = "foreground prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' - skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - iteration = 1 - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $skillDirectory - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('a' * 64) - skillHash = $skillHash - }) - $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } - Write-TestJson -Path (Join-Path $resultDirectory $resultName) -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' }) - $runs[$configuration] = [ordered]@{ - mode = $configuration - run_manifest = "$evalName/$configuration/run.json" - execution_result = "$evalName/results/$executionName" - result = "$evalName/results/$resultName" - } - } - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $tools - Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - configurations = @('with_skill', 'without_skill') + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash + }) + $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + Write-TestJson -Path (Join-Path $resultDirectory $resultName) -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' }) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $tools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Phase 1 lifecycle fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - evals = @($manifestEvals.ToArray()) - }) - - return [pscustomobject]@{ - IterationDirectory = $IterationDirectory - Tools = $tools - LogPath = Join-Path $IterationDirectory 'fixture-events.jsonl' - } -} - -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-foreground-phase1-' + [Guid]::NewGuid().ToString('N')) -$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') -try { - $success = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'success') -EvalCount 2 -Concurrency 2 - foreach ($obsolete in @('phase1-control-common.ps1', 'control-runner-owned-phase1.ps1', 'supervise-runner-owned-phase1.ps1')) { - Assert-True (-not (Test-Path -LiteralPath (Join-Path $success.Tools $obsolete) -PathType Leaf)) "foreground package does not carry obsolete $obsolete" - } - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $success.LogPath) - $first = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory - Assert-Equal 0 $first.ExitCode 'foreground Phase 1 exits successfully' - Assert-Equal 'phase1' ([string]$first.Document.phase) 'foreground Phase 1 returns the fan-out summary directly' - Assert-Equal 'completed' ([string]$first.Document.status) 'foreground Phase 1 completes' - Assert-Equal 4 ([int]$first.Document.expected_count) 'foreground Phase 1 sees four paired arms' - Assert-Equal 4 ([int]$first.Document.terminal_count) 'foreground Phase 1 registers every arm terminal' - Assert-Equal 4 ([int]$first.Document.execution_count) 'foreground Phase 1 executes every compatible arm' - Assert-True ([int]$first.Document.max_observed_active -gt 1) 'foreground Phase 1 honors requested concurrency when capacity permits' - Assert-True (Test-Path -LiteralPath (Join-Path $success.IterationDirectory 'execution-freeze.json') -PathType Leaf) 'foreground Phase 1 writes execution-freeze.json only after terminal arms' - $freeze = Assert-ExecutionFreeze -IterationDirectory $success.IterationDirectory -RequireOrchestrationState - Assert-True ([bool]$freeze.PhaseOneSuccess) 'foreground Phase 1 freeze validates as successful' - Assert-Equal 4 @($freeze.Freeze.executions).Count 'foreground Phase 1 freeze contains every expected arm' - $events = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 8 $events.Count 'foreground Phase 1 invokes each preflight and execution exactly once' - Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflights every arm' - Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 executes every compatible arm' - $firstExecuteIndex = -1 - $lastPreflightIndex = -1 - for ($index = 0; $index -lt $events.Count; $index++) { - if ($events[$index].kind -eq 'preflight') { $lastPreflightIndex = $index } - if ($events[$index].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $index } - } - Assert-True ($firstExecuteIndex -gt $lastPreflightIndex) 'foreground Phase 1 starts zero executions before all preflights pass' - - $second = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory - Assert-Equal 2 $second.ExitCode 'foreground Phase 1 refuses a second invocation after freeze' - Assert-True ([string]$second.Document.error -match 'already frozen|existing orchestration state') 'foreground Phase 1 reports why rerun is refused' - $eventsAfterSecond = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 4 @($eventsAfterSecond | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 rerun starts zero additional executions' - - $interrupted = Join-Path $testRoot 'interrupted' - Copy-Item -LiteralPath $success.IterationDirectory -Destination $interrupted -Recurse -Force - Remove-Item -LiteralPath (Join-Path $interrupted 'execution-freeze.json') -Force - foreach ($raw in @(Get-ChildItem -LiteralPath $interrupted -Recurse -File -Filter '*.execution-result.json')) { - Remove-Item -LiteralPath $raw.FullName -Force - } - $interruptedLog = Join-Path $interrupted 'new-events.jsonl' - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $interruptedLog) - $interruptedResult = Invoke-ForegroundPhaseOne -IterationDirectory $interrupted - Assert-Equal 2 $interruptedResult.ExitCode 'foreground Phase 1 fails closed on interrupted state without a freeze' - Assert-True ([string]$interruptedResult.Document.error -match 'refuses to replace an existing orchestration state') 'foreground Phase 1 does not adopt or rerun incomplete state' - Assert-True (-not (Test-Path -LiteralPath $interruptedLog -PathType Leaf)) 'foreground Phase 1 interrupted-state refusal starts zero executions' - - $preflightGate = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'preflight-gate') -EvalCount 2 -Concurrency 2 - [System.IO.File]::WriteAllText((Join-Path $preflightGate.IterationDirectory 'foreground-eval-02\with_skill\home\preflight-incompatible'), 'fixture', [System.Text.UTF8Encoding]::new($false)) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $preflightGate.LogPath) - $gate = Invoke-ForegroundPhaseOne -IterationDirectory $preflightGate.IterationDirectory - Assert-Equal 2 $gate.ExitCode 'foreground Phase 1 exits non-zero for incompatible preflight' - Assert-Equal 'preflight' ([string]$gate.Document.phase) 'foreground Phase 1 reports preflight phase failure' - Assert-Equal 'preflight_incompatible' ([string]$gate.Document.status) 'foreground Phase 1 reports incompatible preflight' - Assert-Equal 4 ([int]$gate.Document.preflight_count) 'foreground Phase 1 still probes every arm' - Assert-True (-not [bool]$gate.Document.execution_started) 'foreground Phase 1 starts zero executions when any preflight is incompatible' - $gateEvents = @(Get-Content -LiteralPath $preflightGate.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 4 @($gateEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflight gate records every preflight' - Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 preflight gate records zero executions' - Assert-True (-not (Test-Path -LiteralPath (Join-Path $preflightGate.IterationDirectory 'execution-freeze.json') -PathType Leaf)) 'foreground Phase 1 writes no freeze before a failed preflight gate' - - $fanoutText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($fanoutText -notmatch '(?i)Job Object|breakaway|process ancestry|supervisor independence|phase1-control-common|AGENTIC_PHASE1_SUPERVISOR_ID') 'foreground Phase 1 has no Windows host-security/durable-detachment requirement' - - Write-Output 'Runner-owned foreground Phase 1 lifecycle: PASS' -} finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals.ToArray()) + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + Tools = $tools + LogPath = Join-Path $IterationDirectory 'fixture-events.jsonl' + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-foreground-phase1-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + $success = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'success') -EvalCount 2 -Concurrency 2 + foreach ($obsolete in @('phase1-control-common.ps1', 'control-runner-owned-phase1.ps1', 'supervise-runner-owned-phase1.ps1')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $success.Tools $obsolete) -PathType Leaf)) "foreground package does not carry obsolete $obsolete" + } + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $success.LogPath) + $first = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 0 $first.ExitCode 'foreground Phase 1 exits successfully' + Assert-Equal 'phase1' ([string]$first.Document.phase) 'foreground Phase 1 returns the fan-out summary directly' + Assert-Equal 'completed' ([string]$first.Document.status) 'foreground Phase 1 completes' + Assert-Equal 4 ([int]$first.Document.expected_count) 'foreground Phase 1 sees four paired arms' + Assert-Equal 4 ([int]$first.Document.terminal_count) 'foreground Phase 1 registers every arm terminal' + Assert-Equal 4 ([int]$first.Document.execution_count) 'foreground Phase 1 executes every compatible arm' + Assert-True ([int]$first.Document.max_observed_active -gt 1) 'foreground Phase 1 honors requested concurrency when capacity permits' + Assert-True (Test-Path -LiteralPath (Join-Path $success.IterationDirectory 'execution-freeze.json') -PathType Leaf) 'foreground Phase 1 writes execution-freeze.json only after terminal arms' + $freeze = Assert-ExecutionFreeze -IterationDirectory $success.IterationDirectory -RequireOrchestrationState + Assert-True ([bool]$freeze.PhaseOneSuccess) 'foreground Phase 1 freeze validates as successful' + Assert-Equal 4 @($freeze.Freeze.executions).Count 'foreground Phase 1 freeze contains every expected arm' + $events = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 8 $events.Count 'foreground Phase 1 invokes each preflight and execution exactly once' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflights every arm' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 executes every compatible arm' + $firstExecuteIndex = -1 + $lastPreflightIndex = -1 + for ($index = 0; $index -lt $events.Count; $index++) { + if ($events[$index].kind -eq 'preflight') { $lastPreflightIndex = $index } + if ($events[$index].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $index } + } + Assert-True ($firstExecuteIndex -gt $lastPreflightIndex) 'foreground Phase 1 starts zero executions before all preflights pass' + + $second = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 2 $second.ExitCode 'foreground Phase 1 refuses a second invocation after freeze' + Assert-True ([string]$second.Document.error -match 'already frozen|existing orchestration state') 'foreground Phase 1 reports why rerun is refused' + $eventsAfterSecond = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($eventsAfterSecond | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 rerun starts zero additional executions' + + $interrupted = Join-Path $testRoot 'interrupted' + Copy-Item -LiteralPath $success.IterationDirectory -Destination $interrupted -Recurse -Force + Remove-Item -LiteralPath (Join-Path $interrupted 'execution-freeze.json') -Force + foreach ($raw in @(Get-ChildItem -LiteralPath $interrupted -Recurse -File -Filter '*.execution-result.json')) { + Remove-Item -LiteralPath $raw.FullName -Force + } + $interruptedLog = Join-Path $interrupted 'new-events.jsonl' + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $interruptedLog) + $interruptedResult = Invoke-ForegroundPhaseOne -IterationDirectory $interrupted + Assert-Equal 2 $interruptedResult.ExitCode 'foreground Phase 1 fails closed on interrupted state without a freeze' + Assert-True ([string]$interruptedResult.Document.error -match 'refuses to replace an existing orchestration state') 'foreground Phase 1 does not adopt or rerun incomplete state' + Assert-True (-not (Test-Path -LiteralPath $interruptedLog -PathType Leaf)) 'foreground Phase 1 interrupted-state refusal starts zero executions' + + $preflightGate = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'preflight-gate') -EvalCount 2 -Concurrency 2 + [System.IO.File]::WriteAllText((Join-Path $preflightGate.IterationDirectory 'foreground-eval-02\with_skill\home\preflight-incompatible'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $preflightGate.LogPath) + $gate = Invoke-ForegroundPhaseOne -IterationDirectory $preflightGate.IterationDirectory + Assert-Equal 2 $gate.ExitCode 'foreground Phase 1 exits non-zero for incompatible preflight' + Assert-Equal 'preflight' ([string]$gate.Document.phase) 'foreground Phase 1 reports preflight phase failure' + Assert-Equal 'preflight_incompatible' ([string]$gate.Document.status) 'foreground Phase 1 reports incompatible preflight' + Assert-Equal 4 ([int]$gate.Document.preflight_count) 'foreground Phase 1 still probes every arm' + Assert-True (-not [bool]$gate.Document.execution_started) 'foreground Phase 1 starts zero executions when any preflight is incompatible' + $gateEvents = @(Get-Content -LiteralPath $preflightGate.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($gateEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflight gate records every preflight' + Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 preflight gate records zero executions' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $preflightGate.IterationDirectory 'execution-freeze.json') -PathType Leaf)) 'foreground Phase 1 writes no freeze before a failed preflight gate' + + $fanoutText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($fanoutText -notmatch '(?i)Job Object|breakaway|process ancestry|supervisor independence|phase1-control-common|AGENTIC_PHASE1_SUPERVISOR_ID') 'foreground Phase 1 has no Windows host-security/durable-detachment requirement' + + Write-Output 'Runner-owned foreground Phase 1 lifecycle: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 7eaa252..2483b35 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -125,6 +125,12 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldHome = $env:HOME +$recordedOldUserProfile = $env:USERPROFILE +$recordedOldAppData = $env:APPDATA +$recordedOldLocalAppData = $env:LOCALAPPDATA +$recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME +$recordedOldTmpDir = $env:TMPDIR $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -203,7 +209,7 @@ if ($arguments -contains '--help' -and -not ($harness -eq 'codex' -and $argument 'copilot' { if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } - else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } + else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --excluded-tools --available-tools --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } } default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } } @@ -212,6 +218,11 @@ if ($arguments -contains '--help' -and -not ($harness -eq 'codex' -and $argument Write-Output $help exit 0 } +if ($harness -eq 'copilot' -and ($arguments -contains 'skill') -and ($arguments -contains 'list')) { + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'skill_list_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output '[{"name":"customize-cloud-agent","source":"builtin","enabled":true},{"name":"github-pr-media","source":"builtin","enabled":true}]' + exit 0 +} $continuationFlag = $null foreach ($candidate in @('--resume', '--session-id', '--session')) { if ($arguments -contains $candidate -or @($arguments | Where-Object { [string]$_ -like ($candidate + '=*') }).Count -gt 0) { @@ -648,7 +659,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { if ($null -eq $threadStart) { $record.rpc_methods = @($initialize.method, $initialized.method, $skillsList.method) $record.skills_list_params = $skillsList.params - $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -eq 'shell_environment_policy.inherit=none' }) + $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -like 'shell_environment_policy.*' }) [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) exit 0 } @@ -732,7 +743,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { $record.auth_only_home = [bool]$record.parent_auth_file_visible -and -not [bool]$record.parent_config_file_visible -and -not [bool]$record.parent_skills_directory_visible -and -not [bool]$record.parent_agents_directory_visible -and -not [bool]$record.parent_sessions_directory_visible -and -not [bool]$record.parent_memories_directory_visible -and -not [bool]$record.parent_plugins_directory_visible -and -not [bool]$record.parent_mcp_configuration_visible -and -not [bool]$record.parent_agents_file_visible $record.rpc_methods = @($initialize.method, $initialized.method, $skillsList.method, $threadStart.method, $turnStart.method, 'thread/read') $record.skills_list_params = $skillsList.params - $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -eq 'shell_environment_policy.inherit=none' }) + $record.native_skill_config_args = @($arguments | Where-Object { [string]$_ -like 'skills.*' -or [string]$_ -like 'shell_environment_policy.*' }) $record.thread_params = $threadStart.params $record.turn_params = $turnStart.params [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -824,7 +835,7 @@ if ($arguments -contains '--help') { 'copilot' { if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } - else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } + else { '--prompt --output-format --model --allow-all --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --excluded-tools --available-tools --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all-paths --allow-all-urls' } } default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } } @@ -833,6 +844,11 @@ if ($arguments -contains '--help') { Write-Output $help exit 0 } +if ($harness -eq 'copilot' -and ($arguments -contains 'skill') -and ($arguments -contains 'list')) { + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'skill_list_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output '[{"name":"customize-cloud-agent","source":"builtin","enabled":true},{"name":"github-pr-media","source":"builtin","enabled":true}]' + exit 0 +} $stdinMemory = [IO.MemoryStream]::new() [Console]::OpenStandardInput().CopyTo($stdinMemory) $stdinBytes = $stdinMemory.ToArray() @@ -960,9 +976,21 @@ if ($harness -eq 'codex') { [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.cmd"), "@echo off`r`npwsh -NoProfile -NonInteractive -File ""%~dp0$harness.ps1"" %*`r`n", [System.Text.UTF8Encoding]::new($false)) } + $fakeGitBin = Join-Path $recordedRoot 'git-bin' + New-Item -ItemType Directory -Path $fakeGitBin -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fakeGitBin 'git.cmd'), "@echo off`r`necho {""args"":""%*"",""path"":""%PATH%""}>> ""%CD%\git-probe-log.jsonl""`r`necho git version recorded.fixture`r`n", [System.Text.UTF8Encoding]::new($false)) + $fakeGitSh = Join-Path $fakeGitBin 'git' + [System.IO.File]::WriteAllText($fakeGitSh, "#!/usr/bin/env sh`nprintf '{""args"":""%s"",""path"":""%s""}\n' ""`$*"" ""`$PATH"" >> ""`$PWD/git-probe-log.jsonl""`nprintf 'git version recorded.fixture\n'`n", [System.Text.UTF8Encoding]::new($false)) + if (-not $IsWindows) { & chmod +x $fakeGitSh } $fakeGh = @' [CmdletBinding()] param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$logPath = Join-Path (Split-Path -Parent $MyInvocation.MyCommand.Path) 'gh-auth-token-probe.jsonl' +$record = [ordered]@{ + args = @($RemainingArguments | ForEach-Object { [string]$_ }) + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') +} +[System.IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { @@ -976,7 +1004,9 @@ if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and exit 2 '@ [System.IO.File]::WriteAllText((Join-Path $fakeBin 'gh.ps1'), $fakeGh, [System.Text.UTF8Encoding]::new($false)) - $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" + $hostOnlyPath = Join-Path $recordedRoot 'host-only-bin' + New-Item -ItemType Directory -Path $hostOnlyPath -Force | Out-Null + $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$fakeGitBin$([System.IO.Path]::PathSeparator)$hostOnlyPath$([System.IO.Path]::PathSeparator)$recordedOldPath" $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' @@ -987,6 +1017,7 @@ exit 2 New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) $env:GH_CONFIG_DIR = $recordedGhConfig + $ghAuthProbeLogPath = Join-Path $fakeBin 'gh-auth-token-probe.jsonl' $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) @@ -1017,6 +1048,36 @@ exit 2 $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' + $gitWorkspaceIteration = Join-Path $recordedRoot 'iteration-git-workspace' + New-Item -ItemType Directory -Path $gitWorkspaceIteration -Force | Out-Null + $gitWorkspaceRun = New-TestRun -IterationDirectory $gitWorkspaceIteration -Configuration with_skill -EvalName 'codex-git-workspace' + $gitWorkspaceRunJson = Read-RunnerJson -Path $gitWorkspaceRun.Path + $gitWorkspaceRunJson.gitWorkspace = $true + Write-TestJson -Path $gitWorkspaceRun.Path -Value $gitWorkspaceRunJson + $gitWorkspacePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $gitWorkspaceRun.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'compatible' $gitWorkspacePreflight.status 'Codex gitWorkspace preflight passes when git resolves through the sanitized child PATH' + Assert-Equal 1 @($gitWorkspacePreflight.checks | Where-Object { $_.name -eq 'git_workspace_tool_path' -and $_.status -eq 'passed' }).Count 'Codex gitWorkspace preflight records a passed git probe' + $gitProbeLog = Join-Path $gitWorkspaceRun.Root 'repo\git-probe-log.jsonl' + Assert-True (Test-Path -LiteralPath $gitProbeLog -PathType Leaf) 'Codex gitWorkspace preflight runs git --version in the sanitized environment' + $gitProbeText = Get-Content -LiteralPath $gitProbeLog -Raw + Assert-True ([string]$gitProbeText -match [regex]::Escape('--version')) 'Codex gitWorkspace preflight probes git --version' + Assert-True ([string]$gitProbeText -match [regex]::Escape($fakeGitBin)) 'Codex gitWorkspace probe PATH includes the resolved git directory' + Assert-True ([string]$gitProbeText -notmatch [regex]::Escape($hostOnlyPath)) 'Codex gitWorkspace probe PATH excludes arbitrary host-only directories' + $gitWorkspaceShellPath = @($gitWorkspacePreflight.checks | Where-Object { $_.name -eq 'git_workspace_tool_path' } | Select-Object -First 1).detail + Assert-True ([string]$gitWorkspaceShellPath -match [regex]::Escape($fakeGitBin)) 'Codex gitWorkspace preflight reports the git directory in the sanitized PATH' + $pathWithoutGit = "$fakeBin$([System.IO.Path]::PathSeparator)$(Split-Path -Parent (Get-Command pwsh).Source)" + $pathWithGit = $env:PATH + try { + $env:PATH = $pathWithoutGit + $missingGitExecution = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $gitWorkspaceRun.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $missingGitExecution.status 'Codex gitWorkspace execution fails closed when git cannot resolve before model execution' + Assert-Equal 'preflight_incompatible' $missingGitExecution.final_response.reason 'Codex missing-git execution stops at preflight' + $missingGitLog = Join-Path $gitWorkspaceRun.Root 'repo\codex-fake-cli-log.jsonl' + $missingGitRecords = if (Test-Path -LiteralPath $missingGitLog -PathType Leaf) { @(Get-Content -LiteralPath $missingGitLog | ForEach-Object { $_ | ConvertFrom-Json }) } else { @() } + Assert-Equal 0 @($missingGitRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -or @((Get-JsonProperty -Object $_ -Name 'rpc_methods' -Default @()) | Where-Object { [string]$_ -eq 'turn/start' }).Count -gt 0 }).Count 'Codex missing-git preflight starts zero model execution processes' + } finally { + $env:PATH = $pathWithGit + } foreach ($fixtureName in @( 'copilot-scripted-turn-1-events.jsonl', 'copilot-scripted-turn-2-events.jsonl', @@ -1223,6 +1284,11 @@ exit 2 Assert-True ($args -contains '--ask-for-approval') 'Codex uses explicit approval policy' Assert-True ($args -contains 'never') 'Codex approval policy is never' Assert-True ($args -contains '--strict-config') 'Codex CLI uses strict config parsing for session controls' + Assert-True (@($args | Where-Object { $_ -eq 'shell_environment_policy.inherit=none' }).Count -eq 1) 'Codex CLI disables child shell environment inheritance' + $codexShellPathArg = @($args | Where-Object { [string]$_ -like 'shell_environment_policy.set.PATH=*' } | Select-Object -First 1) + Assert-Equal 1 $codexShellPathArg.Count 'Codex CLI sets a sanitized child shell PATH' + Assert-True ([string]$codexShellPathArg[0] -notmatch [regex]::Escape($hostOnlyPath)) 'Codex CLI child shell PATH excludes arbitrary host-only directories' + Assert-True ([string]$codexShellPathArg[0] -notmatch [regex]::Escape($fakeGitBin)) 'Codex CLI child shell PATH does not include git for non-git workspaces' Assert-True ($args -contains '--sandbox' -and $args -contains 'danger-full-access') 'Codex grants full operational sandbox permission' Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' Assert-True (@($args | Where-Object { $_ -eq 'skills.include_instructions=false' }).Count -eq 1) 'Codex CLI disables native skill catalog injection at session scope' @@ -1324,8 +1390,8 @@ exit 2 Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is projection-local' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the projected isolated home' Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' @@ -1966,62 +2032,189 @@ exit 2 Assert-True (-not [bool]$subscriptionRecord.unrelated_present) 'Codex app-server parent excludes unrelated inherited environment variables' Assert-True (-not [bool]$subscriptionRecord.worker_auth_file_visible) 'Codex app-server worker fixture does not receive auth.json' Assert-True (@($subscriptionRecord.args) -contains 'shell_environment_policy.inherit=none') 'Codex app-server disables child shell environment inheritance' + $subscriptionShellPathArg = @($subscriptionRecord.args | Where-Object { [string]$_ -like 'shell_environment_policy.set.PATH=*' } | Select-Object -First 1) + Assert-Equal 1 $subscriptionShellPathArg.Count 'Codex app-server sets a sanitized child shell PATH' + Assert-True ([string]$subscriptionShellPathArg[0] -notmatch [regex]::Escape($hostOnlyPath)) 'Codex app-server child shell PATH excludes arbitrary host-only directories' + Assert-True ([string]$subscriptionShellPathArg[0] -notmatch [regex]::Escape($fakeGitBin)) 'Codex app-server child shell PATH does not include git for non-git workspaces' $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:CODEX_HOME = $recordedOldCodexHome } if ($Suite -in @('All', 'Copilot')) { - # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and - # no-auth fixtures are all deterministic and contain no credential values. + # GitHub Copilot authentication continuity and failure semantics: + # deterministic fixtures only, no live credentials. $env:COPILOT_GITHUB_TOKEN = $null $env:GH_TOKEN = $null $env:GITHUB_TOKEN = $null $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + New-Item -ItemType Directory -Path $missingGhConfig -Force | Out-Null + $copilotLogPath = Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl' - # The fixture marker is fake-CLI input only; it models a positive OS - # keychain lookup without naming or reading a real credential-store file. - $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' - New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + # Fixture marker only: models a keychain-only host with no resolvable + # non-interactive token source. + $copilotKeychainHost = Join-Path $recordedRoot 'copilot-keychain-host' + New-Item -ItemType Directory -Path $copilotKeychainHost -Force | Out-Null New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) - $env:COPILOT_HOME = $copilotKeychainHome + $env:COPILOT_HOME = $copilotKeychainHost $env:GH_CONFIG_DIR = $missingGhConfig + $nativeExecutionsBeforeKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsBeforeKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' - Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' - Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + Assert-Equal 'incompatible' $copilotKeychainPreflight.status 'Copilot keychain-only auth is rejected before Phase 1 execution' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot keychain-only preflight fails authentication deterministically' + Assert-Equal 'copilot_os_keychain_unverified' $copilotKeychainPreflight.protocol_observations.authentication.source 'Copilot keychain-only source is diagnostic only' + Assert-True (-not [bool]$copilotKeychainPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot keychain-only source is not accepted as non-interactive readiness' + Assert-True (@($copilotKeychainPreflight.reasons | Where-Object { $_ -match 'supported non-interactive Copilot authentication source' }).Count -eq 1) 'Copilot keychain-only preflight reports the missing non-interactive source' $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' - $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'copilot_authentication_source' -Default '') -eq 'os_keychain' }) - Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Assert-Equal 'incompatible' $copilotKeychainResult.status 'Copilot execute fails closed when preflight cannot resolve non-interactive auth' + Assert-Equal 'incompatible' ([string]$copilotKeychainResult.evidence.preflight.status) 'Copilot execute returns preflight evidence for keychain-only rejection' + $nativeExecutionsAfterKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsAfterKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } + Assert-Equal $nativeExecutionsBeforeKeychain.Count $nativeExecutionsAfterKeychain.Count 'Copilot keychain-only rejection occurs before model execution' Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + # Fresh-context continuity regression: the first GH config candidate exists + # but is not authenticated; fallback resolves a second supported host + # candidate without exposing auth files or token values to the run. + $copilotFreshBoundaryRoot = Join-Path $recordedRoot 'copilot-fresh-boundary' + $copilotFreshHome = Join-Path $copilotFreshBoundaryRoot 'home' + $copilotFreshXdg = Join-Path $copilotFreshBoundaryRoot 'xdg' + $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' + $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' + $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' + $copilotFreshHostGhConfig = if ($IsWindows) { Join-Path $copilotFreshHostAppData 'GitHub CLI' } else { Join-Path $copilotFreshXdg 'gh' } + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, $copilotFreshHostGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotFreshHome + $env:USERPROFILE = $copilotFreshHome + $env:APPDATA = $copilotFreshHostAppData + $env:LOCALAPPDATA = $copilotFreshLocalAppData + $env:XDG_CONFIG_HOME = $copilotFreshXdg + $env:COPILOT_HOME = (Join-Path $copilotFreshBoundaryRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $null + $copilotFreshBoundaryPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotFreshBoundaryPreflight.status 'Copilot preflight resolves non-interactive auth across fresh-context candidate fallback' + Assert-True (@($copilotFreshBoundaryPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot fresh-context preflight proves authentication readiness' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryPreflight.protocol_observations.authentication.source 'Copilot fresh-context source resolves through GitHub CLI fallback' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot fresh-context source is accepted as non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.github_cli_config_candidate_used) 'Copilot fresh-context preflight records GH config candidate fallback usage' + $copilotFreshBoundaryResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotFreshBoundaryResult.status 'Copilot fresh-context fallback executes after preflight authentication continuity' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryResult.evidence.credential.source 'Copilot fresh-context execution records GitHub CLI fallback source' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.noninteractive_ready) 'Copilot fresh-context execution records non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.github_cli_token_resolved) 'Copilot fresh-context execution resolves a trusted GitHub CLI token' + $freshBoundaryExecutionRecords = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $freshBoundaryExecution = $freshBoundaryExecutionRecords[$freshBoundaryExecutionRecords.Count - 1] + Assert-Equal 'explicit_environment' $freshBoundaryExecution.copilot_authentication_source 'Copilot fresh-context worker receives only protected env-token auth' + Assert-Equal 1 @($freshBoundaryExecution.copilot_auth_names_present).Count 'Copilot fresh-context worker receives one token variable' + Assert-True (@($freshBoundaryExecution.copilot_auth_names_present) -contains 'GH_TOKEN') 'Copilot fresh-context worker receives only GH_TOKEN from trusted fallback' + Assert-True ([string]::IsNullOrWhiteSpace([string]$freshBoundaryExecution.gh_config_dir)) 'Copilot fresh-context worker does not receive GH_CONFIG_DIR' + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $copilotGhFallbackHome $env:GH_CONFIG_DIR = $copilotGhConfig + $ghFallbackMatchesBefore = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }).Count $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + Assert-Equal 'github_cli_token' $copilotGhPreflight.protocol_observations.authentication.source 'Copilot preflight classifies explicit GH config fallback source' + Assert-True ([bool]$copilotGhPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot preflight classifies GH config fallback as non-interactive readiness' $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True ([bool]$copilotGhResult.evidence.credential.noninteractive_ready) 'Copilot GitHub CLI fallback result records non-interactive readiness' Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) - Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' + Assert-True (($copilotGhResult | ConvertTo-Json -Depth 100) -notmatch 'recorded-gh-fallback-token-not-logged') 'Copilot GitHub CLI fallback result never exposes token values' + + # Explicit GH_CONFIG_DIR is authoritative. If it cannot resolve auth, do not + # consult discovered/ambient configurations even when they are valid. + $copilotExplicitMissingConfig = Join-Path $recordedRoot 'copilot-explicit-missing-gh-config' + if (Test-Path -LiteralPath $copilotExplicitMissingConfig) { + Remove-Item -LiteralPath $copilotExplicitMissingConfig -Recurse -Force + } + $copilotExplicitMissingRoot = Join-Path $recordedRoot 'copilot-explicit-missing' + $copilotExplicitMissingHome = Join-Path $copilotExplicitMissingRoot 'home' + $copilotExplicitMissingXdg = Join-Path $copilotExplicitMissingRoot 'xdg' + $copilotExplicitMissingAppData = Join-Path $copilotExplicitMissingRoot 'host-appdata' + $copilotExplicitMissingLocalAppData = Join-Path $copilotExplicitMissingRoot 'localappdata' + $copilotExplicitMissingFallbackConfig = if ($IsWindows) { Join-Path $copilotExplicitMissingAppData 'GitHub CLI' } else { Join-Path $copilotExplicitMissingXdg 'gh' } + New-Item -ItemType Directory -Path $copilotExplicitMissingHome, $copilotExplicitMissingXdg, $copilotExplicitMissingAppData, $copilotExplicitMissingLocalAppData, $copilotExplicitMissingFallbackConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotExplicitMissingFallbackConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotExplicitMissingHome + $env:USERPROFILE = $copilotExplicitMissingHome + $env:APPDATA = $copilotExplicitMissingAppData + $env:LOCALAPPDATA = $copilotExplicitMissingLocalAppData + $env:XDG_CONFIG_HOME = $copilotExplicitMissingXdg + $env:COPILOT_HOME = (Join-Path $copilotExplicitMissingRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $copilotExplicitMissingConfig + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { Remove-Item -LiteralPath $ghAuthProbeLogPath -Force } + $nativeExecutionsBeforeExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $copilotExplicitMissingPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingPreflight.status 'Copilot explicit GH_CONFIG_DIR without a valid token fails closed' + Assert-True (@($copilotExplicitMissingPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure is detected in preflight' + Assert-True ([bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.explicit_gh_config_dir_provided) 'Copilot preflight records that GH_CONFIG_DIR was explicitly provided' + Assert-True (-not [bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot explicit GH_CONFIG_DIR failure does not report non-interactive readiness' + Assert-True (@($copilotExplicitMissingPreflight.reasons | Where-Object { $_ -match 'GH_CONFIG_DIR was explicitly provided' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure reports selected-identity resolution failure' + $copilotExplicitMissingResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingResult.status 'Copilot explicit GH_CONFIG_DIR failure remains terminal in execute' + Assert-Equal 'incompatible' ([string]$copilotExplicitMissingResult.evidence.preflight.status) 'Copilot explicit GH_CONFIG_DIR execute result preserves incompatible preflight evidence' + $nativeExecutionsAfterExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeExplicitMissing.Count $nativeExecutionsAfterExplicitMissing.Count 'Copilot explicit GH_CONFIG_DIR rejection occurs before model execution' + $ghAuthProbeRecords = @() + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { + $ghAuthProbeRecords = @(Get-Content -LiteralPath $ghAuthProbeLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + $ghAuthTokenProbes = @($ghAuthProbeRecords | Where-Object { @($_.args).Count -eq 2 -and $_.args[0] -eq 'auth' -and $_.args[1] -eq 'token' }) + Assert-Equal 2 $ghAuthTokenProbes.Count 'Copilot explicit GH_CONFIG_DIR executes exactly one GitHub CLI token probe per preflight invocation' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -ne $copilotExplicitMissingConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure never probes another GitHub CLI configuration' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -eq $copilotExplicitMissingFallbackConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure does not consult a valid discovered fallback configuration' + $explicitFailureEvidenceText = ($copilotExplicitMissingPreflight | ConvertTo-Json -Depth 100) + ($copilotExplicitMissingResult | ConvertTo-Json -Depth 100) + Assert-True ($explicitFailureEvidenceText -notmatch 'recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot explicit GH_CONFIG_DIR failure never exposes token values' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $copilotNoAuthRoot = Join-Path $recordedRoot 'copilot-no-auth-roots' + New-Item -ItemType Directory -Path (Join-Path $copilotNoAuthRoot 'home'), (Join-Path $copilotNoAuthRoot 'appdata'), (Join-Path $copilotNoAuthRoot 'localappdata'), (Join-Path $copilotNoAuthRoot 'xdg') -Force | Out-Null + $env:HOME = Join-Path $copilotNoAuthRoot 'home' + $env:USERPROFILE = Join-Path $copilotNoAuthRoot 'home' + $env:APPDATA = Join-Path $copilotNoAuthRoot 'appdata' + $env:LOCALAPPDATA = Join-Path $copilotNoAuthRoot 'localappdata' + $env:XDG_CONFIG_HOME = Join-Path $copilotNoAuthRoot 'xdg' + $nativeExecutionsBeforeNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) $env:COPILOT_HOME = $copilotNoAuthHome $env:GH_CONFIG_DIR = $missingGhConfig $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' - Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + Assert-Equal 'incompatible' $copilotNoAuthPreflight.status 'Copilot preflight fails closed when no supported non-interactive auth source is available' + Assert-True (@($copilotNoAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot no-auth preflight records an authentication failure check' + Assert-Equal 'copilot_os_keychain_unverified' $copilotNoAuthPreflight.protocol_observations.authentication.source 'Copilot no-auth preflight reports diagnostic auth source identity' + Assert-True (-not [bool]$copilotNoAuthPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot no-auth preflight reports non-interactive auth readiness as false' $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' - Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' - Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + Assert-Equal 'incompatible' $copilotNoAuthResult.status 'Copilot no-auth execution fails before Phase 1 model execution' + Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' + $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $recordedOldCopilotHome } Write-Output "Real runner deterministic adapter conformance ($Suite): PASS" @@ -2036,6 +2229,12 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome + $env:TMPDIR = $recordedOldTmpDir $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } @@ -2053,6 +2252,17 @@ function Assert-Throws { if (-not $thrown) { throw "ASSERT: $Message" } } +function Assert-ObservedPathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath, + [Parameter(Mandatory = $true)][bool]$Expected, + [Parameter(Mandatory = $true)][string]$Message + ) + + Assert-Equal $Expected ([bool](Test-ObservedPathInside -BasePath $BasePath -CandidatePath $CandidatePath)) $Message +} + function Write-TestJson { param([string]$Path, [object]$Value) New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null @@ -2106,19 +2316,27 @@ function New-TestRun { [System.IO.File]::WriteAllText((Join-Path $repo 'AGENTS.md'), '# repo-owned-agent-instruction', [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $repo '.github\copilot-instructions.md'), '# repo-owned-copilot-instruction', [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $repo 'opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) - $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀. I’m testing Microsoft’s guidance. ÆØÅ`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) - [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'expected-prompt-sha256.txt'), (Get-Sha256HexFromFile -Path (Join-Path $runRoot 'prompt.md')), [System.Text.UTF8Encoding]::new($false)) + # For with_skill the prompt must have the candidate instructions before the + # working-environment boundary so the Copilot runner can verify the hash. + $taskBody = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀. I'm testing Microsoft's guidance. ÆØÅ`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) + $candidateInstructionHash = $null if ($Configuration -eq 'with_skill') { $skill = Join-Path $runRoot 'skill\candidate' New-Item -ItemType Directory -Path $skill -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $skill 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) $skillDirectory = 'skill/candidate' $skillHash = Get-TestTreeHash -Root $skill + $candidateInstructions = "## Skill: candidate`r`n`r`nConformance fixture candidate instructions for testing." + $candidateInstructionsNormalized = $candidateInstructions -replace "`r`n", "`n" -replace "`r", "`n" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.UTF8Encoding]::new($false).GetBytes($candidateInstructionsNormalized)))).ToLowerInvariant() + $prompt = $candidateInstructions + "`r`n`r`n# Working environment`r`n`r`n" + $taskBody } else { $skillDirectory = $null $skillHash = $null + $prompt = $taskBody } + [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'expected-prompt-sha256.txt'), (Get-Sha256HexFromFile -Path (Join-Path $runRoot 'prompt.md')), [System.Text.UTF8Encoding]::new($false)) $run = [ordered]@{ schema = (Get-RunnerSchemaNames).Run evalId = 1 @@ -2138,6 +2356,7 @@ function New-TestRun { inputFiles = @() fixtureHash = Get-TestTreeHash -Root $repo skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash contract = [ordered]@{ sandboxRoot = '.' workingDirectory = 'repo' @@ -2230,6 +2449,24 @@ try { Assert-Equal 'candidate' $withRunContract.skillName 'with_skill run.json names the exposed candidate' Assert-True ($null -eq $withoutRunContract.skillName) 'without_skill run.json keeps skillName null' Assert-True ($null -eq $withoutRunContract.skillDirectory) 'without_skill run.json keeps skillDirectory null' + # Fix 1: candidate instruction hash invariant - with_skill requires hash, without_skill forbids it. + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$withRunContract.candidateInstructionHash)) 'prepared with_skill run.json declares candidateInstructionHash' + Assert-True ($withRunContract.candidateInstructionHash -match '^[0-9a-f]{64}$') 'with_skill candidateInstructionHash is a valid SHA-256' + Assert-True ($null -eq $withoutRunContract.candidateInstructionHash -or [string]::IsNullOrWhiteSpace([string]$withoutRunContract.candidateInstructionHash)) 'without_skill run.json must not carry candidateInstructionHash' + $missingHashDir = Join-Path $iteration 'missing-hash'; New-Item -ItemType Directory -Path (Join-Path $missingHashDir 'repo'), (Join-Path $missingHashDir 'home'), (Join-Path $missingHashDir 'skill/c') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $missingHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $missingHashDir 'skill/c/SKILL.md'), '# c', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $missingHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = 'c'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/c'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64) }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $missingHashDir 'run.json') } 'missing with_skill candidateInstructionHash must fail Resolve-RunContract' + $badHashDir = Join-Path $iteration 'bad-hash'; New-Item -ItemType Directory -Path (Join-Path $badHashDir 'repo'), (Join-Path $badHashDir 'home'), (Join-Path $badHashDir 'skill/c') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $badHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $badHashDir 'skill/c/SKILL.md'), '# c', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $badHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = 'c'; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = 'skill/c'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = ('b' * 64); candidateInstructionHash = 'not-a-sha256' }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $badHashDir 'run.json') } 'malformed with_skill candidateInstructionHash must fail Resolve-RunContract' + $baselineHashDir = Join-Path $iteration 'baseline-with-hash'; New-Item -ItemType Directory -Path (Join-Path $baselineHashDir 'repo'), (Join-Path $baselineHashDir 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $baselineHashDir 'prompt.md'), 'test', [System.Text.UTF8Encoding]::new($false)) + Write-TestJson -Path (Join-Path $baselineHashDir 'run.json') -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = 'h'; candidateSkillName = 'c'; skillName = $null; mode = 'without_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; skillDirectory = $null; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; gitWorkspace = $false; fixtureHash = ('a' * 64); skillHash = $null; candidateInstructionHash = ('a' * 64) }) + Assert-Throws { Resolve-RunContract -RunPath (Join-Path $baselineHashDir 'run.json') } 'without_skill with candidateInstructionHash must fail Resolve-RunContract' $withoutPromptText = [System.IO.File]::ReadAllText((Join-Path $without.Root 'prompt.md'), [System.Text.UTF8Encoding]::new($false)) Assert-True (-not $withoutPromptText.Contains('candidate')) 'candidate control-plane identity is not added to the baseline prompt' $fakePath = Join-Path $runnerRoot 'fake\runner.ps1' @@ -2398,19 +2635,21 @@ try { Assert-True ($orchestrationText -notmatch '(?i)capture-native-results\.ps1|synthesize|worker_authored') 'generic orchestration must not manufacture native terminal envelopes' Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' - Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('validate-eval-grading.ps1') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading validation, grading application, and finalization boundaries' + Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('invoke-phase2-analyzer.ps1') -and $prepareText.Contains('phase2-state.json') -and $prepareText.Contains('grading-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the Phase 1 freeze, Phase 2 analyzer controller, grading freeze, deterministic grading, and finalization boundaries' Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' - Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' - Assert-True ($prepareText.Contains('Write `grading.json`, then validate it before finalization') -and $prepareText.Contains('Grading validation is retryable; finalization is not') -and $prepareText.Contains('only after grading validation succeeds')) 'handoff preparation must place retryable grading validation before exactly-once finalization' + Assert-True ($prepareText.Contains('authentication incompatibility is terminal for this package iteration') -and $prepareText.Contains('Do not suggest switching to another runner') -and $prepareText.Contains('do not suggest starting another Orchestrator') -and $prepareText.Contains('do not rerun Phase 1')) 'handoff preparation must fail closed on Copilot authentication incompatibility without alternate-runner/orchestrator suggestions' + Assert-True ($prepareText.Contains('The Phase 2 controller, not this outer orchestrator') -and $prepareText.Contains('deterministically merges root `grading.json`')) 'handoff preparation must route semantic grading through the package-local Phase 2 controller' + Assert-True ($prepareText.Contains('phase2-state.json') -and $prepareText.Contains('freezes `grading-freeze.json`') -and $prepareText.Contains('After Phase 2 succeeds, invoke finalization exactly once')) 'handoff preparation must require Phase 2 state/freeze before exactly-once finalization' Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' Assert-True ($prepareText.Contains('evaluation is incomplete') -and $prepareText.Contains('Only persisted runner-produced evidence')) 'handoff preparation must fail closed when runner evidence cannot be persisted' Assert-True ($prepareText.Contains('fresh package/code fix is required') -and $prepareText.Contains('Never patch package-local runner code') -and $prepareText.Contains('delete execution results') -and $prepareText.Contains('delete or replace `execution-freeze.json`') -and $prepareText.Contains('rerun Phase 1') -and $prepareText.Contains('manually broaden a capability check')) 'generated handoff must forbid package-local repair, state deletion, retry, and manual capability broadening' $generatedHandoff = Invoke-GeneratedRunnerPrompt $generatedConcurrencyThreeHandoff = Invoke-GeneratedRunnerPrompt -RequestedConcurrency 3 Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' + Assert-True ($generatedHandoff.Contains('authentication incompatibility is terminal for this package iteration') -and $generatedHandoff.Contains('Do not suggest switching to another runner') -and $generatedHandoff.Contains('do not suggest starting another Orchestrator') -and $generatedHandoff.Contains('do not rerun Phase 1')) 'generated handoff output must keep Copilot authentication failures fail-closed without fallback orchestration suggestions' Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' - Assert-True ($generatedHandoff.Contains('validate-eval-grading.ps1') -and $generatedHandoff.Contains('-ShowSkeleton') -and $generatedHandoff.Contains('Write `grading.json`, then validate it before finalization') -and $generatedHandoff.Contains('Grading validation is retryable; finalization is not') -and $generatedHandoff.Contains('only after grading validation succeeds')) 'generated handoff requires author -> validate/retry -> finalize exactly once' + Assert-True ($generatedHandoff.Contains('invoke-phase2-analyzer.ps1') -and $generatedHandoff.Contains('The Phase 2 controller, not this outer orchestrator') -and $generatedHandoff.Contains('phase2-state.json') -and $generatedHandoff.Contains('grading-freeze.json') -and $generatedHandoff.Contains('After Phase 2 succeeds, invoke finalization exactly once')) 'generated handoff requires package-local Phase 2 controller -> freeze -> finalize exactly once' Assert-True ($generatedHandoff.Contains('allowance of 6240 seconds') -and $generatedHandoff.Contains('6 arm(s) × 120-second fixed model-free runner preflight timeout = 720-second serial preflight allowance') -and $generatedHandoff.Contains('5490-second execution allowance across 3 batch(es) at concurrency 2') -and $generatedHandoff.Contains('2 scripted user turn(s) × profile.timeout_seconds 900 + 30 seconds runner grace') -and $generatedHandoff.Contains('+ 30 seconds orchestration grace')) 'generated handoff uses the fixed preflight timeout, scripted-turn model timeout, execution batches, and separate orchestration grace' Assert-True (-not $generatedHandoff.Contains('max(120, profile.timeout_seconds + runner grace)') -and -not $generatedHandoff.Contains('5580-second serial preflight allowance')) 'generated handoff does not budget preflight from the model timeout plus runner grace' Assert-True ($generatedConcurrencyThreeHandoff.Contains('allowance of 4410 seconds') -and $generatedConcurrencyThreeHandoff.Contains('execution allowance across 2 batch(es) at concurrency 3')) 'generated handoff recalculates execution batches when concurrency changes' @@ -2429,6 +2668,20 @@ try { Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' Assert-True ($manifestBridgeText.Contains('Get-ManifestRunRecords') -and $manifestBridgeText.Contains('$record.ResultPath')) 'package-level bridge must resolve exact manifest records' Assert-True ($manifestBridgeText -notmatch 'with[-_]skill\.result\.json|without[-_]skill\.result\.json') 'package-level bridge must not encode arm-derived result filenames' + foreach ($case in @( + @{ Name = 'Windows inside'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection\repo\README.md'; Expected = $true } + @{ Name = 'Windows prefix collision'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection2\repo\README.md'; Expected = $false } + @{ Name = 'Windows dotdot escape'; Base = 'C:\temp\projection'; Candidate = 'C:\temp\projection\..\outside\README.md'; Expected = $false } + @{ Name = 'Windows case-insensitive normalized separators'; Base = 'C:\TEMP\Projection'; Candidate = 'c:/temp/projection/repo/file.txt'; Expected = $true } + @{ Name = 'Unix inside'; Base = '/tmp/projection'; Candidate = '/tmp/projection/repo/README.md'; Expected = $true } + @{ Name = 'Unix prefix collision'; Base = '/tmp/projection'; Candidate = '/tmp/projection2/README.md'; Expected = $false } + @{ Name = 'Unix dotdot escape'; Base = '/tmp/projection'; Candidate = '/tmp/projection/../outside/README.md'; Expected = $false } + @{ Name = 'Unix remains case-sensitive'; Base = '/tmp/Projection'; Candidate = '/tmp/projection/file'; Expected = $false } + @{ Name = 'UNC inside'; Base = '\\server\share\projection'; Candidate = '\\server\share\projection\repo\file.txt'; Expected = $true } + @{ Name = 'UNC share boundary'; Base = '\\server\share\projection'; Candidate = '\\server\share-other\projection\file.txt'; Expected = $false } + )) { + Assert-ObservedPathInside -BasePath ([string]$case.Base) -CandidatePath ([string]$case.Candidate) -Expected ([bool]$case.Expected) -Message ("Observed path containment stays style-aware for {0}" -f [string]$case.Name) + } Assert-Equal 1 ([regex]::Matches($opencodeRunnerText, '\$directoryArgument = Get-SandboxVisiblePath').Count) 'OpenCode CLI argument construction assigns the sandbox directory once' $opencodeAst = Get-OpenCodeRunnerAst $scriptedFunctionAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeScriptedExecute' }, $true) | Select-Object -First 1) @@ -2437,17 +2690,70 @@ try { $continuationParserAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Get-OpenCodeContinuationCapability' }, $true)) $continuationArgumentAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'New-OpenCodeContinuationArguments' }, $true)) $turnProcessAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeTurnProcess' }, $true)) + $openCodeBoundaryHelperNames = @( + 'Add-OpenCodeBoundaryValueCandidate', + 'Get-OpenCodeBoundaryValueCandidates', + 'Get-OpenCodeLogicalPackageRoot', + 'Test-OpenCodeBoundaryPairedArmPath', + 'Test-OpenCodeBoundaryForbiddenGradingPath', + 'Get-OpenCodeBoundaryAssessment', + 'ConvertTo-OpenCodeBoundaryAssessment', + 'Apply-OpenCodeBoundaryAssessment' + ) Assert-Equal 1 $scriptedFunctionAst.Count 'OpenCode has one selected scripted execution function' Assert-Equal 1 $executeFunctionAst.Count 'OpenCode has one execution dispatcher' Assert-Equal 0 $legacyFunctionAst.Count 'OpenCode removes the dead legacy scripted continuation function' Assert-Equal 1 $continuationParserAst.Count 'OpenCode keeps installed-help exact --session parsing' Assert-Equal 1 $continuationArgumentAst.Count 'OpenCode keeps explicit --session argument construction' Assert-Equal 1 $turnProcessAst.Count 'OpenCode uses a direct CLI turn-process helper' + foreach ($helperName in $openCodeBoundaryHelperNames) { + $helperAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $helperName }, $true) | Select-Object -First 1) + Assert-Equal 1 $helperAst.Count "OpenCode keeps boundary helper $helperName" + Invoke-Expression $helperAst[0].Extent.Text + } $scriptedFunctionText = [string]$scriptedFunctionAst[0].Extent.Text $executeFunctionText = [string]$executeFunctionAst[0].Extent.Text Assert-True ($scriptedFunctionText.Contains('Invoke-OpenCodeTurnProcess') -and $scriptedFunctionText.Contains('New-OpenCodeContinuationArguments') -and $scriptedFunctionText.Contains('Get-OpenCodeFutureTurnCanary')) 'selected OpenCode scripted transport uses direct CLI exact-session continuation and future-turn secrecy checks' Assert-True ($opencodeRunnerText -notmatch '(?i)Start-OpenCodeServer|Invoke-OpenCodeHttpRequest|SessionCreatePath|SessionMessagePath|/global/health|/doc|opencode\s+serve|serve_help_probe|opencode-server-synchronous-http') 'OpenCode runner has no eval server transport or loopback API probes' Assert-True ($executeFunctionText.Contains('Invoke-OpenCodeScriptedExecute') -and $executeFunctionText -notmatch '(?i)Invoke-OpenCodeScriptedExecuteLegacy\s+-Inputs') 'OpenCode dispatcher selects the direct CLI scripted transport for interaction runs' + $openCodeBoundaryProjection = [pscustomobject]@{ + Root = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture' + PhysicalWorkingDirectory = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo' + SourceRepositoryRoot = 'C:\Source\GitHub\codebeltnet\agentic' + LogicalRun = [pscustomobject]@{ + RunRoot = 'C:\prepared\iteration-1\eval-01\with_skill' + Mode = 'with_skill' + } + } + Assert-Equal 'C:\prepared\iteration-1' (Get-OpenCodeLogicalPackageRoot -Projection $openCodeBoundaryProjection) 'OpenCode derives the logical package root lexically from observed Windows paths' + Assert-True (Test-OpenCodeBoundaryForbiddenGradingPath -ResolvedPath 'C:\prepared\iteration-1\eval-01\eval-metadata.json' -Projection $openCodeBoundaryProjection) 'OpenCode grading-material detection stays lexical for observed Windows package paths' + Assert-True (Test-OpenCodeBoundaryPairedArmPath -ResolvedPath 'C:\prepared\iteration-1\eval-01\without_skill\repo\README.md' -Projection $openCodeBoundaryProjection) 'OpenCode paired-arm detection stays lexical for observed Windows package paths' + foreach ($event in @( + @{ Name = 'read inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\README.md' } } } + @{ Name = 'write inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'write'; path = 'C:\Users\Administrator\AppData\Local\Temp\agentic-opencode-projection-fixture\repo\out.txt' } } } + @{ Name = 'repo-relative shell inside projection'; Event = @{ type = 'tool_use'; part = @{ tool = 'shell'; command = 'Get-Content README.md' } } } + )) { + $assessment = Get-OpenCodeBoundaryAssessment -Events @($event.Event) -Projection $openCodeBoundaryProjection + Assert-Equal 0 @($assessment.Contradictions).Count "OpenCode boundary allows $($event.Name)" + } + foreach ($event in @( + @{ Name = 'read unrelated Windows repo'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\xunit\README.md' } } } + @{ Name = 'write unrelated Windows repo'; Event = @{ type = 'tool_use'; part = @{ tool = 'write'; path = 'C:\Source\GitHub\xunit\README.md' } } } + @{ Name = 'PowerShell filesystem shell escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'shell'; command = 'Get-ChildItem C:\Users\Administrator -Recurse' } } } + @{ Name = 'Linux absolute path escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = '/home/user/other-repo/file' } } } + @{ Name = 'macOS absolute path escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = '/Users/user/other-repo/file' } } } + @{ Name = 'source repository escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\codebeltnet\agentic\README.md' } } } + @{ Name = 'prepared package escape'; Event = @{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\prepared\iteration-1\eval-01\with_skill\repo\README.md' } } } + )) { + $assessment = Get-OpenCodeBoundaryAssessment -Events @($event.Event) -Projection $openCodeBoundaryProjection + Assert-True ([bool]$assessment.ProjectionEscapeObserved) "OpenCode boundary rejects $($event.Name)" + Assert-True (@($assessment.Contradictions).Count -gt 0) "OpenCode boundary records contradiction evidence for $($event.Name)" + } + $outsideProjectionAssessment = Get-OpenCodeBoundaryAssessment -Events @(@{ type = 'tool_use'; part = @{ tool = 'read'; path = 'C:\Source\GitHub\xunit\README.md' } }) -Projection $openCodeBoundaryProjection + $outsideProjectionOutcome = Apply-OpenCodeBoundaryAssessment -Status 'completed' -Assessment $outsideProjectionAssessment + Assert-Equal 'incompatible' ([string]$outsideProjectionOutcome.Status) 'observed OpenCode projection escape makes a completed arm incompatible' + Assert-Equal 'opencode_projection_escape_observed' ([string]$outsideProjectionOutcome.FailureCode) 'observed OpenCode projection escape surfaces the dedicated failure code' + Assert-True (@($outsideProjectionOutcome.BoundaryEvidence.contradictions).Count -gt 0) 'OpenCode boundary outcome preserves structured contradiction evidence' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $withoutRawPath = Join-Path $iteration 'conformance\results\without-skill.execution-result.json' diff --git a/scripts/eval-runners/tests/test-runner-observability.ps1 b/scripts/eval-runners/tests/test-runner-observability.ps1 index 9e67c1d..d0b696b 100644 --- a/scripts/eval-runners/tests/test-runner-observability.ps1 +++ b/scripts/eval-runners/tests/test-runner-observability.ps1 @@ -1,351 +1,356 @@ -<#! -.SYNOPSIS - Deterministic, model-free live-observability tests for every eval runner. - -.DESCRIPTION - Proves the shared observability contract without any model, network, or - authenticated CLI. Synthetic child processes exercise the shared process - primitive (heartbeats, activity tracking, tee/relay, timeout diagnostics, - secret hygiene), and one real runner-owned fan-out run over the deterministic - fixture proves concurrent-arm attribution, the STDOUT machine contract, and - persisted progress evidence. No test may hang: every scenario is bounded. -#> -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path -. (Join-Path $runnerRoot 'runner-common.ps1') -. (Join-Path $runnerRoot 'runner-progress.ps1') -. (Join-Path $runnerRoot 'fanout-process.ps1') -. (Join-Path $runnerRoot 'manifest-paths.ps1') -. (Join-Path $runnerRoot 'package-integrity.ps1') -. (Join-Path $runnerRoot 'execution-freeze.ps1') - -function Assert-True { - param([bool]$Condition, [string]$Message) - if (-not $Condition) { throw "ASSERT: $Message" } -} - -function Assert-Equal { - param([object]$Expected, [object]$Actual, [string]$Message) - if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } -} - -function Get-Field { - param([object]$Object, [string]$Name, [object]$Default = $null) - if ($null -ne $Object.PSObject.Properties[$Name]) { return $Object.$Name } - return $Default -} - -$pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) -$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-observability-' + [Guid]::NewGuid().ToString('N')) -New-Item -ItemType Directory -Path $testRoot -Force | Out-Null - -function New-SyntheticChildScript { - param( - [Parameter(Mandatory = $true)][string]$Name, - [Parameter(Mandatory = $true)][string]$Body - ) - - $path = Join-Path $testRoot ($Name + '.ps1') - [System.IO.File]::WriteAllText($path, $Body, [System.Text.UTF8Encoding]::new($false)) - return $path -} - -function Invoke-SyntheticChild { - <# - Drives one synthetic child through the shared process primitive exactly as - the fan-out does: Start -> Wait-AnyRunnerChild (heartbeats/relay) -> Complete - (terminal diagnostic). Returns the child, exit code, and the parsed progress - log so a test can assert what an operator would have seen live. - #> - param( - [Parameter(Mandatory = $true)][string]$ScriptPath, - [Parameter(Mandatory = $true)][string]$WorkerId, - [int]$TimeoutSeconds = 30, - [double]$HeartbeatSeconds = 0.3, - [object]$EvalId = 1, - [string]$Configuration = 'with_skill' - ) - - $workDirectory = Join-Path $testRoot ('work-' + [Guid]::NewGuid().ToString('N')) - New-Item -ItemType Directory -Path $workDirectory -Force | Out-Null - $stdoutPath = Join-Path $workDirectory 'result.json' - $stderrPath = Join-Path $workDirectory 'child.stderr' - $logPath = Join-Path $workDirectory 'progress.jsonl' - $operatorErrorWriter = [System.IO.StringWriter]::new([Globalization.CultureInfo]::InvariantCulture) - $originalErrorWriter = [Console]::Error - try { - [Console]::SetError($operatorErrorWriter) +<#! +.SYNOPSIS + Deterministic, model-free live-observability tests for every eval runner. + +.DESCRIPTION + Proves the shared observability contract without any model, network, or + authenticated CLI. Synthetic child processes exercise the shared process + primitive (heartbeats, activity tracking, tee/relay, timeout diagnostics, + secret hygiene), and one real runner-owned fan-out run over the deterministic + fixture proves concurrent-arm attribution, the STDOUT machine contract, and + persisted progress evidence. No test may hang: every scenario is bounded. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'runner-progress.ps1') +. (Join-Path $runnerRoot 'fanout-process.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } +} + +function Get-Field { + param([object]$Object, [string]$Name, [object]$Default = $null) + if ($null -ne $Object.PSObject.Properties[$Name]) { return $Object.$Name } + return $Default +} + +$pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-observability-' + [Guid]::NewGuid().ToString('N')) +New-Item -ItemType Directory -Path $testRoot -Force | Out-Null + +function New-SyntheticChildScript { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][string]$Body + ) + + $path = Join-Path $testRoot ($Name + '.ps1') + [System.IO.File]::WriteAllText($path, $Body, [System.Text.UTF8Encoding]::new($false)) + return $path +} + +function Invoke-SyntheticChild { + <# + Drives one synthetic child through the shared process primitive exactly as + the fan-out does: Start -> Wait-AnyRunnerChild (heartbeats/relay) -> Complete + (terminal diagnostic). Returns the child, exit code, and the parsed progress + log so a test can assert what an operator would have seen live. + #> + param( + [Parameter(Mandatory = $true)][string]$ScriptPath, + [Parameter(Mandatory = $true)][string]$WorkerId, + [int]$TimeoutSeconds = 30, + [double]$HeartbeatSeconds = 0.3, + [object]$EvalId = 1, + [string]$Configuration = 'with_skill' + ) + + $workDirectory = Join-Path $testRoot ('work-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $workDirectory -Force | Out-Null + $stdoutPath = Join-Path $workDirectory 'result.json' + $stderrPath = Join-Path $workDirectory 'child.stderr' + $logPath = Join-Path $workDirectory 'progress.jsonl' + $operatorErrorWriter = [System.IO.StringWriter]::new([Globalization.CultureInfo]::InvariantCulture) + $originalErrorWriter = [Console]::Error + try { + [Console]::SetError($operatorErrorWriter) $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $ScriptPath) -WorkingDirectory $workDirectory -StdoutPath $stdoutPath -StderrPath $stderrPath -TimeoutSeconds $TimeoutSeconds -Runner 'synthetic' -WorkerId $WorkerId -EvalId $EvalId -Configuration $Configuration -Phase 'model-cli' -ProgressLogPath $logPath -HeartbeatSeconds $HeartbeatSeconds - $running = [System.Collections.Generic.List[object]]::new() - $running.Add([pscustomobject]@{ worker_id = $WorkerId; child = $child; Process = $child.Process }) - $index = Wait-AnyRunnerChild -Running $running - $exitCode = Complete-RunnerChildProcess -Child $child - } finally { - try { [Console]::Error.Flush() } catch { } - [Console]::SetError($originalErrorWriter) - } - $events = @() - if (Test-Path -LiteralPath $logPath -PathType Leaf) { - $events = @(Get-Content -LiteralPath $logPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - } - return [pscustomobject]@{ - Child = $child - ExitCode = $exitCode - WaitIndex = $index - Events = $events - StdoutPath = $stdoutPath - StderrPath = $stderrPath - LogPath = $logPath - OperatorStderr = [string]$operatorErrorWriter.ToString() - } -} - -function New-ObservabilityFanoutPackage { - <# - Builds a minimal runner-owned fixture package. The fixture is a protocol - adapter only; it never calls a model or an AI CLI. Each arm is slowed by a - per-run marker so heartbeats fire on the real operator-facing path. - #> - param( - [Parameter(Mandatory = $true)][string]$IterationDirectory, - [int]$EvalCount = 2, - [int]$Concurrency = 4, - [int]$DelayMs = 700 - ) - - $tools = Join-Path $IterationDirectory 'tools\eval-runners' - New-Item -ItemType Directory -Path $tools -Force | Out-Null - foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { - Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force - } - $fixtureRunnerDirectory = Join-Path $tools 'fixture' - New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null - Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force - - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'execution-profile.json'), (([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'fixture' - model = 'fixture-model' - reasoning_effort = $null - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = 30 - concurrency = $Concurrency - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - - $manifestEvals = [System.Collections.Generic.List[object]]::new() - for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { - $evalName = 'obs-eval-{0:d2}' -f $evalId - $evalDirectory = Join-Path $IterationDirectory $evalName - New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $runs = [ordered]@{} - foreach ($configuration in @('with_skill', 'without_skill')) { - $runDirectory = Join-Path $evalDirectory $configuration - $repoDirectory = Join-Path $runDirectory 'repo' - $homeDirectory = Join-Path $runDirectory 'home' - $resultDirectory = Join-Path $evalDirectory 'results' - New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "observability prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) - # Slow each arm past the heartbeat interval so the operator path emits - # heartbeats without any model involvement. - [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), [string]$DelayMs, [System.Text.UTF8Encoding]::new($false)) - $skillDirectory = $null - $skillHash = $null - if ($configuration -eq 'with_skill') { - $skillDirectory = 'skill/candidate' - New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) - $skillHash = ('b' * 64) - } - [System.IO.File]::WriteAllText((Join-Path $runDirectory 'run.json'), (([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = $evalId - evalName = $evalName + $running = [System.Collections.Generic.List[object]]::new() + $running.Add([pscustomobject]@{ worker_id = $WorkerId; child = $child; Process = $child.Process }) + $index = Wait-AnyRunnerChild -Running $running + $exitCode = Complete-RunnerChildProcess -Child $child + } finally { + try { [Console]::Error.Flush() } catch { } + [Console]::SetError($originalErrorWriter) + } + $events = @() + if (Test-Path -LiteralPath $logPath -PathType Leaf) { + $events = @(Get-Content -LiteralPath $logPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + return [pscustomobject]@{ + Child = $child + ExitCode = $exitCode + WaitIndex = $index + Events = $events + StdoutPath = $stdoutPath + StderrPath = $stderrPath + LogPath = $logPath + OperatorStderr = [string]$operatorErrorWriter.ToString() + } +} + +function New-ObservabilityFanoutPackage { + <# + Builds a minimal runner-owned fixture package. The fixture is a protocol + adapter only; it never calls a model or an AI CLI. Each arm is slowed by a + per-run marker so heartbeats fire on the real operator-facing path. + #> + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [int]$EvalCount = 2, + [int]$Concurrency = 4, + [int]$DelayMs = 700 + ) + + $tools = Join-Path $IterationDirectory 'tools\eval-runners' + New-Item -ItemType Directory -Path $tools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $tools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'execution-profile.json'), (([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = $Concurrency + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { + $evalName = 'obs-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $resultDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "observability prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + # Slow each arm past the heartbeat interval so the operator path emits + # heartbeats without any model involvement. + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), [string]$DelayMs, [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + $candidateInstructionHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + $promptContent = "observability prompt $evalName/$configuration" + $candidateInstructionHash = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($promptContent)))).ToLowerInvariant() + } + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'run.json'), (([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName candidateSkillName = 'candidate' - skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } - iteration = 1 - mode = $configuration - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $skillDirectory - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('a' * 64) - skillHash = $skillHash - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } - [System.IO.File]::WriteAllText((Join-Path $resultDirectory $resultName), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - $runs[$configuration] = [ordered]@{ - mode = $configuration - run_manifest = "$evalName/$configuration/run.json" - execution_result = "$evalName/results/$executionName" - result = "$evalName/results/$resultName" - } - } - $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) - } - - $toolIntegrity = Get-PackageTreeIntegrity -Root $tools - [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'manifest.json'), (([ordered]@{ - schema = 'codebeltnet/agentic/eval-package/2' - configurations = @('with_skill', 'without_skill') + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + fixtureHash = ('a' * 64) + skillHash = $skillHash + candidateInstructionHash = $candidateInstructionHash + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + [System.IO.File]::WriteAllText((Join-Path $resultDirectory $resultName), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $tools + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory 'manifest.json'), (([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') execution_selection = [ordered]@{ harness = 'Deterministic runner-owned fixture' runner = 'fixture' model = 'fixture-model' preset = 'Observability fixture' } - execution_profile = 'execution-profile.json' - runner_tools = 'tools/eval-runners' - runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } - execution_freeze = 'execution-freeze.json' - evals = @($manifestEvals.ToArray()) - } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) - return $IterationDirectory -} - -function Invoke-ObservabilityFanout { - param( - [Parameter(Mandatory = $true)][string]$Root, - [int]$EvalCount = 2, - [int]$Concurrency = 4, - [int]$DelayMs = 700 - ) - - New-Item -ItemType Directory -Path $Root -Force | Out-Null - [void](New-ObservabilityFanoutPackage -IterationDirectory $Root -EvalCount $EvalCount -Concurrency $Concurrency -DelayMs $DelayMs) - $fanout = Join-Path $Root 'tools/eval-runners/invoke-runner-owned-arms.ps1' - $stderrPath = Join-Path $Root 'phase1.stderr' - $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', '0.3') - try { + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals.ToArray()) + } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + return $IterationDirectory +} + +function Invoke-ObservabilityFanout { + param( + [Parameter(Mandatory = $true)][string]$Root, + [int]$EvalCount = 2, + [int]$Concurrency = 4, + [int]$DelayMs = 700 + ) + + New-Item -ItemType Directory -Path $Root -Force | Out-Null + [void](New-ObservabilityFanoutPackage -IterationDirectory $Root -EvalCount $EvalCount -Concurrency $Concurrency -DelayMs $DelayMs) + $fanout = Join-Path $Root 'tools/eval-runners/invoke-runner-owned-arms.ps1' + $stderrPath = Join-Path $Root 'phase1.stderr' + $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', '0.3') + try { $output = & $pwshPath -NoProfile -NonInteractive -File $fanout -IterationDirectory $Root 2>$stderrPath - $exitCode = $LASTEXITCODE - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) - } - $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - return [pscustomobject]@{ - IterationDirectory = $Root - ExitCode = $exitCode - Stdout = $stdout - Stderr = $stderr - } -} - -function Write-TestJson { - param( - [Parameter(Mandatory = $true)][string]$Path, - [Parameter(Mandatory = $true)][object]$Value - ) - - [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) -} - -function Get-RelayedProgressEventsFromText { - param([AllowEmptyString()][string]$Text) - - $sentinel = Get-RunnerProgressSentinel - return @($Text -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) } | ForEach-Object { - $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $payload | ConvertFrom-Json -Depth 50 } catch { $null } - } | Where-Object { $null -ne $_ }) -} - -function New-CodexObservabilityFixture { - param( - [Parameter(Mandatory = $true)][string]$Root, - [ValidateSet('success', 'timeout')][string]$Mode = 'success', - [int]$TimeoutSeconds = 5 - ) - - New-Item -ItemType Directory -Path $Root -Force | Out-Null - $fakeBin = Join-Path $Root 'fake-bin' - New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null - $fakeCodexPath = Join-Path $fakeBin 'codex.ps1' - [System.IO.File]::WriteAllText($fakeCodexPath, @' -param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest - -$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) -$heartbeatMs = 150 -$heartbeatOverride = [Environment]::GetEnvironmentVariable('AGENTIC_CODEX_OBS_HEARTBEAT_MS') -if (-not [string]::IsNullOrWhiteSpace($heartbeatOverride)) { - $parsedHeartbeat = 0 - if ([int]::TryParse($heartbeatOverride, [ref]$parsedHeartbeat) -and $parsedHeartbeat -gt 0) { - $heartbeatMs = $parsedHeartbeat - } -} -$homeRoot = [Environment]::GetEnvironmentVariable('HOME') + $exitCode = $LASTEXITCODE + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) + } + $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + return [pscustomobject]@{ + IterationDirectory = $Root + ExitCode = $exitCode + Stdout = $stdout + Stderr = $stderr + } +} + +function Write-TestJson { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$Value + ) + + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Get-RelayedProgressEventsFromText { + param([AllowEmptyString()][string]$Text) + + $sentinel = Get-RunnerProgressSentinel + return @($Text -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) } | ForEach-Object { + $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $payload | ConvertFrom-Json -Depth 50 } catch { $null } + } | Where-Object { $null -ne $_ }) +} + +function New-CodexObservabilityFixture { + param( + [Parameter(Mandatory = $true)][string]$Root, + [ValidateSet('success', 'timeout')][string]$Mode = 'success', + [int]$TimeoutSeconds = 5 + ) + + New-Item -ItemType Directory -Path $Root -Force | Out-Null + $fakeBin = Join-Path $Root 'fake-bin' + New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null + $fakeCodexPath = Join-Path $fakeBin 'codex.ps1' + [System.IO.File]::WriteAllText($fakeCodexPath, @' +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) +$heartbeatMs = 150 +$heartbeatOverride = [Environment]::GetEnvironmentVariable('AGENTIC_CODEX_OBS_HEARTBEAT_MS') +if (-not [string]::IsNullOrWhiteSpace($heartbeatOverride)) { + $parsedHeartbeat = 0 + if ([int]::TryParse($heartbeatOverride, [ref]$parsedHeartbeat) -and $parsedHeartbeat -gt 0) { + $heartbeatMs = $parsedHeartbeat + } +} +$homeRoot = [Environment]::GetEnvironmentVariable('HOME') $isProjectedExecution = [string](Get-Location).Path -match 'agentic-codex-projection-' $nativeSuppressionEnabled = @($arguments | Where-Object { [string]$_ -eq 'skills.include_instructions=false' -or [string]$_ -eq 'skills.bundled.enabled=false' }).Count -gt 0 $mode = if ($isProjectedExecution -and $nativeSuppressionEnabled -and ( - (-not [string]::IsNullOrWhiteSpace($homeRoot) -and (Test-Path -LiteralPath (Join-Path $homeRoot 'codex-observability-timeout') -PathType Leaf)) -or - (Test-Path -LiteralPath (Join-Path (Get-Location).Path 'codex-observability-timeout') -PathType Leaf) + (-not [string]::IsNullOrWhiteSpace($homeRoot) -and (Test-Path -LiteralPath (Join-Path $homeRoot 'codex-observability-timeout') -PathType Leaf)) -or + (Test-Path -LiteralPath (Join-Path (Get-Location).Path 'codex-observability-timeout') -PathType Leaf) )) { 'timeout' } else { 'success' } - -function Get-DelayMilliseconds { - param([double]$Multiplier) - - return [int][Math]::Max(50, [Math]::Ceiling($heartbeatMs * $Multiplier)) -} - -function Read-AppServerMessage { + +function Get-DelayMilliseconds { + param([double]$Multiplier) + + return [int][Math]::Max(50, [Math]::Ceiling($heartbeatMs * $Multiplier)) +} + +function Read-AppServerMessage { param([switch]$AllowEndOfStream) - $line = [Console]::In.ReadLine() + $line = [Console]::In.ReadLine() if ($null -eq $line) { if ($AllowEndOfStream) { return $null } throw 'observability fake app-server reached EOF before the expected request' } - return ($line | ConvertFrom-Json -Depth 50) -} - -function Write-AppServerMessage { - param([Parameter(Mandatory = $true)][object]$Value) - - [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) - [Console]::Out.Flush() -} - -function Write-CodexSchemas { - param([Parameter(Mandatory = $true)][string]$SchemaDirectory) - - New-Item -ItemType Directory -Path $SchemaDirectory -Force | Out-Null - foreach ($existing in @(Get-ChildItem -LiteralPath $SchemaDirectory -Force -ErrorAction SilentlyContinue)) { - Remove-Item -LiteralPath $existing.FullName -Recurse -Force - } - - $schema = 'http://json-schema.org/draft-07/schema#' - $definitions = [ordered]@{ - AbsolutePathBuf = [ordered]@{ type = 'string' } - LegacyAppPathString = [ordered]@{ type = 'string' } - SandboxMode = [ordered]@{ type = 'string'; enum = @('read-only', 'workspace-write', 'danger-full-access') } - AskForApproval = [ordered]@{ oneOf = @([ordered]@{ type = 'string'; enum = @('untrusted', 'on-request', 'never') }) } - ReasoningEffort = [ordered]@{ type = 'string'; minLength = 1 } - ModelRerouteReason = [ordered]@{ type = 'string'; enum = @('highRiskCyberActivity') } - TurnStatus = [ordered]@{ type = 'string'; enum = @('inProgress', 'completed', 'failed', 'interrupted') } - UserInput = [ordered]@{ oneOf = @([ordered]@{ type = 'object'; required = @('text', 'type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('text') }; text = [ordered]@{ type = 'string' } } }) } - SandboxPolicy = [ordered]@{ oneOf = @( - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('dangerFullAccess') } } } - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('readOnly') }; networkAccess = [ordered]@{ type = 'boolean' } } } - [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('workspaceWrite') }; writableRoots = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } }; networkAccess = [ordered]@{ type = 'boolean' } } } - ) } - Thread = [ordered]@{ type = 'object'; required = @('id', 'cwd', 'ephemeral', 'sessionId', 'turns'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; cwd = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }) }; ephemeral = [ordered]@{ type = 'boolean' }; sessionId = [ordered]@{ type = 'string' }; turns = [ordered]@{ type = 'array' } } } - Turn = [ordered]@{ type = 'object'; required = @('id', 'items', 'status'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; items = [ordered]@{ type = 'array' }; status = [ordered]@{ '$ref' = '#/definitions/TurnStatus' } } } + return ($line | ConvertFrom-Json -Depth 50) +} + +function Write-AppServerMessage { + param([Parameter(Mandatory = $true)][object]$Value) + + [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + [Console]::Out.Flush() +} + +function Write-CodexSchemas { + param([Parameter(Mandatory = $true)][string]$SchemaDirectory) + + New-Item -ItemType Directory -Path $SchemaDirectory -Force | Out-Null + foreach ($existing in @(Get-ChildItem -LiteralPath $SchemaDirectory -Force -ErrorAction SilentlyContinue)) { + Remove-Item -LiteralPath $existing.FullName -Recurse -Force + } + + $schema = 'http://json-schema.org/draft-07/schema#' + $definitions = [ordered]@{ + AbsolutePathBuf = [ordered]@{ type = 'string' } + LegacyAppPathString = [ordered]@{ type = 'string' } + SandboxMode = [ordered]@{ type = 'string'; enum = @('read-only', 'workspace-write', 'danger-full-access') } + AskForApproval = [ordered]@{ oneOf = @([ordered]@{ type = 'string'; enum = @('untrusted', 'on-request', 'never') }) } + ReasoningEffort = [ordered]@{ type = 'string'; minLength = 1 } + ModelRerouteReason = [ordered]@{ type = 'string'; enum = @('highRiskCyberActivity') } + TurnStatus = [ordered]@{ type = 'string'; enum = @('inProgress', 'completed', 'failed', 'interrupted') } + UserInput = [ordered]@{ oneOf = @([ordered]@{ type = 'object'; required = @('text', 'type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('text') }; text = [ordered]@{ type = 'string' } } }) } + SandboxPolicy = [ordered]@{ oneOf = @( + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('dangerFullAccess') } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('readOnly') }; networkAccess = [ordered]@{ type = 'boolean' } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('workspaceWrite') }; writableRoots = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } }; networkAccess = [ordered]@{ type = 'boolean' } } } + ) } + Thread = [ordered]@{ type = 'object'; required = @('id', 'cwd', 'ephemeral', 'sessionId', 'turns'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; cwd = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }) }; ephemeral = [ordered]@{ type = 'boolean' }; sessionId = [ordered]@{ type = 'string' }; turns = [ordered]@{ type = 'array' } } } + Turn = [ordered]@{ type = 'object'; required = @('id', 'items', 'status'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; items = [ordered]@{ type = 'array' }; status = [ordered]@{ '$ref' = '#/definitions/TurnStatus' } } } SkillMetadata = [ordered]@{ type = 'object'; required = @('name', 'path', 'enabled'); properties = [ordered]@{ name = [ordered]@{ type = 'string' }; path = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }; enabled = [ordered]@{ type = 'boolean' }; scope = [ordered]@{ type = 'string'; enum = @('user', 'repo', 'system', 'admin') }; description = [ordered]@{ type = 'string' } } } SkillsListEntry = [ordered]@{ type = 'object'; required = @('cwd', 'errors', 'skills'); properties = [ordered]@{ cwd = [ordered]@{ type = 'string' }; errors = [ordered]@{ type = 'array' }; skills = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/SkillMetadata' } } } } Config = [ordered]@{ type = 'object'; additionalProperties = $true } @@ -381,159 +386,159 @@ function Write-CodexSchemas { type = 'object' required = @('data') properties = [ordered]@{ data = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/SkillsListEntry' } } } - } - $definitions.ThreadStartParams = [ordered]@{ - '$schema' = $schema - title = 'ThreadStartParams' - type = 'object' - properties = [ordered]@{ - model = [ordered]@{ type = @('string', 'null') } - cwd = [ordered]@{ type = @('string', 'null') } - approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } - sandbox = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxMode' }, [ordered]@{ type = 'null' }) } - ephemeral = [ordered]@{ type = @('boolean', 'null') } - } - } - $definitions.ThreadStartResponse = [ordered]@{ - '$schema' = $schema - title = 'ThreadStartResponse' - type = 'object' - required = @('approvalPolicy', 'approvalsReviewer', 'cwd', 'model', 'modelProvider', 'sandbox', 'thread') - properties = [ordered]@{ - approvalPolicy = [ordered]@{ '$ref' = '#/definitions/AskForApproval' } - cwd = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } - instructionSources = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/LegacyAppPathString' } } - model = [ordered]@{ type = 'string' } - sandbox = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }) } - thread = [ordered]@{ '$ref' = '#/definitions/Thread' } - } - } - $definitions.TurnStartParams = [ordered]@{ - '$schema' = $schema - title = 'TurnStartParams' - type = 'object' - required = @('input', 'threadId') - properties = [ordered]@{ - threadId = [ordered]@{ type = 'string' } - input = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/UserInput' } } - cwd = [ordered]@{ type = @('string', 'null') } - model = [ordered]@{ type = @('string', 'null') } - effort = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/ReasoningEffort' }, [ordered]@{ type = 'null' }) } - approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } - sandboxPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }, [ordered]@{ type = 'null' }) } - } - } - $definitions.TurnStartResponse = [ordered]@{ - '$schema' = $schema - title = 'TurnStartResponse' - type = 'object' - required = @('turn') - properties = [ordered]@{ turn = [ordered]@{ '$ref' = '#/definitions/Turn' } } - } - $definitions.ThreadReadParams = [ordered]@{ - '$schema' = $schema - title = 'ThreadReadParams' - type = 'object' - required = @('threadId') - properties = [ordered]@{ threadId = [ordered]@{ type = 'string' }; includeTurns = [ordered]@{ type = 'boolean' } } - } - $definitions.ThreadReadResponse = [ordered]@{ - '$schema' = $schema - title = 'ThreadReadResponse' - type = 'object' - required = @('thread') - properties = [ordered]@{ thread = [ordered]@{ '$ref' = '#/definitions/Thread' } } - } - $definitions.ModelReroutedNotification = [ordered]@{ - '$schema' = $schema - title = 'ModelReroutedNotification' - type = 'object' - required = @('fromModel', 'reason', 'threadId', 'toModel', 'turnId') - properties = [ordered]@{ - fromModel = [ordered]@{ type = 'string' } - reason = [ordered]@{ '$ref' = '#/definitions/ModelRerouteReason' } - threadId = [ordered]@{ type = 'string' } - toModel = [ordered]@{ type = 'string' } - turnId = [ordered]@{ type = 'string' } - } - } - - $schemaFiles = [ordered]@{ - 'codex_app_server_protocol.v2.schemas.json' = [ordered]@{ '$schema' = $schema; title = 'codex_app_server_protocol.v2.schemas'; type = 'object'; definitions = $definitions } - } + } + $definitions.ThreadStartParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartParams' + type = 'object' + properties = [ordered]@{ + model = [ordered]@{ type = @('string', 'null') } + cwd = [ordered]@{ type = @('string', 'null') } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandbox = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxMode' }, [ordered]@{ type = 'null' }) } + ephemeral = [ordered]@{ type = @('boolean', 'null') } + } + } + $definitions.ThreadStartResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartResponse' + type = 'object' + required = @('approvalPolicy', 'approvalsReviewer', 'cwd', 'model', 'modelProvider', 'sandbox', 'thread') + properties = [ordered]@{ + approvalPolicy = [ordered]@{ '$ref' = '#/definitions/AskForApproval' } + cwd = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } + instructionSources = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/LegacyAppPathString' } } + model = [ordered]@{ type = 'string' } + sandbox = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }) } + thread = [ordered]@{ '$ref' = '#/definitions/Thread' } + } + } + $definitions.TurnStartParams = [ordered]@{ + '$schema' = $schema + title = 'TurnStartParams' + type = 'object' + required = @('input', 'threadId') + properties = [ordered]@{ + threadId = [ordered]@{ type = 'string' } + input = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/UserInput' } } + cwd = [ordered]@{ type = @('string', 'null') } + model = [ordered]@{ type = @('string', 'null') } + effort = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/ReasoningEffort' }, [ordered]@{ type = 'null' }) } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandboxPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }, [ordered]@{ type = 'null' }) } + } + } + $definitions.TurnStartResponse = [ordered]@{ + '$schema' = $schema + title = 'TurnStartResponse' + type = 'object' + required = @('turn') + properties = [ordered]@{ turn = [ordered]@{ '$ref' = '#/definitions/Turn' } } + } + $definitions.ThreadReadParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadParams' + type = 'object' + required = @('threadId') + properties = [ordered]@{ threadId = [ordered]@{ type = 'string' }; includeTurns = [ordered]@{ type = 'boolean' } } + } + $definitions.ThreadReadResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadResponse' + type = 'object' + required = @('thread') + properties = [ordered]@{ thread = [ordered]@{ '$ref' = '#/definitions/Thread' } } + } + $definitions.ModelReroutedNotification = [ordered]@{ + '$schema' = $schema + title = 'ModelReroutedNotification' + type = 'object' + required = @('fromModel', 'reason', 'threadId', 'toModel', 'turnId') + properties = [ordered]@{ + fromModel = [ordered]@{ type = 'string' } + reason = [ordered]@{ '$ref' = '#/definitions/ModelRerouteReason' } + threadId = [ordered]@{ type = 'string' } + toModel = [ordered]@{ type = 'string' } + turnId = [ordered]@{ type = 'string' } + } + } + + $schemaFiles = [ordered]@{ + 'codex_app_server_protocol.v2.schemas.json' = [ordered]@{ '$schema' = $schema; title = 'codex_app_server_protocol.v2.schemas'; type = 'object'; definitions = $definitions } + } foreach ($schemaName in @('ConfigReadParams', 'ConfigReadResponse', 'SkillsListParams', 'SkillsListResponse', 'ThreadStartParams', 'ThreadStartResponse', 'TurnStartParams', 'TurnStartResponse', 'ThreadReadParams', 'ThreadReadResponse', 'ModelReroutedNotification')) { - $source = $definitions[$schemaName] - $individual = [ordered]@{ '$schema' = $schema } - foreach ($propertyName in @('title', 'type', 'properties', 'required')) { - if ($source.Contains($propertyName)) { $individual[$propertyName] = $source[$propertyName] } - } - $individual.definitions = $definitions - $schemaFiles[('v2\{0}.json' -f $schemaName)] = $individual - } - foreach ($schemaName in $schemaFiles.Keys) { - $schemaPath = Join-Path $SchemaDirectory $schemaName - New-Item -ItemType Directory -Path (Split-Path -Parent $schemaPath) -Force | Out-Null - [System.IO.File]::WriteAllText($schemaPath, ([string]($schemaFiles[$schemaName] | ConvertTo-Json -Depth 100)), [System.Text.UTF8Encoding]::new($false)) - } -} - -if ($arguments -contains '--version') { - Write-Output 'recorded-codex 9.9' - exit 0 -} -if ($arguments -contains '--help' -and -not ($arguments -contains 'app-server')) { + $source = $definitions[$schemaName] + $individual = [ordered]@{ '$schema' = $schema } + foreach ($propertyName in @('title', 'type', 'properties', 'required')) { + if ($source.Contains($propertyName)) { $individual[$propertyName] = $source[$propertyName] } + } + $individual.definitions = $definitions + $schemaFiles[('v2\{0}.json' -f $schemaName)] = $individual + } + foreach ($schemaName in $schemaFiles.Keys) { + $schemaPath = Join-Path $SchemaDirectory $schemaName + New-Item -ItemType Directory -Path (Split-Path -Parent $schemaPath) -Force | Out-Null + [System.IO.File]::WriteAllText($schemaPath, ([string]($schemaFiles[$schemaName] | ConvertTo-Json -Depth 100)), [System.Text.UTF8Encoding]::new($false)) + } +} + +if ($arguments -contains '--version') { + Write-Output 'recorded-codex 9.9' + exit 0 +} +if ($arguments -contains '--help' -and -not ($arguments -contains 'app-server')) { Write-Output '--ask-for-approval never --strict-config --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox danger-full-access --cd --model --config' - exit 0 -} -if ($arguments -contains 'features' -and $arguments -contains 'list') { - Write-Output 'multi_agent stable true' - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains '--help') { - Write-Output 'generate-json-schema' - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { - $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) - if ($outArgument.Count -eq 0) { exit 2 } - $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) - Write-CodexSchemas -SchemaDirectory $schemaDirectory - exit 0 -} -if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { - $initialize = Read-AppServerMessage - if ($mode -eq 'timeout') { - Start-Sleep -Milliseconds (Get-DelayMilliseconds 20) - exit 0 - } - - $repoAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' - $threadId = 'obs-thread' - $turnId = 'obs-turn' - $threadObject = [ordered]@{ - id = $threadId - sessionId = 'obs-session' - ephemeral = $true - cwd = (Get-Location).Path - cliVersion = '9.9' - createdAt = 1 - updatedAt = 1 - modelProvider = 'recorded-provider' - preview = $false - projectId = $null - source = 'startup' - status = [ordered]@{ type = 'idle' } - turns = @() - } - - Start-Sleep -Milliseconds (Get-DelayMilliseconds 3.2) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $initialize.id - result = [ordered]@{ serverInfo = [ordered]@{ name = 'observability-codex'; version = '9.9' } } - }) - - $null = Read-AppServerMessage + exit 0 +} +if ($arguments -contains 'features' -and $arguments -contains 'list') { + Write-Output 'multi_agent stable true' + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains '--help') { + Write-Output 'generate-json-schema' + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { + $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) + if ($outArgument.Count -eq 0) { exit 2 } + $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) + Write-CodexSchemas -SchemaDirectory $schemaDirectory + exit 0 +} +if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { + $initialize = Read-AppServerMessage + if ($mode -eq 'timeout') { + Start-Sleep -Milliseconds (Get-DelayMilliseconds 20) + exit 0 + } + + $repoAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' + $threadId = 'obs-thread' + $turnId = 'obs-turn' + $threadObject = [ordered]@{ + id = $threadId + sessionId = 'obs-session' + ephemeral = $true + cwd = (Get-Location).Path + cliVersion = '9.9' + createdAt = 1 + updatedAt = 1 + modelProvider = 'recorded-provider' + preview = $false + projectId = $null + source = 'startup' + status = [ordered]@{ type = 'idle' } + turns = @() + } + + Start-Sleep -Milliseconds (Get-DelayMilliseconds 3.2) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $initialize.id + result = [ordered]@{ serverInfo = [ordered]@{ name = 'observability-codex'; version = '9.9' } } + }) + + $null = Read-AppServerMessage $skillsList = Read-AppServerMessage $candidateName = 'candidate' $ambientCandidatePath = "C:\Users\some-user\.agents\skills\$candidateName\SKILL.md" @@ -547,650 +552,650 @@ if ($arguments -contains 'app-server' -and $arguments -contains '--stdio') { }) $threadStart = Read-AppServerMessage -AllowEndOfStream if ($null -eq $threadStart) { exit 0 } - Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.8) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $threadStart.id - result = [ordered]@{ - approvalPolicy = 'never' - approvalsReviewer = 'user' - cwd = (Get-Location).Path - model = [string]$threadStart.params.model - modelProvider = 'recorded-provider' - sandbox = [ordered]@{ type = 'readOnly' } - instructionSources = @($repoAgentsPath) - thread = $threadObject - } - }) - - $turnStart = Read-AppServerMessage - Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.4) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $turnStart.id - result = [ordered]@{ turn = [ordered]@{ id = $turnId; status = 'inProgress'; items = @() } } - }) - - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'thread/started' - params = [ordered]@{ thread = [ordered]@{ id = $threadId } } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'item/completed' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - completedAtMs = 1 - item = [ordered]@{ - type = 'commandExecution' - id = 'cmd-1' - command = 'echo observability' - commandActions = @() - cwd = (Get-Location).Path - status = 'completed' - exitCode = 0 - aggregatedOutput = 'OBSERVABILITY_PROTOCOL_OUTPUT_CANARY' - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'item/completed' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - completedAtMs = 2 - item = [ordered]@{ - type = 'agentMessage' - id = 'message-1' - text = 'OBSERVABILITY_MODEL_CONTENT_CANARY' - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'thread/tokenUsage/updated' - params = [ordered]@{ - threadId = $threadId - turnId = $turnId - tokenUsage = [ordered]@{ - total = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } - last = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } - } - } - }) - Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - method = 'turn/completed' - params = [ordered]@{ - threadId = $threadId - turn = [ordered]@{ id = $turnId; status = 'completed'; items = @() } - } - }) - - $threadRead = Read-AppServerMessage - Write-AppServerMessage ([ordered]@{ - jsonrpc = '2.0' - id = $threadRead.id - result = [ordered]@{ thread = $threadObject } - }) - exit 0 -} - -exit 0 -'@, [System.Text.UTF8Encoding]::new($false)) - - $runRoot = Join-Path $Root 'codex-run' - $repoRoot = Join-Path $runRoot 'repo' - $homeRoot = Join-Path $runRoot 'home' - New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null - if ($Mode -eq 'timeout') { - [System.IO.File]::WriteAllText((Join-Path $homeRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $repoRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) - } - [System.IO.File]::WriteAllText((Join-Path $repoRoot 'AGENTS.md'), '# codex observability repo instruction', [System.Text.UTF8Encoding]::new($false)) - [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), 'codex observability prompt', [System.Text.UTF8Encoding]::new($false)) - $runPath = Join-Path $runRoot 'run.json' - Write-TestJson -Path $runPath -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Run - evalId = 99 - evalName = 'codex-observability' + Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.8) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $threadStart.id + result = [ordered]@{ + approvalPolicy = 'never' + approvalsReviewer = 'user' + cwd = (Get-Location).Path + model = [string]$threadStart.params.model + modelProvider = 'recorded-provider' + sandbox = [ordered]@{ type = 'readOnly' } + instructionSources = @($repoAgentsPath) + thread = $threadObject + } + }) + + $turnStart = Read-AppServerMessage + Start-Sleep -Milliseconds (Get-DelayMilliseconds 2.4) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $turnStart.id + result = [ordered]@{ turn = [ordered]@{ id = $turnId; status = 'inProgress'; items = @() } } + }) + + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'thread/started' + params = [ordered]@{ thread = [ordered]@{ id = $threadId } } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'item/completed' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + completedAtMs = 1 + item = [ordered]@{ + type = 'commandExecution' + id = 'cmd-1' + command = 'echo observability' + commandActions = @() + cwd = (Get-Location).Path + status = 'completed' + exitCode = 0 + aggregatedOutput = 'OBSERVABILITY_PROTOCOL_OUTPUT_CANARY' + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'item/completed' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + completedAtMs = 2 + item = [ordered]@{ + type = 'agentMessage' + id = 'message-1' + text = 'OBSERVABILITY_MODEL_CONTENT_CANARY' + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'thread/tokenUsage/updated' + params = [ordered]@{ + threadId = $threadId + turnId = $turnId + tokenUsage = [ordered]@{ + total = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } + last = [ordered]@{ inputTokens = 2; cachedInputTokens = 0; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } + } + } + }) + Start-Sleep -Milliseconds (Get-DelayMilliseconds 0.5) + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + method = 'turn/completed' + params = [ordered]@{ + threadId = $threadId + turn = [ordered]@{ id = $turnId; status = 'completed'; items = @() } + } + }) + + $threadRead = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ + jsonrpc = '2.0' + id = $threadRead.id + result = [ordered]@{ thread = $threadObject } + }) + exit 0 +} + +exit 0 +'@, [System.Text.UTF8Encoding]::new($false)) + + $runRoot = Join-Path $Root 'codex-run' + $repoRoot = Join-Path $runRoot 'repo' + $homeRoot = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repoRoot, $homeRoot -Force | Out-Null + if ($Mode -eq 'timeout') { + [System.IO.File]::WriteAllText((Join-Path $homeRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repoRoot 'codex-observability-timeout'), '1', [System.Text.UTF8Encoding]::new($false)) + } + [System.IO.File]::WriteAllText((Join-Path $repoRoot 'AGENTS.md'), '# codex observability repo instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runRoot 'prompt.md'), 'codex observability prompt', [System.Text.UTF8Encoding]::new($false)) + $runPath = Join-Path $runRoot 'run.json' + Write-TestJson -Path $runPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 99 + evalName = 'codex-observability' candidateSkillName = 'candidate' - skillName = $null - iteration = 1 - mode = 'without_skill' - promptFile = 'prompt.md' - workingDirectory = 'repo' - homeDirectory = 'home' - skillDirectory = $null - freshContextRequired = $true - filesystemIsolationRequired = $true - isolatedHomeRequired = $true - fixtureHash = ('c' * 64) - }) - $profilePath = Join-Path $Root 'execution-profile.json' - Write-TestJson -Path $profilePath -Value ([ordered]@{ - schema = (Get-RunnerSchemaNames).Profile - runner = 'codex' - model = 'gpt-5.6-luna' - reasoning_effort = 'medium' - configuration_profile = 'isolated-default' - tool_profile = 'default' - timeout_seconds = $TimeoutSeconds - concurrency = 1 - }) - - $ambientCodexHome = Join-Path $Root 'ambient-codex-home' - New-Item -ItemType Directory -Path $ambientCodexHome -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $ambientCodexHome 'auth.json'), '{"access_token":"fixture"}', [System.Text.UTF8Encoding]::new($false)) - - return [pscustomobject]@{ - Root = $Root - FakeBin = $fakeBin - RunRoot = $runRoot - RunPath = $runPath - ProfilePath = $profilePath - AmbientCodexHome = $ambientCodexHome - RunnerStderrPath = Join-Path $Root 'runner.stderr' - } -} - -function Invoke-CodexAppServerFixture { - param( - [ValidateSet('success', 'timeout')][string]$Mode, - [double]$HeartbeatSeconds = 0.15, - [int]$TimeoutSeconds = 5 - ) - - $fixture = New-CodexObservabilityFixture -Root (Join-Path $testRoot ('codex-app-server-' + $Mode + '-' + [Guid]::NewGuid().ToString('N'))) -Mode $Mode -TimeoutSeconds $TimeoutSeconds - $runnerPath = Join-Path $runnerRoot 'codex\runner.ps1' - $previousPath = [Environment]::GetEnvironmentVariable('PATH') - $previousCodexHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') - $previousProgress = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') - $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') - $previousOpenAiKey = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') - $heartbeatText = $HeartbeatSeconds.ToString([Globalization.CultureInfo]::InvariantCulture) - $clock = [System.Diagnostics.Stopwatch]::StartNew() - try { - [Environment]::SetEnvironmentVariable('PATH', $fixture.FakeBin + [System.IO.Path]::PathSeparator + $previousPath) - [Environment]::SetEnvironmentVariable('CODEX_HOME', $fixture.AmbientCodexHome) - [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $null) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $heartbeatText) + skillName = $null + iteration = 1 + mode = 'without_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $null + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + fixtureHash = ('c' * 64) + }) + $profilePath = Join-Path $Root 'execution-profile.json' + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'codex' + model = 'gpt-5.6-luna' + reasoning_effort = 'medium' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = $TimeoutSeconds + concurrency = 1 + }) + + $ambientCodexHome = Join-Path $Root 'ambient-codex-home' + New-Item -ItemType Directory -Path $ambientCodexHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $ambientCodexHome 'auth.json'), '{"access_token":"fixture"}', [System.Text.UTF8Encoding]::new($false)) + + return [pscustomobject]@{ + Root = $Root + FakeBin = $fakeBin + RunRoot = $runRoot + RunPath = $runPath + ProfilePath = $profilePath + AmbientCodexHome = $ambientCodexHome + RunnerStderrPath = Join-Path $Root 'runner.stderr' + } +} + +function Invoke-CodexAppServerFixture { + param( + [ValidateSet('success', 'timeout')][string]$Mode, + [double]$HeartbeatSeconds = 0.15, + [int]$TimeoutSeconds = 5 + ) + + $fixture = New-CodexObservabilityFixture -Root (Join-Path $testRoot ('codex-app-server-' + $Mode + '-' + [Guid]::NewGuid().ToString('N'))) -Mode $Mode -TimeoutSeconds $TimeoutSeconds + $runnerPath = Join-Path $runnerRoot 'codex\runner.ps1' + $previousPath = [Environment]::GetEnvironmentVariable('PATH') + $previousCodexHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $previousProgress = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') + $previousHeartbeat = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS') + $previousOpenAiKey = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') + $heartbeatText = $HeartbeatSeconds.ToString([Globalization.CultureInfo]::InvariantCulture) + $clock = [System.Diagnostics.Stopwatch]::StartNew() + try { + [Environment]::SetEnvironmentVariable('PATH', $fixture.FakeBin + [System.IO.Path]::PathSeparator + $previousPath) + [Environment]::SetEnvironmentVariable('CODEX_HOME', $fixture.AmbientCodexHome) + [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $null) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $heartbeatText) $output = & $pwshPath -NoProfile -NonInteractive -File $runnerPath execute -Run $fixture.RunPath -Profile $fixture.ProfilePath 2>$fixture.RunnerStderrPath - $exitCode = $LASTEXITCODE - } finally { - $clock.Stop() - [Environment]::SetEnvironmentVariable('PATH', $previousPath) - [Environment]::SetEnvironmentVariable('CODEX_HOME', $previousCodexHome) - [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $previousOpenAiKey) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousProgress) - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) - } - - $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) - $stderr = if (Test-Path -LiteralPath $fixture.RunnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($fixture.RunnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } - $progressEvents = @(Get-RelayedProgressEventsFromText -Text $stderr) - return [pscustomobject]@{ - Fixture = $fixture - ExitCode = $exitCode - ElapsedSeconds = [Math]::Round($clock.Elapsed.TotalSeconds, 3) - Stdout = $stdout - Stderr = $stderr - Result = $result - ProgressEvents = $progressEvents - AppServerEvents = @($progressEvents | Where-Object { [string]$_.phase -eq 'codex-app-server' }) - RawEventsPath = Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl' - RawEventsText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl'), [System.Text.UTF8Encoding]::new($false)) } else { '' } - RawStderrPath = Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt' - RawStderrText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt'), [System.Text.UTF8Encoding]::new($false)) } else { '' } - } -} - -try { - # Pre-warm the shared activity type so its one-time JIT/compile cost is paid - # before any timed scenario. Otherwise, under heavy machine load, that cost - # inflates the first child's launch timestamp and compresses its heartbeat - # window, making timing-sensitive assertions flaky. - Initialize-RunnerActivityType - - # ------------------------------------------------------------------ - # Test 1 - quiet but alive: a healthy but silent process must still be - # visibly alive. Heartbeats appear even though the process emits nothing. - # ------------------------------------------------------------------ - $quietScript = New-SyntheticChildScript -Name 'quiet' -Body @' -Start-Sleep -Milliseconds 1500 -[Console]::Out.Write('{"status":"completed"}') -'@ - $quiet = Invoke-SyntheticChild -ScriptPath $quietScript -WorkerId 'arm-1-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $quiet.ExitCode 'quiet-but-alive child exits cleanly' - $quietHeartbeats = @($quiet.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' -and $null -ne $_.PSObject.Properties['pid'] }) - Assert-True ($quietHeartbeats.Count -ge 2) "a quiet but alive process still emits heartbeats (got $($quietHeartbeats.Count))" - $quietElapsed = @($quiet.Events | Where-Object { $null -ne $_.PSObject.Properties['elapsedSeconds'] } | ForEach-Object { [double]$_.elapsedSeconds }) - $quietElapsedSpan = (($quietElapsed | Measure-Object -Maximum).Maximum) - (($quietElapsed | Measure-Object -Minimum).Minimum) - Assert-True ($quietElapsedSpan -ge 0.2) "quiet progress shows elapsed runtime advancing (span=$([Math]::Round($quietElapsedSpan,3))s)" - Assert-True (@($quiet.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ne 0 }).Count -eq 0) 'quiet process reports zero real stderr activity' - Assert-True (@($quiet.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'quiet process reports a terminal completed state' - - # ------------------------------------------------------------------ - # Test 2 - active process: periodic output is observable as live activity - # without corrupting the captured machine result or leaking into it. - # ------------------------------------------------------------------ - $activeScript = New-SyntheticChildScript -Name 'active' -Body @' -for ($i = 0; $i -lt 4; $i++) { - [Console]::Error.WriteLine("chunk $i produced") - Start-Sleep -Milliseconds 250 -} -[Console]::Error.WriteLine('@@AGENTIC-PROGRESS@@ {"state":"active","phase":"model-cli","detail":"turn-1","stdoutEvents":3}') -[Console]::Out.Write('{"status":"completed"}') -'@ - $active = Invoke-SyntheticChild -ScriptPath $activeScript -WorkerId 'arm-2-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $active.ExitCode 'active child exits cleanly' - Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $active.StdoutPath -Raw).Trim()) 'active child result STDOUT is captured exactly, uncorrupted by activity' - Assert-True (@($active.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ge 1 }).Count -ge 1) 'active process real stderr activity is observable in heartbeats' - $activeWithAge = @($active.Events | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) - Assert-True ($activeWithAge.Count -ge 1) 'active process reports the age of its most recent activity' - $relayed = @($active.Events | Where-Object { [string]$_.origin -eq 'relay' }) - Assert-True ($relayed.Count -ge 1) 'structured child progress is relayed through the parent' - Assert-Equal 'arm-2-with_skill' ([string]$relayed[0].worker) 'a relayed event is attributed to the emitting worker' - Assert-Equal 'turn-1' ([string]$relayed[0].detail) 'a relayed event preserves the safe child-provided detail' - - # ------------------------------------------------------------------ - # Test 3 - hanging process: the watchdog must fire, terminate the child, - # and produce a final diagnostic. There must be no indefinite hang. - # ------------------------------------------------------------------ - $hangScript = New-SyntheticChildScript -Name 'hang' -Body @' -while ($true) { Start-Sleep -Milliseconds 150 } -'@ - $hangClock = [System.Diagnostics.Stopwatch]::StartNew() - $hang = Invoke-SyntheticChild -ScriptPath $hangScript -WorkerId 'arm-3-with_skill' -TimeoutSeconds 2 -HeartbeatSeconds 0.3 - $hangClock.Stop() - Assert-True ($hangClock.Elapsed.TotalSeconds -lt 15) ("hanging process reaches a terminal state promptly; elapsed={0:N2}s" -f $hangClock.Elapsed.TotalSeconds) - Assert-True ([bool]$hang.Child.TimedOut) 'hanging process is reported as timed out' - Assert-True ([bool]$hang.Child.TerminationObserved) 'hanging process is actually terminated' - Assert-True ($null -eq $hang.ExitCode) 'timed-out child has no synthesized success exit code' - $hangHeartbeats = @($hang.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) - Assert-True ($hangHeartbeats.Count -ge 2) 'heartbeats continue while a process hangs' - $timedOut = @($hang.Events | Where-Object { [string]$_.state -eq 'timed-out' }) - Assert-Equal 1 $timedOut.Count 'a hanging process produces exactly one timed-out diagnostic' - Assert-True ([bool]$timedOut[0].terminationObserved) 'the timeout diagnostic records that termination was observed' - Assert-True ($null -ne $timedOut[0].PSObject.Properties['elapsedSeconds'] -and [double]$timedOut[0].elapsedSeconds -gt 0) 'the timeout diagnostic records elapsed runtime' - Assert-True (@($hang.Events | Where-Object { [string]$_.state -eq 'terminating' }).Count -ge 1) 'a terminating lifecycle state precedes termination' - - # ------------------------------------------------------------------ - # Test 6 - fast successful runner: observability must not disturb a normal - # short execution or its captured result. - # ------------------------------------------------------------------ - $fastScript = New-SyntheticChildScript -Name 'fast' -Body @' -[Console]::Out.Write('{"status":"completed"}') -'@ - $fast = Invoke-SyntheticChild -ScriptPath $fastScript -WorkerId 'arm-6-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 0 $fast.ExitCode 'fast successful child exits cleanly' - Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $fast.StdoutPath -Raw).Trim()) 'fast child result is captured exactly' - Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'running' -and $null -ne $_.PSObject.Properties['pid'] }).Count -ge 1) 'fast child still records a launch lifecycle event' - Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'fast child records exactly one completed state' - - # ------------------------------------------------------------------ - # Test 7 - failure diagnostics: a runner that fails after some activity must - # yield a diagnostic identifying meaningful last-known state. - # ------------------------------------------------------------------ - $failScript = New-SyntheticChildScript -Name 'fail' -Body @' -[Console]::Error.WriteLine('preparing request') -Start-Sleep -Milliseconds 400 -[Console]::Error.WriteLine('harness aborted unexpectedly') -exit 17 -'@ - $fail = Invoke-SyntheticChild -ScriptPath $failScript -WorkerId 'arm-7-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - Assert-Equal 17 $fail.ExitCode 'failing child reports its real non-zero exit code' - $failed = @($fail.Events | Where-Object { [string]$_.state -eq 'failed' }) - Assert-Equal 1 $failed.Count 'a failing child produces exactly one failure diagnostic' - Assert-Equal 17 ([int]$failed[0].exitCode) 'the failure diagnostic records the exit code' - Assert-True ([int64](Get-Field $failed[0] 'stderrEvents' 0) -ge 1) 'the failure diagnostic records observed stderr activity' - Assert-Equal 'process exited with non-zero status' ([string]$failed[0].detail) 'the failure diagnostic keeps a structured detail only' - Assert-True ([bool](Get-Field $failed[0] 'outputDrainCompleted' $false)) 'the failure diagnostic records bounded output draining' - $failLogText = if (Test-Path -LiteralPath $fail.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($fail.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-True (-not $failLogText.Contains('harness aborted unexpectedly')) 'raw stderr text is absent from persisted progress diagnostics' - Assert-True (-not ([string]$fail.OperatorStderr).Contains('harness aborted unexpectedly')) 'raw stderr text is absent from live operator diagnostics' - Assert-True ((Get-Content -LiteralPath $fail.StderrPath -Raw).Contains('harness aborted unexpectedly')) 'raw stderr evidence remains available in its file' - - # ------------------------------------------------------------------ - # Test 8 - sensitive value hygiene: a recognizable secret in the process - # environment and stderr must never surface in operator progress output. - # ------------------------------------------------------------------ - $secret = 'topsecret-' + [Guid]::NewGuid().ToString('N') - $oldSecret = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET') - [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $secret) - try { - $secretScript = New-SyntheticChildScript -Name 'secret' -Body @' -[Console]::Error.WriteLine('AUTH_TOKEN=' + [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET')) -Start-Sleep -Milliseconds 300 -exit 9 -'@ - $secretRun = Invoke-SyntheticChild -ScriptPath $secretScript -WorkerId 'arm-8-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $oldSecret) - } - $secretLog = if (Test-Path -LiteralPath $secretRun.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($secretRun.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-True (-not $secretLog.Contains($secret)) 'the raw secret value never appears in persisted progress output' - Assert-True (-not ([string]$secretRun.OperatorStderr).Contains($secret)) 'the raw secret value never appears in live operator progress' - $secretFailed = @($secretRun.Events | Where-Object { [string]$_.state -eq 'failed' }) - Assert-Equal 1 $secretFailed.Count 'the secret-bearing child still produces a failure diagnostic' - Assert-Equal 'process exited with non-zero status' ([string]$secretFailed[0].detail) 'secret-bearing failures keep structured detail only' - Assert-True ([bool](Test-Path -LiteralPath $secretRun.StderrPath -PathType Leaf)) 'raw child stderr evidence is still captured to its file' - Assert-True ((Get-Content -LiteralPath $secretRun.StderrPath -Raw).Contains($secret)) 'raw child stderr evidence still retains the secret for deliberate forensic inspection' - - # ------------------------------------------------------------------ - # Tests 4 and 5 - one real runner-owned fan-out run over the deterministic - # fixture proves concurrent-arm attribution, the STDOUT machine contract, - # and persisted progress evidence, all on the true operator-facing path. - # ------------------------------------------------------------------ - $fanoutResult = Invoke-ObservabilityFanout -Root (Join-Path $testRoot 'fanout') -EvalCount 2 -Concurrency 4 -DelayMs 700 - Assert-Equal 0 $fanoutResult.ExitCode 'observability fan-out completes successfully' - - # Test 5 - the STDOUT contract: exactly one machine-readable terminal JSON. - $stdoutTrimmed = ([string]$fanoutResult.Stdout).Trim() - $terminal = $null - $terminal = $stdoutTrimmed | ConvertFrom-Json -Depth 100 - Assert-Equal 'phase1' ([string]$terminal.phase) 'STDOUT still carries exactly one machine-readable terminal summary' - Assert-Equal 'completed' ([string]$terminal.status) 'the terminal summary reports completion' - Assert-True (-not $stdoutTrimmed.Contains((Get-RunnerProgressSentinel))) 'no relay sentinel ever leaks onto STDOUT' - Assert-True (-not ($stdoutTrimmed -match '(?m)^\[synthetic\]|(?m)^\[fixture\]')) 'no operator progress line contaminates STDOUT' - Assert-True ($stdoutTrimmed.StartsWith('{') -and $stdoutTrimmed.EndsWith('}')) 'STDOUT is a single JSON object with no surrounding progress text' - - # Progress is also persisted for post-mortem inspection at the advertised path. - Assert-Equal 'progress/phase1-progress.jsonl' ([string]$terminal.progress_log) 'the terminal summary advertises where progress is persisted' - $persistedLog = Join-Path $fanoutResult.IterationDirectory ([string]$terminal.progress_log) - Assert-True (Test-Path -LiteralPath $persistedLog -PathType Leaf) 'the persisted progress log exists at the advertised path' - $persistedEvents = @(Get-Content -LiteralPath $persistedLog | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-True ($persistedEvents.Count -ge 4) 'the persisted progress log retains events for post-mortem inspection' - - # Test 4 - concurrent-arm attribution: every progress line is attributable. - $stderrLines = @(([string]$fanoutResult.Stderr) -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) - $progressLines = @($stderrLines | Where-Object { $_ -match '^\[(synthetic|fixture)\]' }) - Assert-True ($progressLines.Count -ge 2) 'the concurrent fan-out emits live operator progress on STDERR' - $unlabeled = @($progressLines | Where-Object { -not ($_ -match '\[arm-\d+-(with_skill|without_skill)\]') }) - Assert-Equal 0 $unlabeled.Count 'every operator progress line carries a resolvable worker identity' - $workerIds = @($persistedEvents | ForEach-Object { [string]$_.worker } | Where-Object { $_ -match '^arm-\d+-' } | Sort-Object -Unique) - Assert-True ($workerIds.Count -ge 2) "concurrent arms are individually attributable (distinct workers: $($workerIds.Count))" - foreach ($workerId in @('arm-1-with_skill', 'arm-1-without_skill', 'arm-2-with_skill', 'arm-2-without_skill')) { - Assert-True ($workerIds -contains $workerId) "progress is attributed to worker $workerId" - } - # Attribution must be exclusive: an event's fields belong to exactly its worker. - $misattributed = @($persistedEvents | Where-Object { - [string]$_.state -in @('running', 'active', 'completed', 'timed-out', 'failed') -and - $null -ne $_.PSObject.Properties['configuration'] -and - -not ([string]$_.worker).EndsWith([string]$_.configuration) - }) - Assert-Equal 0 $misattributed.Count 'no progress event mixes one worker identity with another configuration' - - # ------------------------------------------------------------------ - # Test 9 - the shared process primitive: runner model-CLI progress is opt-in - # via the orchestration environment, relays through STDERR, and never - # contaminates the captured model result on STDOUT. - # ------------------------------------------------------------------ - Assert-True ($null -eq (Get-RunnerModelProgressContext -Runner 'opencode')) 'runner model-CLI progress is silent without the orchestration flag' - $previousFlag = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') - try { - $enabledContext = Get-RunnerModelProgressContext -Runner 'opencode' -Phase 'opencode-cli' - Assert-True ($null -ne $enabledContext) 'the orchestration flag enables a runner model-CLI progress context' - Assert-Equal 'Relayable' ([string]$enabledContext['channel']) 'runner model-CLI progress uses the relayable channel' - Assert-Equal 'opencode' ([string]$enabledContext['runner']) 'runner model-CLI progress context carries the runner identity' - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousFlag) - } - - $primitiveDriver = New-SyntheticChildScript -Name 'primitive-driver' -Body @' -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') -$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source -$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.3 } -$grandchild = 'Start-Sleep -Milliseconds 900; [Console]::Out.Write(''grandchild-done'')' -$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchild) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx -[Console]::Out.Write([string]$r.Stdout) -'@ - $driverStderrPath = Join-Path $testRoot 'primitive.stderr' - $previousRoot = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - try { + $exitCode = $LASTEXITCODE + } finally { + $clock.Stop() + [Environment]::SetEnvironmentVariable('PATH', $previousPath) + [Environment]::SetEnvironmentVariable('CODEX_HOME', $previousCodexHome) + [Environment]::SetEnvironmentVariable('OPENAI_API_KEY', $previousOpenAiKey) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousProgress) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_HEARTBEAT_SECONDS', $previousHeartbeat) + } + + $stdout = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $stderr = if (Test-Path -LiteralPath $fixture.RunnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($fixture.RunnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $result = if ([string]::IsNullOrWhiteSpace($stdout)) { $null } else { $stdout | ConvertFrom-Json -Depth 100 } + $progressEvents = @(Get-RelayedProgressEventsFromText -Text $stderr) + return [pscustomobject]@{ + Fixture = $fixture + ExitCode = $exitCode + ElapsedSeconds = [Math]::Round($clock.Elapsed.TotalSeconds, 3) + Stdout = $stdout + Stderr = $stderr + Result = $result + ProgressEvents = $progressEvents + AppServerEvents = @($progressEvents | Where-Object { [string]$_.phase -eq 'codex-app-server' }) + RawEventsPath = Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl' + RawEventsText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-app-server-events.jsonl'), [System.Text.UTF8Encoding]::new($false)) } else { '' } + RawStderrPath = Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt' + RawStderrText = if (Test-Path -LiteralPath (Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt') -PathType Leaf) { [System.IO.File]::ReadAllText((Join-Path $fixture.RunRoot 'evidence\codex-stderr.txt'), [System.Text.UTF8Encoding]::new($false)) } else { '' } + } +} + +try { + # Pre-warm the shared activity type so its one-time JIT/compile cost is paid + # before any timed scenario. Otherwise, under heavy machine load, that cost + # inflates the first child's launch timestamp and compresses its heartbeat + # window, making timing-sensitive assertions flaky. + Initialize-RunnerActivityType + + # ------------------------------------------------------------------ + # Test 1 - quiet but alive: a healthy but silent process must still be + # visibly alive. Heartbeats appear even though the process emits nothing. + # ------------------------------------------------------------------ + $quietScript = New-SyntheticChildScript -Name 'quiet' -Body @' +Start-Sleep -Milliseconds 1500 +[Console]::Out.Write('{"status":"completed"}') +'@ + $quiet = Invoke-SyntheticChild -ScriptPath $quietScript -WorkerId 'arm-1-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $quiet.ExitCode 'quiet-but-alive child exits cleanly' + $quietHeartbeats = @($quiet.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' -and $null -ne $_.PSObject.Properties['pid'] }) + Assert-True ($quietHeartbeats.Count -ge 2) "a quiet but alive process still emits heartbeats (got $($quietHeartbeats.Count))" + $quietElapsed = @($quiet.Events | Where-Object { $null -ne $_.PSObject.Properties['elapsedSeconds'] } | ForEach-Object { [double]$_.elapsedSeconds }) + $quietElapsedSpan = (($quietElapsed | Measure-Object -Maximum).Maximum) - (($quietElapsed | Measure-Object -Minimum).Minimum) + Assert-True ($quietElapsedSpan -ge 0.2) "quiet progress shows elapsed runtime advancing (span=$([Math]::Round($quietElapsedSpan,3))s)" + Assert-True (@($quiet.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ne 0 }).Count -eq 0) 'quiet process reports zero real stderr activity' + Assert-True (@($quiet.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'quiet process reports a terminal completed state' + + # ------------------------------------------------------------------ + # Test 2 - active process: periodic output is observable as live activity + # without corrupting the captured machine result or leaking into it. + # ------------------------------------------------------------------ + $activeScript = New-SyntheticChildScript -Name 'active' -Body @' +for ($i = 0; $i -lt 4; $i++) { + [Console]::Error.WriteLine("chunk $i produced") + Start-Sleep -Milliseconds 250 +} +[Console]::Error.WriteLine('@@AGENTIC-PROGRESS@@ {"state":"active","phase":"model-cli","detail":"turn-1","stdoutEvents":3}') +[Console]::Out.Write('{"status":"completed"}') +'@ + $active = Invoke-SyntheticChild -ScriptPath $activeScript -WorkerId 'arm-2-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $active.ExitCode 'active child exits cleanly' + Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $active.StdoutPath -Raw).Trim()) 'active child result STDOUT is captured exactly, uncorrupted by activity' + Assert-True (@($active.Events | Where-Object { [int64](Get-Field $_ 'stderrEvents' 0) -ge 1 }).Count -ge 1) 'active process real stderr activity is observable in heartbeats' + $activeWithAge = @($active.Events | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) + Assert-True ($activeWithAge.Count -ge 1) 'active process reports the age of its most recent activity' + $relayed = @($active.Events | Where-Object { [string]$_.origin -eq 'relay' }) + Assert-True ($relayed.Count -ge 1) 'structured child progress is relayed through the parent' + Assert-Equal 'arm-2-with_skill' ([string]$relayed[0].worker) 'a relayed event is attributed to the emitting worker' + Assert-Equal 'turn-1' ([string]$relayed[0].detail) 'a relayed event preserves the safe child-provided detail' + + # ------------------------------------------------------------------ + # Test 3 - hanging process: the watchdog must fire, terminate the child, + # and produce a final diagnostic. There must be no indefinite hang. + # ------------------------------------------------------------------ + $hangScript = New-SyntheticChildScript -Name 'hang' -Body @' +while ($true) { Start-Sleep -Milliseconds 150 } +'@ + $hangClock = [System.Diagnostics.Stopwatch]::StartNew() + $hang = Invoke-SyntheticChild -ScriptPath $hangScript -WorkerId 'arm-3-with_skill' -TimeoutSeconds 2 -HeartbeatSeconds 0.3 + $hangClock.Stop() + Assert-True ($hangClock.Elapsed.TotalSeconds -lt 15) ("hanging process reaches a terminal state promptly; elapsed={0:N2}s" -f $hangClock.Elapsed.TotalSeconds) + Assert-True ([bool]$hang.Child.TimedOut) 'hanging process is reported as timed out' + Assert-True ([bool]$hang.Child.TerminationObserved) 'hanging process is actually terminated' + Assert-True ($null -eq $hang.ExitCode) 'timed-out child has no synthesized success exit code' + $hangHeartbeats = @($hang.Events | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) + Assert-True ($hangHeartbeats.Count -ge 2) 'heartbeats continue while a process hangs' + $timedOut = @($hang.Events | Where-Object { [string]$_.state -eq 'timed-out' }) + Assert-Equal 1 $timedOut.Count 'a hanging process produces exactly one timed-out diagnostic' + Assert-True ([bool]$timedOut[0].terminationObserved) 'the timeout diagnostic records that termination was observed' + Assert-True ($null -ne $timedOut[0].PSObject.Properties['elapsedSeconds'] -and [double]$timedOut[0].elapsedSeconds -gt 0) 'the timeout diagnostic records elapsed runtime' + Assert-True (@($hang.Events | Where-Object { [string]$_.state -eq 'terminating' }).Count -ge 1) 'a terminating lifecycle state precedes termination' + + # ------------------------------------------------------------------ + # Test 6 - fast successful runner: observability must not disturb a normal + # short execution or its captured result. + # ------------------------------------------------------------------ + $fastScript = New-SyntheticChildScript -Name 'fast' -Body @' +[Console]::Out.Write('{"status":"completed"}') +'@ + $fast = Invoke-SyntheticChild -ScriptPath $fastScript -WorkerId 'arm-6-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 0 $fast.ExitCode 'fast successful child exits cleanly' + Assert-Equal '{"status":"completed"}' ((Get-Content -LiteralPath $fast.StdoutPath -Raw).Trim()) 'fast child result is captured exactly' + Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'running' -and $null -ne $_.PSObject.Properties['pid'] }).Count -ge 1) 'fast child still records a launch lifecycle event' + Assert-True (@($fast.Events | Where-Object { [string]$_.state -eq 'completed' }).Count -eq 1) 'fast child records exactly one completed state' + + # ------------------------------------------------------------------ + # Test 7 - failure diagnostics: a runner that fails after some activity must + # yield a diagnostic identifying meaningful last-known state. + # ------------------------------------------------------------------ + $failScript = New-SyntheticChildScript -Name 'fail' -Body @' +[Console]::Error.WriteLine('preparing request') +Start-Sleep -Milliseconds 400 +[Console]::Error.WriteLine('harness aborted unexpectedly') +exit 17 +'@ + $fail = Invoke-SyntheticChild -ScriptPath $failScript -WorkerId 'arm-7-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + Assert-Equal 17 $fail.ExitCode 'failing child reports its real non-zero exit code' + $failed = @($fail.Events | Where-Object { [string]$_.state -eq 'failed' }) + Assert-Equal 1 $failed.Count 'a failing child produces exactly one failure diagnostic' + Assert-Equal 17 ([int]$failed[0].exitCode) 'the failure diagnostic records the exit code' + Assert-True ([int64](Get-Field $failed[0] 'stderrEvents' 0) -ge 1) 'the failure diagnostic records observed stderr activity' + Assert-Equal 'process exited with non-zero status' ([string]$failed[0].detail) 'the failure diagnostic keeps a structured detail only' + Assert-True ([bool](Get-Field $failed[0] 'outputDrainCompleted' $false)) 'the failure diagnostic records bounded output draining' + $failLogText = if (Test-Path -LiteralPath $fail.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($fail.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-True (-not $failLogText.Contains('harness aborted unexpectedly')) 'raw stderr text is absent from persisted progress diagnostics' + Assert-True (-not ([string]$fail.OperatorStderr).Contains('harness aborted unexpectedly')) 'raw stderr text is absent from live operator diagnostics' + Assert-True ((Get-Content -LiteralPath $fail.StderrPath -Raw).Contains('harness aborted unexpectedly')) 'raw stderr evidence remains available in its file' + + # ------------------------------------------------------------------ + # Test 8 - sensitive value hygiene: a recognizable secret in the process + # environment and stderr must never surface in operator progress output. + # ------------------------------------------------------------------ + $secret = 'topsecret-' + [Guid]::NewGuid().ToString('N') + $oldSecret = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET') + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $secret) + try { + $secretScript = New-SyntheticChildScript -Name 'secret' -Body @' +[Console]::Error.WriteLine('AUTH_TOKEN=' + [Environment]::GetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET')) +Start-Sleep -Milliseconds 300 +exit 9 +'@ + $secretRun = Invoke-SyntheticChild -ScriptPath $secretScript -WorkerId 'arm-8-with_skill' -TimeoutSeconds 30 -HeartbeatSeconds 0.3 + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_FAKE_SECRET', $oldSecret) + } + $secretLog = if (Test-Path -LiteralPath $secretRun.LogPath -PathType Leaf) { [System.IO.File]::ReadAllText($secretRun.LogPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-True (-not $secretLog.Contains($secret)) 'the raw secret value never appears in persisted progress output' + Assert-True (-not ([string]$secretRun.OperatorStderr).Contains($secret)) 'the raw secret value never appears in live operator progress' + $secretFailed = @($secretRun.Events | Where-Object { [string]$_.state -eq 'failed' }) + Assert-Equal 1 $secretFailed.Count 'the secret-bearing child still produces a failure diagnostic' + Assert-Equal 'process exited with non-zero status' ([string]$secretFailed[0].detail) 'secret-bearing failures keep structured detail only' + Assert-True ([bool](Test-Path -LiteralPath $secretRun.StderrPath -PathType Leaf)) 'raw child stderr evidence is still captured to its file' + Assert-True ((Get-Content -LiteralPath $secretRun.StderrPath -Raw).Contains($secret)) 'raw child stderr evidence still retains the secret for deliberate forensic inspection' + + # ------------------------------------------------------------------ + # Tests 4 and 5 - one real runner-owned fan-out run over the deterministic + # fixture proves concurrent-arm attribution, the STDOUT machine contract, + # and persisted progress evidence, all on the true operator-facing path. + # ------------------------------------------------------------------ + $fanoutResult = Invoke-ObservabilityFanout -Root (Join-Path $testRoot 'fanout') -EvalCount 2 -Concurrency 4 -DelayMs 700 + Assert-Equal 0 $fanoutResult.ExitCode 'observability fan-out completes successfully' + + # Test 5 - the STDOUT contract: exactly one machine-readable terminal JSON. + $stdoutTrimmed = ([string]$fanoutResult.Stdout).Trim() + $terminal = $null + $terminal = $stdoutTrimmed | ConvertFrom-Json -Depth 100 + Assert-Equal 'phase1' ([string]$terminal.phase) 'STDOUT still carries exactly one machine-readable terminal summary' + Assert-Equal 'completed' ([string]$terminal.status) 'the terminal summary reports completion' + Assert-True (-not $stdoutTrimmed.Contains((Get-RunnerProgressSentinel))) 'no relay sentinel ever leaks onto STDOUT' + Assert-True (-not ($stdoutTrimmed -match '(?m)^\[synthetic\]|(?m)^\[fixture\]')) 'no operator progress line contaminates STDOUT' + Assert-True ($stdoutTrimmed.StartsWith('{') -and $stdoutTrimmed.EndsWith('}')) 'STDOUT is a single JSON object with no surrounding progress text' + + # Progress is also persisted for post-mortem inspection at the advertised path. + Assert-Equal 'progress/phase1-progress.jsonl' ([string]$terminal.progress_log) 'the terminal summary advertises where progress is persisted' + $persistedLog = Join-Path $fanoutResult.IterationDirectory ([string]$terminal.progress_log) + Assert-True (Test-Path -LiteralPath $persistedLog -PathType Leaf) 'the persisted progress log exists at the advertised path' + $persistedEvents = @(Get-Content -LiteralPath $persistedLog | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-True ($persistedEvents.Count -ge 4) 'the persisted progress log retains events for post-mortem inspection' + + # Test 4 - concurrent-arm attribution: every progress line is attributable. + $stderrLines = @(([string]$fanoutResult.Stderr) -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) + $progressLines = @($stderrLines | Where-Object { $_ -match '^\[(synthetic|fixture)\]' }) + Assert-True ($progressLines.Count -ge 2) 'the concurrent fan-out emits live operator progress on STDERR' + $unlabeled = @($progressLines | Where-Object { -not ($_ -match '\[arm-\d+-(with_skill|without_skill)\]') }) + Assert-Equal 0 $unlabeled.Count 'every operator progress line carries a resolvable worker identity' + $workerIds = @($persistedEvents | ForEach-Object { [string]$_.worker } | Where-Object { $_ -match '^arm-\d+-' } | Sort-Object -Unique) + Assert-True ($workerIds.Count -ge 2) "concurrent arms are individually attributable (distinct workers: $($workerIds.Count))" + foreach ($workerId in @('arm-1-with_skill', 'arm-1-without_skill', 'arm-2-with_skill', 'arm-2-without_skill')) { + Assert-True ($workerIds -contains $workerId) "progress is attributed to worker $workerId" + } + # Attribution must be exclusive: an event's fields belong to exactly its worker. + $misattributed = @($persistedEvents | Where-Object { + [string]$_.state -in @('running', 'active', 'completed', 'timed-out', 'failed') -and + $null -ne $_.PSObject.Properties['configuration'] -and + -not ([string]$_.worker).EndsWith([string]$_.configuration) + }) + Assert-Equal 0 $misattributed.Count 'no progress event mixes one worker identity with another configuration' + + # ------------------------------------------------------------------ + # Test 9 - the shared process primitive: runner model-CLI progress is opt-in + # via the orchestration environment, relays through STDERR, and never + # contaminates the captured model result on STDOUT. + # ------------------------------------------------------------------ + Assert-True ($null -eq (Get-RunnerModelProgressContext -Runner 'opencode')) 'runner model-CLI progress is silent without the orchestration flag' + $previousFlag = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS') + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', '1') + try { + $enabledContext = Get-RunnerModelProgressContext -Runner 'opencode' -Phase 'opencode-cli' + Assert-True ($null -ne $enabledContext) 'the orchestration flag enables a runner model-CLI progress context' + Assert-Equal 'Relayable' ([string]$enabledContext['channel']) 'runner model-CLI progress uses the relayable channel' + Assert-Equal 'opencode' ([string]$enabledContext['runner']) 'runner model-CLI progress context carries the runner identity' + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_PROGRESS', $previousFlag) + } + + $primitiveDriver = New-SyntheticChildScript -Name 'primitive-driver' -Body @' +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') +$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source +$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.3 } +$grandchild = 'Start-Sleep -Milliseconds 900; [Console]::Out.Write(''grandchild-done'')' +$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchild) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx +[Console]::Out.Write([string]$r.Stdout) +'@ + $driverStderrPath = Join-Path $testRoot 'primitive.stderr' + $previousRoot = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + try { $driverOut = & $pwshPath -NoProfile -NonInteractive -File $primitiveDriver 2>$driverStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot) - } - $driverStdout = ([string]::Join('', @($driverOut | ForEach-Object { [string]$_ }))).Trim() - $driverStderr = if (Test-Path -LiteralPath $driverStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($driverStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-Equal 'grandchild-done' $driverStdout 'the model result on STDOUT passes through the shared primitive uncorrupted' - Assert-True (-not $driverStdout.Contains((Get-RunnerProgressSentinel))) 'no relayable sentinel leaks onto the shared primitive STDOUT' - Assert-True ($driverStderr.Contains((Get-RunnerProgressSentinel))) 'the shared primitive relays model-process progress on STDERR' - Assert-True ($driverStderr -match '"state":"running"') 'the relayed model-process progress reports a running lifecycle state' - - # ------------------------------------------------------------------ - # Test 9b - active inner model process: incremental output advances - # stdout/stderr event and byte counters BEFORE the process completes. - # A sleeping grandchild with one final write does NOT satisfy this test. - # ------------------------------------------------------------------ - $activeInnerDriver = New-SyntheticChildScript -Name 'active-inner-driver' -Body @' -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') -. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-progress.ps1') -$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source -$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.15 } -# Grandchild emits 6 stderr lines at 130ms intervals, then the final stdout result. -$grandchildBody = 'for ($i = 0; $i -lt 6; $i++) { [Console]::Error.WriteLine("event-" + $i); [System.Threading.Thread]::Sleep(130) }; [Console]::Out.Write("{""result"":""active-inner-done""}")' -$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchildBody) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx -# Forward captured stdout verbatim so the outer driver can verify fidelity. -[Console]::Out.Write([string]$r.Stdout) -'@ - $activeInnerStderrPath = Join-Path $testRoot 'active-inner-driver.stderr' - $previousRoot2 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - try { + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot) + } + $driverStdout = ([string]::Join('', @($driverOut | ForEach-Object { [string]$_ }))).Trim() + $driverStderr = if (Test-Path -LiteralPath $driverStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($driverStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-Equal 'grandchild-done' $driverStdout 'the model result on STDOUT passes through the shared primitive uncorrupted' + Assert-True (-not $driverStdout.Contains((Get-RunnerProgressSentinel))) 'no relayable sentinel leaks onto the shared primitive STDOUT' + Assert-True ($driverStderr.Contains((Get-RunnerProgressSentinel))) 'the shared primitive relays model-process progress on STDERR' + Assert-True ($driverStderr -match '"state":"running"') 'the relayed model-process progress reports a running lifecycle state' + + # ------------------------------------------------------------------ + # Test 9b - active inner model process: incremental output advances + # stdout/stderr event and byte counters BEFORE the process completes. + # A sleeping grandchild with one final write does NOT satisfy this test. + # ------------------------------------------------------------------ + $activeInnerDriver = New-SyntheticChildScript -Name 'active-inner-driver' -Body @' +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-common.ps1') +. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT 'runner-progress.ps1') +$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source +$ctx = @{ enabled = $true; runner = 'opencode'; phase = 'opencode-cli'; channel = 'Relayable'; heartbeatSeconds = 0.15 } +# Grandchild emits 6 stderr lines at 130ms intervals, then the final stdout result. +$grandchildBody = 'for ($i = 0; $i -lt 6; $i++) { [Console]::Error.WriteLine("event-" + $i); [System.Threading.Thread]::Sleep(130) }; [Console]::Out.Write("{""result"":""active-inner-done""}")' +$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @('-NoProfile', '-Command', $grandchildBody) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx +# Forward captured stdout verbatim so the outer driver can verify fidelity. +[Console]::Out.Write([string]$r.Stdout) +'@ + $activeInnerStderrPath = Join-Path $testRoot 'active-inner-driver.stderr' + $previousRoot2 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + try { $activeInnerOut = & $pwshPath -NoProfile -NonInteractive -File $activeInnerDriver 2>$activeInnerStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot2) - } - $activeInnerStdout = ([string]::Join('', @($activeInnerOut | ForEach-Object { [string]$_ }))).Trim() - $activeInnerStderr = if (Test-Path -LiteralPath $activeInnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($activeInnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - # Machine STDOUT must be the exact captured model result, uncorrupted. - Assert-Equal '{"result":"active-inner-done"}' $activeInnerStdout 'active inner process: captured stdout passes through uncorrupted' - Assert-True (-not $activeInnerStdout.Contains((Get-RunnerProgressSentinel))) 'active inner process: no sentinel leaks onto captured STDOUT' - # Parse the relayed sentinel lines to verify live activity tracking. - $sentinel = Get-RunnerProgressSentinel - $relayedLines = @($activeInnerStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) - Assert-True ($relayedLines.Count -ge 2) "active inner process: at least two relayed heartbeats (got $($relayedLines.Count))" - $relayedEvents = @($relayedLines | ForEach-Object { - $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $payload | ConvertFrom-Json } catch { $null } - } | Where-Object { $null -ne $_ }) - # Counters must advance: the last heartbeat must show more events than the first. - $eventCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) - Assert-True ($eventCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrEvents' - $firstCount = ($eventCounts | Measure-Object -Minimum).Minimum - $lastCount = ($eventCounts | Measure-Object -Maximum).Maximum - Assert-True ($lastCount -gt $firstCount) "active inner process: stderrEvents increase across heartbeats (first=$firstCount last=$lastCount)" - # Byte counters must also advance. - $byteCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrBytes'] } | ForEach-Object { [int64]$_.stderrBytes }) - Assert-True ($byteCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrBytes' - Assert-True (($byteCounts | Measure-Object -Maximum).Maximum -gt ($byteCounts | Measure-Object -Minimum).Minimum) 'active inner process: stderrBytes increase across heartbeats' - # lastActivity must appear once any real output has arrived. - $withLastActivity = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) - Assert-True ($withLastActivity.Count -ge 1) 'active inner process: lastActivity is present once real output is received' - - # ------------------------------------------------------------------ - # Test 9c - OpenCode-shaped streaming: a grandchild emitting structured - # JSONL/event-like output (simulating OpenCode session events) advances - # activity metadata safely without echoing model content to the operator. - # ------------------------------------------------------------------ - $openCodeGrandchild = New-SyntheticChildScript -Name 'oc-stream-grandchild' -Body @' -$events = @( - '{"type":"session.start","session_id":"abc123","model":"claude-3-5-haiku"}', - '{"type":"assistant.delta","session_id":"abc123","content":"I will"}', - '{"type":"assistant.delta","session_id":"abc123","content":"analyze"}', - '{"type":"tool.use","tool":"read_file","path":"input.txt"}', - '{"type":"assistant.delta","session_id":"abc123","content":"the result"}', - '{"type":"session.complete","session_id":"abc123","cost":0.002}' -) -foreach ($ev in $events) { - [Console]::Error.WriteLine($ev) - [System.Threading.Thread]::Sleep(100) -} -[Console]::Out.Write('{"status":"completed","session_id":"abc123"}') -'@ - $openCodeStreamDriver = New-SyntheticChildScript -Name 'opencode-stream-driver' -Body ( -'$ErrorActionPreference = "Stop"; Set-StrictMode -Version Latest' + "`n" + -'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-common.ps1")' + "`n" + -'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-progress.ps1")' + "`n" + -'$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source' + "`n" + -'$ctx = @{ enabled = $true; runner = "opencode"; phase = "opencode-cli"; channel = "Relayable"; heartbeatSeconds = 0.15 }' + "`n" + -('$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @("-NoProfile", "-File", $env:AGENTIC_OBS_GRANDCHILD) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx') + "`n" + -'[Console]::Out.Write([string]$r.Stdout)' -) - $ocStreamStderrPath = Join-Path $testRoot 'oc-stream-driver.stderr' - $previousRoot3 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') - $previousGrandchild = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD') - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $openCodeGrandchild) - try { + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot2) + } + $activeInnerStdout = ([string]::Join('', @($activeInnerOut | ForEach-Object { [string]$_ }))).Trim() + $activeInnerStderr = if (Test-Path -LiteralPath $activeInnerStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($activeInnerStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + # Machine STDOUT must be the exact captured model result, uncorrupted. + Assert-Equal '{"result":"active-inner-done"}' $activeInnerStdout 'active inner process: captured stdout passes through uncorrupted' + Assert-True (-not $activeInnerStdout.Contains((Get-RunnerProgressSentinel))) 'active inner process: no sentinel leaks onto captured STDOUT' + # Parse the relayed sentinel lines to verify live activity tracking. + $sentinel = Get-RunnerProgressSentinel + $relayedLines = @($activeInnerStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) + Assert-True ($relayedLines.Count -ge 2) "active inner process: at least two relayed heartbeats (got $($relayedLines.Count))" + $relayedEvents = @($relayedLines | ForEach-Object { + $payload = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $payload | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ }) + # Counters must advance: the last heartbeat must show more events than the first. + $eventCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) + Assert-True ($eventCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrEvents' + $firstCount = ($eventCounts | Measure-Object -Minimum).Minimum + $lastCount = ($eventCounts | Measure-Object -Maximum).Maximum + Assert-True ($lastCount -gt $firstCount) "active inner process: stderrEvents increase across heartbeats (first=$firstCount last=$lastCount)" + # Byte counters must also advance. + $byteCounts = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrBytes'] } | ForEach-Object { [int64]$_.stderrBytes }) + Assert-True ($byteCounts.Count -ge 2) 'active inner process: multiple heartbeats carry stderrBytes' + Assert-True (($byteCounts | Measure-Object -Maximum).Maximum -gt ($byteCounts | Measure-Object -Minimum).Minimum) 'active inner process: stderrBytes increase across heartbeats' + # lastActivity must appear once any real output has arrived. + $withLastActivity = @($relayedEvents | Where-Object { $null -ne $_.PSObject.Properties['lastActivitySeconds'] }) + Assert-True ($withLastActivity.Count -ge 1) 'active inner process: lastActivity is present once real output is received' + + # ------------------------------------------------------------------ + # Test 9c - OpenCode-shaped streaming: a grandchild emitting structured + # JSONL/event-like output (simulating OpenCode session events) advances + # activity metadata safely without echoing model content to the operator. + # ------------------------------------------------------------------ + $openCodeGrandchild = New-SyntheticChildScript -Name 'oc-stream-grandchild' -Body @' +$events = @( + '{"type":"session.start","session_id":"abc123","model":"claude-3-5-haiku"}', + '{"type":"assistant.delta","session_id":"abc123","content":"I will"}', + '{"type":"assistant.delta","session_id":"abc123","content":"analyze"}', + '{"type":"tool.use","tool":"read_file","path":"input.txt"}', + '{"type":"assistant.delta","session_id":"abc123","content":"the result"}', + '{"type":"session.complete","session_id":"abc123","cost":0.002}' +) +foreach ($ev in $events) { + [Console]::Error.WriteLine($ev) + [System.Threading.Thread]::Sleep(100) +} +[Console]::Out.Write('{"status":"completed","session_id":"abc123"}') +'@ + $openCodeStreamDriver = New-SyntheticChildScript -Name 'opencode-stream-driver' -Body ( +'$ErrorActionPreference = "Stop"; Set-StrictMode -Version Latest' + "`n" + +'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-common.ps1")' + "`n" + +'. (Join-Path $env:AGENTIC_OBS_RUNNER_ROOT "runner-progress.ps1")' + "`n" + +'$pwsh = (Get-Command pwsh -CommandType Application | Select-Object -First 1).Source' + "`n" + +'$ctx = @{ enabled = $true; runner = "opencode"; phase = "opencode-cli"; channel = "Relayable"; heartbeatSeconds = 0.15 }' + "`n" + +('$r = Invoke-RunnerProcess -FileName $pwsh -ArgumentList @("-NoProfile", "-File", $env:AGENTIC_OBS_GRANDCHILD) -WorkingDirectory $env:AGENTIC_OBS_RUNNER_ROOT -TimeoutSeconds 20 -ProgressContext $ctx') + "`n" + +'[Console]::Out.Write([string]$r.Stdout)' +) + $ocStreamStderrPath = Join-Path $testRoot 'oc-stream-driver.stderr' + $previousRoot3 = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT') + $previousGrandchild = [Environment]::GetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD') + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $runnerRoot) + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $openCodeGrandchild) + try { $ocStreamOut = & $pwshPath -NoProfile -NonInteractive -File $openCodeStreamDriver 2>$ocStreamStderrPath - } finally { - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot3) - [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $previousGrandchild) - } - $ocStreamStdout = ([string]::Join('', @($ocStreamOut | ForEach-Object { [string]$_ }))).Trim() - $ocStreamStderr = if (Test-Path -LiteralPath $ocStreamStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($ocStreamStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } - Assert-Equal '{"status":"completed","session_id":"abc123"}' $ocStreamStdout 'opencode streaming: final stdout captured exactly' - Assert-True (-not $ocStreamStdout.Contains('assistant.delta')) 'opencode streaming: model content not present on stdout' - Assert-True (-not $ocStreamStderr.Contains('"content":"I will"')) 'opencode streaming: model delta content not echoed to operator stderr' - Assert-True (-not $ocStreamStderr.Contains('"content":"analyze"')) 'opencode streaming: second delta not echoed to operator stderr' - $ocRelayedLines = @($ocStreamStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) - Assert-True ($ocRelayedLines.Count -ge 2) "opencode streaming: operator receives multiple heartbeats (got $($ocRelayedLines.Count))" - $ocRelayedEvents = @($ocRelayedLines | ForEach-Object { - $p = $_.TrimStart().Substring($sentinel.Length).TrimStart() - try { $p | ConvertFrom-Json } catch { $null } - } | Where-Object { $null -ne $_ }) - $ocEventCounts = @($ocRelayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) - Assert-True ($ocEventCounts.Count -ge 2) 'opencode streaming: multiple heartbeats carry stderrEvents' - Assert-True (($ocEventCounts | Measure-Object -Maximum).Maximum -gt ($ocEventCounts | Measure-Object -Minimum).Minimum) 'opencode streaming: stderrEvents advance as events are received' - - # ------------------------------------------------------------------ - # Test 9d - Codex app-server real transport: the actual Invoke-CodexAppServer - # path must emit heartbeats while blocked waiting for protocol input, then - # surface real protocol counters and last-activity aging without echoing - # JSON-RPC payloads or model content to operator progress. - # ------------------------------------------------------------------ - $codexApp = Invoke-CodexAppServerFixture -Mode 'success' -HeartbeatSeconds 0.15 -TimeoutSeconds 6 - Assert-Equal 0 $codexApp.ExitCode 'codex app-server: runner process exits cleanly' - $codexStdout = ([string]$codexApp.Stdout).Trim() - Assert-True ($codexStdout.StartsWith('{') -and $codexStdout.EndsWith('}')) 'codex app-server: STDOUT remains one terminal JSON object' - Assert-True (-not $codexStdout.Contains((Get-RunnerProgressSentinel))) 'codex app-server: no progress sentinel contaminates STDOUT' - Assert-True (-not ($codexStdout -match '(?m)^\[codex\]')) 'codex app-server: operator progress never contaminates STDOUT' - Assert-Equal 'completed' ([string]$codexApp.Result.status) 'codex app-server: execution result completes successfully' - Assert-Equal 'OBSERVABILITY_MODEL_CONTENT_CANARY' ([string]$codexApp.Result.final_response.text) 'codex app-server: machine output preserves the final response text' - Assert-True ([bool](Test-Path -LiteralPath $codexApp.RawEventsPath -PathType Leaf)) 'codex app-server: raw protocol evidence is retained on disk' - Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: raw protocol evidence retains model content' - Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: raw protocol evidence retains protocol payload content' - Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: model content is not echoed to operator progress' - Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: protocol payload content is not echoed to operator progress' - $codexAppEvents = @($codexApp.AppServerEvents) - Assert-True ($codexAppEvents.Count -ge 6) "codex app-server: real transport emitted observable progress events (got $($codexAppEvents.Count))" - $firstProtocolIndex = -1 - for ($eventIndex = 0; $eventIndex -lt $codexAppEvents.Count; $eventIndex++) { - if ([int64](Get-Field $codexAppEvents[$eventIndex] 'stdoutEvents' 0) -gt 0) { - $firstProtocolIndex = $eventIndex - break - } - } - Assert-True ($firstProtocolIndex -gt 0) 'codex app-server: at least one heartbeat occurs before the first protocol message is observed' - $preProtocolEvents = @($codexAppEvents[0..($firstProtocolIndex - 1)]) - $quietPreProtocolHeartbeats = @($preProtocolEvents | Where-Object { - [string]$_.state -eq 'running' -and - [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and - $null -eq $_.PSObject.Properties['lastActivitySeconds'] -and - [string]::IsNullOrWhiteSpace([string](Get-Field $_ 'detail' '')) - }) - Assert-True ($quietPreProtocolHeartbeats.Count -ge 1) 'codex app-server: quiet heartbeats are visible before any protocol message arrives' - $firstActivityEvent = $codexAppEvents[$firstProtocolIndex] - $firstActivityCount = [int64](Get-Field $firstActivityEvent 'stdoutEvents' 0) - $firstActivityBytes = [int64](Get-Field $firstActivityEvent 'stdoutBytes' 0) - Assert-True ($firstActivityCount -gt 0 -and $firstActivityBytes -gt 0) 'codex app-server: the first observed protocol heartbeat carries non-zero event and byte counters' - Assert-True ($null -ne $firstActivityEvent.PSObject.Properties['lastActivitySeconds']) 'codex app-server: lastActivity appears after the first real protocol message' - $sameCounterQuietEvents = @($codexAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -eq $firstActivityCount -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq $firstActivityBytes -and - $null -ne $_.PSObject.Properties['lastActivitySeconds'] - }) - Assert-True ($sameCounterQuietEvents.Count -ge 2) 'codex app-server: quiet heartbeats preserve counters after the first protocol message' - $sameCounterAges = @($sameCounterQuietEvents | ForEach-Object { [double]$_.lastActivitySeconds }) - $sameCounterMinAge = ($sameCounterAges | Measure-Object -Minimum).Minimum - $sameCounterMaxAge = ($sameCounterAges | Measure-Object -Maximum).Maximum - Assert-True ($sameCounterMaxAge -gt $sameCounterMinAge) 'codex app-server: lastActivitySeconds ages during a quiet period' - $nextActivityEvent = @($codexAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -gt $firstActivityCount -and - [int64](Get-Field $_ 'stdoutBytes' 0) -gt $firstActivityBytes -and - $null -ne $_.PSObject.Properties['lastActivitySeconds'] - } | Select-Object -First 1) - Assert-True ($nextActivityEvent.Count -eq 1) 'codex app-server: later protocol traffic advances counters again' - Assert-True ([double]$nextActivityEvent[0].lastActivitySeconds -lt $sameCounterMaxAge) 'codex app-server: lastActivitySeconds resets after fresh protocol traffic' - - # ------------------------------------------------------------------ - # Test 9e - Codex app-server timeout: repeated quiet heartbeats must not - # extend the total timeout. The real transport still fails closed, bounded. - # ------------------------------------------------------------------ - $codexTimeout = Invoke-CodexAppServerFixture -Mode 'timeout' -HeartbeatSeconds 0.15 -TimeoutSeconds 2 - Assert-Equal 0 $codexTimeout.ExitCode 'codex app-server timeout: runner still returns a terminal result object' + } finally { + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_RUNNER_ROOT', $previousRoot3) + [Environment]::SetEnvironmentVariable('AGENTIC_OBS_GRANDCHILD', $previousGrandchild) + } + $ocStreamStdout = ([string]::Join('', @($ocStreamOut | ForEach-Object { [string]$_ }))).Trim() + $ocStreamStderr = if (Test-Path -LiteralPath $ocStreamStderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($ocStreamStderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + Assert-Equal '{"status":"completed","session_id":"abc123"}' $ocStreamStdout 'opencode streaming: final stdout captured exactly' + Assert-True (-not $ocStreamStdout.Contains('assistant.delta')) 'opencode streaming: model content not present on stdout' + Assert-True (-not $ocStreamStderr.Contains('"content":"I will"')) 'opencode streaming: model delta content not echoed to operator stderr' + Assert-True (-not $ocStreamStderr.Contains('"content":"analyze"')) 'opencode streaming: second delta not echoed to operator stderr' + $ocRelayedLines = @($ocStreamStderr -split "`r?`n" | Where-Object { $_.TrimStart().StartsWith($sentinel) }) + Assert-True ($ocRelayedLines.Count -ge 2) "opencode streaming: operator receives multiple heartbeats (got $($ocRelayedLines.Count))" + $ocRelayedEvents = @($ocRelayedLines | ForEach-Object { + $p = $_.TrimStart().Substring($sentinel.Length).TrimStart() + try { $p | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ }) + $ocEventCounts = @($ocRelayedEvents | Where-Object { $null -ne $_.PSObject.Properties['stderrEvents'] } | ForEach-Object { [int64]$_.stderrEvents }) + Assert-True ($ocEventCounts.Count -ge 2) 'opencode streaming: multiple heartbeats carry stderrEvents' + Assert-True (($ocEventCounts | Measure-Object -Maximum).Maximum -gt ($ocEventCounts | Measure-Object -Minimum).Minimum) 'opencode streaming: stderrEvents advance as events are received' + + # ------------------------------------------------------------------ + # Test 9d - Codex app-server real transport: the actual Invoke-CodexAppServer + # path must emit heartbeats while blocked waiting for protocol input, then + # surface real protocol counters and last-activity aging without echoing + # JSON-RPC payloads or model content to operator progress. + # ------------------------------------------------------------------ + $codexApp = Invoke-CodexAppServerFixture -Mode 'success' -HeartbeatSeconds 0.15 -TimeoutSeconds 6 + Assert-Equal 0 $codexApp.ExitCode 'codex app-server: runner process exits cleanly' + $codexStdout = ([string]$codexApp.Stdout).Trim() + Assert-True ($codexStdout.StartsWith('{') -and $codexStdout.EndsWith('}')) 'codex app-server: STDOUT remains one terminal JSON object' + Assert-True (-not $codexStdout.Contains((Get-RunnerProgressSentinel))) 'codex app-server: no progress sentinel contaminates STDOUT' + Assert-True (-not ($codexStdout -match '(?m)^\[codex\]')) 'codex app-server: operator progress never contaminates STDOUT' + Assert-Equal 'completed' ([string]$codexApp.Result.status) 'codex app-server: execution result completes successfully' + Assert-Equal 'OBSERVABILITY_MODEL_CONTENT_CANARY' ([string]$codexApp.Result.final_response.text) 'codex app-server: machine output preserves the final response text' + Assert-True ([bool](Test-Path -LiteralPath $codexApp.RawEventsPath -PathType Leaf)) 'codex app-server: raw protocol evidence is retained on disk' + Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: raw protocol evidence retains model content' + Assert-True ($codexApp.RawEventsText.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: raw protocol evidence retains protocol payload content' + Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_MODEL_CONTENT_CANARY')) 'codex app-server: model content is not echoed to operator progress' + Assert-True (-not $codexApp.Stderr.Contains('OBSERVABILITY_PROTOCOL_OUTPUT_CANARY')) 'codex app-server: protocol payload content is not echoed to operator progress' + $codexAppEvents = @($codexApp.AppServerEvents) + Assert-True ($codexAppEvents.Count -ge 6) "codex app-server: real transport emitted observable progress events (got $($codexAppEvents.Count))" + $firstProtocolIndex = -1 + for ($eventIndex = 0; $eventIndex -lt $codexAppEvents.Count; $eventIndex++) { + if ([int64](Get-Field $codexAppEvents[$eventIndex] 'stdoutEvents' 0) -gt 0) { + $firstProtocolIndex = $eventIndex + break + } + } + Assert-True ($firstProtocolIndex -gt 0) 'codex app-server: at least one heartbeat occurs before the first protocol message is observed' + $preProtocolEvents = @($codexAppEvents[0..($firstProtocolIndex - 1)]) + $quietPreProtocolHeartbeats = @($preProtocolEvents | Where-Object { + [string]$_.state -eq 'running' -and + [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and + $null -eq $_.PSObject.Properties['lastActivitySeconds'] -and + [string]::IsNullOrWhiteSpace([string](Get-Field $_ 'detail' '')) + }) + Assert-True ($quietPreProtocolHeartbeats.Count -ge 1) 'codex app-server: quiet heartbeats are visible before any protocol message arrives' + $firstActivityEvent = $codexAppEvents[$firstProtocolIndex] + $firstActivityCount = [int64](Get-Field $firstActivityEvent 'stdoutEvents' 0) + $firstActivityBytes = [int64](Get-Field $firstActivityEvent 'stdoutBytes' 0) + Assert-True ($firstActivityCount -gt 0 -and $firstActivityBytes -gt 0) 'codex app-server: the first observed protocol heartbeat carries non-zero event and byte counters' + Assert-True ($null -ne $firstActivityEvent.PSObject.Properties['lastActivitySeconds']) 'codex app-server: lastActivity appears after the first real protocol message' + $sameCounterQuietEvents = @($codexAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -eq $firstActivityCount -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq $firstActivityBytes -and + $null -ne $_.PSObject.Properties['lastActivitySeconds'] + }) + Assert-True ($sameCounterQuietEvents.Count -ge 2) 'codex app-server: quiet heartbeats preserve counters after the first protocol message' + $sameCounterAges = @($sameCounterQuietEvents | ForEach-Object { [double]$_.lastActivitySeconds }) + $sameCounterMinAge = ($sameCounterAges | Measure-Object -Minimum).Minimum + $sameCounterMaxAge = ($sameCounterAges | Measure-Object -Maximum).Maximum + Assert-True ($sameCounterMaxAge -gt $sameCounterMinAge) 'codex app-server: lastActivitySeconds ages during a quiet period' + $nextActivityEvent = @($codexAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -gt $firstActivityCount -and + [int64](Get-Field $_ 'stdoutBytes' 0) -gt $firstActivityBytes -and + $null -ne $_.PSObject.Properties['lastActivitySeconds'] + } | Select-Object -First 1) + Assert-True ($nextActivityEvent.Count -eq 1) 'codex app-server: later protocol traffic advances counters again' + Assert-True ([double]$nextActivityEvent[0].lastActivitySeconds -lt $sameCounterMaxAge) 'codex app-server: lastActivitySeconds resets after fresh protocol traffic' + + # ------------------------------------------------------------------ + # Test 9e - Codex app-server timeout: repeated quiet heartbeats must not + # extend the total timeout. The real transport still fails closed, bounded. + # ------------------------------------------------------------------ + $codexTimeout = Invoke-CodexAppServerFixture -Mode 'timeout' -HeartbeatSeconds 0.15 -TimeoutSeconds 2 + Assert-Equal 0 $codexTimeout.ExitCode 'codex app-server timeout: runner still returns a terminal result object' Assert-True ($codexTimeout.ElapsedSeconds -lt 20) ("codex app-server timeout: transport remains bounded after discovery/verification probes; elapsed={0:N3}s" -f $codexTimeout.ElapsedSeconds) - Assert-Equal 'incompatible' ([string]$codexTimeout.Result.status) 'codex app-server timeout: native evidence still fails closed after the bounded timeout' + Assert-Equal 'incompatible' ([string]$codexTimeout.Result.status) 'codex app-server timeout: native evidence still fails closed after the bounded timeout' Assert-Equal 'native_skill_isolation_unverified' ([string]$codexTimeout.Result.exit.failure.code) 'codex app-server timeout: the failure remains structured' - Assert-True ([string]$codexTimeout.Result.exit.failure.message -match 'Codex did not finish before timeout_seconds') 'codex app-server timeout: the failure message preserves the bounded timeout detail' - $timeoutAppEvents = @($codexTimeout.AppServerEvents) - Assert-True ($timeoutAppEvents.Count -ge 2) 'codex app-server timeout: quiet heartbeats occur before the timeout result' - $timeoutQuietEvents = @($timeoutAppEvents | Where-Object { - [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and - [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and - $null -eq $_.PSObject.Properties['lastActivitySeconds'] - }) - Assert-True ($timeoutQuietEvents.Count -ge 2) 'codex app-server timeout: a silent server stays externally observable without inventing activity' - - # ------------------------------------------------------------------ - # Test 10 - synchronous wait/preflight heartbeat: Complete-RunnerChildProcess - # must emit heartbeats while waiting even when the caller is not using the - # concurrent Wait-AnyRunnerChild loop. - # ------------------------------------------------------------------ - $slowPreflightScript = New-SyntheticChildScript -Name 'slow-preflight' -Body @' -Start-Sleep -Milliseconds 1500 -[Console]::Out.Write('{"status":"compatible"}') -'@ - $preflightStdoutPath = Join-Path $testRoot 'preflight.stdout' - $preflightStderrPath = Join-Path $testRoot 'preflight.stderr' - $preflightLogPath = Join-Path $testRoot 'preflight-progress.jsonl' + Assert-True ([string]$codexTimeout.Result.exit.failure.message -match 'Codex did not finish before timeout_seconds') 'codex app-server timeout: the failure message preserves the bounded timeout detail' + $timeoutAppEvents = @($codexTimeout.AppServerEvents) + Assert-True ($timeoutAppEvents.Count -ge 2) 'codex app-server timeout: quiet heartbeats occur before the timeout result' + $timeoutQuietEvents = @($timeoutAppEvents | Where-Object { + [int64](Get-Field $_ 'stdoutEvents' 0) -eq 0 -and + [int64](Get-Field $_ 'stdoutBytes' 0) -eq 0 -and + $null -eq $_.PSObject.Properties['lastActivitySeconds'] + }) + Assert-True ($timeoutQuietEvents.Count -ge 2) 'codex app-server timeout: a silent server stays externally observable without inventing activity' + + # ------------------------------------------------------------------ + # Test 10 - synchronous wait/preflight heartbeat: Complete-RunnerChildProcess + # must emit heartbeats while waiting even when the caller is not using the + # concurrent Wait-AnyRunnerChild loop. + # ------------------------------------------------------------------ + $slowPreflightScript = New-SyntheticChildScript -Name 'slow-preflight' -Body @' +Start-Sleep -Milliseconds 1500 +[Console]::Out.Write('{"status":"compatible"}') +'@ + $preflightStdoutPath = Join-Path $testRoot 'preflight.stdout' + $preflightStderrPath = Join-Path $testRoot 'preflight.stderr' + $preflightLogPath = Join-Path $testRoot 'preflight-progress.jsonl' $preflightChild = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $slowPreflightScript) -WorkingDirectory $testRoot -StdoutPath $preflightStdoutPath -StderrPath $preflightStderrPath -TimeoutSeconds 30 -Runner 'opencode' -WorkerId 'preflight-arm-99' -EvalId 99 -Configuration 'with_skill' -Phase 'preflight' -ProgressLogPath $preflightLogPath -HeartbeatSeconds 0.3 - # Call Complete-RunnerChildProcess directly (the synchronous preflight path), - # without using Wait-AnyRunnerChild. - $preflightExit = Complete-RunnerChildProcess -Child $preflightChild - $preflightEvents = @() - if (Test-Path -LiteralPath $preflightLogPath -PathType Leaf) { - $preflightEvents = @(Get-Content -LiteralPath $preflightLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) - } - Assert-Equal 0 $preflightExit 'synchronous preflight completes cleanly' - $preflightHeartbeats = @($preflightEvents | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) - Assert-True ($preflightHeartbeats.Count -ge 2) "synchronous Complete-RunnerChildProcess emits heartbeats during wait (got $($preflightHeartbeats.Count))" - $preflightCompleted = @($preflightEvents | Where-Object { [string]$_.state -eq 'completed' }) - Assert-Equal 1 $preflightCompleted.Count 'synchronous preflight reports exactly one completed state' - - Write-Output 'Runner observability: PASS' -} finally { - if (Test-Path -LiteralPath $testRoot) { - Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue - } -} + # Call Complete-RunnerChildProcess directly (the synchronous preflight path), + # without using Wait-AnyRunnerChild. + $preflightExit = Complete-RunnerChildProcess -Child $preflightChild + $preflightEvents = @() + if (Test-Path -LiteralPath $preflightLogPath -PathType Leaf) { + $preflightEvents = @(Get-Content -LiteralPath $preflightLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + Assert-Equal 0 $preflightExit 'synchronous preflight completes cleanly' + $preflightHeartbeats = @($preflightEvents | Where-Object { [string]$_.state -eq 'running' -and [string]$_.origin -eq 'parent' }) + Assert-True ($preflightHeartbeats.Count -ge 2) "synchronous Complete-RunnerChildProcess emits heartbeats during wait (got $($preflightHeartbeats.Count))" + $preflightCompleted = @($preflightEvents | Where-Object { [string]$_.state -eq 'completed' }) + Assert-Equal 1 $preflightCompleted.Count 'synchronous preflight reports exactly one completed state' + + Write-Output 'Runner observability: PASS' +} finally { + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index cdc8436..50644dd 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -59,6 +59,7 @@ $OutputEncoding = $utf8NoBom . (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') . (Join-Path $PSScriptRoot 'eval-runners/execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/phase2-grading.ps1') function Set-BenchmarkTokenMetrics { param([object]$Benchmark, [object[]]$ManifestRecords) @@ -155,6 +156,21 @@ function Get-Property { return $Default } +function Get-AssertionDisplayText { + param([object]$Assertion) + + if ($Assertion -is [string]) { return [string]$Assertion } + $text = [string](Get-Property -Object $Assertion -Name 'assertion' -Default (Get-Property -Object $Assertion -Name 'text' -Default '')) + if ([string]::IsNullOrWhiteSpace($text)) { return [string]$Assertion } + return $text +} + +function Get-AssertionDisplayTexts { + param([object[]]$Assertions) + + return @($Assertions | ForEach-Object { Get-AssertionDisplayText -Assertion $_ }) +} + function Get-SafeSegment { param([string]$Value) @@ -376,18 +392,19 @@ function Get-ReportOutputFiles { function Get-ReportGrades { param( [object]$Result, - [string[]]$Assertions + [object[]]$Assertions ) + $assertionTexts = @(Get-AssertionDisplayTexts -Assertions $Assertions) $grading = @(Get-Property -Object $Result -Name 'grading' -Default @()) - $count = [Math]::Max($grading.Count, $Assertions.Count) + $count = [Math]::Max($grading.Count, $assertionTexts.Count) $grades = [System.Collections.Generic.List[object]]::new() for ($index = 0; $index -lt $count; $index++) { $grade = if ($index -lt $grading.Count) { $grading[$index] } else { $null } $text = [string](Get-Property -Object $grade -Name 'text' -Default '') $generic = [string]::IsNullOrWhiteSpace($text) -or $text -match '^(Passed|Failed|Assertion\s+\d+)$' - if ($generic -and $index -lt $Assertions.Count) { - $text = [string]$Assertions[$index] + if ($generic -and $index -lt $assertionTexts.Count) { + $text = [string]$assertionTexts[$index] } if ([string]::IsNullOrWhiteSpace($text)) { $text = 'Assertion' @@ -419,11 +436,89 @@ function Get-ReportMetric { if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { return $value } + } return $null } +function Get-MetricTotalOrNull { + param([object[]]$Items, [string]$Name) + + if (@($Items).Count -eq 0) { return $null } + $total = 0.0 + foreach ($item in @($Items)) { + $value = Get-Property -Object $item -Name $Name -Default $null + if ($null -eq $value -or [string]::IsNullOrWhiteSpace([string]$value)) { return $null } + $total += [double]$value + } + return $total +} + +function Get-ExecutionUsageSummary { + param([object[]]$ManifestRecords) + + $results = @($ManifestRecords | ForEach-Object { Read-JsonFile -Path $_.ResultPath }) + return [ordered]@{ + runner = $null + model = if ($results.Count -eq 0) { $null } else { ([string]::Join(', ', @($results | ForEach-Object { [string](Get-Property -Object $_ -Name 'model' -Default '') } | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Sort-Object -Unique))) } + duration_seconds = Get-MetricTotalOrNull -Items $results -Name 'duration_seconds' + input_tokens = Get-MetricTotalOrNull -Items $results -Name 'base_input_tokens' + output_tokens = Get-MetricTotalOrNull -Items $results -Name 'output_tokens' + cache_read_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_read_tokens' + cache_write_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_write_tokens' + total_tokens = Get-MetricTotalOrNull -Items $results -Name 'total_tokens' + cost = Get-MetricTotalOrNull -Items $results -Name 'estimated_cost_usd' + worker_count = $results.Count + } +} + +function Get-AnalyzerUsageSummary { + param([string]$IterationPath, [object]$AnalyzerProfile) + + $statePath = Join-Path $IterationPath 'phase2-state.json' + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + return [ordered]@{ + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + duration_seconds = $null + input_tokens = $null + output_tokens = $null + cache_read_tokens = $null + cache_write_tokens = $null + total_tokens = $null + cost = $null + worker_count = $null + failed_worker_count = $null + } + } + $state = Read-JsonFile -Path $statePath + $entries = @(Get-Property -Object $state -Name 'analyzer_results' -Default @()) + $results = @($entries | ForEach-Object { + $relative = [string](Get-Property -Object $_ -Name 'path' -Default '') + if ([string]::IsNullOrWhiteSpace($relative)) { return } + Read-JsonFile -Path (Join-Path $IterationPath ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar)) + }) + $durationMs = Get-MetricTotalOrNull -Items $results -Name 'duration_ms' + return [ordered]@{ + runner = [string]$AnalyzerProfile.Runner + model = [string]$AnalyzerProfile.Model + profile_sha256 = [string]$AnalyzerProfile.Hash + duration_seconds = if ($results.Count -eq 0) { 0.0 } elseif ($null -eq $durationMs) { $null } else { [Math]::Round(($durationMs / 1000.0), 4) } + input_tokens = Get-MetricTotalOrNull -Items $results -Name 'input_tokens' + output_tokens = Get-MetricTotalOrNull -Items $results -Name 'output_tokens' + cache_read_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_read_tokens' + cache_write_tokens = Get-MetricTotalOrNull -Items $results -Name 'cache_write_tokens' + total_tokens = Get-MetricTotalOrNull -Items $results -Name 'total_tokens' + cost = Get-MetricTotalOrNull -Items $results -Name 'cost' + worker_count = @($state.expected_worker_ids).Count + failed_worker_count = @((Get-JsonPropertyNames -Object (Get-Property -Object $state -Name 'completed' -Default ([ordered]@{}))) | Where-Object { + $completed = Get-Property -Object $state -Name 'completed' -Default ([ordered]@{}) + [string](Get-JsonProperty -Object (Get-JsonProperty -Object $completed -Name ([string]$_) -Default $null) -Name 'status' -Default '') -ne 'completed' + }).Count + } +} + function Get-ReportRun { param( [object]$Result, @@ -521,7 +616,8 @@ function Write-FirstPartyReport { [object]$Validation, [string]$IterationPath, [string]$OutputPath, - [object]$Benchmark + [object]$Benchmark, + [object]$Usage = $null ) $evals = [System.Collections.Generic.List[object]]::new() @@ -535,7 +631,7 @@ function Write-FirstPartyReport { $evalDirectory = [string]$entryRecords[0].EvalDirectory $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $runMap = [ordered]@{} - $assertions = @($metadata.assertions | ForEach-Object { [string]$_ }) + $assertions = @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) foreach ($configuration in @('with_skill', 'without_skill')) { $records = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration @@ -572,6 +668,7 @@ function Write-FirstPartyReport { completed_runs = $completedRuns expected_runs = @($Manifest.evals).Count * 2 generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') + usage = $Usage } $reportData = [ordered]@{ skill_name = [string]$Manifest.skill_name @@ -685,7 +782,7 @@ function New-UpstreamWorkspace { eval_name = [string]$metadata.eval_name prompt = [string]$metadata.prompt expected_output = [string](Get-Property -Object $metadata -Name 'expected_output' -Default '') - expectations = @($metadata.assertions) + expectations = @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) } Write-JsonFile -Path (Join-Path $evalFolder 'eval_metadata.json') -Value $upstreamMetadata $workspaceEntries.Add([pscustomobject]@{ EvalId = [int]$entry.eval_id; EvalName = [string]$entry.eval_name }) @@ -727,7 +824,7 @@ function New-UpstreamWorkspace { $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) Copy-RecordedOutputFiles -Result $result -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory - Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @($metadata.assertions | ForEach-Object { [string]$_ }) + Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @(Get-AssertionDisplayTexts -Assertions @($metadata.assertions)) } } @@ -749,6 +846,7 @@ foreach ($warning in @($validation.Warnings)) { if (-not $validation.Complete) { throw "Evaluation completion gate failed: expected $($validation.ExpectedArmCount) bridged terminal arms, found $($validation.BridgedResults)." } +$gradingFreeze = Assert-GradingFreeze -IterationDirectory $iterationPath $skillCreatorPathResolved = Resolve-SkillCreatorPath -RequestedPath $SkillCreatorPath $pythonCommand = Resolve-PythonCommand $workspacePath = Join-Path $iterationPath '.skill-creator-report' @@ -792,7 +890,33 @@ foreach ($entry in @($manifest.evals)) { $benchmark.metadata.runs_per_configuration = 1 $benchmark.metadata.executor_model = if ($models.Count -eq 0) { 'model not recorded' } else { $models -join ', ' } -$benchmark.metadata.analyzer_model = 'external skill-creator evaluator' +# Name the actual, validated analyzer/grader from the persisted analyzer profile rather than a generic label. Executor +# identity and analyzer identity remain independently attributable in the report. +$analyzerProfilePath = Join-Path $iterationPath 'analyzer-profile.json' +if (Test-Path -LiteralPath $analyzerProfilePath -PathType Leaf) { + $analyzerProfile = $gradingFreeze.Analyzer.Profile + $analyzerRunner = [string]$gradingFreeze.Analyzer.Runner + $analyzerModelName = [string]$gradingFreeze.Analyzer.Model + $analyzerReasoning = [string]$gradingFreeze.Analyzer.ReasoningEffort + $analyzerIdentity = ((@($analyzerRunner, $analyzerModelName) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) -join ' / ') + if ([string]::IsNullOrWhiteSpace($analyzerIdentity)) { $analyzerIdentity = 'analyzer profile incomplete' } + if (-not [string]::IsNullOrWhiteSpace($analyzerReasoning)) { $analyzerIdentity += " (reasoning: $analyzerReasoning)" } + $benchmark.metadata.analyzer_model = $analyzerIdentity + $benchmark.metadata | Add-Member -NotePropertyName analyzer_runner -NotePropertyValue $analyzerRunner -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_reasoning_effort -NotePropertyValue $analyzerReasoning -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_contract_version -NotePropertyValue ([string](Get-Property -Object $analyzerProfile -Name 'contract_version' -Default '')) -Force + $benchmark.metadata | Add-Member -NotePropertyName analyzer_profile_sha256 -NotePropertyValue ([string]$gradingFreeze.Analyzer.Hash) -Force +} else { + $benchmark.metadata.analyzer_model = 'analyzer profile not recorded' +} +$usageSummary = [ordered]@{ + execution = Get-ExecutionUsageSummary -ManifestRecords $manifestRecords + analyzer = Get-AnalyzerUsageSummary -IterationPath $iterationPath -AnalyzerProfile $gradingFreeze.Analyzer +} +$usageSummary.execution.runner = [string](Get-Property -Object $manifest.execution_selection -Name 'runner' -Default '') +$usageSummary.execution.model = [string](Get-Property -Object $manifest.execution_selection -Name 'model' -Default $usageSummary.execution.model) +$benchmark.metadata | Add-Member -NotePropertyName execution_usage -NotePropertyValue $usageSummary.execution -Force +$benchmark.metadata | Add-Member -NotePropertyName analyzer_usage -NotePropertyValue $usageSummary.analyzer -Force $benchmark.metadata.evals_run = @($workspaceEntries | ForEach-Object { $_.EvalId }) foreach ($run in @($benchmark.runs)) { $match = @($workspaceEntries | Where-Object { $_.EvalId -eq [int]$run.eval_id }) | Select-Object -First 1 @@ -824,7 +948,7 @@ $viewerArguments = @( ) Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $viewerPath -Arguments $viewerArguments -Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark +Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark -Usage $usageSummary if ($RequireComplete) { foreach ($output in @($benchmarkOutputPath, $benchmarkMarkdownOutputPath, $upstreamHtmlOutputPath, $htmlOutputPath)) { diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 358e697..adf94b5 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -44,6 +44,9 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER PassThru + Also return each successfully prepared absolute RUN-THIS.prompt.md path on the success stream. + .PARAMETER Runner Package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. GitHub Copilot and Codex can resolve repository-defined default models; OpenCode requires an explicit model selector. @@ -60,6 +63,10 @@ .PARAMETER ModelCatalogPath Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. +.PARAMETER AnalyzerModelCatalogPath + Optional deterministic analyzer catalog JSON used by the model discovery helper. Intended for tests and offline + validation of the analyzer profile independently from executor model validation. + .PARAMETER ReasoningEffort Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to low when this is omitted. @@ -129,6 +136,10 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$PassThru, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$Runner, @@ -145,10 +156,26 @@ param( [Parameter(ParameterSetName = 'Changed')] [string]$ModelCatalogPath, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerModelCatalogPath, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$ReasoningEffort, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerRunner, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerModel, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$AnalyzerReasoningEffort, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$ConfigurationProfile = 'isolated-default', @@ -184,6 +211,7 @@ $OutputEncoding = $utf8NoBom . (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') . (Join-Path $PSScriptRoot 'eval-runners/package-integrity.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/phase2-grading.ps1') $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' @@ -191,6 +219,8 @@ $resultSchema = 'codebeltnet/agentic/eval-result/2' $runSchema = 'codebeltnet/agentic/eval-run/1' $executionProfileSchema = 'codebeltnet/agentic/eval-execution-profile/1' $executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' +$analyzerProfileSchema = 'codebeltnet/agentic/eval-analyzer-profile/1' +$analyzerContractVersion = 'codebeltnet/agentic/eval-grading/1' $runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 @@ -413,7 +443,8 @@ function Confirm-HarnessModel { param( [Parameter(Mandatory = $true)][string]$RepoRoot, [Parameter(Mandatory = $true)][string]$RunnerName, - [Parameter(Mandatory = $true)][string]$ModelName + [Parameter(Mandatory = $true)][string]$ModelName, + [AllowNull()][string]$CatalogPath = $null ) $discoveryScript = Join-Path $RepoRoot 'scripts/Get-HarnessModels.ps1' @@ -422,8 +453,8 @@ function Confirm-HarnessModel { } $arguments = @('-Runner', $RunnerName, '-RequireModel', $ModelName) - if (-not [string]::IsNullOrWhiteSpace($ModelCatalogPath)) { - $arguments += @('-CatalogPath', $ModelCatalogPath) + if (-not [string]::IsNullOrWhiteSpace($CatalogPath)) { + $arguments += @('-CatalogPath', $CatalogPath) } $discoveryOutput = & pwsh -NoProfile -NonInteractive -File $discoveryScript @arguments 2>&1 if ($LASTEXITCODE -ne 0) { @@ -464,7 +495,7 @@ function Resolve-ExecutionSelection { if ($supportedRunners -notcontains $referenceRunner) { throw "Codebelt Reference requires runner '$referenceRunner', but it is unavailable. Supported runner IDs: $supportedText." } - [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel) + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel -CatalogPath $ModelCatalogPath) return [pscustomobject]@{ Runner = $referenceRunner @@ -482,6 +513,11 @@ function Resolve-ExecutionSelection { if (-not $hasRunner -and $hasModel) { throw 'Runner/model selection requires -Runner when -Model is supplied.' } + # Normalize the user-facing names before discovery; model policy remains runner-owned here. + $Runner = switch ($Runner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } $resolvedRunnerMatch = @($supportedRunners | Where-Object { [string]::Equals($_, $Runner, [StringComparison]::OrdinalIgnoreCase) }) if ($resolvedRunnerMatch.Count -ne 1) { throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." @@ -508,7 +544,7 @@ function Resolve-ExecutionSelection { } } - [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $resolvedModel) + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $resolvedModel -CatalogPath $ModelCatalogPath) return [pscustomobject]@{ Runner = $resolvedRunner @@ -690,14 +726,17 @@ function New-ResultStub { [object]$EvalEntry, [string]$EvalName, [string]$Configuration, - [string[]]$Assertions + [object[]]$Assertions ) $grading = foreach ($assertion in $Assertions) { [ordered]@{ - text = $assertion + text = Get-AssertionText -Assertion $assertion passed = $null evidence = '' + evidence_domain = [string](Get-JsonProperty -Object $assertion -Name 'evidence_domain' -Default 'output') + evidence_refs = @() + reason = '' } } @@ -765,6 +804,67 @@ function New-ExecutionProfile { } } +function Resolve-AnalyzerSelection { + param( + [Parameter(Mandatory = $true)][string]$RepoRoot, + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [string]$ExecutionReasoningEffort + ) + + # The analyzer/grader is a distinct, persisted, reproducible profile - never "whichever model happens to host the + # outer orchestrator" and never silently matched to the executor. The repository-owned default reuses the existing + # Codebelt Reference policy as a stable analyzer stratum across executor providers. + $referenceRunner = 'github-copilot' + $referenceModel = 'claude-haiku-4.5' + $hasRunner = -not [string]::IsNullOrWhiteSpace($AnalyzerRunner) + $hasModel = -not [string]::IsNullOrWhiteSpace($AnalyzerModel) + if ($hasModel -and -not $hasRunner) { throw 'Analyzer selection requires -AnalyzerRunner when -AnalyzerModel is supplied.' } + + if (-not $hasRunner -and -not $hasModel) { + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $referenceRunner -ModelName $referenceModel -CatalogPath $AnalyzerModelCatalogPath) + return [pscustomobject]@{ + Runner = $referenceRunner + Model = $referenceModel + ReasoningEffort = $null + Harness = Get-HarnessName -RunnerName $referenceRunner + Source = 'codebelt-reference' + } + } + + if (-not $hasModel) { throw 'Analyzer selection requires -AnalyzerModel when -AnalyzerRunner is supplied.' } + $resolvedRunner = switch ($AnalyzerRunner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } + $supportedRunners = @(Get-SupportedRunnerIds -RepoRoot $RepoRoot) + if ($supportedRunners -notcontains $resolvedRunner) { + throw "Analyzer runner '$resolvedRunner' is not a supported runner id ($($supportedRunners -join ', '))." + } + [void](Confirm-HarnessModel -RepoRoot $RepoRoot -RunnerName $resolvedRunner -ModelName $AnalyzerModel -CatalogPath $AnalyzerModelCatalogPath) + $reasoning = if (-not [string]::IsNullOrWhiteSpace($AnalyzerReasoningEffort)) { $AnalyzerReasoningEffort } elseif ($resolvedRunner -eq 'codex') { 'low' } else { $null } + return [pscustomobject]@{ + Runner = $resolvedRunner + Model = $AnalyzerModel + ReasoningEffort = $reasoning + Harness = Get-HarnessName -RunnerName $resolvedRunner + Source = 'explicit' + } +} + +function New-AnalyzerProfile { + param([Parameter(Mandatory = $true)][object]$AnalyzerSelection) + + return [ordered]@{ + schema = $analyzerProfileSchema + contract_version = $analyzerContractVersion + runner = $AnalyzerSelection.Runner + harness = $AnalyzerSelection.Harness + model = $AnalyzerSelection.Model + reasoning_effort = $AnalyzerSelection.ReasoningEffort + selection_source = $AnalyzerSelection.Source + } +} + function Resolve-EffectiveConcurrency { param( [Parameter(Mandatory = $true)][string]$RunnerName, @@ -833,7 +933,7 @@ function Get-Assertions { param([object]$EvalEntry) if ($EvalEntry.PSObject.Properties.Name -contains 'expectations' -and $null -ne $EvalEntry.expectations) { - return @($EvalEntry.expectations | ForEach-Object { [string]$_ }) + return @(Get-NormalizedAssertions -Assertions @($EvalEntry.expectations)) } return @() @@ -923,6 +1023,8 @@ function Get-EvalWorkspaceOption { if ($EvalEntry.PSObject.Properties.Name -contains 'workspace' -and $null -ne $EvalEntry.workspace) { $workspace = $EvalEntry.workspace if ($workspace.PSObject.Properties.Name -contains 'git' -and $null -ne $workspace.git) { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $workspace.git $wantsGit = [bool]$workspace.git } } @@ -1031,8 +1133,12 @@ function Initialize-GitWorkspace { '-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', + '-c', 'core.hooksPath=', + '-c', 'init.templateDir=', '-c', 'core.autocrlf=false' ) + $previousAuthorDate = $env:GIT_AUTHOR_DATE + $previousCommitterDate = $env:GIT_COMMITTER_DATE $env:GIT_AUTHOR_DATE = '2020-01-01T00:00:00Z' $env:GIT_COMMITTER_DATE = '2020-01-01T00:00:00Z' try { @@ -1050,8 +1156,8 @@ function Initialize-GitWorkspace { } & git @identity -C $RepoDirectory tag 'v1.0.0' 2>$null | Out-Null } finally { - Remove-Item Env:GIT_AUTHOR_DATE -ErrorAction SilentlyContinue - Remove-Item Env:GIT_COMMITTER_DATE -ErrorAction SilentlyContinue + $env:GIT_AUTHOR_DATE = $previousAuthorDate + $env:GIT_COMMITTER_DATE = $previousCommitterDate } } @@ -1251,6 +1357,7 @@ function New-RunManifest { [string[]]$RepoFiles, [string]$FixtureHash, [string]$SkillHash, + [string]$CandidateInstructionHash = $null, [bool]$GitWorkspace, [string]$InteractionFile = '', [string]$InteractionHash = '' @@ -1278,6 +1385,7 @@ function New-RunManifest { inputFiles = @($RepoFiles) fixtureHash = $FixtureHash skillHash = if ($Configuration -eq 'with_skill') { $SkillHash } else { $null } + candidateInstructionHash = if ($Configuration -eq 'with_skill') { $CandidateInstructionHash } else { $null } contract = [ordered]@{ sandboxRoot = '.' workingDirectory = $runDirectoryNames.Working @@ -1425,6 +1533,10 @@ function Invoke-PrepareMode { $executionSelection = Resolve-ExecutionSelection -RepoRoot $repoRoot $effectiveConcurrency = Resolve-EffectiveConcurrency -RunnerName ([string]$executionSelection.Runner) -RequestedConcurrency $Concurrency -ConcurrencyWasExplicit ($scriptBoundParameters.ContainsKey('Concurrency')) + $executionProfile = New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value) + # Validate the analyzer/grader profile before any output directory is created so an unknown analyzer model leaves no + # partial package behind. The profile is persisted later once the iteration root exists. + $analyzerSelection = Resolve-AnalyzerSelection -RepoRoot $repoRoot -ExecutionSelection $executionSelection -ExecutionReasoningEffort ([string]$executionProfile.reasoning_effort) $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" @@ -1453,6 +1565,12 @@ function Invoke-PrepareMode { if (-not $Force) { throw "'$iterationDirectory' already exists. Pass -Force to replace it, or -Iteration to write a new one." } + # Force must not erase a one-shot reservation or evidence of an active/completed run. + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $marker)) { + throw "Cannot replace '$iterationDirectory': handoff or execution has already started ($marker). Preserve this package and use a fresh iteration." + } + } Remove-Item -LiteralPath $iterationDirectory -Recurse -Force } New-Item -ItemType Directory -Path $iterationDirectory -Force | Out-Null @@ -1466,7 +1584,13 @@ function Invoke-PrepareMode { sha256 = Get-TreeHash -Root $copiedRunnerTools file_count = @(Get-ChildItem -LiteralPath $copiedRunnerTools -Recurse -File -Force).Count } - ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value)) + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value $executionProfile + # The analyzer/grader profile is validated during preparation exactly like the execution profile and persisted + # separately so executor identity and analyzer identity stay independently attributable in evidence and reports. + $analyzerProfile = New-AnalyzerProfile -AnalyzerSelection $analyzerSelection + $analyzerProfilePath = Join-Path $iterationDirectory 'analyzer-profile.json' + ConvertTo-JsonFile -Path $analyzerProfilePath -Value $analyzerProfile + $analyzerProfileHash = Get-FileSha256 -Path $analyzerProfilePath $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1477,6 +1601,14 @@ function Invoke-PrepareMode { $generatedUtc = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ') $withSkillInstructions = New-SkillInstructionSection -SkillName $Skill -SkillBody $skillBody -Inventory $inventory + # The candidate-instruction identity proof hashes exactly these canonical bytes (LF-normalized to match the written + # prompt). The runner and bridge isolate them as the prompt prefix before the working-environment marker, so the + # candidate instructions must never themselves contain that marker, or the proof boundary would be ambiguous. + $candidateInstructionCanonical = $withSkillInstructions -replace "`r`n", "`n" -replace "`r", "`n" + if ($candidateInstructionCanonical.Contains("`n`n# Working environment")) { + throw 'Candidate instructions must not contain the working-environment boundary marker; the candidate-instruction identity proof would be ambiguous.' + } + $candidateInstructionHash = Get-Sha256Hex -Bytes ([System.Text.Encoding]::UTF8.GetBytes($candidateInstructionCanonical)) $manifestEvals = [System.Collections.Generic.List[object]]::new() foreach ($evalEntry in $selectedEvals) { @@ -1532,6 +1664,8 @@ function Invoke-PrepareMode { } if ($workspaceOption.Git) { Initialize-GitWorkspace -RepoDirectory $repoDir + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Add-EvalGitScenario -RepoDirectory $repoDir -Scenario $evalEntry.workspace.git } $homeDir = Join-Path $runDir $runDirectoryNames.Home @@ -1548,13 +1682,34 @@ function Invoke-PrepareMode { # runs would otherwise differ, while the tracked fixture content is the same. $fixtureHash = Get-TreeHash -Root (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Working) -ExcludeSegments @('.git') - $inputFilesSection = New-InputFilesSection -Fixtures @($fixtures) + # Git-workspace evals must never inline fixture source into the prompt: the Git scenario mutates the working tree + # after fixtures are materialized (for example a feature commit that removes an API), so an inlined base copy would + # disagree with the final staged tree the worker actually reads. Omit inline source for these evals so the worker + # inspects the repository; the working-environment section already tells it to treat the repo as the source of + # truth. For non-Git evals, materialized fixtures still match their inline copies, so inlining is safe. + $inputFilesSection = if ($workspaceOption.Git) { $null } else { New-InputFilesSection -Fixtures @($fixtures) } $assertions = Get-Assertions -EvalEntry $evalEntry $interactionDocument = New-InteractionDocument -EvalEntry $evalEntry $withSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withSkillInstructions -InputFilesSection $inputFilesSection $withoutSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withoutSkillPreamble -InputFilesSection $inputFilesSection + # Deterministic guard: a Git-workspace prompt must not carry inlined fixture content that could disagree with the + # final working tree. It holds by omission above (no inline section is emitted for Git evals); this guard also + # protects against future regressions and, if inline source is ever re-enabled for a Git eval, requires + # byte-equality with the materialized working-tree file rendered after the Git scenario ran. + if ($workspaceOption.Git -and -not [string]::IsNullOrWhiteSpace($inputFilesSection)) { + $finalWorkingTree = Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Working + foreach ($fixture in @($fixtures)) { + if (-not $fixture.Inlined) { continue } + $materializedPath = Join-Path $finalWorkingTree ($fixture.RepoRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $materialized = if (Test-Path -LiteralPath $materializedPath -PathType Leaf) { [System.IO.File]::ReadAllText($materializedPath, $utf8NoBom) } else { $null } + if ($null -eq $materialized -or ($materialized.TrimEnd() -ne ([string]$fixture.Content).TrimEnd())) { + throw "Git-workspace eval '$evalName' inlines fixture '$($fixture.RepoRelative)', but the inline content disagrees with the final staged working tree. Omit inline source for Git-workspace evals so the prompt cannot contradict the repository." + } + } + } + Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Prompt) -Content $withSkillPrompt Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Prompt) -Content $withoutSkillPrompt @@ -1569,8 +1724,8 @@ function Invoke-PrepareMode { if ((Get-FileSha256 -Path $interactionPath) -ne $interactionHash) { throw "Scripted interaction sidecar diverged between configurations for '$evalName'." } } } - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -CandidateInstructionHash $candidateInstructionHash -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -CandidateSkillName $candidateSkillName -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -CandidateInstructionHash $null -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) $assumptions = [System.Collections.Generic.List[string]]::new() $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') @@ -1582,6 +1737,7 @@ function Invoke-PrepareMode { } if ($workspaceOption.Git) { $assumptions.Add('This eval stages a real .git in repo/ so repository-root detection and version-deriving tools behave as on a developer machine.') + $assumptions.Add('Input files are intentionally not inlined for this Git-workspace eval; the worker must inspect the staged repository (branches, commits, and diff) rather than rely on any prompt copy, which could disagree with the final working tree.') } $assumptions.Add('The expected output and assertions in this file are the grading key. They live outside every run directory and must never reach a worker.') if ($null -ne $interactionDocument) { @@ -1670,6 +1826,11 @@ function Invoke-PrepareMode { }) } + Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -EffectiveConcurrency $effectiveConcurrency) + $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' + Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency ([int]$effectiveConcurrency.Value) -PerArmTimeoutSeconds $TimeoutSeconds) + $runnerPromptHash = Get-FileSha256 -Path $runnerPath + $manifest = [ordered]@{ schema = $packageSchema skill_name = $Skill @@ -1685,13 +1846,28 @@ function Invoke-PrepareMode { preset = $executionSelection.Preset } runner_prompt = 'RUN-THIS.prompt.md' + runner_prompt_sha256 = $runnerPromptHash execution_profile = 'execution-profile.json' + analyzer_profile = 'analyzer-profile.json' + analyzer_profile_sha256 = $analyzerProfileHash + analyzer_selection = [ordered]@{ + runner = $analyzerSelection.Runner + harness = $analyzerSelection.Harness + model = $analyzerSelection.Model + reasoning_effort = $analyzerSelection.ReasoningEffort + selection_source = $analyzerSelection.Source + contract_version = $analyzerContractVersion + analyzer_profile_sha256 = $analyzerProfileHash + } runner_protocol = $runnerProtocolSchema runner_tools = $evalRunnerToolRelativePath runner_tools_integrity = $runnerToolsIntegrity execution_result_schema = $executionResultSchema execution_freeze = 'execution-freeze.json' + phase2_controller = "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1" + phase2_state = 'phase2-state.json' grading = 'grading.json' + grading_freeze = 'grading-freeze.json' grading_validator = "$evalRunnerToolRelativePath/validate-eval-grading.ps1" grading_contract = "$evalRunnerToolRelativePath/contracts/grading.schema.json" finalizer = "$evalRunnerToolRelativePath/finalize-eval-package.ps1" @@ -1736,10 +1912,6 @@ function Invoke-PrepareMode { evals = @($manifestEvals) } ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'manifest.json') -Value $manifest - - Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -EffectiveConcurrency $effectiveConcurrency) - $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' - Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency ([int]$effectiveConcurrency.Value) -PerArmTimeoutSeconds $TimeoutSeconds) [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iterationDirectory -Manifest $manifest) [void](Assert-PackageRunnerIdentity -IterationDirectory $iterationDirectory -Manifest $manifest -ExpectedRunner ([string]$executionSelection.Runner)) @@ -1755,6 +1927,13 @@ function Invoke-PrepareMode { Write-Host " Preset: $($executionSelection.Preset)" } Write-Host '' + Write-Host 'Analyzer:' + Write-Host " Harness: $($analyzerSelection.Harness)" + Write-Host " Runner: $($analyzerSelection.Runner)" + Write-Host " Model: $($analyzerSelection.Model)" + Write-Host " Source: $($analyzerSelection.Source)" + Write-Host " Profile: analyzer-profile.json ($analyzerProfileHash)" + Write-Host '' Write-Host "Cases: $($manifestEvals.Count)" Write-Host "Arms: $($manifestEvals.Count * 2)" Write-Host '' @@ -1781,6 +1960,7 @@ function Invoke-PrepareMode { Write-Host 'The selected evaluator must write valid runner-produced execution results back into this package.' Write-Host 'If it cannot, the evaluation is incomplete and must fail closed; only persisted runner-produced evidence at' Write-Host 'the manifest-declared paths may proceed.' + if ($PassThru) { Write-Output $runnerPath } } function New-RunnerPrompt { @@ -1798,7 +1978,7 @@ function New-RunnerPrompt { $profilePath = Join-Path $IterationDirectory 'execution-profile.json' $runnerOwnedFanoutPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1" $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" - $gradingValidatorPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/validate-eval-grading.ps1" + $phase2AnalyzerPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1" $finalizerPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1" $maxScriptedUserTurns = 1 foreach ($manifestEval in @($ManifestEvals)) { @@ -1841,20 +2021,20 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Read the selected runner descriptor and its `delegation.dispatch_owner`. For runner-owned behavioral transport, invoke:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$runnerOwnedFanoutPath`" -IterationDirectory `"$IterationDirectory`"") [void]$builder.AppendLine('It performs every preflight before any execute process, preserves exact manifest paths, owns concurrency/backpressure, timeout/watchdog handling, terminal registration, orchestration evidence, and immutable `execution-freeze.json` before Phase 2. Consume its terminal JSON summary. If Phase 1 reports incompatible or fails, stop: the evaluation is incomplete and must fail closed. The evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check. Do not create outer workers, execute an arm yourself, write orchestration state, or edit raw result/evidence files. If dispatch ownership is orchestrator-owned, use only the descriptor-declared native worker transport, the exact manifest paths, and then run the shared freeze boundary; do not synthesize or repair transport evidence. Only persisted runner-produced evidence at the manifest-declared paths may proceed.') + [void]$builder.AppendLine('For `github-copilot`, authentication incompatibility is terminal for this package iteration: report the missing supported non-interactive source (explicit `COPILOT_GITHUB_TOKEN`/`GH_TOKEN`/`GITHUB_TOKEN` or trusted `gh auth token` fallback), preserve the package, and stop. Do not suggest switching to another runner, do not suggest starting another Orchestrator, and do not rerun Phase 1.') [void]$builder.AppendLine('Workers receive only their isolated run directory. Keep the paired arm, metadata, expected output, assertions, grading, reports, and orchestration files out of Phase 1. Preserve runner-owned terminal results and all referenced raw transcript/event artifacts exactly as written.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2 — grading and finalization') [void]$builder.AppendLine() [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") - [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') - [void]$builder.AppendLine('To display the authoritative top-level grading skeleton, run:') - [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -ShowSkeleton") - [void]$builder.AppendLine('Write `grading.json`, then validate it before finalization:') - [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -IterationDirectory `"$IterationDirectory`" -GradingPath `"grading.json`"") - [void]$builder.AppendLine('Grading validation is retryable; finalization is not. If validation fails, correct `grading.json` and rerun the validation command as many times as required. Do not invoke the application helper separately; the finalizer invokes `apply-eval-grading.ps1` deterministically after revalidating grading. Invoke finalization exactly once, and only after grading validation succeeds:') + [void]$builder.AppendLine('Only if that bridge succeeds, invoke the package-local Phase 2 analyzer controller:') + [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$phase2AnalyzerPath`" -IterationDirectory `"$IterationDirectory`"") + [void]$builder.AppendLine('The Phase 2 controller, not this outer orchestrator, reads the grading key, resolves validator-domain assertions deterministically, creates one isolated analyzer run per semantic arm, dispatches those analyzer runs with bounded concurrency, validates observed analyzer runner/model/session identity, writes `phase2-state.json`, freezes `grading-freeze.json`, and deterministically merges root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`. Each grade is keyed by eval/configuration/assertion_index and carries passed, evidence_domain, evidence_refs, reason, and upstream-compatible evidence. If it fails, stop: do not author or repair grading.json, do not retry an analyzer worker, do not change analyzer-profile.json, do not switch providers, and do not grade in this context.') + [void]$builder.AppendLine('The analyzer profile persisted at `analyzer-profile.json` is the only grading stratum. The analyzer is independent from the executor and must not inherit executor HOME, provider configuration, session, native tools, skill catalogs, plugins, paired-arm output, sibling evals, source checkout, or candidate skill content. The controller supplies only one arm''s allowed evidence bundle and the packaged `tools/skill-creator/agents/grader.md` contract to each analyzer worker.') + [void]$builder.AppendLine('After Phase 2 succeeds, invoke finalization exactly once:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$finalizerPath`" -IterationDirectory `"$IterationDirectory`"") - [void]$builder.AppendLine('The finalizer revalidates the manifest, profile, terminal orchestration/concurrency evidence, immutable freeze, raw artifacts, bridge, canonical results, and grading; it then generates and verifies all required reports. Return only its machine-readable JSON summary and artifact paths. A non-zero exit, missing artifact, integrity error, or report error means the evaluation is incomplete. Never repair, re-freeze, re-bridge a changed raw result, or report prose success.') + [void]$builder.AppendLine('The finalizer revalidates the manifest, profile, terminal orchestration/concurrency evidence, immutable Phase 1 freeze, bridge, Phase 2 state, analyzer evidence, grading freeze, deterministic grading hash/cardinality, canonical results, and reports. Return only its machine-readable JSON summary and artifact paths. A non-zero exit, missing artifact, integrity error, or report error means the evaluation is incomplete. Never repair, re-freeze, re-bridge a changed raw result, or report prose success.') [void]$builder.AppendLine() [void]$builder.AppendLine('The four required package-root artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`. Same-session scripted evals, when present in a run, are handled by the selected runner only if its descriptor/preflight proves `scripted_multi_turn_same_session`; otherwise preflight fails before execution. The paired configurations receive identical scripted user turns.') [void]$builder.AppendLine() @@ -1896,7 +2076,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports, then grades and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports. Phase 2 grading is owned by the package-local analyzer controller, not by the outer orchestrator.') [void]$builder.AppendLine() [void]$builder.AppendLine("Execution selection: runner=$($ExecutionSelection.Runner); model=$($ExecutionSelection.Model); timeout_seconds=$TimeoutSeconds; concurrency=$($EffectiveConcurrency.Value); concurrency_source=$($EffectiveConcurrency.Source).") [void]$builder.AppendLine() @@ -1907,7 +2087,7 @@ function New-PackageReadme { } [void]$builder.AppendLine() [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') - [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, `analyzer-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Orchestration topology') [void]$builder.AppendLine() @@ -1934,9 +2114,9 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine(('1. Read `execution-profile.json` and the selected runner descriptor. If `runner` or `model` is missing or unsupported, fail clearly instead of guessing. For `delegation.dispatch_owner=runner`, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1") + ' exactly once with the caller shell/tool timeout set to at least the package-computed Phase 1 allowance. It performs all preflight, native dispatch, concurrency, terminal registration, timeout handling, and raw-evidence freezing. For `delegation.dispatch_owner=orchestrator`, use only the descriptor-declared native worker mechanism, then invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/freeze-execution-evidence.ps1") + ' after every arm is terminal.')) [void]$builder.AppendLine('2. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. Do not execute an arm in the parent context, create a second worker for a runner-owned arm, expose grading material during execution, or author/repair raw evidence. If Phase 1 reports incompatible or Phase 1/freezing fails, stop: the evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check.') - [void]$builder.AppendLine(('3. After the freeze succeeds, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1") + ' with `-RequireComplete -RequireParallelDispatch` and add `-RequireNativeDelegation` when the selected descriptor has `dispatch_owner=runner`. Only after that deterministic bridge succeeds, give the grading key to the Grader. The Grader writes only the package-root `grading.json` grading-only artifact. It must not modify execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.')) - [void]$builder.AppendLine(('4. To display the authoritative grading skeleton, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/validate-eval-grading.ps1") + ' with `-ShowSkeleton`. After writing `grading.json`, invoke the same helper with `-IterationDirectory "' + $IterationDirectory + '" -GradingPath "grading.json"` before finalization. Grading validation is retryable; finalization is not. Correct and rerun validation until it succeeds.')) - [void]$builder.AppendLine(('5. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1") + ' exactly once, only after grading validation succeeds. It invokes the deterministic apply-eval-grading boundary, validates the frozen evidence, idempotent bridge, complete grading, and report outputs. Return only its machine-readable summary.')) + [void]$builder.AppendLine(('3. After the freeze succeeds, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1") + ' with `-RequireComplete -RequireParallelDispatch` and add `-RequireNativeDelegation` when the selected descriptor has `dispatch_owner=runner`. This deterministic bridge validates the frozen execution evidence and canonical result paths before any grading begins.')) + [void]$builder.AppendLine(('4. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-phase2-analyzer.ps1") + ' once. It resolves validator assertions, starts one fresh analyzer worker per remaining semantic arm using `analyzer-profile.json`, persists `phase2-state.json`, freezes `grading-freeze.json`, and deterministically merges package-root `grading.json`. Do not grade in the outer orchestrator, retry failed analyzer workers, edit analyzer-profile.json, or manually repair grading.json.')) + [void]$builder.AppendLine(('5. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1") + ' exactly once, only after Phase 2 succeeds. It validates the frozen Phase 1 evidence, bridge, Phase 2 state/freeze, deterministic grading hash/cardinality, canonical grading application, and report outputs. Return only its machine-readable summary.')) [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It never executes an eval arm in its own model context. Same-session scripted interactions are allowed only when the selected runner proves that capability; paired runs receive identical deterministic turns. The package is complete only when the finalizer exits successfully.') [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6858d9c..976fe81 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -463,17 +463,24 @@ function Invoke-ValidationScriptJobs { $dotnetHome = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-dotnet-home-' + [Guid]::NewGuid().ToString('N')) $oldDotNetCliHome = $env:DOTNET_CLI_HOME $oldXdgDataHome = $env:XDG_DATA_HOME + $oldTemp = $env:TEMP + $oldTmp = $env:TMP try { New-Item -ItemType Directory -Path $dotnetHome -Force | Out-Null $env:DOTNET_CLI_HOME = $dotnetHome $env:XDG_DATA_HOME = Join-Path $dotnetHome 'share' + $env:TEMP = Join-Path $dotnetHome 'tmp' + $env:TMP = $env:TEMP + New-Item -ItemType Directory -Path $env:TEMP -Force | Out-Null $output = & $Script.Path 2>&1 $exitCode = $LASTEXITCODE } finally { $env:DOTNET_CLI_HOME = $oldDotNetCliHome $env:XDG_DATA_HOME = $oldXdgDataHome + $env:TEMP = $oldTemp + $env:TMP = $oldTmp if (Test-Path $dotnetHome) { Remove-Item -Path $dotnetHome -Recurse -Force -ErrorAction SilentlyContinue } @@ -648,8 +655,9 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills include va if ($eval.workspace -isnot [System.Management.Automation.PSCustomObject]) { throw "$relativeEvalPath eval $($eval.id) has a non-object 'workspace'" } - if ($eval.workspace.PSObject.Properties.Name -contains 'git' -and $eval.workspace.git -isnot [bool]) { - throw "$relativeEvalPath eval $($eval.id) must declare 'workspace.git' as a boolean" + if ($eval.workspace.PSObject.Properties.Name -contains 'git') { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $eval.workspace.git } } } @@ -1277,6 +1285,54 @@ Add-ValidationResult -Results $results -Name 'Token normalization and benchmark if ($LASTEXITCODE -ne 0) { throw "Token reporting regression failed: $($output -join [Environment]::NewLine)" } } +Add-ValidationResult -Results $results -Name 'Copilot physical boundaries and checkpoint telemetry remain deterministic' -Group 'Runners' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-copilot-boundaries.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Copilot boundary regression failed: $($output -join [Environment]::NewLine)" } +} + +Add-ValidationResult -Results $results -Name 'Change-impact default resolution has paired deterministic Git history' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-change-impact-workspace.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Git scenario regression failed: $($output -join [Environment]::NewLine)" } +} + +Add-ValidationResult -Results $results -Name 'Git-workspace evals omit stale inline source and prove candidate-instruction identity' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-change-impact-9-' + [Guid]::NewGuid().ToString('N')) + try { + New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null + $catalogPath = Join-Path $packageRoot 'models.json' + [System.IO.File]::WriteAllText($catalogPath, '{"models":[{"id":"claude-haiku-4.5","operation":"language"}]}', $utf8NoBom) + $output = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-change-impact' -Runner 'github-copilot' -Eval 9 -OutputRoot $packageRoot -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for change-impact eval 9: $($output -join [Environment]::NewLine)" } + $evalDir = @(Get-ChildItem -Path $packageRoot -Recurse -Directory | Where-Object { $_.Name -like 'eval-09*' } | Select-Object -First 1) + if ($evalDir.Count -ne 1) { throw 'Prepared change-impact eval 9 directory was not found.' } + $withRun = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/run.json'), $utf8NoBom) | ConvertFrom-Json + $withoutRun = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'without_skill/run.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$withRun.candidateInstructionHash -notmatch '^[0-9a-f]{64}$') { throw 'with_skill run.json must declare a SHA-256 candidateInstructionHash.' } + if (-not [string]::IsNullOrWhiteSpace([string]$withoutRun.candidateInstructionHash)) { throw 'without_skill run.json must not declare a candidateInstructionHash.' } + $withPrompt = ([System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/prompt.md'), $utf8NoBom)) -replace "`r`n", "`n" -replace "`r", "`n" + $withoutPrompt = ([System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'without_skill/prompt.md'), $utf8NoBom)) -replace "`r`n", "`n" -replace "`r", "`n" + foreach ($pair in @(@('with_skill', $withPrompt), @('without_skill', $withoutPrompt))) { + if ($pair[1] -match '(?m)^# Input files') { throw "$($pair[0]) Git-workspace prompt must omit inline fixture source that could disagree with the final working tree." } + if ($pair[1] -match 'public static Widget Parse') { throw "$($pair[0]) prompt inlines the stale pre-removal Widget.Parse source." } + if ($pair[1] -match 'Use dotnet-change-impact') { throw "$($pair[0]) task must be skill-neutral and must not name the candidate skill." } + } + $marker = "`n`n# Working environment" + $idx = $withPrompt.IndexOf($marker, [System.StringComparison]::Ordinal) + if ($idx -lt 0) { throw 'with_skill prompt is missing the working-environment boundary.' } + $instruction = $withPrompt.Substring(0, $idx) + $injected = ([Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData([System.Text.Encoding]::UTF8.GetBytes($instruction)))).ToLowerInvariant() + if ($injected -ne [string]$withRun.candidateInstructionHash) { throw 'with_skill injected candidate instructions must hash to the frozen candidateInstructionHash.' } + $finalWidget = [System.IO.File]::ReadAllText((Join-Path $evalDir[0].FullName 'with_skill/repo/src/Widget.cs'), $utf8NoBom) + if ($finalWidget -match 'Parse') { throw 'The final staged working tree must reflect the feature commit that removed Widget.Parse.' } + } finally { + if (Test-Path -LiteralPath $packageRoot) { Remove-Item -LiteralPath $packageRoot -Recurse -Force -ErrorAction SilentlyContinue } + } +} + Add-ValidationResult -Results $results -Name 'Runner-owned orchestration remains deterministic' -Group 'Runners' -Action { if (-not [string]::IsNullOrWhiteSpace($Ref)) { return @@ -1430,11 +1486,18 @@ Add-ValidationResult -Results $results -Name 'Progress coalescing renders select } } +Add-ValidationResult -Results $results -Name 'Explicit eval requests preserve preparation and reserve one external handoff' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-eval-request.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Eval request regressions failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Group 'Preparation' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref $runnerReadme = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-runners/README.md' -GitRef $Ref + $evalRequest = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-request.ps1' -GitRef $Ref $prepare = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/prepare-skill-evals.ps1' -GitRef $Ref Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Portable Eval Handoff' @@ -1447,6 +1510,11 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '-CanDelegateFreshOrchestrator' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'authorization is already complete (`confirmation_required = false`, `dispatch_immediately = true`, `max_new_external_orchestrators = 1`)' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'A bounded native wait that returns no completed agent or otherwise no terminal result means still running, not failure; wait again on that same handle.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery mirrors every model exposed by all configured OpenCode providers' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'availability is presentation metadata only and never filters the selectable catalog' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made.' @@ -1477,15 +1545,28 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability' + Assert-Contains -Name 'README.md' -Content $readme -Needle '`external_handoff` means authorization is complete: do not ask again.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'an empty wait or timeout means still running, so wait again on the same handle' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Discovery is an availability check, not model ranking' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'choose a Harness + Model when the user did not already do so' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'OpenCode mirrors every model exposed by all configured providers' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'ask the user to choose one, and wait' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle '`external_handoff` means authorization is complete: do not ask again.' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'a wait that returns no completed agent or otherwise no terminal result means still running' Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' Assert-Contains -Name 'README.md' -Content $readme -Needle 'OpenCode discovery mirrors every model exposed by all configured providers' Assert-Contains -Name 'README.md' -Content $readme -Needle 'the user must choose from the discovered selectors before package preparation' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'OpenCode through `opencode models --verbose`' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'without filtering the selectable catalog' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle "[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator" + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'dispatch_immediately' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'max_new_external_orchestrators' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'wait_same_handle_again' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'New-ExternalEvalOrchestratorState' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'Update-ExternalEvalOrchestratorState' + Assert-NotContains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'claude-opus-4.7' Assert-Contains -Name 'README.md' -Content $readme -Needle 'full operational permission inside each isolated behavioral harness configuration' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Hard filesystem confinement is a separate optional outer capability' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'full harness operational permission inside each isolated eval boundary' @@ -1506,6 +1587,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`.bot/-workspace/` — the default.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Anywhere else inside the repository is forbidden' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Deterministic synthetic Git history is allowed only when it is the fixture being tested' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'throwaway Git history is still forbidden in the real source working tree' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'It did not run them, and nothing here will.' if (-not [string]::IsNullOrWhiteSpace($Ref)) { @@ -1535,6 +1618,16 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable if ([string]$manifest.execution_selection.runner -ne $ExpectedRunner -or [string]$manifest.execution_selection.model -ne $ExpectedModel) { throw "$Name manifest.execution_selection must preserve requested runner/model '$ExpectedRunner'/'$ExpectedModel'." } + if ([string]$manifest.runner_prompt -ne 'RUN-THIS.prompt.md') { + throw "$Name manifest.runner_prompt must declare RUN-THIS.prompt.md." + } + if ([string]$manifest.runner_prompt_sha256 -notmatch '^[0-9a-f]{64}$') { + throw "$Name manifest.runner_prompt_sha256 must be a lowercase SHA-256." + } + $actualPromptHash = (Get-FileHash -Algorithm SHA256 -LiteralPath $promptPath).Hash.ToLowerInvariant() + if ([string]$manifest.runner_prompt_sha256 -ne $actualPromptHash) { + throw "$Name manifest.runner_prompt_sha256 must match RUN-THIS.prompt.md byte-for-byte." + } $runnerTools = Join-Path $IterationDirectory ([string]$manifest.runner_tools) . (Join-Path $runnerTools 'runner-common.ps1') @@ -1587,6 +1680,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } try { $catalogPath = Join-Path $packageRoot 'fake-model-catalog.json' + $analyzerCatalogPath = Join-Path $packageRoot 'fake-analyzer-model-catalog.json' New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null [System.IO.File]::WriteAllText($catalogPath, (@' { @@ -1601,6 +1695,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable ] } '@), $utf8NoBom) + Copy-Item -LiteralPath $catalogPath -Destination $analyzerCatalogPath -Force $expectedSelectors = @( 'claude-haiku-4.5', @@ -1730,7 +1825,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $referencePackageRoot = Join-Path $packageRoot 'reference-package' - $referencePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath 2>&1 + $referencePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -CodebeltReference failed against the fake current catalog: $($referencePrepareOutput -join [Environment]::NewLine)" } @@ -1741,7 +1836,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PreparedRunnerIdentity -Name 'GitHub Copilot Codebelt Reference package' -IterationDirectory (Join-Path $referencePackageRoot 'iteration-1') -ExpectedRunner 'github-copilot' -ExpectedModel 'claude-haiku-4.5' $codexPackageRoot = Join-Path $packageRoot 'codex-package' - $codexPrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -ModelCatalogPath $catalogPath 2>&1 + $codexPrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for the Codex default fixture: $($codexPrepareOutput -join [Environment]::NewLine)" } @@ -1763,7 +1858,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $opencodePackageRoot = Join-Path $packageRoot 'opencode-package' - $opencodePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $opencodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath 2>&1 + $opencodePrepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $opencodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for the OpenCode fixture: $($opencodePrepareOutput -join [Environment]::NewLine)" } @@ -1775,7 +1870,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PreparedRunnerIdentity -Name 'OpenCode package' -IterationDirectory (Join-Path $opencodePackageRoot 'iteration-1') -ExpectedRunner 'opencode' -ExpectedModel 'provider-paid/Paid.Model' $explicitGithubPackageRoot = Join-Path $packageRoot 'github-explicit-runner-package' - $explicitGithubOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitGithubPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath 2>&1 + $explicitGithubOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitGithubPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for explicit GitHub Copilot selection: $($explicitGithubOutput -join [Environment]::NewLine)" } $explicitGithubProfile = [System.IO.File]::ReadAllText((Join-Path $explicitGithubPackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json if ([string]$explicitGithubProfile.runner -ne 'github-copilot' -or [string]$explicitGithubProfile.model -ne 'gpt-5.6-luna' -or $null -ne $explicitGithubProfile.reasoning_effort) { @@ -1783,6 +1878,51 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } Assert-PreparedRunnerIdentity -Name 'Explicit GitHub Copilot package' -IterationDirectory (Join-Path $explicitGithubPackageRoot 'iteration-1') -ExpectedRunner 'github-copilot' -ExpectedModel 'gpt-5.6-luna' + $referenceAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $referencePackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $codexAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $codexPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $opencodeAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $opencodePackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + $explicitGithubAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $explicitGithubPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + if ($referenceAnalyzerHash -notmatch '^[0-9a-f]{64}$' -or $referenceAnalyzerHash -ne $codexAnalyzerHash -or $referenceAnalyzerHash -ne $opencodeAnalyzerHash) { + throw 'The same repository analyzer policy must produce the same analyzer_profile_sha256 across GitHub Copilot, Codex, and OpenCode executor packages.' + } + if ($referenceAnalyzerHash -ne $explicitGithubAnalyzerHash) { + throw 'Changing the executor model must not change the analyzer_profile_sha256.' + } + foreach ($iterationToInspect in @((Join-Path $referencePackageRoot 'iteration-1'), (Join-Path $codexPackageRoot 'iteration-1'), (Join-Path $opencodePackageRoot 'iteration-1'), (Join-Path $explicitGithubPackageRoot 'iteration-1'))) { + $profileToInspect = [System.IO.File]::ReadAllText((Join-Path $iterationToInspect 'analyzer-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$profileToInspect.selection_source -eq 'executor-matched') { + throw 'Analyzer selection must not silently fall back to executor-matched.' + } + } + + $alternateAnalyzerPackageRoot = Join-Path $packageRoot 'alternate-analyzer-package' + $alternateAnalyzerOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $alternateAnalyzerPackageRoot -Runner 'github-copilot' -Model 'gpt-5.6-luna' -ModelCatalogPath $catalogPath -AnalyzerRunner 'github-copilot' -AnalyzerModel 'gpt-5.6-luna' -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for alternate analyzer selection: $($alternateAnalyzerOutput -join [Environment]::NewLine)" } + $alternateAnalyzerHash = [string](([System.IO.File]::ReadAllText((Join-Path $alternateAnalyzerPackageRoot 'iteration-1\manifest.json'), $utf8NoBom) | ConvertFrom-Json).analyzer_profile_sha256) + if ($alternateAnalyzerHash -eq $referenceAnalyzerHash) { + throw 'Changing the analyzer model must change analyzer_profile_sha256.' + } + + $invalidAnalyzerRoot = Join-Path $packageRoot 'invalid-analyzer-model-package' + $invalidAnalyzerOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $invalidAnalyzerRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' -ModelCatalogPath $catalogPath -AnalyzerRunner 'github-copilot' -AnalyzerModel 'missing-analyzer-model' -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 + if ($LASTEXITCODE -eq 0) { + throw 'Unknown analyzer model must fail preparation through analyzer model discovery.' + } + $invalidAnalyzerDiagnostic = $invalidAnalyzerOutput -join [Environment]::NewLine + if ($invalidAnalyzerDiagnostic -notmatch 'missing-analyzer-model') { + throw "Invalid analyzer rejection must name the missing analyzer model in the diagnostic; got: $invalidAnalyzerDiagnostic" + } + if ($invalidAnalyzerDiagnostic -notmatch '(?i)analyzer') { + throw "Invalid analyzer rejection must identify the failure as an analyzer model validation/discovery error; got: $invalidAnalyzerDiagnostic" + } + if ($invalidAnalyzerDiagnostic -notmatch '(?i)(github-copilot|analyzer.*runner|runner.*analyzer)') { + throw "Invalid analyzer rejection must identify the analyzer runner; got: $invalidAnalyzerDiagnostic" + } + if (Test-Path -LiteralPath $invalidAnalyzerRoot) { + Remove-Item -LiteralPath $invalidAnalyzerRoot -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create a package when analyzer model validation fails.' + } + $mismatchIteration = Join-Path $packageRoot 'mismatched-profile-package' Copy-Item -LiteralPath (Join-Path $referencePackageRoot 'iteration-1') -Destination $mismatchIteration -Recurse $mismatchProfilePath = Join-Path $mismatchIteration 'execution-profile.json' @@ -1794,7 +1934,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' Assert-PackageIdentityValidationFails -Name 'mismatched runner package' -IterationDirectory $mismatchIteration -ExpectedRunner 'github-copilot' -ExpectedMessagePattern 'execution-profile\.json runner .+ does not match manifest\.execution_selection\.runner' $explicitOpenCodePackageRoot = Join-Path $packageRoot 'opencode-explicit-concurrency-package' - $explicitOpenCodeOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitOpenCodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -Concurrency 16 2>&1 + $explicitOpenCodeOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitOpenCodePackageRoot -Runner 'opencode' -Model 'provider-paid/Paid.Model' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath -Concurrency 16 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for explicit OpenCode concurrency: $($explicitOpenCodeOutput -join [Environment]::NewLine)" } $explicitOpenCodeProfile = [System.IO.File]::ReadAllText((Join-Path $explicitOpenCodePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json if ([int]$explicitOpenCodeProfile.concurrency -ne 16) { throw 'Explicit OpenCode -Concurrency 16 must be honored without clamping.' } @@ -1833,7 +1973,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw 'OpenCode preparation must not create a package without an explicit model choice.' } - $prepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $prepareOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" } @@ -1865,6 +2005,19 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' [string]$manifest.report.benchmark_markdown -ne 'benchmark.md') { throw 'The prepared package manifest must declare the first-party report, upstream skill-creator tools, and output artifacts.' } + if ([string]$manifest.analyzer_profile -ne 'analyzer-profile.json') { + throw 'The prepared package manifest must declare the analyzer_profile artifact.' + } + $analyzerProfile = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'analyzer-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$analyzerProfile.schema -ne 'codebeltnet/agentic/eval-analyzer-profile/1') { + throw "analyzer-profile.json must declare the analyzer profile schema; got '$($analyzerProfile.schema)'." + } + if ([string]::IsNullOrWhiteSpace([string]$analyzerProfile.runner) -or [string]::IsNullOrWhiteSpace([string]$analyzerProfile.model) -or [string]::IsNullOrWhiteSpace([string]$analyzerProfile.contract_version)) { + throw 'analyzer-profile.json must persist a validated grader runner, model, and contract version distinct from the executor.' + } + if ([string]$manifest.analyzer_selection.model -ne [string]$analyzerProfile.model -or [string]$manifest.analyzer_selection.runner -ne [string]$analyzerProfile.runner) { + throw 'manifest.analyzer_selection must match the persisted analyzer-profile.json for independent executor/analyzer attribution.' + } if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/generate-eval-report.ps1')) -or -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/eval-report-template.html')) -or -not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'tools/skill-creator/scripts/aggregate_benchmark.py')) -or @@ -1899,11 +2052,13 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' 'assertion_index', 'passed', 'evidence', - 'validate-eval-grading.ps1', - 'Write `grading.json`, then validate it before finalization', - 'Grading validation is retryable; finalization is not', - 'only after grading validation succeeds', - 'apply-eval-grading.ps1', + 'evidence_domain', + 'evidence_refs', + 'invoke-phase2-analyzer.ps1', + 'phase2-state.json', + 'grading-freeze.json', + 'The Phase 2 controller, not this outer orchestrator', + 'After Phase 2 succeeds, invoke finalization exactly once', 'finalize-eval-package.ps1', 'machine-readable JSON summary', 'report.html', @@ -1959,10 +2114,11 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' 'Do not create outer workers', 'execution-freeze.json', 'grading.json', - 'validate-eval-grading.ps1', - 'Grading validation is retryable; finalization is not', - 'only after grading validation succeeds', - 'apply-eval-grading.ps1', + 'invoke-phase2-analyzer.ps1', + 'phase2-state.json', + 'grading-freeze.json', + 'evidence_refs', + 'After Phase 2 succeeds, invoke finalization exactly once', 'finalize-eval-package.ps1', 'evaluation is incomplete and must fail closed', 'Only persisted runner-produced evidence at the manifest-declared paths may proceed' @@ -2157,8 +2313,9 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw "$($entry.eval_name) prompts must not carry the expected output; that is the grading key." } foreach ($assertion in @($metadata.assertions)) { - if ($prompt.Contains([string]$assertion)) { - throw "$($entry.eval_name) prompts must not carry assertion '$assertion'; that is the grading key." + $assertionText = if ($assertion -is [string]) { [string]$assertion } elseif ($assertion.PSObject.Properties.Name -contains 'assertion') { [string]$assertion.assertion } else { '' } + if (-not [string]::IsNullOrWhiteSpace($assertionText) -and $prompt.Contains($assertionText)) { + throw "$($entry.eval_name) prompts must not carry assertion '$assertionText'; that is the grading key." } } } @@ -2290,6 +2447,22 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' $manifest.execution_selection.model = 'fixture-model' $manifest.execution_selection.harness = 'Deterministic runner-owned fixture' $manifest.execution_selection.preset = 'Deterministic validator' + $analyzerProfilePath = Join-Path $iterationDirectory ([string]$manifest.analyzer_profile) + $deterministicAnalyzerProfile = [System.IO.File]::ReadAllText($analyzerProfilePath, $utf8NoBom) | ConvertFrom-Json + $deterministicAnalyzerProfile.runner = 'fixture' + $deterministicAnalyzerProfile.model = 'fixture-model' + $deterministicAnalyzerProfile.harness = 'deterministic runner-owned fixture' + $deterministicAnalyzerProfile.reasoning_effort = $null + $deterministicAnalyzerProfile.selection_source = 'explicit' + [System.IO.File]::WriteAllText($analyzerProfilePath, (($deterministicAnalyzerProfile | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $analyzerProfileHash = Get-FileHash -Algorithm SHA256 -LiteralPath $analyzerProfilePath + $manifest.analyzer_profile_sha256 = $analyzerProfileHash.Hash.ToLowerInvariant() + $manifest.analyzer_selection.runner = 'fixture' + $manifest.analyzer_selection.model = 'fixture-model' + $manifest.analyzer_selection.harness = 'deterministic runner-owned fixture' + $manifest.analyzer_selection.reasoning_effort = $null + $manifest.analyzer_selection.selection_source = 'explicit' + $manifest.analyzer_selection.analyzer_profile_sha256 = $manifest.analyzer_profile_sha256 [System.IO.File]::WriteAllText((Join-Path $iterationDirectory 'manifest.json'), (($manifest | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) $fixtureMetricEnvironment = [ordered]@{ AGENTIC_RUNNER_FIXTURE_FINAL_RESPONSE = 'validator output' @@ -2325,43 +2498,21 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' [Environment]::SetEnvironmentVariable($environmentName, $fixtureMetricEnvironmentBefore[$environmentName]) } } - $executionCollectOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -CollectResults $iterationDirectory 2>&1 + $bridgePath = Join-Path $iterationDirectory 'tools/eval-runners/bridge-manifest-results.ps1' + $bridgeOutput = & pwsh -NoProfile -NonInteractive -File $bridgePath -IterationDirectory $iterationDirectory -RequireComplete -RequireParallelDispatch -RequireNativeDelegation 2>&1 if ($LASTEXITCODE -ne 0) { - throw "prepare-skill-evals.ps1 -CollectResults failed while bridging the complete deterministic fixture: $($executionCollectOutput -join [Environment]::NewLine)" - } - # The only post-execution artifact authored by this validator is a - # grading-only document with exact metadata identities. Canonical - # result grading is projected by the deterministic application helper. - $gradingEntries = [System.Collections.Generic.List[object]]::new() - foreach ($entryToGrade in @($manifest.evals)) { - $metadataPath = Join-Path $iterationDirectory ([string]$entryToGrade.metadata) - $metadataForGrade = [System.IO.File]::ReadAllText($metadataPath, $utf8NoBom) | ConvertFrom-Json - foreach ($configuration in @('with_skill', 'without_skill')) { - for ($assertionIndex = 0; $assertionIndex -lt @($metadataForGrade.assertions).Count; $assertionIndex++) { - $gradingEntries.Add([ordered]@{ - eval_id = [int]$entryToGrade.eval_id - eval_name = [string]$entryToGrade.eval_name - configuration = $configuration - assertion_index = $assertionIndex - assertion = [string]$metadataForGrade.assertions[$assertionIndex] - passed = $true - evidence = 'validator evidence' - }) - } - } + throw "The deterministic manifest bridge failed: $($bridgeOutput -join [Environment]::NewLine)" + } + $phase2Path = Join-Path $iterationDirectory 'tools/eval-runners/invoke-phase2-analyzer.ps1' + $phase2Output = & pwsh -NoProfile -NonInteractive -File $phase2Path -IterationDirectory $iterationDirectory -Concurrency 4 -TimeoutSeconds 60 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "The deterministic Phase 2 analyzer controller failed: $($phase2Output -join [Environment]::NewLine)" } - $gradingPath = Join-Path $iterationDirectory ([string]$manifest.grading) - [System.IO.File]::WriteAllText($gradingPath, (([ordered]@{ schema = 'codebeltnet/agentic/eval-grading/1'; grading = @($gradingEntries.ToArray()) } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) $validateGradingPath = Join-Path $iterationDirectory 'tools/eval-runners/validate-eval-grading.ps1' $validateGradingOutput = & pwsh -NoProfile -NonInteractive -File $validateGradingPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 if ($LASTEXITCODE -ne 0) { throw "The deterministic grading validation failed: $($validateGradingOutput -join [Environment]::NewLine)" } - $applyGradingPath = Join-Path $iterationDirectory 'tools/eval-runners/apply-eval-grading.ps1' - $applyOutput = & pwsh -NoProfile -NonInteractive -File $applyGradingPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "The deterministic grading-only application failed: $($applyOutput -join [Environment]::NewLine)" - } $finalizerPath = Join-Path $iterationDirectory 'tools/eval-runners/finalize-eval-package.ps1' $finalizerOutput = & pwsh -NoProfile -NonInteractive -File $finalizerPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 if ($LASTEXITCODE -ne 0) { @@ -2394,7 +2545,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' throw "benchmark.json must preserve recorded token metrics through the upstream skill-creator aggregation (with_skill=$($benchmark.run_summary.with_skill.tokens.mean), without_skill=$($benchmark.run_summary.without_skill.tokens.mean))." } - $changedOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Changed -Base 'HEAD' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $changedOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Changed -Base 'HEAD' -OutputRoot $packageRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -Changed failed: $($changedOutput -join [Environment]::NewLine)" } @@ -2403,7 +2554,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' } $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' - $isolationOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $isolationOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -eq 0) { throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' } @@ -2418,7 +2569,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' try { - $botOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath 2>&1 + $botOutput = & pwsh -NoProfile -NonInteractive -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -ModelCatalogPath $catalogPath -AnalyzerModelCatalogPath $analyzerCatalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" } @@ -2747,6 +2898,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces $commitLanguage = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/commit-language.md' -GitRef $Ref $subjectValidator = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/validate-commit-subject.ps1' -GitRef $Ref $subjectTests = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/test-commit-subject.ps1' -GitRef $Ref + $groupingExamples = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/grouping-examples.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'automatic trigger for this skill, not as a casual hint.' @@ -2811,17 +2963,17 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Single-category context quality gate' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Exactly one changed file is the only fast-path exception; skip this gate for that case.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then refresh the Step 1 inventory and review tracked diffs and untracked contents' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Put a visible line in the commit plan such as `Quality gate:' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` and `auto` do not bypass this gate.' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Release-adjacent splitting rule' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '#### Release-adjacent splitting rule' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'do not collapse "new skill introduced" and "existing skill refactored" into one commit' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '**New repo-managed skill**' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '**New repo-managed skill**' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Allowed Prefixes' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Emoji Selection' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle 'Gitmoji First, Fallback Second' diff --git a/skills/dotnet-change-impact/SKILL.md b/skills/dotnet-change-impact/SKILL.md index be53457..2089fda 100644 --- a/skills/dotnet-change-impact/SKILL.md +++ b/skills/dotnet-change-impact/SKILL.md @@ -328,11 +328,21 @@ there is truly no concrete change to list.> ## Compatibility impact -- Behavioral change: — -- Binary compatibility: — -- Source compatibility: — -- Design-time compatibility: — -- Backwards compatibility: — +State each category with one unambiguous categorical value — `Compatible`, `Breaking`, or +`Potentially breaking` — followed by the reason. Never answer these with `yes`/`no`: "yes" is +ambiguous about whether it means "yes, compatible" or "yes, it breaks", which has produced +self-contradictory answers such as "Binary compatibility: yes — compiled consumers may fail". + +- Behavioral compatibility: — +- Binary compatibility: — +- Source compatibility: — +- Design-time compatibility: — +- Backwards compatibility: — + +Read the values consistently: `Compatible` means existing consumers are unaffected in that +category, `Breaking` means they are broken, and `Potentially breaking` means the outcome depends +on a stated condition (for example, whether an API is public or whether consumers rely on the old +behavior). ## Reasoning diff --git a/skills/dotnet-change-impact/evals/evals.json b/skills/dotnet-change-impact/evals/evals.json index 9794de7..8cf472b 100644 --- a/skills/dotnet-change-impact/evals/evals.json +++ b/skills/dotnet-change-impact/evals/evals.json @@ -1,19 +1,19 @@ -{ - "skill_name": "dotnet-change-impact", - "evals": [ - { - "id": 1, - "prompt": "In my .NET library I removed the public method `HttpRetryHandler.Send(HttpRequestMessage)` and renamed the public property `Client.BaseUri` to `Client.BaseAddress`. Should I bump major, minor, or patch?", +{ + "skill_name": "dotnet-change-impact", + "evals": [ + { + "id": 1, + "prompt": "In my .NET library I removed the public method `HttpRetryHandler.Send(HttpRequestMessage)` and renamed the public property `Client.BaseUri` to `Client.BaseAddress`. Should I bump major, minor, or patch?", "expected_output": "Structured answer recommending Major with compatibility reasoning.", "expectations": [ "Recommends Major in the Recommendation section", "Includes Key changes identified, Compatibility impact, Reasoning, and Deterministic decision sections", "Treats removing a public method and renaming a public property as binary and source breaking" ] - }, - { - "id": 2, - "prompt": "I added a brand-new public class `RetryPolicy` and a new `WithRetry(this HttpClient client, int attempts)` extension method to my NuGet package. No existing public APIs were touched. What's the version bump?", + }, + { + "id": 2, + "prompt": "I added a brand-new public class `RetryPolicy` and a new `WithRetry(this HttpClient client, int attempts)` extension method to my NuGet package. No existing public APIs were touched. What's the version bump?", "expected_output": "Structured answer recommending Minor with compatibility reasoning.", "expectations": [ "Recommends Minor in the Recommendation section", @@ -21,10 +21,10 @@ "Recognizes purely additive, backward-compatible public surface as Minor", "Does not escalate to Major when nothing existing was changed or removed" ] - }, - { - "id": 3, - "prompt": "This release only fixes a NullReferenceException inside an internal cache class (no public API or observable behavior change), updates a transitive dependency from 8.0.1 to 8.0.3, and refreshes the README. Major, minor, or patch?", + }, + { + "id": 3, + "prompt": "This release only fixes a NullReferenceException inside an internal cache class (no public API or observable behavior change), updates a transitive dependency from 8.0.1 to 8.0.3, and refreshes the README. Major, minor, or patch?", "expected_output": "Structured answer recommending Patch with compatibility reasoning.", "expectations": [ "Recommends Patch in the Recommendation section", @@ -32,22 +32,22 @@ "Treats internal-only fix, compatible patch dependency bump, and docs as non-breaking", "Does not inflate an internal, non-observable refactor to Minor or Major" ] - }, - { - "id": 4, - "prompt": "I changed `DateParser.Parse` to throw a FormatException on empty input instead of returning null. Should this be a patch? It's just a bug fix.", + }, + { + "id": 4, + "prompt": "I changed `DateParser.Parse` to throw a FormatException on empty input instead of returning null. Should this be a patch? It's just a bug fix.", "expected_output": "Structured answer: ambiguous because it depends on whether Parse is public and whether consumers rely on the null-return behavior. Should not just say Patch.", "expectations": [ "Uses structured reasoning instead of emitting a bare single word", "Uses the structured template with Recommendation, Compatibility impact, Reasoning, and Deterministic decision sections", "Flags this as a behavioral change that can be breaking even though it compiles", "Does not accept the user's framing that a bug fix is automatically a Patch", - "Identifies the deciding fact: whether Parse is public and whether the old null-return behavior is relied upon or documented" - ] - }, - { - "id": 5, - "prompt": "Big release: I added two new public extension methods, fixed a couple of internal bugs, AND removed a public `ILogger` overload that nobody should be using. One command answer please: major, minor, or patch?", + "Identifies the deciding fact: whether Parse is public and whether the old null-return behavior is relied upon or documented" + ] + }, + { + "id": 5, + "prompt": "Big release: I added two new public extension methods, fixed a couple of internal bugs, AND removed a public `ILogger` overload that nobody should be using. One command answer please: major, minor, or patch?", "expected_output": "Structured answer recommending Major, because precedence picks the highest required bump and the public removal is breaking.", "expectations": [ "Recommends Major in the Recommendation section", @@ -55,29 +55,29 @@ "Applies precedence: the breaking public removal dominates the additive and internal changes", "Does not downgrade to Minor or Patch despite the user implying the removed overload is unused" ] - }, - { - "id": 6, - "prompt": "I added a new member `Task FlushAsync()` to my public interface `IMessageSink`. Existing implementations are out in the wild via my NuGet package. Why would this be more than a patch?", + }, + { + "id": 6, + "prompt": "I added a new member `Task FlushAsync()` to my public interface `IMessageSink`. Existing implementations are out in the wild via my NuGet package. Why would this be more than a patch?", "expected_output": "Structured answer recommending Major, explaining that adding a member to a public interface breaks existing implementers (source compatibility).", "expectations": [ "Recommends Major", "Explains that adding a member to an existing public interface breaks existing implementers' source compilation", "Mentions default interface members as a nuance without treating them as automatically safe", - "Uses the structured explanation template" - ] - }, - { - "id": 7, - "prompt": "I added a new enum value `LogLevel.Trace` to an existing public enum. My library serializes these values to JSON and some consumers switch over them. Patch, minor, or major?", + "Uses the structured explanation template" + ] + }, + { + "id": 7, + "prompt": "I added a new enum value `LogLevel.Trace` to an existing public enum. My library serializes these values to JSON and some consumers switch over them. Patch, minor, or major?", "expected_output": "Structured answer: usually Minor but flag the exhaustive-switch and serialization-contract risk that could make it behavioral/source-impacting.", "expectations": [ "Does not emit a bare one-word answer given the explicit switch and serialization caveats", "Notes that adding an enum value is usually Minor", - "Flags exhaustive consumer switch statements and serialization contracts as escalation risks", - "Uses the structured explanation template with a clear recommendation" - ] - }, + "Flags exhaustive consumer switch statements and serialization contracts as escalation risks", + "Uses the structured explanation template with a clear recommendation" + ] + }, { "id": 8, "prompt": "My library dropped support for net6.0 and now targets only net8.0 and net9.0. Existing consumers on net6.0 will be affected. Version bump?", @@ -91,15 +91,61 @@ }, { "id": 9, - "prompt": "Use dotnet-change-impact to classify the version bump for this branch.", + "files": ["evals/files/default-resolution/src/Widget.cs", "evals/files/default-resolution/src/Example.csproj"], + "workspace": { + "git": { + "base_branch": "trunk", + "feature_branch": "feature/remove-legacy-api", + "commits": [ + { + "message": "Remove the legacy public Parse overload", + "files": { "src/Widget.cs": "namespace Example;\n\npublic class Widget\n{\n public string Name { get; set; } = string.Empty;\n}\n" } + } + ] + } + }, + "prompt": "Classify the version bump for this branch.", "expected_output": "Current-branch default-resolution behavior: inspect the local Git repository, resolve the current branch against the upstream/default base branch, collect commits and net diff, then classify those changes with structured reasoning. Must not ask the user to provide change details first.", "expectations": [ - "Follows Default Resolution Behavior when no explicit change details or compare range are provided", - "Detects the current branch and resolves a base branch using local Git state, following the documented fallback order", - "Collects both branch commits and net diff before classifying compatibility impact", - "Returns structured reasoning with the recommendation instead of a bare Major, Minor, or Patch", - "Does not respond with a generic request for release notes, diffs, PR summaries, or bug descriptions when the current repository can be inspected", - "Asks only for the missing base branch or compare range if local default resolution cannot identify one" + { + "assertion": "Current Git branch was resolved before classifying", + "evidence_domain": "validator", + "validator": "git.current_branch_observed" + }, + { + "assertion": "Base/default branch was resolved using allowed local Git state", + "evidence_domain": "validator", + "validator": "git.default_branch_resolved" + }, + { + "assertion": "Branch-only commits were collected before classifying", + "evidence_domain": "validator", + "validator": "git.branch_commits_collected" + }, + { + "assertion": "Three-dot net diff was collected before classifying", + "evidence_domain": "validator", + "validator": "git.three_dot_diff_observed" + }, + { + "assertion": "No search was performed for a package or tool named dotnet-change-impact", + "evidence_domain": "validator", + "validator": "skill.no_candidate_package_search" + }, + { + "assertion": "No installation attempt was made for dotnet-change-impact", + "evidence_domain": "validator", + "validator": "skill.no_candidate_install_attempt" + }, + { + "assertion": "No unnecessary project, package, or tool mutation was performed for this read-only classification task", + "evidence_domain": "validator", + "validator": "workspace.no_unnecessary_mutation" + }, + { + "assertion": "Returns structured reasoning with the recommendation instead of a bare Major, Minor, or Patch", + "evidence_domain": "output" + } ] } ] diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj new file mode 100644 index 0000000..ec2cce1 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj @@ -0,0 +1,5 @@ + + + net8.0 + + diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs new file mode 100644 index 0000000..184db81 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs @@ -0,0 +1,7 @@ +namespace Example; + +public class Widget +{ + public string Name { get; set; } = string.Empty; + public static Widget Parse(string value) => new Widget { Name = value }; +} diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index 0700436..547cbbd 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -81,7 +81,7 @@ If the user did **not** say `yolo` or `auto`, and session-level auto mode is not If the user says `git bot commit`, `git commit`, or `git our commit` without narrowing language, treat the request as covering the full current worktree. -- The default scope is **all current changes visible in git status**. +- The default scope is **all current changes visible in git status**: staged, unstaged, deleted, renamed, and non-ignored untracked files, including files inside new directories. - Your job is to group that full worktree into the right number of commits by semantic intent. - Never silently narrow the scope to "just the files from the last thing I worked on", "just the files I touched", or "just the newest skill" unless the user explicitly said to do that. - `yolo` keeps this same full-worktree default. It removes the approval wait; it does not narrow scope. @@ -93,6 +93,8 @@ Narrow scope only when the user explicitly does one of these: - asks for a review/plan for a subset before committing If the user did not narrow scope, do not invent a narrower scope on their behalf. + +Use the expanded status inventory in Step 1 as the scope of record. `git diff`, `git diff --stat`, and `git diff --name-only` omit untracked files; none can replace that inventory. Untracked status alone is neither an exclusion nor a reason to ask permission. Respect Git's ignore rules for untracked files; do not force-add ignored files. Already tracked files remain in scope even if an ignore pattern now matches them. ### Recovery Safety Rule @@ -238,7 +240,9 @@ Commits include a body by default. `no-body` or `tmi` suppresses it for one requ ### Step 1: Review changes -Run `git status` and `git diff` (and `git diff --staged` if there are staged changes) to understand what has changed. +From the repository root, run `git status --short --untracked-files=all` to enumerate individual paths even when Git configuration hides untracked files. Keep this inventory through final verification. For programmatic parsing, use `git status --porcelain=v1 --untracked-files=all -z` and parse NUL-delimited records, including rename pairs; do not split filenames on whitespace. + +Review `git diff` and `git diff --staged` for tracked changes and inspect every untracked file directly with an appropriate text or binary viewer. An empty diff does not mean a clean worktree. Do not stage files just to make them appear in a diff. Unless the user explicitly narrowed scope, inspect the **entire current worktree** and build the commit plan from that full set of changes. Do not default to the last task only. @@ -250,7 +254,7 @@ When resolving that reference, prefer the bundled skill path first instead of tr ### Step 2: Classify changes -Before composing any commit message, bucket every changed file by its **semantic intent** — not just its file type. Read the actual diff for each file and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. +Before composing any commit message, bucket every path in the Step 1 inventory by its **semantic intent** — not its tracking status or file type. Read the actual diff for tracked files and contents of untracked files and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. Use the inspected commit-language reference as the meaning source, not your gut. For example, restructuring an existing skill's `SKILL.md`, `FORMS.md`, `references/`, or `evals/` is normally refactor intent and should map to `♻️`; configuration-file changes map to `🔧`; truly new repo or application capabilities map to `✨`. @@ -298,36 +302,9 @@ When more than one file is changed and your first classification puts every chan Ask yourself explicitly: **“Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?”** A metadata preview, remembered rule, or partial read is a failed answer. If the answer is no or uncertain, read `SKILL.md` from its first line through EOF and restart Step 1 and Step 2. -Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. +Then refresh the Step 1 inventory and review tracked diffs and untracked contents; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. Only keep one category after this audit if every path still has one rationale. Put a visible line in the commit plan such as `Quality gate: 3 files, one category retained; full skill read, full diff review, per-file rationale check, and alternative-category check confirmed.` If any check fails or any file has a materially different intent, split the groups and rerun the normal validation. `yolo` and `auto` do not bypass this gate. - -#### Commit body guidance - -Unless **no-body mode** is active, every commit includes a body explaining the *why*: - -- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") -- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") -- **Refactors** → explain what motivated the restructuring -- **New features** → explain the purpose and scope -- **Bug fixes** → explain what was broken and how this fixes it - -Common groupings: -- New repo-managed skill or workflow introduction together -- Existing skill refactor or extraction together -- Dependency/version baseline updates together -- Package/publish metadata together -- Config/setup files together (app host, bootstrapping) -- Environment and infrastructure config together (test runners, CI matrix, container settings) -- Documentation publishing fixes together -- Community health or release communication docs together -- New feature or module code together -- Data contracts, types, and interfaces together -- Database models, migrations, and schema changes together -- Test logic and assertions together (when they share the same rationale) -- Documentation and inline comments together - -When in doubt, one commit per "thing that changes" is better than one big commit. #### Mixed-scope guard @@ -339,40 +316,9 @@ This guard runs unconditionally — including in auto-approval mode. Documentation files (`CHANGELOG.md`, `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, release notes) are **separate-by-default**. They only belong in the same commit as non-doc files when the commit is explicitly documentation-focused (e.g. `📝 add api usage guide` where the docs are the point, not a side effect). -#### Release-adjacent splitting rule - -Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: - -- **Dependency/version baselines** — version alignment or runner baseline changes -- **Community health/release communication** — changelogs and human-facing repo health docs -- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` -- **Documentation publishing** — DocFX navigation, branding, or publishing assets -- **CI/automation** — workflows and helper scripts used only by automation - -These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. - -Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: - -- **Build system / dependency baseline** -- **CI or automation** -- **Release communication plus package metadata** - -Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. - -#### Repo-aligned grouping example - -When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: +#### Detailed grouping rules -- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` -- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill -- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` -- **Template/runtime files** — `assets/`, scaffold helper scripts -- **Validation/tooling** — validator scripts, repo checks -- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` - -Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". - -If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. +Read [grouping-examples.md](references/grouping-examples.md) before finalizing groups. Apply its release-adjacent splitting rule and repository examples alongside the guards above. #### Rename vs removal distinction @@ -409,7 +355,7 @@ Even in auto-approval mode, surface the commit buckets explicitly before committ The summary is status output, not a review request. Step 5 is mandatory in the same turn once its preconditions pass: never ask "Proceed with committing these groups?" (or an equivalent question), wait for a reply, or finish with a pending commit plan. -If the user did not narrow scope, the plan you surface must account for the full worktree rather than an arbitrarily chosen subset. +Reconcile the plan's exact path lists against a fresh Step 1 inventory before presenting it. Every in-scope path must belong to a commit group; splitting a file across groups must account for all its changes. Resolve missing paths before proceeding. Show a coverage line such as `Scope: 7 changed paths (3 untracked), 7 planned, 0 omitted.` Counts supplement the path comparison; equal counts alone do not prove coverage. Record explicit user exclusions separately. Apply this check in every mode, including single-file and auto-approved requests. **Otherwise**, wait for the user to confirm or adjust. They may say things like: - "Looks good" → proceed to stage and commit @@ -432,7 +378,8 @@ Before committing, validate each message against its file list: ### Step 5: Stage and commit each group For each group: -1. `git add ` — be precise, don't use `git add .` unless everything belongs in one commit +1. Refresh the inventory after approval and reconcile any changes with the plan. Stage each group's exact paths, including new files and deletions, with `git add -A -- `. Use literal pathspecs for names containing Git wildcard syntax. Do not use tracked-only staging such as `git add -u` or `git commit -a` as a substitute. + Compare `git diff --staged --name-status` and the full staged diff against the group's planned changes before committing. Account for pre-existing staged changes too; resolve missing or extra staged paths without discarding their contents. Keep staging and commits sequential because they share the index. 2. Compose the commit message (see format above) 3. Run `scripts/validate-commit-subject.ps1` again against the exact subject that will be passed to Git. Add `-PrefixMode Required` only for an explicitly requested combo. Do not run Git unless the validator exits successfully. 4. Run the appropriate commit command: @@ -449,35 +396,14 @@ When the body is just one short explanatory paragraph, prefer a single natural p ### Step 6: Verify +After each commit, compare its recorded paths (`git diff-tree --root --no-commit-id --name-status -r HEAD`) with the group, accounting for both sides of renames. After the last group, rerun `git status --short --untracked-files=all` and reconcile remaining paths with the original inventory and explicit exclusions. A full-worktree request is complete only when no in-scope changes remain, including untracked files. If a path was missed, review and include it through the same workflow under the existing authorization; do not silently leave it behind or claim completion. Report any concrete blocker or newly arriving change that prevents completion. + After committing, run `git log --oneline -5` to confirm the commit looks right. Read the stored subject with `git log -1 --format=%s` and run `scripts/validate-commit-subject.ps1` against that exact value. Then always run `git log -1 --format="%an <%ae>"` and verify that the author matches the requested identity mode before reporting success. Also run `git log -1 --format=%B` and verify the stored body contains readable prose with real line breaks, not literal escape sequences such as `\n`, and is not hard-wrapped mid-sentence just to satisfy a column limit. If any verification fails, amend the commit immediately instead of merely warning about it. If verification fails because the commit path used the wrong author or ignored the requested alias, stop treating it as a message-tweaking problem. Correct the tool path first, preserve the worktree safely, and only then repair the commit. --- -## Good Examples - -``` -🎉 begin api project -✨ add submission endpoint module -🐛 handle null optional fields in dto -➕ add validation library -🐛 fix: handle null optional fields in dto ← only when combo mode was requested -``` - -## Bad Examples (and why) - -``` -feat: add submission endpoint ← "feat:" is not an allowed prefix -✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed -💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning -💬 update changelog for v10.0.10 ← more than one separator space -📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table -🎉 initial commit with all files ← vague, bundles everything -⚙️ config: setup api ← "config:" is not an allowed prefix -♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo -``` - ## Branching (for reference) When explicitly requested, use `[version]/[description]` (for example `v1.2.0/admin-dashboard`). Do not create, rename, or delete branches unless the user explicitly asks. diff --git a/skills/git-visual-commits/evals/evals.json b/skills/git-visual-commits/evals/evals.json index 6ab5043..7fbda95 100644 --- a/skills/git-visual-commits/evals/evals.json +++ b/skills/git-visual-commits/evals/evals.json @@ -283,6 +283,41 @@ "Treats the full current worktree as scope because the user did not narrow it", "Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer" ] + }, + { + "id": 26, + "prompt": "Please git commit, yolo. The worktree has an unstaged change to src/Router.cs, a staged change to config/routes.json, and new untracked src/RoutePolicy.cs and assets/route-map.svg. git diff --stat only lists src/Router.cs. All four files belong to this routing change.", + "expected_output": "A complete commit workflow accounting for all four paths, inspecting untracked contents and reconciling inventory, plan, index, recorded commits, and final status.", + "expectations": [ + "Uses git status --short --untracked-files=all or its NUL-delimited porcelain equivalent as the scope inventory instead of a diff summary", + "Inspects and includes both src/RoutePolicy.cs and assets/route-map.svg without asking permission merely because they are untracked", + "Accounts for the pre-existing staged config/routes.json change when verifying each commit group", + "Compares exact planned paths with the inventory and reports no omitted in-scope paths", + "Checks staged contents and recorded commit paths against each group and verifies no in-scope tracked or untracked changes remain" + ] + }, + { + "id": 27, + "prompt": "Please review my working tree and show the commit plan only. I changed README.md and added a new assets folder containing logo.svg and dark theme/banner.svg. Git is configured with status.showUntrackedFiles=no. A scratch file under ignored .cache/ also exists.", + "expected_output": "A read-only plan that explicitly discovers both new files, handles the path containing spaces, covers the README change, and excludes the ignored scratch file.", + "expectations": [ + "Overrides hidden untracked status with --untracked-files=all and enumerates individual files in new directories", + "Inspects both untracked assets and includes assets/logo.svg and assets/dark theme/banner.svg in the plan", + "Preserves the space-containing path as one filename", + "Excludes the ignored .cache file without force-adding it", + "Does not stage or commit for a plan-only request" + ] + }, + { + "id": 28, + "prompt": "Please git commit only the README.md change, yolo. src/Router.cs is modified and src/RoutePolicy.cs is untracked too; leave both alone.", + "expected_output": "A README-only commit that records the explicit exclusions and reports the remaining source changes accurately.", + "expectations": [ + "Limits the commit to README.md because the user explicitly narrowed scope", + "Leaves both src/Router.cs and src/RoutePolicy.cs out of the commit and preserves their contents", + "Reconciles final status against the explicit exclusions instead of requiring an entirely clean worktree", + "Does not claim the entire worktree is clean when excluded paths remain" + ] } ] } diff --git a/skills/git-visual-commits/references/grouping-examples.md b/skills/git-visual-commits/references/grouping-examples.md new file mode 100644 index 0000000..689962f --- /dev/null +++ b/skills/git-visual-commits/references/grouping-examples.md @@ -0,0 +1,89 @@ +# Commit grouping examples + +#### Release-adjacent splitting rule + +Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: + +- **Dependency/version baselines** — version alignment or runner baseline changes +- **Community health/release communication** — changelogs and human-facing repo health docs +- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` +- **Documentation publishing** — DocFX navigation, branding, or publishing assets +- **CI/automation** — workflows and helper scripts used only by automation + +These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. + +Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: + +- **Build system / dependency baseline** +- **CI or automation** +- **Release communication plus package metadata** + +Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. + +#### Repo-aligned grouping example + +When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: + +- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` +- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill +- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` +- **Template/runtime files** — `assets/`, scaffold helper scripts +- **Validation/tooling** — validator scripts, repo checks +- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` + +Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". + +If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. + + +#### Commit body guidance + +Unless **no-body mode** is active, every commit includes a body explaining the *why*: + +- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") +- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") +- **Refactors** → explain what motivated the restructuring +- **New features** → explain the purpose and scope +- **Bug fixes** → explain what was broken and how this fixes it + +Common groupings: +- New repo-managed skill or workflow introduction together +- Existing skill refactor or extraction together +- Dependency/version baseline updates together +- Package/publish metadata together +- Config/setup files together (app host, bootstrapping) +- Environment and infrastructure config together (test runners, CI matrix, container settings) +- Documentation publishing fixes together +- Community health or release communication docs together +- New feature or module code together +- Data contracts, types, and interfaces together +- Database models, migrations, and schema changes together +- Test logic and assertions together (when they share the same rationale) +- Documentation and inline comments together + +When in doubt, one commit per "thing that changes" is better than one big commit. + + +## Good Examples + +``` +🎉 begin api project +✨ add submission endpoint module +🐛 handle null optional fields in dto +➕ add validation library +🐛 fix: handle null optional fields in dto ← only when combo mode was requested +``` + +## Bad Examples (and why) + +``` +feat: add submission endpoint ← "feat:" is not an allowed prefix +✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed +💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning +💬 update changelog for v10.0.10 ← more than one separator space +📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table +🎉 initial commit with all files ← vague, bundles everything +⚙️ config: setup api ← "config:" is not an allowed prefix +♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo +``` +
MetricWith skillWithout skillDelta