diff --git a/AGENTS.md b/AGENTS.md index 80d4201..28b2b53 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,7 +44,7 @@ Where `skill-creator` says to spawn with-skill and baseline subagents in the sam ### Asking for an eval -`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, `evaluate using the existing evals`, `please do an eval`, and similar explicit evaluation requests all invoke this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. @@ -91,21 +91,13 @@ Useful switches: `-Eval ` to prepare a subset, `-Iteration ` plus `-Fo The expected output and the assertions are the grading key. They belong in `eval-metadata.json`, outside every run directory, and must never appear in either prompt — a baseline handed the answer key is not a baseline. -### Eval preparation is a completion gate +### Optional eval preparation -Adding or modifying any repo-managed skill triggers this workflow. It is not something the user asks for separately, and "the change is small" or "the evals did not change" does not exempt it. Touching `SKILL.md`, `FORMS.md`, `references/`, `scripts/`, `assets/`, or `evals/` under `skills//` is a skill change. +Eval package preparation is optional. A skill change does not require package preparation, and it does not require choosing a harness or model. Prepare a package only when the user explicitly asks for an evaluation, such as `eval `, `evaluate `, `please do an eval`, or a similar request. A request to create, modify, validate, or release a skill does not implicitly ask for eval preparation. -After the final skill edit is in place, run: +When the user explicitly asks for an eval, follow [Asking for an eval](#asking-for-an-eval), resolve the Harness + Model choice, and run the preparation command described there. Preparation remains model-free and never executes the generated prompts. If a package is prepared, name its prompt path in the completion message and hand it over as described below. Do not ask for a harness, prepare a package, or report an eval handoff solely because a skill changed. -``` -pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model -``` - -It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate. - -Then name the prepared prompt paths in the completion message so the user knows what is waiting for them. Preparing and reporting satisfies this gate. Executing a prompt never does, and an agent that runs one has broken the Priority 1 rule rather than completed the gate. - -Run it before `scripts/sync-skill-install.ps1`, which stays the last gate because it must observe the final state of every file. See [Blocking Completion Gates](#blocking-completion-gates). +`scripts/sync-skill-install.ps1` remains the last gate when a repo-managed skill was edited. An explicitly requested eval adds package preparation before handoff, but package preparation is not a completion gate. See [Blocking Completion Gates](#blocking-completion-gates). ### Manual execution boundary @@ -181,13 +173,16 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - When `files` is present, keep the paths relative to `skills//` and validate that every fixture exists - Treat eval prompts, expected outcomes, and assertions as versioned review specifications; their presence never authorizes automated model execution - Start with `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -MetadataOnly` for a sub-second repository-wide metadata and fixture check -- Run only the changed skill's deterministic validator and focused regression scripts during iteration; independent read-only checks may use bounded local parallelism, while shared-file mutations stay sequential -- Run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` once before completion for the repository gate +- Run only the changed skill's deterministic validator and focused regression scripts during iteration; run independent checks concurrently with bounded local parallelism, while shared-file mutations stay sequential. +- For the repository completion gate, run the complete CI matrix locally with the permanent parallel scheduler: `pwsh -NoProfile -File ./scripts/validate-local.ps1`. It derives every suite from `.github/workflows/validate-skill-templates.yml`, so local and CI coverage stay aligned, and that matrix remains the source of truth for script paths and `-Suite` values, including the transport-specific conformance suites and phase-specific integrity suites. The scheduler runs each suite in a separate PowerShell 7 process with bounded concurrency, streams a log per suite, kills timed-out process trees, isolates temporary fixtures, and exits non-zero unless every suite passed and printed its terminal success marker. It writes `summary.json` beside the logs with per-suite status, exit codes, timings, and coverage counts. The DocFX suite always receives `-Full`. A timeout or a skipped check is a failure, never a pass. +- `scripts/validate-local.ps1` is the only sanctioned local full-run entry point. Do not hand-roll process fan-out, and do not rely on ad hoc, under-parallelized dispatch. For focused iteration, run a single suite or group directly (`-Suite Templates`, `-Suite Preparation`, `-Suite Runners`, `-Full -Suite Docfx`, plus the conformance and integrity `-Suite` values), and use `-MetadataOnly` for the sub-second metadata and fixture check. Do not invoke the sequential aggregate `scripts/validate-skill-templates.ps1` with no `-Suite` or with `-Suite All` for agent-driven validation, and do not run the CI suite matrix in a sequential loop. If concurrent execution is unavailable, report the limitation instead of silently falling back to sequential execution. +- `scripts/validate-local.ps1` is covered by `scripts/tests/test-validate-local.ps1`, which asserts scheduling, concurrency bounding, failure propagation, silent-skip detection, timeout and process-tree teardown, deadline cancellation, coverage parity, and output-root safety against fixture workflows. Update those tests when the scheduler's contract changes. +- An explicit user instruction to skip or stop testing overrides these validation gates for that session. Stop any tests you started, run no further validation, and report the unvalidated work. - Follow the top-level **AI/LLM Evaluation Automation Prohibition** for every eval. No per-skill or third-party requirement overrides it. - To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` before considering the work complete. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. +If you add a new skill or modify an existing repo-managed skill, keep that skill's `evals/evals.json` present and update it when the eval definitions or fixtures change. Package preparation is optional and is triggered only by an explicit eval request; when requested, use `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model `. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. ## Git Identity @@ -282,6 +277,15 @@ When a skill needs time-sensitive or environment-sensitive values, prefer comput - When a dynamic default exists, describe both the source and the fallback behavior in `FORMS.md` / `SKILL.md` - If a value changes over time (supported frameworks, current versions, generated paths, repo-derived names), assume hardcoding will drift and design for refreshable computation +## File-Based App Invocation + +A .NET file-based app (`dotnet run --file .cs`) re-runs SDK, restore, and MSBuild evaluation on every call. On current hardware that fixed cost is several seconds even when the built output is already up to date, and the deterministic suites invoke the same app dozens of times per run. + +- Test code that invokes a file-based app must resolve the built assembly once with `scripts/file-based-app.ps1` (`Get-FileBasedAppCommand`) and execute that assembly, instead of calling `dotnet run --file` per invocation. +- The helper caches a build artifact keyed by source hash and SDK version under the user temp directory and outside the repository. It never caches a validation result: every assertion still runs on every execution. +- When the helper cannot prepare or locate a cached build it falls back to `dotnet run --file`, so behavior is never worse than the unoptimized path, only slower. +- Do not reintroduce per-call `dotnet run --file` in a deterministic suite. Measure the suite before and after any change to how it invokes a file-based app. + ## Scaffold Invariants For repo-managed .NET scaffolding skills, preserve semantic versioning infrastructure unless you are replacing it end-to-end in the same change. @@ -317,7 +321,7 @@ Before any completion message, reread the skill instructions and the current con For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. -Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` (or `-CodebeltReference` after dynamic availability verification) runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). +When a repo-managed skill was edited, `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report its actual output; an earlier run in the same session satisfies neither. If the user explicitly requested an eval, prepare the package before handing it over as described in [Optional eval preparation](#optional-eval-preparation), but do not treat that preparation as a completion gate. See [Local Install Sync](#local-install-sync). ## User Input UX diff --git a/CHANGELOG.md b/CHANGELOG.md index 21f0132..49cb321 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.10.0] - 2026-09-11 + +This is a minor release that introduces `dotnet-nuget-update`, a deterministic NuGet dependency audit and update workflow for .NET repositories. The skill owns the complete-audit invariant, ensuring every declared package version is accounted for before any update is applied. It handles both central package management (`Directory.Packages.props`) and project-level `PackageReference` versioning, tracks stable and prerelease intent, preserves TFM-band pins (keeping `net9` or `net10` packages within their matching major when that major is the compatibility signal), and supports both normal mode (auto-applies patch/minor/revision, batches majors for approval) and yolo mode (auto-applies safe classes only, reports held majors). All scripts are deterministic and offline-testable via bundled fixtures. Alongside that, `git-visual-commits` gained refactoring and three new eval cases for edge-case coverage, `git-keep-a-changelog` gained improved entity-classification infrastructure and a dedicated reference document for section validation, and repository-level validation enhancements ensure all skills conform to updated testing requirements. + +### Added + +- `dotnet-nuget-update` skill for auditing and updating NuGet dependencies with complete declaration accounting, supporting both central package management and project-level versioning, two interactive modes (normal with approval batching, yolo for safe updates only), stable/prerelease intent inference, and TFM-band awareness so conditional `net9`/`net10` package declarations stay within their matching major when that major is the compatibility signal rather than jumping to the newest overall release, +- bundled deterministic scripts for `dotnet-nuget-update`: `Get-DependencyAudit.ps1` for complete graph enumeration before any edit, `Get-PackageGraph.ps1` for central-package condition resolution, `Get-TargetFrameworks.ps1` for TFM matrix discovery, `Resolve-NuGetVersion.ps1` and `Compare-Version.ps1` for version investigation, `Apply-PackageUpdates.ps1` for minimal structural XML edits preserving comments and line endings, and `Get-NuGetSources.ps1` for feed configuration visibility, +- comprehensive test coverage for `dotnet-nuget-update`: regression harnesses for dependency audit, package graph, TFM-band logic, project-level package references, version comparison, and update application, together with offline-testable fixtures covering central-package scenarios, mixed stable/prerelease intent, multi-TFM bands, plain project references, and XML comment pinning, +- per-process memoization for live or offline flat-container NuGet version feeds in `dotnet-nuget-update`, with filesystem flat-container fixtures supporting deterministic offline testing, +- complete-audit invariant validation in `dotnet-nuget-update` ensuring `current + auto + approval + unresolved == declared` before reporting the repository as updated, +- history-aware update tracking in `dotnet-nuget-update` with adjacent XML comments surfacing as `note` fields so pin rationale remains visible during audits and auto-updates marked with `READ THE NOTE before applying` receive explicit attention, +- README catalog entry, installation snippet, and "Why dotnet-nuget-update?" community health section explaining the need for complete dependency graph auditing, TFM-band awareness, and preservation of intentional pins and compatibility markers, +- `references/section-validation.md` in `git-keep-a-changelog` documenting the section-classification contract for resolved release entities, +- test harness `scripts/test-resolve-release-entity.ps1` in `git-keep-a-changelog` validating entity-classification behavior across multiple release scenarios. + +### Changed + +- `git-visual-commits` SKILL.md refactored for improved clarity, with single-category quality-gate guidance hardened and commit-language best practices restructured to emphasize inventory refresh and per-path explanation before classification, +- `git-visual-commits` evals expanded with three new eval cases (26, 27, 28) covering untracked file discovery, path handling with spaces and untracked directories, and explicit scope narrowing, +- `git-keep-a-changelog` SKILL.md improved with clarifications on entity classification workflow, pending-change handling, and section-validation procedures, +- `git-keep-a-changelog` evals updated to reflect improved entity-classification guidance and concurrent validation requirements, +- Repository validation infrastructure in `scripts/validate-skill-templates.ps1` hardened with concurrent suite dispatch, bounded concurrency of at least two workers, per-suite exit codes, and explicit prohibition on sequential aggregate runs, +- Repository agent guidelines in `AGENTS.md` clarified with explicit concurrent execution guidance, per-suite validation, and user-instruction overrides for testing suspension, +- README.md documentation updated with concurrent validation procedures and explicit messaging about running independent validation suites in parallel with bounded concurrency. + ## [0.9.1] - 2026-09-07 This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. @@ -638,6 +664,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.10.0]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.10.0 [0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8df4768..dd78199 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -86,13 +86,15 @@ Evals let you verify the skill works and measure improvement over a baseline. Ev Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something. -Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra: +Evals are prepared, not executed, from this repository. Package preparation is optional and happens only after an explicit eval request, such as `eval `, `evaluate `, or `please do an eval`. Adding or modifying a repo-managed skill does not require a package, a harness choice, or a model choice. + +When an eval is requested, prepare the changed packages with: ```console pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot ``` -Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use: +For a single skill on demand, use: ```console pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model @@ -135,6 +137,14 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. +That command runs one script's groups in order, which is fine for a focused check but slow for the whole repository. For the complete gate, run the CI matrix in parallel instead: + +```console +pwsh -NoProfile -File ./scripts/validate-local.ps1 +``` + +The scheduler reads every suite from `.github/workflows/validate-skill-templates.yml`, runs each in a separate PowerShell 7 process with bounded concurrency and a streamed log, enforces per-suite timeouts by killing the process tree, and keeps going after a failure so a single run reports every problem. It exits non-zero unless every suite exited zero and printed its terminal success marker, and it writes `summary.json` with the coverage counts beside the logs. Use `-ListSuites` to see what will run, and `-MaxConcurrency`, `-TimeoutSeconds`, and `-DeadlineSeconds` to tune. The scheduler's own behavior is covered by `scripts/tests/test-validate-local.ps1`. + To compare a change against the initial imported version, run the same harness against a git ref: ```console @@ -149,9 +159,10 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD - [ ] At least one eval in `evals/evals.json` - [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name - [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported +- [ ] If an eval was explicitly requested, `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` or `-CodebeltReference` was run, and the prepared prompt paths were reported - [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only for explicitly authorized forensic recovery of an existing package -- [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior +- [ ] `pwsh -NoProfile -File ./scripts/validate-local.ps1` passes for the current working tree when changing scaffold or template behavior, with every CI-matrix suite reported as passed and verified +- [ ] Focused iteration used `scripts/validate-skill-templates.ps1 -Suite ` or another single suite rather than a sequential loop over the matrix - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too - [ ] Eval packages live in `.bot/-workspace/` or a temp path, never anywhere else in the working tree - [ ] Changed skill files are synced across `skills//`, `~/.claude/skills//`, and `~/.agents/skills//` diff --git a/README.md b/README.md index b2dce65..005d237 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Package preparation is optional and happens only when the user explicitly asks for an evaluation, such as `eval `, `evaluate `, or `please do an eval`; changing a skill alone does not require choosing a harness or preparing a package: Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. @@ -56,9 +56,15 @@ Use the metadata-only mode for the fastest feedback on every skill manifest, fix pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -MetadataOnly ``` -During iteration, run the changed skill's bundled deterministic validator and focused regression scripts. Before completion, run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1`; use `-Full` when the slower DocFX suites are relevant. GitHub Actions supplies the same deterministic safety net. This layered path catches structural and behavioral regressions quickly without hidden model traffic. +During iteration, run the changed skill's bundled deterministic validator and focused regression scripts, overlapping independent checks. For the repository completion gate, run the whole CI matrix locally with the parallel scheduler: -CI runs the complete validation set in 13 independent jobs with a five-minute budget per job. Template checks, package preparation, runner regressions, and DocFX have separate jobs; runner conformance is split by transport and integrity tests by phase. Every job uses isolated temporary fixtures, and the existing `validate-skill-templates` required check passes only when every job succeeds. New commits cancel obsolete runs for the same pull request. Each job records its elapsed time in the Actions summary. Local validation still defaults to all checks; `-Suite Templates`, `-Suite Preparation`, or `-Suite Runners` selects a focused group, and `-Full -Suite Docfx` selects DocFX. The conformance and integrity test scripts also accept the `-Suite` values listed in the workflow. This partition changes scheduling, not the model-free validation contract. +```powershell +pwsh -NoProfile -File ./scripts/validate-local.ps1 +``` + +The scheduler reads every suite from `.github/workflows/validate-skill-templates.yml`, so local and CI coverage stay aligned, and runs each one in a separate PowerShell 7 process with bounded concurrency, a streamed per-suite log, isolated fixtures, and per-suite exit codes. It enforces a per-suite timeout by killing the whole process tree, keeps running the remaining suites after a failure so one run reports every problem, and writes `summary.json` with the coverage counts next to the logs. It exits non-zero unless every CI-matrix suite exited zero and printed its terminal success marker, so a timeout or a silently skipped check is never reported as a pass. Use `-ListSuites` to print the resolved matrix, and `-MaxConcurrency`, `-TimeoutSeconds`, and `-DeadlineSeconds` to tune a run. Sequentially invoking the aggregate script or looping over the matrix one entry at a time is not permitted for agent-driven validation. If concurrent execution is unavailable, report the limitation. An explicit user instruction to skip or stop testing suspends validation for that session; report the work as unvalidated. + +CI runs the complete validation set in 13 independent jobs with a five-minute budget per job. Template checks, package preparation, runner regressions, and DocFX have separate jobs; runner conformance is split by transport and integrity tests by phase. Every job uses isolated temporary fixtures, and the existing `validate-skill-templates` required check passes only when every job succeeds. New commits cancel obsolete runs for the same pull request. Each job records its elapsed time in the Actions summary. For focused local validation, `-Suite Templates`, `-Suite Preparation`, or `-Suite Runners` selects a group, and `-Full -Suite Docfx` selects DocFX. The conformance and integrity test scripts also accept the `-Suite` values listed in the workflow. Agents use `scripts/validate-local.ps1` for the full gate and these explicit suite selections for focused work, rather than the script's sequential default. This partition changes scheduling, not the model-free validation contract. ## Install a skill @@ -145,6 +151,7 @@ Each `SKILL.md` description is lean activation metadata. The catalog below expla | [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | +| [dotnet-nuget-update](skills/dotnet-nuget-update/SKILL.md) | Audits and updates NuGet dependencies in .NET repositories with complete declaration accounting before any edit. It supports both `Directory.Packages.props` and project-level `PackageReference` versions, preserves XML structure and line endings, deduplicates package IDs, resolves independent live or offline flat-container version feeds with bounded parallel lookups and per-process memoization, uses bounded network timeouts, and merges results deterministically. Its single-process update runner keeps the audit, in-memory safe-update plan, and structural apply together for fast yolo passes. It keeps stable pins on stable candidates unless prerelease intent is explicit, and applies the TFM-band rule so conditional `net9`/`net10` package declarations stay within their matching major when that major is the compatibility signal rather than jumping to the newest overall release. Normal mode auto-applies revision/patch/minor and same-major prerelease updates, then batches majors for one approval decision; yolo mode applies only the auto classes and reports held majors without asking. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | | [dotnet-test](skills/dotnet-test/SKILL.md) | Moves xUnit projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's ASP.NET-only `WebApplicationFactory`—and the hand-rolled `HostBuilder` that console and worker tests reach for because Microsoft ships no equivalent—with one family of abstractions where the application's own entry point owns startup. Invocation is the request: it inspects and refactors immediately instead of opening with a menu or a questionnaire. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | | [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | @@ -253,6 +260,11 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-diges ```bash npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark ``` +`dotnet-nuget-update` + +```bash +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-nuget-update +``` `dotnet-test` ```bash @@ -343,6 +355,8 @@ Sometimes the history is already written and the only thing you need is the fina ### Why git-keep-a-changelog? +On repeated runs, the entity resolver checks each proposed bullet section against the full release baseline. A new unreleased capability permits only `Added`, so later fixes and refinements stay within its introduction instead of accumulating under `Changed` or `Fixed`. + Writing `CHANGELOG.md` well is harder than it looks. Raw commit subjects are too noisy, PR titles often miss migration context, and release notes get much better when the writer actually reads the commit bodies and understands the net diff. That is where **git-keep-a-changelog** fits: it turns the current branch into a curated Keep a Changelog entry and creates or updates the file directly for review. - **Keep a Changelog first** — writes `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, and `Security` sections in the expected style @@ -621,6 +635,20 @@ Picking the wrong version number is one of the easiest ways to break downstream - **Precedence-aware** — mixed releases take the highest required bump, - **Special-case savvy** — dependency updates, bug fixes, new overloads, interface and enum changes, analyzers/source generators, TFM/platform support, and performance changes each get the right default and the right escalation triggers. +### Why dotnet-nuget-update? + +Dependency updates look simple until a repository encodes compatibility in its package graph. A `Directory.Packages.props` file can pin the same package differently for `net9` and `net10`, hold a package back with an inline comment because a newer release dropped a target framework, or mix stable and prerelease intent on purpose. A shallow “latest package wins” pass breaks those repos quietly. + +**dotnet-nuget-update** makes the complete audit the first-class artifact. It enumerates every declaration before editing, keeps `current + auto + approval + unresolved == declared` as a hard invariant, resolves versions from a live or offline flat-container feed with per-process memoization, and preserves XML comments, spacing, encoding, and line endings when it writes changes back. + +- **Complete graph first** — every `` declaration, and explicit project-level `PackageReference` when CPM is absent, becomes an audit row before any update is applied, +- **TFM-band aware** — a package pinned under `net9` or `net10` stays inside that matching major when the pinned major itself is the compatibility signal, while mismatched majors remain free to move, +- **Stable/prerelease intent inference** — stable pins stay on stable candidates; prerelease pins may move within prerelease lines; same-major prerelease movement is auto, not approval, +- **History-first notes** — adjacent XML comments surface as `note` fields, and note-bearing auto updates are explicitly marked `READ THE NOTE before applying`, +- **Normal and yolo modes** — normal mode batches majors into one approval decision after the full audit, while yolo mode applies only the safe classes and reports held majors without asking, +- **Structural edits only** — version updates touch only the targeted attribute or element and leave comments, blank lines, unrelated files, and existing line endings intact, +- **Offline-testable** — the bundled scripts accept filesystem flat-container fixtures so audit, comparison, and update logic can be regression-tested without network dependency. + ### Why dotnet-docfx-digest? API documentation rots the moment code changes. A new public type ships without a namespace page, an extension method never makes it into the `Extension Members` table, a copy/paste example silently stops compiling, and "availability" drifts away from the real target frameworks. The usual fix — telling an agent to "remember to update the docs" — relies on AI memory, which is exactly the thing that fails on the next change. diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 89b26be..be01bd6 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -269,20 +269,22 @@ function Expand-WindowsShortPath { # is the final component. Walk existing components so a short # parent such as ADMINI~1 is expanded before comparing a nested # runner path. + # Use filesystem APIs in this hot loop; PowerShell provider calls + # multiply across every arm's repeated integrity checks. $current = $root $remaining = $fullPath.Substring($root.Length) -split '[\\/]' for ($componentIndex = 0; $componentIndex -lt $remaining.Count; $componentIndex++) { $component = [string]$remaining[$componentIndex] if ([string]::IsNullOrWhiteSpace($component)) { continue } - $next = Join-Path -Path $current -ChildPath $component - if (Test-Path -LiteralPath $next -PathType Container) { + $next = [System.IO.Path]::Combine($current, $component) + if ([System.IO.Directory]::Exists($next)) { $current = ([System.IO.DirectoryInfo]::new($next)).FullName - } elseif (Test-Path -LiteralPath $next -PathType Leaf) { + } elseif ([System.IO.File]::Exists($next)) { $current = ([System.IO.FileInfo]::new($next)).FullName } else { - $current = Join-Path -Path $current -ChildPath $component + $current = $next if ($componentIndex + 1 -lt $remaining.Count) { - $current = Join-Path -Path $current -ChildPath ([string]::Join([System.IO.Path]::DirectorySeparatorChar, @($remaining[($componentIndex + 1)..($remaining.Count - 1)] | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) }))) + $current = [System.IO.Path]::Combine($current, [string]::Join([System.IO.Path]::DirectorySeparatorChar, @($remaining[($componentIndex + 1)..($remaining.Count - 1)] | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) }))) } break } diff --git a/scripts/eval-runners/tests/test-codex-paths.ps1 b/scripts/eval-runners/tests/test-codex-paths.ps1 index 0f31a1b..c869c16 100644 --- a/scripts/eval-runners/tests/test-codex-paths.ps1 +++ b/scripts/eval-runners/tests/test-codex-paths.ps1 @@ -24,6 +24,51 @@ if ((ConvertTo-CodexComparablePath $comparable) -cne $comparable) { throw 'Path if (-not (Test-CodexPathInsideComparableRoot $staged (Join-Path $staged 'FORMS.md'))) { throw 'Staged descendants must remain inside their root.' } if (Test-CodexPathInsideComparableRoot $staged ($staged + '-other/FORMS.md')) { throw 'A sibling prefix is not a descendant.' } +if ($IsWindows) { + Add-Type -Namespace AgenticPathTests -Name Native -MemberDefinition @' +[System.Runtime.InteropServices.DllImport("kernel32.dll", CharSet = System.Runtime.InteropServices.CharSet.Unicode, SetLastError = true)] +public static extern uint GetShortPathName(string path, System.Text.StringBuilder buffer, uint size); +[System.Runtime.InteropServices.DllImport("kernel32.dll", CharSet = System.Runtime.InteropServices.CharSet.Unicode, SetLastError = true)] +public static extern uint GetLongPathName(string path, System.Text.StringBuilder buffer, uint size); +'@ + # TEMP itself can contain a short user-profile component. Derive the + # expected long spelling with Win32, independently of the helper under test. + $buffer = [Text.StringBuilder]::new(32768) + $length = [AgenticPathTests.Native]::GetLongPathName([IO.Path]::GetTempPath(), $buffer, $buffer.Capacity) + if ($length -eq 0 -or $length -ge $buffer.Capacity) { throw 'Cannot resolve the temporary directory long path.' } + $tempRoot = $buffer.ToString().TrimEnd('\', '/') + $pathRoot = Join-Path $tempRoot ('agentic-paths-' + [guid]::NewGuid().ToString('N')) + try { + $directory = [IO.Directory]::CreateDirectory((Join-Path $pathRoot 'long directory [literal] æ')).FullName + $file = Join-Path $directory 'long filename [literal].json' + [IO.File]::WriteAllText($file, '{}') + $missing = Join-Path $directory 'missing parent/child/file.json' + foreach ($path in @($directory, $file, $missing, [IO.Path]::GetPathRoot($directory))) { + if ((Expand-WindowsShortPath $path) -cne [IO.Path]::GetFullPath($path)) { throw "Long path spelling changed: $path" } + } + if ((Expand-WindowsShortPath (Join-Path $directory '../long directory [literal] æ')) -cne $directory) { throw 'Relative components must normalize.' } + + $shortCases = 0 + foreach ($path in @($directory, $file, $env:ProgramFiles)) { + $buffer = [Text.StringBuilder]::new(32768) + $length = [AgenticPathTests.Native]::GetShortPathName($path, $buffer, $buffer.Capacity) + if ($length -eq 0 -or $length -ge $buffer.Capacity) { throw "Cannot query short path: $path" } + $short = $buffer.ToString() + if ($short -ieq $path) { continue } # Volumes may have 8.3 creation disabled. + if (-not (Test-ExactObservedPath -Expected $path -Observed $short)) { throw "Short path must match its long spelling: $short" } + if ([IO.Directory]::Exists($path)) { + $suffix = 'missing parent/child/file.json' + if (-not (Test-ExactObservedPath -Expected (Join-Path $path $suffix) -Observed (Join-Path $short $suffix))) { throw 'A missing tail must retain expansion of its existing short parent.' } + } + $shortCases++ + } + Write-Output "Windows path compatibility: PASS; $shortCases native short-path cases" + } finally { + if ([IO.Path]::GetDirectoryName([IO.Path]::GetFullPath($pathRoot)) -ne $tempRoot) { throw 'Path fixture cleanup must stay under the temporary directory.' } + if (Test-Path -LiteralPath $pathRoot) { Remove-Item -LiteralPath $pathRoot -Recurse -Force } + } +} + foreach ($access in @('command', 'file')) { $isolation = [ordered]@{ ambient_skill_paths_observed = @($ambient); failures = @() } $parameters = @{ NativeSkillIsolation = $isolation; AllowedStagedSkillRoot = $staged } diff --git a/scripts/file-based-app.ps1 b/scripts/file-based-app.ps1 new file mode 100644 index 0000000..6ade020 --- /dev/null +++ b/scripts/file-based-app.ps1 @@ -0,0 +1,217 @@ +<# +.SYNOPSIS + Executes .NET file-based apps without paying the `dotnet run --file` CLI overhead per call. + +.DESCRIPTION + `dotnet run --file` re-evaluates the SDK, restore graph, and MSBuild pipeline on every + invocation. On a current SDK that costs several seconds even when the built output is already + up to date and the app itself runs in a fraction of a second. Deterministic validation suites + invoke the same file-based app dozens of times per run, so that fixed cost dominates their + runtime. + + Get-FileBasedAppCommand builds the app once per source revision and returns the command that + executes the built assembly directly. The executed code, the arguments, the current directory, + and the environment are identical to `dotnet run --file`; only the redundant SDK startup is + removed. The build cache is content-addressed by source hash plus SDK version and lives outside + the repository, under the user temp directory. + + This helper caches a build artifact, never a validation result. Every assertion still runs on + every execution. When the cache cannot be prepared for any reason the helper falls back to + `dotnet run --file`, so a caller is never worse off than before. + + Callers keep full control of redirection and exit codes: + + $app = Get-FileBasedAppCommand -SourcePath $ValidatorPath + $output = & $app.Executable @($app.ArgumentPrefix + @('--repo-root', $Workspace, '--json')) 2>$null +#> + +$script:FileBasedAppCache = @{} +$script:FileBasedAppSdkVersion = $null + +function Get-FileBasedAppCacheRoot { + return (Join-Path ([System.IO.Path]::GetTempPath()) 'agentic-file-based-app-cache') +} + +function Get-FileBasedAppSdkVersion { + if ($null -ne $script:FileBasedAppSdkVersion) { + return $script:FileBasedAppSdkVersion + } + + $previousErrorAction = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + try { + $version = & dotnet --version 2>$null | Select-Object -First 1 + } finally { + $ErrorActionPreference = $previousErrorAction + } + + $script:FileBasedAppSdkVersion = if ([string]::IsNullOrWhiteSpace([string]$version)) { 'unknown' } else { ([string]$version).Trim() } + return $script:FileBasedAppSdkVersion +} + +function Get-FileBasedAppFallbackCommand { + param([Parameter(Mandatory = $true)][string]$SourcePath) + + return [pscustomobject]@{ + Executable = 'dotnet' + ArgumentPrefix = @('run', '--file', $SourcePath, '--') + Mode = 'dotnet-run' + Assembly = $null + } +} + +function Find-FileBasedAppAssembly { + param( + [Parameter(Mandatory = $true)][string]$Root, + [Parameter(Mandatory = $true)][string]$AssemblyName + ) + + if (-not (Test-Path -LiteralPath $Root -PathType Container)) { + return $null + } + + $assembly = Get-ChildItem -LiteralPath $Root -Recurse -File -Filter $AssemblyName -ErrorAction SilentlyContinue | + Sort-Object FullName | + Select-Object -First 1 + + if ($null -eq $assembly) { + return $null + } + + return $assembly.FullName +} + +function Build-FileBasedAppRevision { + param( + [Parameter(Mandatory = $true)][string]$SourcePath, + [Parameter(Mandatory = $true)][string]$RevisionDirectory, + [Parameter(Mandatory = $true)][string]$AssemblyName + ) + + $cacheRoot = Split-Path -Parent $RevisionDirectory + $staging = Join-Path $cacheRoot ('.staging-' + [Guid]::NewGuid().ToString('N')) + + $previousErrorAction = $ErrorActionPreference + $nativePreferenceVariable = Get-Variable -Name 'PSNativeCommandUseErrorActionPreference' -ErrorAction SilentlyContinue + if ($null -ne $nativePreferenceVariable) { + $previousNativePreference = $nativePreferenceVariable.Value + } + $ErrorActionPreference = 'Continue' + if ($null -ne $nativePreferenceVariable) { + $PSNativeCommandUseErrorActionPreference = $false + } + try { + New-Item -ItemType Directory -Path $staging -Force | Out-Null + & dotnet build $SourcePath --artifacts-path $staging --nologo -v quiet 1>$null 2>$null + $buildExitCode = $LASTEXITCODE + } finally { + $ErrorActionPreference = $previousErrorAction + if ($null -ne $nativePreferenceVariable) { + $PSNativeCommandUseErrorActionPreference = $previousNativePreference + } + } + + if ($buildExitCode -ne 0) { + Remove-Item -LiteralPath $staging -Recurse -Force -ErrorAction SilentlyContinue + return $null + } + + if ($null -eq (Find-FileBasedAppAssembly -Root $staging -AssemblyName $AssemblyName)) { + Remove-Item -LiteralPath $staging -Recurse -Force -ErrorAction SilentlyContinue + return $null + } + + if (Test-Path -LiteralPath $RevisionDirectory) { + Remove-Item -LiteralPath $staging -Recurse -Force -ErrorAction SilentlyContinue + return $RevisionDirectory + } + + try { + [System.IO.Directory]::Move($staging, $RevisionDirectory) + } catch { + # A concurrent suite built the same revision first; prefer that copy. + Remove-Item -LiteralPath $staging -Recurse -Force -ErrorAction SilentlyContinue + if (-not (Test-Path -LiteralPath $RevisionDirectory)) { + return $null + } + } + + return $RevisionDirectory +} + +function Remove-StaleFileBasedAppRevisions { + param( + [Parameter(Mandatory = $true)][string]$AppDirectory, + [Parameter(Mandatory = $true)][string]$KeepRevision + ) + + Get-ChildItem -LiteralPath $AppDirectory -Directory -ErrorAction SilentlyContinue | + Where-Object { $_.Name -ne $KeepRevision } | + ForEach-Object { Remove-Item -LiteralPath $_.FullName -Recurse -Force -ErrorAction SilentlyContinue } +} + +function Get-FileBasedAppCommand { + <# + .SYNOPSIS + Resolves the command that runs a file-based app, building it once per revision. + .OUTPUTS + An object with Executable, ArgumentPrefix, Mode, and Assembly. Append the app's own + arguments to ArgumentPrefix and invoke Executable, e.g. + `& $command.Executable @($command.ArgumentPrefix + $arguments)`. + #> + [CmdletBinding()] + param( + [Parameter(Mandatory = $true)][string]$SourcePath, + [switch]$NoCache + ) + + if ($NoCache) { + return (Get-FileBasedAppFallbackCommand -SourcePath $SourcePath) + } + + try { + $source = (Resolve-Path -LiteralPath $SourcePath -ErrorAction Stop).Path + } catch { + return (Get-FileBasedAppFallbackCommand -SourcePath $SourcePath) + } + + $cacheKey = $source + '|' + (Get-FileBasedAppSdkVersion) + if ($script:FileBasedAppCache.ContainsKey($cacheKey)) { + return $script:FileBasedAppCache[$cacheKey] + } + + $fallback = Get-FileBasedAppFallbackCommand -SourcePath $source + + try { + $assemblyName = [System.IO.Path]::GetFileNameWithoutExtension($source) + '.dll' + $sdkTag = [regex]::Replace((Get-FileBasedAppSdkVersion), '[^A-Za-z0-9._-]', '_') + $revision = (Get-FileHash -LiteralPath $source -Algorithm SHA256).Hash + '-' + $sdkTag + $appDirectory = Join-Path (Get-FileBasedAppCacheRoot) ([System.IO.Path]::GetFileNameWithoutExtension($source)) + $revisionDirectory = Join-Path $appDirectory $revision + + $assembly = Find-FileBasedAppAssembly -Root $revisionDirectory -AssemblyName $assemblyName + if ($null -eq $assembly) { + [void](Build-FileBasedAppRevision -SourcePath $source -RevisionDirectory $revisionDirectory -AssemblyName $assemblyName) + $assembly = Find-FileBasedAppAssembly -Root $revisionDirectory -AssemblyName $assemblyName + } + + if ($null -eq $assembly) { + $script:FileBasedAppCache[$cacheKey] = $fallback + return $fallback + } + + Remove-StaleFileBasedAppRevisions -AppDirectory $appDirectory -KeepRevision $revision + + $command = [pscustomobject]@{ + Executable = 'dotnet' + ArgumentPrefix = @($assembly) + Mode = 'cached-assembly' + Assembly = $assembly + } + $script:FileBasedAppCache[$cacheKey] = $command + return $command + } catch { + $script:FileBasedAppCache[$cacheKey] = $fallback + return $fallback + } +} diff --git a/scripts/test-validation-suites.ps1 b/scripts/test-validation-suites.ps1 index 5fb3470..6440a52 100644 --- a/scripts/test-validation-suites.ps1 +++ b/scripts/test-validation-suites.ps1 @@ -37,3 +37,40 @@ foreach ($script in @('scripts/validate-skill-templates.ps1', 'scripts/eval-runn } if ($workflow -notmatch '(?m)^ validate-skill-templates:' -or $workflow -notmatch 'needs: validate' -or $workflow -notmatch "VALIDATION_RESULT -ne 'success'") { throw 'The required aggregate check must reject failed, skipped, or cancelled suites.' } Write-Output 'CI suite coverage: PASS' + +# Exercise the production validation block against both supported content layouts. +& { + $tokens = $null + $parseErrors = $null + $validatorAst = [Management.Automation.Language.Parser]::ParseFile((Join-Path $repoRoot 'scripts/validate-skill-templates.ps1'), [ref]$tokens, [ref]$parseErrors) + foreach ($name in @('Assert-Contains', 'Assert-NotContains')) { + $definition = $validatorAst.Find({ param($node) $node -is [Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq $name }, $true) + . ([scriptblock]::Create($definition.Extent.Text)) + } + $command = $validatorAst.Find({ param($node) $node -is [Management.Automation.Language.CommandAst] -and $node.GetCommandName() -eq 'Add-ValidationResult' -and $node.Extent.Text.Contains("-Name 'Git visual commits skill enforces subject, identity, and grouping locks'") }, $true) + $action = @($command.CommandElements | Where-Object { $_ -is [Management.Automation.Language.ScriptBlockExpressionAst] })[0].ScriptBlock.GetScriptBlock() + $skillPath = 'skills/git-visual-commits/SKILL.md' + $examplesPath = 'skills/git-visual-commits/references/grouping-examples.md' + $currentSkill = [IO.File]::ReadAllText((Join-Path $repoRoot $skillPath)) + $examples = [IO.File]::ReadAllText((Join-Path $repoRoot $examplesPath)) + function Get-FileText { + param($RepoRoot, $RelativePath, $GitRef) + if ($RelativePath -eq $skillPath) { return $fixtureSkill } + if ($RelativePath -eq $examplesPath -and $missingExamples) { throw 'Missing grouping examples fixture' } + return [IO.File]::ReadAllText((Join-Path $RepoRoot $RelativePath)) + } + foreach ($case in @('current', 'historical-inline', 'historical-linked', 'broken-historical-link', 'missing-current-reference', 'missing-historical-content')) { + $Ref = if ($case -in @('current', 'missing-current-reference')) { '' } else { 'fixture-ref' } + $fixtureSkill = $currentSkill + $missingExamples = $case -in @('historical-inline', 'broken-historical-link', 'missing-current-reference', 'missing-historical-content') + if ($case -in @('historical-inline', 'missing-historical-content')) { + $fixtureSkill = $currentSkill.Replace('[grouping-examples.md](references/grouping-examples.md)', '') + if ($case -eq 'historical-inline') { $fixtureSkill += "`n$examples" } + } + $failure = $null + try { & $action } catch { $failure = $_ } + $shouldFail = $case -in @('broken-historical-link', 'missing-current-reference', 'missing-historical-content') + if ($shouldFail -ne ($null -ne $failure)) { throw "Grouping reference regression '$case' failed: $failure" } + } + Write-Output 'Grouping reference layouts: PASS (6 cases)' +} diff --git a/scripts/tests/test-validate-local.ps1 b/scripts/tests/test-validate-local.ps1 new file mode 100644 index 0000000..dd075f9 --- /dev/null +++ b/scripts/tests/test-validate-local.ps1 @@ -0,0 +1,462 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Deterministic regression tests for scripts/validate-local.ps1. + +.DESCRIPTION + Exercises the local full-validation scheduler against fixture workflows and fixture suites so + that scheduling, failure propagation, silent-skip detection, timeout handling, process-tree + teardown, cancellation, concurrency bounding, complete coverage, and output-root safety are all + verified without depending on the real validation suites or on any model. + + Every scenario runs the real scheduler in a child process against an isolated fixture + repository under the user temp directory, so nothing here touches the working tree. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +# Fixture suites intentionally exit non-zero and the scheduler reports failures on stderr; never +# promote native command output into a terminating error in this harness. +$PSNativeCommandUseErrorActionPreference = $false + +$scheduler = Join-Path (Resolve-Path (Join-Path $PSScriptRoot '..')).Path 'validate-local.ps1' +$root = Join-Path ([System.IO.Path]::GetTempPath()) ('validate-local-tests-' + [Guid]::NewGuid().ToString('N')) +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +$failures = [System.Collections.Generic.List[string]]::new() + +function Write-FixtureFile { + param([string]$Path, [string]$Content) + + $directory = Split-Path -Parent $Path + if (-not (Test-Path -LiteralPath $directory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + [System.IO.File]::WriteAllText($Path, $Content, $utf8NoBom) +} + +# --------------------------------------------------------------------------- +# Fixture repository. Script names mirror the real matrix paths so the scheduler +# applies the same terminal-marker expectations it uses in production. +# --------------------------------------------------------------------------- + +$fixtureTemplateValidator = @' +param( + [ValidateSet('All', 'Good', 'Bad', 'Silent', 'Hang', 'SlowA', 'SlowB', 'SlowC', 'SlowD', 'Sleepy', 'Tree', 'Skip')] + [string]$Suite = 'All' +) + +$ErrorActionPreference = 'Stop' +$probe = $env:VALIDATION_TEST_PROBE_DIR + +function Write-Probe { + param([string]$Suffix, [string]$Value = '') + if ([string]::IsNullOrWhiteSpace($probe)) { return } + New-Item -ItemType Directory -Path $probe -Force | Out-Null + $text = if ([string]::IsNullOrEmpty($Value)) { [string][DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds() } else { $Value } + [System.IO.File]::WriteAllText((Join-Path $probe ("$Suite-$Suffix")), $text) +} + +if ($Suite -eq 'Skip') { + Write-Host '[SKIP] fixture suite skipped' + exit 0 +} + +Write-Probe 'start' + +switch -Regex ($Suite) { + '^Good$' { + Write-Host 'Passed: 3' + Write-Host 'Failed: 0' + Write-Probe 'end' + exit 0 + } + '^Bad$' { + Write-Host '[FAIL] fixture check failed' + Write-Host 'Passed: 1' + Write-Host 'Failed: 1' + Write-Probe 'end' + exit 1 + } + '^Silent$' { + Write-Probe 'end' + exit 0 + } + '^Sleepy$' { + Start-Sleep -Milliseconds 400 + Write-Host 'Passed: 2' + Write-Host 'Failed: 0' + Write-Probe 'end' + exit 0 + } + '^Slow' { + # Prove real overlap without depending on wall-clock luck. Staying open until a peer suite + # is also running makes concurrency structural: a scheduler that serialized independent + # suites could never satisfy this, so the overlap assertion detects the bug instead of a + # startup-time race. The wait is bounded and never changes the suite's own duration logic. + if (-not [string]::IsNullOrWhiteSpace($probe)) { + $rendezvousDeadline = (Get-Date).AddSeconds(20) + while ((Get-Date) -lt $rendezvousDeadline) { + $startedCount = @(Get-ChildItem -LiteralPath $probe -Filter '*-start' -ErrorAction SilentlyContinue).Count + $finishedCount = @(Get-ChildItem -LiteralPath $probe -Filter '*-end' -ErrorAction SilentlyContinue).Count + if (($startedCount - $finishedCount) -ge 2) { break } + Start-Sleep -Milliseconds 25 + } + } + Start-Sleep -Milliseconds 400 + Write-Host 'Passed: 2' + Write-Host 'Failed: 0' + Write-Probe 'end' + exit 0 + } + '^Tree$' { + $child = Start-Process -FilePath (Join-Path $PSHOME 'pwsh') -PassThru -WindowStyle Hidden -ArgumentList @( + '-NoProfile', '-NonInteractive', '-Command', 'Start-Sleep -Seconds 180' + ) + Write-Probe 'child.pid' ([string]$child.Id) + Start-Sleep -Seconds 180 + exit 0 + } + '^Hang$' { + Start-Sleep -Seconds 180 + exit 0 + } +} +'@ + +$fixtureConformance = @' +param( + [ValidateSet('All', 'Protocol')] + [string]$Suite = 'All' +) +Write-Host "Real runner deterministic adapter conformance ($Suite): PASS" +'@ + +$fixtureIntegrity = @' +param( + [ValidateSet('All', 'Bridge')] + [string]$Suite = 'All' +) +Write-Host "Eval package integrity and finalization: PASS ($Suite)" +'@ + +function New-FixtureMatrix { + param([object[]]$Entries) + + $lines = @('matrix:', ' include:') + foreach ($entry in $Entries) { + $lines += (' - {{ name: {0}, script: {1}, suite: {2} }}' -f $entry.Name, $entry.Script, $entry.Suite) + } + return ($lines -join [Environment]::NewLine) + [Environment]::NewLine +} + +function New-FixtureRepo { + param( + [string]$Name, + [object[]]$Entries + ) + + $repo = Join-Path $root ('repo-' + $Name) + Write-FixtureFile -Path (Join-Path $repo 'scripts/validate-skill-templates.ps1') -Content $fixtureTemplateValidator + Write-FixtureFile -Path (Join-Path $repo 'scripts/eval-runners/tests/test-runner-conformance.ps1') -Content $fixtureConformance + Write-FixtureFile -Path (Join-Path $repo 'scripts/eval-runners/tests/test-integrity-finalization.ps1') -Content $fixtureIntegrity + $workflow = Join-Path $root ('workflow-' + $Name + '.yml') + Write-FixtureFile -Path $workflow -Content (New-FixtureMatrix -Entries $Entries) + + return [pscustomobject]@{ + RepositoryRoot = $repo + WorkflowPath = $workflow + OutputRoot = Join-Path $root ('out-' + $Name) + ProbeRoot = Join-Path $root ('probe-' + $Name) + } +} + +function Invoke-Scheduler { + param( + [object]$Fixture, + [int]$MaxConcurrency = 4, + [int]$TimeoutSeconds = 30, + [int]$DeadlineSeconds = 0 + ) + + $arguments = @( + '-NoProfile', '-NonInteractive', '-File', $scheduler, + '-RepositoryRoot', $Fixture.RepositoryRoot, + '-WorkflowPath', $Fixture.WorkflowPath, + '-OutputRoot', $Fixture.OutputRoot, + '-MaxConcurrency', [string]$MaxConcurrency, + '-TimeoutSeconds', [string]$TimeoutSeconds, + '-DeadlineSeconds', [string]$DeadlineSeconds + ) + + $previousProbe = $env:VALIDATION_TEST_PROBE_DIR + $previousErrorAction = $ErrorActionPreference + $env:VALIDATION_TEST_PROBE_DIR = $Fixture.ProbeRoot + $ErrorActionPreference = 'Continue' + try { + $output = & pwsh @arguments 2>&1 + $exitCode = $LASTEXITCODE + } finally { + $ErrorActionPreference = $previousErrorAction + $env:VALIDATION_TEST_PROBE_DIR = $previousProbe + } + + $summaryPath = Get-ChildItem -LiteralPath $Fixture.OutputRoot -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'summary.json' } | + Where-Object { Test-Path -LiteralPath $_ -PathType Leaf } | + Select-Object -First 1 + + $summary = if ($null -ne $summaryPath) { + [System.IO.File]::ReadAllText($summaryPath, $utf8NoBom) | ConvertFrom-Json -Depth 8 + } else { + $null + } + + return [pscustomobject]@{ + ExitCode = $exitCode + Output = ($output | ForEach-Object { [string]$_ }) + Summary = $summary + Path = $summaryPath + } +} + +function Get-Result { + param([object]$Summary, [string]$Name) + + return @($Summary.Results | Where-Object { $_.Name -eq $Name })[0] +} + +function Assert-True { + param([bool]$Condition, [string]$Message) + + if (-not $Condition) { + throw $Message + } +} + +function Invoke-Case { + param([string]$Name, [scriptblock]$Body) + + try { + & $Body + Write-Host "[PASS] $Name" + } catch { + Write-Host "[FAIL] $Name" + Write-Host " $($_.Exception.Message)" + $failures.Add($Name) + } +} + +New-Item -ItemType Directory -Path $root -Force | Out-Null + +try { + Invoke-Case 'all-passing suites report complete verified coverage' { + $fixture = New-FixtureRepo -Name 'allpass' -Entries @( + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Good' } + [pscustomobject]@{ Name = 'beta'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Sleepy' } + [pscustomobject]@{ Name = 'gamma'; Script = 'scripts/eval-runners/tests/test-runner-conformance.ps1'; Suite = 'Protocol' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 3 + + Assert-True -Condition ($result.ExitCode -eq 0) -Message "expected exit 0, got $($result.ExitCode)" + Assert-True -Condition ($result.Summary.Coverage.Expected -eq 3) -Message 'expected 3 scheduled suites' + Assert-True -Condition ($result.Summary.Coverage.Executed -eq 3) -Message 'expected 3 executed suites' + Assert-True -Condition ($result.Summary.Coverage.Passed -eq 3) -Message 'expected 3 passing suites' + Assert-True -Condition ($result.Summary.Coverage.Verified) -Message 'expected coverage to be marked verified' + foreach ($name in @('alpha', 'beta', 'gamma')) { + $entry = Get-Result -Summary $result.Summary -Name $name + Assert-True -Condition ($entry.Status -eq 'PASS') -Message "expected $name to pass" + Assert-True -Condition (Test-Path -LiteralPath $entry.Log -PathType Leaf) -Message "expected a log for $name" + } + } + + Invoke-Case 'a failing suite fails the run without cancelling the others' { + $fixture = New-FixtureRepo -Name 'failure' -Entries @( + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Good' } + [pscustomobject]@{ Name = 'beta'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Bad' } + [pscustomobject]@{ Name = 'gamma'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Sleepy' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 3 + + Assert-True -Condition ($result.ExitCode -eq 1) -Message 'expected a non-zero exit' + Assert-True -Condition ((Get-Result -Summary $result.Summary -Name 'beta').Status -eq 'FAIL') -Message 'expected the failing suite to be reported FAIL' + Assert-True -Condition ((Get-Result -Summary $result.Summary -Name 'alpha').Status -eq 'PASS') -Message 'expected the independent suite to still pass' + Assert-True -Condition ((Get-Result -Summary $result.Summary -Name 'gamma').Status -eq 'PASS') -Message 'expected the independent suite to still pass' + Assert-True -Condition ($result.Summary.Coverage.Verified -eq $false) -Message 'coverage must not be reported verified' + } + + Invoke-Case 'a suite that exits zero without its terminal markers is not a pass' { + $fixture = New-FixtureRepo -Name 'silent' -Entries @( + [pscustomobject]@{ Name = 'silent'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Silent' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 1 + + Assert-True -Condition ($result.ExitCode -eq 1) -Message 'expected silent skipping to fail the run' + $entry = Get-Result -Summary $result.Summary -Name 'silent' + Assert-True -Condition ($entry.Status -eq 'FAIL') -Message 'expected FAIL for a suite with no terminal marker' + Assert-True -Condition ($entry.Verified -eq $false) -Message 'expected the suite to be unverified' + } + + Invoke-Case 'a suite that reports a skipped check is not a pass' { + $fixture = New-FixtureRepo -Name 'skipped' -Entries @( + [pscustomobject]@{ Name = 'skipped'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Skip' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 1 + + Assert-True -Condition ($result.ExitCode -eq 1) -Message 'expected a skipped suite to fail the run' + Assert-True -Condition ((Get-Result -Summary $result.Summary -Name 'skipped').Status -eq 'FAIL') -Message 'expected FAIL for a skipped suite' + } + + Invoke-Case 'a hanging suite times out and its process tree is killed' { + $fixture = New-FixtureRepo -Name 'timeout' -Entries @( + [pscustomobject]@{ Name = 'longrunner'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Tree' } + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Good' } + ) + $started = [DateTimeOffset]::UtcNow + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 2 -TimeoutSeconds 3 + $elapsed = ([DateTimeOffset]::UtcNow - $started).TotalSeconds + + Assert-True -Condition ($result.ExitCode -eq 1) -Message 'expected a timeout to fail the run' + $entry = Get-Result -Summary $result.Summary -Name 'longrunner' + Assert-True -Condition ($entry.Status -eq 'TIMEOUT') -Message "expected TIMEOUT, got $($entry.Status)" + Assert-True -Condition ($result.Summary.Coverage.TimedOut -eq 1) -Message 'expected one timed-out suite' + Assert-True -Condition ($elapsed -lt 30) -Message "timeout was not enforced promptly ($elapsed seconds)" + Assert-True -Condition ((Get-Result -Summary $result.Summary -Name 'alpha').Status -eq 'PASS') -Message 'expected the healthy suite to complete' + + $childPidFile = Join-Path $fixture.ProbeRoot 'Tree-child.pid' + Assert-True -Condition (Test-Path -LiteralPath $childPidFile -PathType Leaf) -Message 'expected the fixture to record its grandchild process' + $childPid = [int]([System.IO.File]::ReadAllText($childPidFile)) + $survivor = $true + for ($attempt = 0; $attempt -lt 40 -and $survivor; $attempt++) { + Start-Sleep -Milliseconds 100 + $survivor = $null -ne (Get-Process -Id $childPid -ErrorAction SilentlyContinue) + } + Assert-True -Condition (-not $survivor) -Message "grandchild process $childPid survived the timeout" + } + + Invoke-Case 'a deadline cancels the run, reports survivors, and still writes a summary' { + $fixture = New-FixtureRepo -Name 'deadline' -Entries @( + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Hang' } + [pscustomobject]@{ Name = 'beta'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Hang' } + [pscustomobject]@{ Name = 'gamma'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Hang' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 1 -TimeoutSeconds 60 -DeadlineSeconds 3 + + Assert-True -Condition ($result.ExitCode -eq 1) -Message 'expected a cancelled run to fail' + Assert-True -Condition ($result.Summary.Interrupted) -Message 'expected the summary to record the interruption' + Assert-True -Condition ($result.Summary.Coverage.Expected -eq 3) -Message 'expected all scheduled suites to be accounted for' + Assert-True -Condition ($result.Summary.Coverage.Passed -eq 0) -Message 'no suite may be reported as passing' + Assert-True -Condition (-not $result.Summary.Coverage.Verified) -Message 'an incomplete run must not be verified' + Assert-True -Condition (@($result.Summary.Results).Count -eq 3) -Message 'expected one result row per scheduled suite' + Assert-True -Condition (@($result.Summary.Results | Where-Object { $_.Status -eq 'NOT RUN' }).Count -ge 1) -Message 'expected at least one suite to be reported as not run' + } + + Invoke-Case 'concurrency is bounded and independent suites actually overlap' { + $fixture = New-FixtureRepo -Name 'concurrency' -Entries @( + [pscustomobject]@{ Name = 'omega1'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'SlowA' } + [pscustomobject]@{ Name = 'omega2'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'SlowB' } + [pscustomobject]@{ Name = 'omega3'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'SlowC' } + [pscustomobject]@{ Name = 'omega4'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'SlowD' } + ) + $result = Invoke-Scheduler -Fixture $fixture -MaxConcurrency 2 -TimeoutSeconds 30 + + Assert-True -Condition ($result.ExitCode -eq 0) -Message 'expected the bounded run to pass' + + $intervals = foreach ($suite in @('SlowA', 'SlowB', 'SlowC', 'SlowD')) { + $startPath = Join-Path $fixture.ProbeRoot "$suite-start" + $endPath = Join-Path $fixture.ProbeRoot "$suite-end" + Assert-True -Condition ((Test-Path -LiteralPath $startPath) -and (Test-Path -LiteralPath $endPath)) -Message "expected $suite to record a run interval" + [pscustomobject]@{ + Start = [long]([System.IO.File]::ReadAllText($startPath)) + End = [long]([System.IO.File]::ReadAllText($endPath)) + } + } + Assert-True -Condition (@($intervals).Count -eq 4) -Message 'expected four run intervals' + + # Reconstruct the maximum number of suites that were executing at the same instant. + $events = foreach ($interval in $intervals) { + [pscustomobject]@{ At = $interval.Start; Delta = 1 } + [pscustomobject]@{ At = $interval.End; Delta = -1 } + } + $running = 0 + $maxOverlap = 0 + foreach ($event in @($events | Sort-Object At, Delta)) { + $running += $event.Delta + if ($running -gt $maxOverlap) { $maxOverlap = $running } + } + + Assert-True -Condition ($maxOverlap -le 2) -Message "concurrency bound was exceeded (observed $maxOverlap)" + Assert-True -Condition ($maxOverlap -ge 2) -Message "independent suites never overlapped (observed $maxOverlap)" + foreach ($interval in $intervals) { + $duration = [Math]::Round(($interval.End - $interval.Start) / 1000.0, 2) + Assert-True -Condition ($duration -ge 0.35) -Message "expected the fixture to honour its programmed duration (got $duration)" + } + Assert-True -Condition (@($result.Summary.Results | Where-Object { $_.Seconds -lt 0 }).Count -eq 0) -Message 'suite timings must be non-negative' + } + + Invoke-Case 'output inside the repository is refused outside .bot' { + $fixture = New-FixtureRepo -Name 'guard' -Entries @( + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Good' } + ) + $repoRoot = Resolve-Path (Join-Path $PSScriptRoot '..\..') + $output = & pwsh -NoProfile -NonInteractive -File $scheduler ` + -RepositoryRoot $repoRoot.Path ` + -OutputRoot (Join-Path $repoRoot.Path 'scripts/validate-local-output') 2>&1 + $exitCode = $LASTEXITCODE + + Assert-True -Condition ($exitCode -ne 0) -Message 'expected an unauthorized output root to be refused' + Assert-True -Condition ((($output | ForEach-Object { [string]$_ }) -join ' ') -match 'must stay under') -Message 'expected an explanatory refusal message' + } + + Invoke-Case 'matrix problems are rejected before any suite runs' { + $duplicate = New-FixtureRepo -Name 'duplicate' -Entries @( + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Good' } + [pscustomobject]@{ Name = 'alpha'; Script = 'scripts/validate-skill-templates.ps1'; Suite = 'Slow' } + ) + $duplicateRun = Invoke-Scheduler -Fixture $duplicate -MaxConcurrency 2 + Assert-True -Condition ($duplicateRun.ExitCode -ne 0) -Message 'expected duplicate suite names to be rejected' + Assert-True -Condition ($null -eq $duplicateRun.Summary) -Message 'no summary should be produced for an invalid matrix' + + $missing = New-FixtureRepo -Name 'missing' -Entries @( + [pscustomobject]@{ Name = 'ghost'; Script = 'scripts/does-not-exist.ps1'; Suite = 'Good' } + ) + $missingRun = Invoke-Scheduler -Fixture $missing -MaxConcurrency 1 + Assert-True -Condition ($missingRun.ExitCode -ne 0) -Message 'expected a missing script to be rejected' + + $emptyRoot = Join-Path $root 'empty' + New-Item -ItemType Directory -Path $emptyRoot -Force | Out-Null + $emptyWorkflow = Join-Path $root 'workflow-empty.yml' + Write-FixtureFile -Path $emptyWorkflow -Content ("matrix:" + [Environment]::NewLine + " include: []" + [Environment]::NewLine) + $emptyOutput = & pwsh -NoProfile -NonInteractive -File $scheduler -RepositoryRoot $emptyRoot -WorkflowPath $emptyWorkflow -OutputRoot (Join-Path $root 'out-empty') 2>&1 + Assert-True -Condition ($LASTEXITCODE -ne 0) -Message 'expected an empty matrix to be rejected' + Assert-True -Condition ((($emptyOutput | ForEach-Object { [string]$_ }) -join ' ') -match 'no suite entries') -Message 'expected an explanatory empty-matrix message' + } + + Invoke-Case 'the matrix matches the checked-in CI workflow' { + $repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path + $listed = & pwsh -NoProfile -NonInteractive -File $scheduler -RepositoryRoot $repoRoot -ListSuites 2>&1 + Assert-True -Condition ($LASTEXITCODE -eq 0) -Message 'expected suite listing to succeed' + $entries = @($listed | Where-Object { [string]$_ -match ' -> ' }) + Assert-True -Condition ($entries.Count -ge 1) -Message 'expected at least one CI suite' + foreach ($entry in $entries) { + $parts = [string]$entry -split ' -> ' + $scriptArguments = $parts[1] -split ' -Suite ' + $scriptPath = Join-Path $repoRoot $scriptArguments[0] + Assert-True -Condition (Test-Path -LiteralPath $scriptPath -PathType Leaf) -Message "missing scheduled script $($scriptArguments[0])" + } + } +} finally { + if (Test-Path -LiteralPath $root) { + Remove-Item -LiteralPath $root -Recurse -Force -ErrorAction SilentlyContinue + } +} + +if ($failures.Count -gt 0) { + Write-Host '' + Write-Host ("Failed cases: {0}" -f ($failures -join ', ')) + exit 1 +} + +Write-Host '' +Write-Host 'Local validation scheduler: PASS' diff --git a/scripts/validate-local.ps1 b/scripts/validate-local.ps1 new file mode 100644 index 0000000..ca04a59 --- /dev/null +++ b/scripts/validate-local.ps1 @@ -0,0 +1,476 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Runs the complete deterministic validation matrix locally, in parallel. + +.DESCRIPTION + The CI workflow (.github/workflows/validate-skill-templates.yml) is the coverage source of + truth: every `{ name, script, suite }` entry is one independently scheduled suite. This script + runs exactly those suites as separate processes, so local execution and CI cover the same + checks, and it keeps a bounded number of them in flight at once. + + Improvements over running the suites by hand: + + * Coverage parity with CI is derived from the workflow, never from a hand-kept list. + * Each suite is a separate process, so a crash or hang cannot take the run down with it. + * Per-suite timeouts kill the whole process tree; a timeout is a failure, never a pass. + * A failing suite does not stop the others, so one run reports every problem at once. + * Each suite's output is streamed to its own log for post-mortem inspection. + * Every suite must produce its terminal success marker. A suite that exits 0 while silently + skipping its checks is reported as a failure instead of a pass. + * Ctrl+C cancels the run, tears down every child process tree, and still writes the summary. + + A pass therefore means: every CI-matrix suite ran to completion, exited 0, reported no failing + check, and printed its terminal success marker. + +.PARAMETER MaxConcurrency + Maximum number of suites executing at the same time. Defaults to the processor count. + +.PARAMETER TimeoutSeconds + Per-suite wall-clock budget. A suite that exceeds it has its process tree killed and is + reported as TIMEOUT. Use 0 to disable. + +.PARAMETER DeadlineSeconds + Wall-clock budget for the entire run. Suites that have not finished are killed and reported, + and the run fails. Use 0 to disable. + +.PARAMETER OutputRoot + Directory that receives one log per suite plus summary.json. Defaults to + /.bot/validation-workspace. Paths inside the repository must stay under `.bot/`. + +.PARAMETER WorkflowPath + Workflow file that defines the suite matrix. Defaults to the repository CI workflow. This + exists so the scheduler can be exercised deterministically against fixture workflows. + +.PARAMETER RepositoryRoot + Repository root the matrix script paths are resolved against. Defaults to the parent of the + script directory. + +.PARAMETER ListSuites + Print the resolved suite matrix and exit without running anything. + +.EXAMPLE + pwsh -NoProfile -File ./scripts/validate-local.ps1 + +.EXAMPLE + pwsh -NoProfile -File ./scripts/validate-local.ps1 -MaxConcurrency 24 -TimeoutSeconds 900 +#> +[CmdletBinding()] +param( + [ValidateRange(1, 128)][int]$MaxConcurrency = [Environment]::ProcessorCount, + [ValidateRange(0, 86400)][int]$TimeoutSeconds = 900, + [ValidateRange(0, 86400)][int]$DeadlineSeconds = 0, + [string]$OutputRoot, + [string]$WorkflowPath, + [string]$RepositoryRoot, + [switch]$ListSuites +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +# Terminal markers that prove a suite executed its checks instead of exiting early. These are the +# public summary lines each scheduled script already prints; they are asserted on top of the exit +# code so that a no-op or partially skipped suite can never be reported as a pass. +$suiteExpectations = @{ + 'scripts/validate-skill-templates.ps1' = @( + [pscustomobject]@{ Pattern = '(?m)^Failed: 0\s*$'; Description = "reports 'Failed: 0'" } + [pscustomobject]@{ Pattern = '(?m)^Passed: [1-9][0-9]*\s*$'; Description = 'reports at least one passing check' } + ) + 'scripts/eval-runners/tests/test-runner-conformance.ps1' = @( + [pscustomobject]@{ Pattern = '(?m)conformance( \([A-Za-z]+\))?: PASS\s*$'; Description = 'reports conformance PASS' } + ) + 'scripts/eval-runners/tests/test-integrity-finalization.ps1' = @( + [pscustomobject]@{ Pattern = '(?m)Eval package integrity and finalization: PASS \([A-Za-z]+\)\s*$'; Description = 'reports integrity PASS' } + ) +} + +$failureMarkerPattern = '(?m)^\[FAIL\]|:\s*FAIL\s*$|^Failed: [1-9][0-9]*\s*$' + +function Get-RepoRoot { + if (-not [string]::IsNullOrWhiteSpace($RepositoryRoot)) { + return ([System.IO.Path]::GetFullPath($RepositoryRoot)).TrimEnd('\', '/') + } + + return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path.TrimEnd('\', '/') +} + +function Get-SuiteMatrix { + param([string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Validation workflow was not found at '$Path'." + } + + $workflow = [System.IO.File]::ReadAllText($Path) + $entries = @([regex]::Matches($workflow, '\{ name: (?[\w-]+), script: (?