Fix skill evaluation authentication and runtime

[PureWeen]

Fixes the authentication and runtime failures seen while evaluating #69028 in https://github.com/dotnet/aspnetcore/actions/runs/33777017172.

Two independent issues prevented the model evaluation from completing:

1. The workflow assumed COPILOT_PAT_0 was populated even though the shared copilot-pat-pool environment can expose any populated slot.
2. The transient Vally/Copilot installation could resolve an incompatible Copilot platform package whose SDK entry point was unavailable.

This change follows the trusted PAT-pool and pinned-toolchain design used by the existing ASP.NET Core agentic workflows and dotnet/skills:

• A dedicated trusted job receives COPILOT_PAT_0 through COPILOT_PAT_9, finds the populated slots, and randomly selects one.
• The selector emits only the selected numeric slot, never a secret value.
• The model step resolves that slot through the repository's canonical static case(...) secret mapping, with COPILOT_GITHUB_TOKEN as the empty-pool fallback.
• The selected credential remains scoped only to the trusted Vally execution step.
• Vally and Copilot are installed from a checked-in manifest and lockfile before credentials enter scope.
• The Copilot dependency graph is constrained to SDK-compatible version 1.0.80, and the workflow verifies Linux platform SDK resolution from the consuming SDK package's module path before running a model.
• Workflow tests execute the real selector against empty, one-slot, and multi-slot synthetic pools and reject credential disclosure or conflicting nested Copilot packages.

The existing authorization, same-repository restriction, exact-commit binding, trusted default-branch workflow, and trusted-file staging boundaries are unchanged.

Validated locally with:

• pwsh ./eng/skill-evals/run.ps1 Test
• pwsh ./eng/skill-evals/run.ps1 Validate
• an isolated npm ci from the checked-in lockfile and SDK-relative platform-resolution probe
• git diff --check

End-to-end validation used PureWeen/aspnetcore, whose copilot-pat-pool environment contained only COPILOT_PAT_2: https://github.com/PureWeen/aspnetcore/actions/runs/33795737099

• The selector reported pool size 1 and selected slot 2.
• The model step received a non-empty masked credential.
• Vally 0.14.0, Copilot CLI 1.0.80, and the Linux x64 SDK platform entry point resolved successfully.
• All five baseline and five skilled executions completed successfully.
• The skilled variant scored 0.79 against the 0.6 threshold; baseline scored 0.555.
• The uploaded artifact contained provenance, report, summaries, results, metadata, and event logs for both variants.

The temporary fork commits were restored with a normal revert after the run.

[dotnet-policy-service]

Hey @dotnet/aspnet-build, looks like this PR is something you want to take a look at.

[Copilot]

Copilot review overview

🟢 Approval recommended

The workflow change is narrowly scoped, includes an explicit regression test for the credential selection expression, and the documentation is updated to match the new behavior.

Review tier: Lite
Findings: None

What changed in this PR

Updates the skill-evaluation GitHub Actions workflow to reliably select a usable Copilot credential when the copilot-pat-pool environment doesn’t provide COPILOT_PAT_0, preventing authentication failures during Vally execution.

Changes:

• Update .github/workflows/skill-evals.yml to select the first available COPILOT_PAT_0..COPILOT_PAT_9, falling back to COPILOT_GITHUB_TOKEN, and improve the failure message when none are configured.
• Add a regression guard in eng/skill-evals/test_workflow.ps1 to ensure the ordered PAT-pool selection + fallback expression remains present exactly once.
• Update eng/skill-evals/README.md to document the PAT pool selection and fallback behavior.

File	Description
eng/​skill-evals/​test_workflow.ps1	Adds a workflow self-test to assert the ordered PAT-pool selection and repository fallback are present.
eng/​skill-evals/​README.md	Updates hosted-evaluation docs to reflect PAT pool selection across COPILOT_PAT_0..9 with repo fallback.
.github/​workflows/​skill-evals.yml	Selects the first configured PAT from the pool with fallback, scoped to the Vally execution step, and improves the configuration error message.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[PureWeen]

Adversarial review

No correctness or security defect survived consensus. The trusted selector matches ASP.NET Core’s existing .github/workflows/shared/pat_pool.md design: it receives the pool only in an isolated trusted job, emits only a numeric slot, and keeps candidate-controlled execution away from the pool.

One testing gap remains: the regression test verifies YAML text but does not execute the empty-pool, slot-2-only, multi-slot, or repository-fallback paths. This is especially relevant because the pull-request event skips the credential-bearing jobs.

The selected-secret expression is valid GitHub Actions syntax because steps.env supports both needs and secrets, and expression index syntax accepts a formatted key. However, it is not the exact consumer expression used by the repository’s existing PAT-pool workflows, which use a static case(...) mapping. That is a consistency difference, not a demonstrated runtime bug.

Methodology: 3 independent reviewers with adversarial consensus. The dynamic-lookup, fallback-provisioning, and selector-exposure concerns were independently disputed and discarded. Existing Copilot feedback reviewed the prior commit and did not cover the current random-selector implementation.

Test coverage assessment: structural coverage is present; the model-bearing authentication path lacks execution coverage in this PR.

[PureWeen]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.