Skip to content

Fix skill evaluation token selection - #69029

Closed
PureWeen wants to merge 1 commit into
pureween-investigate-issue-skillfrom
main
Closed

PureWeen wants to merge 1 commit into
pureween-investigate-issue-skillfrom
main

Conversation

@PureWeen

@PureWeen PureWeen commented Sep 3, 2026

Copy link
Copy Markdown
Member

Fixes the authentication failure seen while evaluating #69028 in https://github.com/dotnet/aspnetcore/actions/runs/33777017172.

The skill-evaluation workflow assumed COPILOT_PAT_0 was populated. The copilot-pat-pool environment currently has no environment-level secrets, and a successful shared PAT-pool run showed that only COPILOT_PAT_2 was available. As a result, COPILOT_GITHUB_TOKEN was empty before Vally started.

This change:

  • Selects the first configured COPILOT_PAT_0 through COPILOT_PAT_9 entry.
  • Falls back to the existing repository COPILOT_GITHUB_TOKEN.
  • Keeps the selected credential scoped only to the trusted Vally execution step.
  • Adds a workflow regression test and updates the hosted-evaluation documentation.

The existing authorization, same-repository restriction, exact-commit binding, trusted default-branch workflow, and trusted-file staging boundaries are unchanged.

Validated with:

  • pwsh ./eng/skill-evals/run.ps1 Test
  • pwsh ./eng/skill-evals/run.ps1 Validate
  • git diff --check

* Update issue triage safeguards

Rebase the issue triage type policy onto current main, preserve existing issue types, and enforce per-output staged mutations with gh-aw v0.87.10.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: bcb017f1-64bc-4c44-81c0-96d2b6cc3bce

* Add fail-closed metadata lookup for issue type

The issue_context job's type lookup conflates 'lookup failed' with
'issue is untyped', so a transient API error (502, network timeout)
cascades through the job graph and kills the entire triage run.

Add an explicit lookup_succeeded output that separates the trust
channel from the value channel:

- Failed/impossible lookup: lookup_succeeded=false, issue_type=''
  Type mutation fails closed (staged); area/labels/comment continue.
- Successful untyped lookup: lookup_succeeded=true, issue_type=''
  Type assignment proceeds normally.
- Successful typed lookup: lookup_succeeded=true, issue_type=<name>
  Existing type preserved (staged).

The step never fails: the gh api call is inside an if-compound that
suppresses errexit. Input is sanitized (single-line enforcement).

The set-issue-type staged expression gains a leading
'lookup_succeeded != true' clause so the handler blocks type writes
whenever the lookup is untrusted. The prompt gains a three-way rule
matching the same states and instructs the agent not to call
set-issue-type when the lookup failed.

Compiled with gh aw v0.87.10 strict, 0 warnings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7098ed82-dcc7-447e-8dd0-09e68f10ce22

---------

Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb017f1-64bc-4c44-81c0-96d2b6cc3bce
Copilot-Session: 7098ed82-dcc7-447e-8dd0-09e68f10ce22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants