V0.9.2/one prompt eval

[gimlichael]

This pull request updates the AGENTS.md documentation to clarify and refine the rules around evaluation package execution, especially regarding explicit user-directed evals using the yolo or auto modifiers. The changes introduce a narrowly-scoped, one-shot external Eval Orchestrator handoff flow, and tighten the requirements for model/harness selection and agent roles during eval preparation and execution. The documentation now more clearly distinguishes between routine/manual and explicitly authorized automated eval handoffs, and details the technical and policy boundaries for each.

Explicit Eval Handoff Flow:

• Added a new section describing an optional, explicitly authorized one-shot external Eval Orchestrator handoff for eval requests with yolo or auto, detailing when and how this handoff is allowed, and the technical steps for invoking it using scripts/eval-request.ps1 and Invoke-EvalRequest.

Eval Preparation and Execution Rules:

• Clarified that normal eval requests result in package preparation and manual handoff, while explicit yolo/auto requests authorize the one-shot external handoff. Updated the preparation and execution sections to reflect this distinction and to specify that only the explicit modifier can trigger automatic handoff. [1] [2] [3]
• Updated agent role definitions and restrictions to specify that the preparer context cannot execute its own package, but a fresh external Orchestrator authorized by the user may do so for that package only.

Model/Harness Selection and Discovery:

• Tightened requirements for harness/model selection: discovery is only performed when the user has not supplied a harness, and never without an explicit runner. The documentation now prohibits reconfirmation of supplied runners and clarifies the handling of OpenCode selectors and repository defaults.

Skill-Creator Integration:

• Updated the skill-creator section to reflect the new explicit handoff flow, clarifying that repository-side validation remains deterministic, and only the explicitly user-directed external executor performs model-backed evaluation and reporting.

[greptile-apps]

Confidence Score: 4/5

The PR is not yet safe to merge because a forced retry can still erase the handoff reservation and authorize a second external evaluation.

Summary

• Introduces deterministic eval-request preparation and handoff reservation logic for explicit yolo/auto requests.
• Adds declarative synthetic Git histories for isolated evaluation fixtures.
• Tightens Copilot workspace projection, credential selection, execution evidence, grading, and finalization boundaries.
• Expands validation and conformance coverage for the new workflows.
• Updates repository documentation, release notes, and affected skill evaluations.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User requests an evaluation] --> B[Prepare and validate package]
    B --> C{Explicit yolo or auto?}
    C -- No --> D[Return absolute RUN-THIS.prompt.md path]
    C -- Yes --> E{Fresh external orchestrator available?}
    E -- No --> D
    E -- Yes --> F{Handoff already reserved or execution started?}
    F -- Yes --> G[Return already_started and observe existing execution]
    F -- No --> H[Create .external-handoff-started]
    H --> I[Delegate one fresh external orchestrator]
    I --> J[Execute package phases and return report]

Loading

[greptile-apps]

Forced Retry Erases Reservation

When Invoke-EvalRequest is retried with the same Iteration and Force preparation options, it reruns preparation before checking the handoff reservation. Because Force deletes and recreates the iteration directory containing .external-handoff-started, Get-EvalHandoff returns external_handoff again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-request.ps1
Line: 68

Comment:
**Forced Retry Erases Reservation**

When `Invoke-EvalRequest` is retried with the same `Iteration` and `Force` preparation options, it reruns preparation before checking the handoff reservation. Because `Force` deletes and recreates the iteration directory containing `.external-handoff-started`, `Get-EvalHandoff` returns `external_handoff` again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Prohibited Temporary Git History

This helper renames a branch, checks out a temporary feature branch, stages changes, and creates throwaway commits. The repository directive explicitly forbids temporary Git repositories, test branches, and throwaway commits. This repository requirement must be satisfied before merging, either by representing the fixture without this prohibited workflow or by intentionally narrowing the directive.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-git-workspace.ps1
Line: 35-49

Comment:
**Prohibited Temporary Git History**

This helper renames a branch, checks out a temporary feature branch, stages changes, and creates throwaway commits. The repository directive explicitly forbids temporary Git repositories, test branches, and throwaway commits. This repository requirement must be satisfied before merging, either by representing the fixture without this prohibited workflow or by intentionally narrowing the directive.

**Context Used:** AGENTS.md ([source](https://github.com/codebeltnet/agentic/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!