Conversation
…ning one
`best_detail` describes whichever run ranked highest, so a partial pass exposes
only the attempt that worked. On a 1-of-3 item every visible verdict belongs to
the run that passed, and the two that failed leave no trace at all -- their
`detail` is computed inside the run loop and then dropped on the floor.
That makes a partial pass undiagnosable after the fact. The only recourse is to
re-run the question and hope it fails the same way, which for a nondeterministic
agent is not a given. In a real corpus this is not an edge case: on one eval day
half of all lost runs sat on items whose recorded detail was entirely green.
`failed_runs` records one entry per run that did not pass, in run order:
run_index, passed, error, detail,
conversation_id, response_id,
stream_ended, turn_wall_clock_sec, latency_s,
reasoning_step_count, reasoning_steps
Three properties worth calling out.
**Kind-agnostic.** `detail` is opaque to the runner -- it never inspects its
shape -- so this covers every test kind and any kind added later, with no
per-evaluator work.
**Failing runs only.** A fully-passing item records nothing, so the cost tracks
how broken the corpus is rather than how large it is, and shrinks as quality
improves. `detail` and the top-level conversation/response ids keep their exact
current meaning, so existing consumers of the report are unaffected.
**Each entry carries its own conversation ids.** This also fixes a latent
mismatch: the report's top-level `conversation_id`/`response_id` are overwritten
on every iteration and end up describing the LAST run, while `best_detail` and
`reasoning_steps` describe the BEST one. When those differ, the ids point at a
different conversation than the detail beside them. `best_chat_result` already
exists to keep reasoning_steps aligned with best_detail; the ids were never
given the same treatment. Per-run ids make the pairing correct by construction
instead of adding a fourth field to keep in sync.
`stream_ended` is included because a stalled turn leaves the evaluator's gated
checks False even though none of them ran, which reads as a content failure in
every downstream rate. Recording it at the source removes the need for consumers
to infer stalls from the shape of the detail block.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 1 billable file and costs up to $0.25. Or wait 45 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (1)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (4)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe runner now records diagnostics for each failed or ungraded run. JSON reports expose these records while preserving the winning run’s existing detail and identifiers. ChangesFailed Run Reporting
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~12 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant _run_one_item
participant _failed_run_record
participant ItemReport
participant _build_run_dict
participant JSONReport
_run_one_item->>_failed_run_record: Build diagnostic for a non-passing run
_failed_run_record->>ItemReport: Append failed-run record
ItemReport->>_build_run_dict: Provide winning detail and failed_runs
_build_run_dict->>JSONReport: Emit both winning and failed-run data
Merge Risk: ⚪ Minimal · up to The failed-run diagnostics are emitted alongside existing report fields without a demonstrated regression, so the change is ready to merge. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
A rabbit logs each failed hop Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1816 +/- ##
=======================================
Coverage 82.49% 82.50%
=======================================
Files 283 283
Lines 20448 20452 +4
=======================================
+ Hits 16869 16873 +4
Misses 3579 3579 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…y ruff format Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
best_detaildescribes whichever run ranked highest. On a partial pass that means every visible verdict belongs to the attempt that worked, and the runs that failed leave no trace — theirdetailis computed inside the run loop and then dropped.So a 1-of-3 item is undiagnosable after the fact. The only recourse is re-running the question and hoping it fails the same way, which for a nondeterministic agent is not a given.
This isn't an edge case. On one evaluation day in our corpus, half of all lost runs sat on items whose recorded detail was entirely green — every criterion passing, the item still failing 2 of 3 times, and nothing anywhere explaining why.
The change
One new field on
ItemReport, emitted besidedetailin the JSON report:Kind-agnostic.
detailis opaque to the runner — it never inspects its shape — so this covers all test kinds and any added later, with no per-evaluator work.Failing runs only. A fully-passing item records nothing, so the cost tracks how broken the corpus is rather than how large it is, and shrinks as quality improves.
Nothing existing changes.
detailand the top-level ids keep their exact current meaning, so consumers of this report are unaffected.It also fixes a latent mismatch
The top-level
conversation_id/response_idare overwritten on every iteration and end up describing the last run, whilebest_detailandreasoning_stepsdescribe the best one. When those differ, the ids point at a different conversation than the detail beside them.best_chat_resultalready exists precisely to keepreasoning_stepsaligned withbest_detail(see the comment at its declaration) — the ids were never given the same treatment. Per-run ids make the pairing correct by construction rather than adding a fourth field to keep in sync.Why
stream_endedis in thereA stalled turn leaves the evaluator's gated checks
Falseeven though none of them ran, which reads as a content failure in every downstream rate. Recording it at the source removes the need for consumers to infer stalls from the shape of the detail block.Tests
Eight new tests.
uv run pytest— 976 passed, 0 failed.best_detailstays the winnerstream_endedandreasoning_step_countare recordedpass_power_k: falseon an item whose graded runs all passedfailed_runsbesidedetail, and[]for a clean item🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests