feat(gooddata-eval): record the turns and steps each agentic run took - #1813
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (10)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour. 📝 WalkthroughWalkthroughAgentic conversation and skill results now preserve turn and reasoning-step counts as integers. Langfuse scoring receives these numeric totals. Tests cover clarification, failed-send, and visualization cases. ChangesAgentic metrics
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~15 minutes Change: Feature Merge Risk: ⚪ Minimal · up to The counter updates consistently use integer values across results and trace scoring, with corresponding test coverage. No merge-blocking risk remains. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
A rabbit counts each turn with care Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1813 +/- ##
==========================================
+ Coverage 82.30% 82.49% +0.19%
==========================================
Files 283 283
Lines 20421 20448 +27
==========================================
+ Hits 16807 16869 +62
+ Misses 3614 3579 -35 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d3ea699 to
7543628
Compare
myhoai
changed the title
The effort comparison in gdc-nas reads these. A pass/fail bit cannot separate two reasoning efforts, while the step count moves with the effort, and the turn count tells a wrong answer from a run max_iterations cut short. alert_skill and kda_skill gained the counters; metric_skill and conversation already had them and only needed the write. conversation also logs clarification_turns, since its total includes the turns its fixture asks for. The tests cover the counters on every skill and assert the counts reach Langfuse — the dataclass fields alone would pass with the score writes deleted.
7543628 to
fa82a6a
Compare
| total_turns: float = 0.0 | ||
| total_steps: float = 0.0 |
There was a problem hiding this comment.
Nit: does this needs to be float?
Review nit on alert_skill: these are counts, and a float reads as though a fraction of a turn or of a reasoning step were possible. conversation already counted its clarification turns as int. Applied to every skill, including the two fields visualization and metric_skill already had — leaving those as float would have split the same two score names across two types. Test literals drop the .0, and one score test pins the type with isinstance: `2 == 2.0` is True, so the literals alone would not catch a regression.
2 participants
The effort comparison in gdc-nas reads these. A pass/fail bit cannot separate two reasoning efforts, while the step count moves with the effort, and the turn count tells a wrong answer from a run max_iterations cut short.
alert_skill and kda_skill gained the counters; metric_skill and conversation already had them and only needed the write. conversation also logs clarification_turns, since its total includes the turns its fixture asks for.
Summary by CodeRabbit
Bug Fixes
Tests