Commit f6cde10
committed
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py to gooddata-eval, evaluating the chatbot's
create_key_driver_analysis/execute_key_driver_analysis tool calls against the
agent_kda_skill Langfuse dataset. Scope is completion (kda_triggered ->
executed -> success -> turn_completed), not per-field correctness -- per-field
checks are computed and logged as kda_-prefixed informational scores for a
follow-up ticket (QA-28699), and are None (not False) both when
expected_output has no key for a field AND when that field's own
precondition (kda_triggered, or executed+success for Summary) isn't met.
Latency is measured directly by the harness, not re-derived from Langfuse
after the fact: ChatResult.turn_wall_clock_sec is set by ChatClient around
the whole request (from just before opening the stream, so it includes
connection/server setup time the caller actually waits through, excluding
only a transient retry's own backoff sleep), stamped on both the successful
result and any ChatError.partial_result so a turn that dies mid-stream after
KDA already succeeded still gets a number. KdaRunResult.turn_wall_clock_sec
captures it in _run_once at the exact point create_args gets set -- atomic
with the kda_triggered signal, not a separate step that can fail
independently. Logged as the kda_turn_wall_clock_sec Langfuse score;
combo_report.py (gdc-nas) reads it directly, with no trace re-querying or
observation-shape matching needed on that side anymore.
Adds ChatResult.stream_ended (from the SSE response_ended event, derived --
not a raw server field, see sse_client.py's _RESPONSE_ENDED_EVENT comment).
turn_completed requires both stream_ended AND a non-empty text_response -- a
stream that ends cleanly but delivers nothing to the user isn't a completed
turn either, and a turn cut off mid-answer can still emit partial, non-empty
text before dying.
Fixes from review:
- kda_ prefix pass_at_k/pass_power_k Langfuse scores -- unprefixed, "pass_at_2"
at k=2 collides with visualization.py's own score name that gdc-nas's
combo_report.py.verdict() checks first, silently misfiling every KDA record
as visualization once KDA_RUN_K=2 is ever set.
- Relative (not absolute) tolerance for Summary's revenue-scale values by
default; change is checked against reference_value's scale, not against
itself. Also recognizes an explicit absolute_tolerance key -- every real
agent_kda_skill dataset item uses it, which this module never read before,
silently running ~3000x looser than the dataset author intended. Warns on
any other *tolerance* key so a future typo surfaces instead of repeating.
- Filters compared as canonicalized sets, not order-sensitive lists.
- except Exception (not except ChatError) around send_message -- a stream cut
off mid-turn raises a raw httpx transport error, which a narrower catch
would let escape uncaught, skipping Langfuse scoring entirely for that run.
- ChatError/TransientChatError carry partial_result so tool calls that
already succeeded before a later, unrelated stream error aren't discarded
and misreported as "the agent never called KDA at all".
- turn_completed resets to False on an exception, so a crash on a later
disambiguation iteration can't leave a stale True from an earlier iteration.
- kda_disambiguated logged and immediately nulls the six *_correct
informational fields: the simulated user reply names the acceptable
candidate(s) drawn from expected_output itself, so those fields aren't an
independent signal once a run went through disambiguation.
- KDA-specific _is_asking_clarification instead of a heuristic shared with
metric_skill.py/conversation.py, which had silently drifted apart; strips a
leading "to clarify, " discourse marker before its substring checks, since
that phrase means "in other words" in a final answer, not a request for one.
- run_agentic_kda_skill rejects k < 1 -- the single initial run happens
unconditionally regardless of k, so a bad env-driven KDA_RUN_K value (0, a
typo, negative) previously ran silently once instead of surfacing the bad
config, indistinguishable from a deliberate k=1.
- Regression tests for find_traces_per_conversation's None-safety and
_filters_match's isinstance guard.
- _MAX_LATENCY_SEC (this module's own hardcoded 60s, used only for
value_score's speed component) now cross-references gdc-nas's independently
configured _KDA_MAX_TURN_SEC in a comment -- the two aren't shared code and
nothing enforces them staying in sync if one changes without the other.
- stream_ended is now set the moment the response_ended event: line itself is
parsed, not its data: line -- an event with no data payload at all
previously left the flag unset.
JIRA: QA-288001 parent 676da68 commit f6cde10
7 files changed
Lines changed: 1955 additions & 7 deletions
File tree
- packages/gooddata-eval
- src/gooddata_eval/core
- agentic
- chat
- tests
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
33 | 41 | | |
34 | 42 | | |
35 | 43 | | |
| |||
56 | 64 | | |
57 | 65 | | |
58 | 66 | | |
| 67 | + | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
69 | 78 | | |
70 | 79 | | |
71 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
72 | 84 | | |
73 | 85 | | |
74 | 86 | | |
| |||
81 | 93 | | |
82 | 94 | | |
83 | 95 | | |
| 96 | + | |
84 | 97 | | |
85 | 98 | | |
86 | 99 | | |
87 | 100 | | |
88 | 101 | | |
89 | 102 | | |
90 | 103 | | |
| 104 | + | |
91 | 105 | | |
92 | 106 | | |
93 | 107 | | |
| |||
0 commit comments