Commit 8a7de88
committed
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py to gooddata-eval, evaluating the chatbot's
create_key_driver_analysis/execute_key_driver_analysis tool calls against the
agent_kda_skill Langfuse dataset. Scope is strictly completion:
strict_pass = triggered AND executed AND success AND turn_completed
Per-field correctness (Measure/Date Attribute/Periods/Filters/Summary
matching expected values) is intentionally out of scope for this PR --
deferred entirely to a follow-up ticket (QA-28699) rather than half-computed
here as unused informational scores.
Latency is measured directly by the harness, not re-derived from Langfuse
after the fact: ChatResult.turn_wall_clock_sec is set by ChatClient before
opening the SSE stream (so it includes connection/server setup time the
caller actually waits through, excluding only a transient retry's own
backoff sleep), stamped on both the successful result and any
ChatError.partial_result so a turn that dies mid-stream after KDA already
succeeded still gets a number. Only the turn that actually completes KDA
(both create AND execute) counts toward KdaRunResult.turn_wall_clock_sec --
not any earlier disambiguation/continuation turn, and not the simulated-
reply generation call itself (a test-harness OpenAI call, not gen-ai's own
time). Logged as the kda_turn_wall_clock_sec Langfuse score; combo_report.py
(gdc-nas) reads it directly, with no trace re-querying needed on that side.
create_key_driver_analysis and execute_key_driver_analysis are tracked
independently across turns, not required to land in the same turn's tool
calls -- if the agent calls create but not execute within one turn, a plain
continuation nudge ("Please proceed.") gives it another turn rather than the
run being scored as if execute never happened at all.
Adds ChatResult.stream_ended (from the SSE response_ended event).
turn_completed requires both stream_ended AND a non-empty text_response -- a
stream that ends cleanly but delivers nothing to the user isn't a completed
turn either, and a turn cut off mid-answer can still emit partial, non-empty
text before dying.
If the agent asks a clarifying question instead of triggering KDA directly
(a title collision, or a metric-vs-fact form choice), a simulated user reply
(gpt-4o-mini, 30s timeout) nudges it forward, bounded to max_iterations.
Fixes from review:
- kda_ prefix pass_at_k/pass_power_k Langfuse scores -- unprefixed, "pass_at_2"
at k=2 collides with visualization.py's own score name that gdc-nas's
combo_report.py.verdict() checks first, silently misfiling every KDA record
as visualization once KDA_RUN_K=2 is ever set.
- except Exception (not except ChatError) around send_message -- a stream cut
off mid-turn raises a raw httpx transport error, which a narrower catch
would let escape uncaught, skipping Langfuse scoring entirely for that run.
- ChatError/TransientChatError carry partial_result so tool calls that
already succeeded before a later, unrelated stream error aren't discarded
and misreported as "the agent never called KDA at all".
- turn_completed resets to False on an exception, so a crash on a later
iteration can't leave a stale True from an earlier one.
- KDA-specific _is_asking_kda_clarification instead of a heuristic shared with
metric_skill.py/conversation.py, which had silently drifted apart; strips a
leading "to clarify, " discourse marker before its substring checks, since
that phrase means "in other words" in a final answer, not a request for one.
- run_agentic_kda_skill rejects k < 1 -- a bad env-driven KDA_RUN_K value (0, a
typo, negative) previously ran silently once instead of surfacing the bad
config, indistinguishable from a deliberate k=1.
- stream_ended is now set the moment the response_ended event: line itself is
parsed, not its data: line -- an event with no data payload at all
previously left the flag unset.
- t0 is set before opening the SSE stream, not after -- it was missing the
request/connection/server-setup time a caller actually waits through.
- Disambiguated-turn latency no longer double-counts: it used to sum every
turn's time plus the simulated-reply OpenAI call itself, inflating exactly
the cases that needed disambiguation (usually the harder ones) with time
that has nothing to do with gen-ai or KDA.
- The 6 per-field informational correctness scores (measure/date_attribute/
analyzed_period/reference_period/filters/summary_correct) and their support
code (_measure_matches, _filters_match, _within_relative_tolerance,
_within_absolute_tolerance, _resolve_summary_tolerance, _normalize_measure,
_to_number) are removed -- team confirmed this PR's scope is trigger+
complete only; correctness is QA-28699's job, not a half-implemented,
unread-by-anything set of scores here. (One of these, _filters_match(actual,
None), silently mismatched an explicit "expect no filters" case -- moot now
that the function is gone.)
- create/execute tracked independently across turns instead of required to
pair from the same turn's tool calls -- a run that calls create in turn 1
and only completes execute in turn 2 was previously scored as if execute
never happened.
- KdaEvaluation.kda_triggered renamed to triggered, matching the other three
core fields (executed/success/turn_completed), none of which carry the
kda_ prefix -- the Langfuse score name kda_triggered is unchanged.
- generate_simulated_kda_response's OpenAI call now has a 30s timeout -- it
previously had none, so a stalled OpenAI request could hang the run
indefinitely.
JIRA: QA-288001 parent a1ccba8 commit 8a7de88
7 files changed
Lines changed: 1526 additions & 7 deletions
File tree
- packages/gooddata-eval
- src/gooddata_eval/core
- agentic
- chat
- tests
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
33 | 41 | | |
34 | 42 | | |
35 | 43 | | |
| |||
56 | 64 | | |
57 | 65 | | |
58 | 66 | | |
| 67 | + | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
69 | 78 | | |
70 | 79 | | |
71 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
72 | 84 | | |
73 | 85 | | |
74 | 86 | | |
| |||
81 | 93 | | |
82 | 94 | | |
83 | 95 | | |
| 96 | + | |
84 | 97 | | |
85 | 98 | | |
86 | 99 | | |
87 | 100 | | |
88 | 101 | | |
89 | 102 | | |
90 | 103 | | |
| 104 | + | |
91 | 105 | | |
92 | 106 | | |
93 | 107 | | |
| |||
0 commit comments