Skip to content

Commit f6cde10

Browse files
committed
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py to gooddata-eval, evaluating the chatbot's create_key_driver_analysis/execute_key_driver_analysis tool calls against the agent_kda_skill Langfuse dataset. Scope is completion (kda_triggered -> executed -> success -> turn_completed), not per-field correctness -- per-field checks are computed and logged as kda_-prefixed informational scores for a follow-up ticket (QA-28699), and are None (not False) both when expected_output has no key for a field AND when that field's own precondition (kda_triggered, or executed+success for Summary) isn't met. Latency is measured directly by the harness, not re-derived from Langfuse after the fact: ChatResult.turn_wall_clock_sec is set by ChatClient around the whole request (from just before opening the stream, so it includes connection/server setup time the caller actually waits through, excluding only a transient retry's own backoff sleep), stamped on both the successful result and any ChatError.partial_result so a turn that dies mid-stream after KDA already succeeded still gets a number. KdaRunResult.turn_wall_clock_sec captures it in _run_once at the exact point create_args gets set -- atomic with the kda_triggered signal, not a separate step that can fail independently. Logged as the kda_turn_wall_clock_sec Langfuse score; combo_report.py (gdc-nas) reads it directly, with no trace re-querying or observation-shape matching needed on that side anymore. Adds ChatResult.stream_ended (from the SSE response_ended event, derived -- not a raw server field, see sse_client.py's _RESPONSE_ENDED_EVENT comment). turn_completed requires both stream_ended AND a non-empty text_response -- a stream that ends cleanly but delivers nothing to the user isn't a completed turn either, and a turn cut off mid-answer can still emit partial, non-empty text before dying. Fixes from review: - kda_ prefix pass_at_k/pass_power_k Langfuse scores -- unprefixed, "pass_at_2" at k=2 collides with visualization.py's own score name that gdc-nas's combo_report.py.verdict() checks first, silently misfiling every KDA record as visualization once KDA_RUN_K=2 is ever set. - Relative (not absolute) tolerance for Summary's revenue-scale values by default; change is checked against reference_value's scale, not against itself. Also recognizes an explicit absolute_tolerance key -- every real agent_kda_skill dataset item uses it, which this module never read before, silently running ~3000x looser than the dataset author intended. Warns on any other *tolerance* key so a future typo surfaces instead of repeating. - Filters compared as canonicalized sets, not order-sensitive lists. - except Exception (not except ChatError) around send_message -- a stream cut off mid-turn raises a raw httpx transport error, which a narrower catch would let escape uncaught, skipping Langfuse scoring entirely for that run. - ChatError/TransientChatError carry partial_result so tool calls that already succeeded before a later, unrelated stream error aren't discarded and misreported as "the agent never called KDA at all". - turn_completed resets to False on an exception, so a crash on a later disambiguation iteration can't leave a stale True from an earlier iteration. - kda_disambiguated logged and immediately nulls the six *_correct informational fields: the simulated user reply names the acceptable candidate(s) drawn from expected_output itself, so those fields aren't an independent signal once a run went through disambiguation. - KDA-specific _is_asking_clarification instead of a heuristic shared with metric_skill.py/conversation.py, which had silently drifted apart; strips a leading "to clarify, " discourse marker before its substring checks, since that phrase means "in other words" in a final answer, not a request for one. - run_agentic_kda_skill rejects k < 1 -- the single initial run happens unconditionally regardless of k, so a bad env-driven KDA_RUN_K value (0, a typo, negative) previously ran silently once instead of surfacing the bad config, indistinguishable from a deliberate k=1. - Regression tests for find_traces_per_conversation's None-safety and _filters_match's isinstance guard. - _MAX_LATENCY_SEC (this module's own hardcoded 60s, used only for value_score's speed component) now cross-references gdc-nas's independently configured _KDA_MAX_TURN_SEC in a comment -- the two aren't shared code and nothing enforces them staying in sync if one changes without the other. - stream_ended is now set the moment the response_ended event: line itself is parsed, not its data: line -- an event with no data payload at all previously left the flag unset. JIRA: QA-28800
1 parent 676da68 commit f6cde10

7 files changed

Lines changed: 1955 additions & 7 deletions

File tree

packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@
3030
evaluate_agentic_guardrail,
3131
run_agentic_guardrail,
3232
)
33+
from gooddata_eval.core.agentic.kda_skill import (
34+
AgenticKdaSummary,
35+
KdaEvaluation,
36+
KdaRunResult,
37+
KdaSkillAssertionError,
38+
evaluate_agentic_kda_skill,
39+
run_agentic_kda_skill,
40+
)
3341
from gooddata_eval.core.agentic.metric_skill import (
3442
AgenticMetricSummary,
3543
MetricRunResult,
@@ -56,6 +64,7 @@
5664
"AgenticAlertSummary",
5765
"AgenticGeneralQuestionSummary",
5866
"AgenticGuardrailSummary",
67+
"AgenticKdaSummary",
5968
"AgenticMetricSummary",
6069
"AgenticSearchSummary",
6170
"AgenticRunSummary",
@@ -69,6 +78,9 @@
6978
"GeneralQuestionResult",
7079
"GuardrailAssertionError",
7180
"GuardrailResult",
81+
"KdaEvaluation",
82+
"KdaRunResult",
83+
"KdaSkillAssertionError",
7284
"MetricRunResult",
7385
"MetricSkillAssertionError",
7486
"RunResult",
@@ -81,13 +93,15 @@
8193
"evaluate_agentic_conversation",
8294
"evaluate_agentic_general_question",
8395
"evaluate_agentic_guardrail",
96+
"evaluate_agentic_kda_skill",
8497
"evaluate_agentic_metric_skill",
8598
"evaluate_agentic_search_tool",
8699
"evaluate_agentic_visualization",
87100
"run_agentic_alert_skill",
88101
"run_agentic_conversation",
89102
"run_agentic_general_question",
90103
"run_agentic_guardrail",
104+
"run_agentic_kda_skill",
91105
"run_agentic_metric_skill",
92106
"run_agentic_search_tool",
93107
"run_agentic_visualization",

0 commit comments

Comments
 (0)