Skip to content

Commit 8a7de88

Browse files
committed
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py to gooddata-eval, evaluating the chatbot's create_key_driver_analysis/execute_key_driver_analysis tool calls against the agent_kda_skill Langfuse dataset. Scope is strictly completion: strict_pass = triggered AND executed AND success AND turn_completed Per-field correctness (Measure/Date Attribute/Periods/Filters/Summary matching expected values) is intentionally out of scope for this PR -- deferred entirely to a follow-up ticket (QA-28699) rather than half-computed here as unused informational scores. Latency is measured directly by the harness, not re-derived from Langfuse after the fact: ChatResult.turn_wall_clock_sec is set by ChatClient before opening the SSE stream (so it includes connection/server setup time the caller actually waits through, excluding only a transient retry's own backoff sleep), stamped on both the successful result and any ChatError.partial_result so a turn that dies mid-stream after KDA already succeeded still gets a number. Only the turn that actually completes KDA (both create AND execute) counts toward KdaRunResult.turn_wall_clock_sec -- not any earlier disambiguation/continuation turn, and not the simulated- reply generation call itself (a test-harness OpenAI call, not gen-ai's own time). Logged as the kda_turn_wall_clock_sec Langfuse score; combo_report.py (gdc-nas) reads it directly, with no trace re-querying needed on that side. create_key_driver_analysis and execute_key_driver_analysis are tracked independently across turns, not required to land in the same turn's tool calls -- if the agent calls create but not execute within one turn, a plain continuation nudge ("Please proceed.") gives it another turn rather than the run being scored as if execute never happened at all. Adds ChatResult.stream_ended (from the SSE response_ended event). turn_completed requires both stream_ended AND a non-empty text_response -- a stream that ends cleanly but delivers nothing to the user isn't a completed turn either, and a turn cut off mid-answer can still emit partial, non-empty text before dying. If the agent asks a clarifying question instead of triggering KDA directly (a title collision, or a metric-vs-fact form choice), a simulated user reply (gpt-4o-mini, 30s timeout) nudges it forward, bounded to max_iterations. Fixes from review: - kda_ prefix pass_at_k/pass_power_k Langfuse scores -- unprefixed, "pass_at_2" at k=2 collides with visualization.py's own score name that gdc-nas's combo_report.py.verdict() checks first, silently misfiling every KDA record as visualization once KDA_RUN_K=2 is ever set. - except Exception (not except ChatError) around send_message -- a stream cut off mid-turn raises a raw httpx transport error, which a narrower catch would let escape uncaught, skipping Langfuse scoring entirely for that run. - ChatError/TransientChatError carry partial_result so tool calls that already succeeded before a later, unrelated stream error aren't discarded and misreported as "the agent never called KDA at all". - turn_completed resets to False on an exception, so a crash on a later iteration can't leave a stale True from an earlier one. - KDA-specific _is_asking_kda_clarification instead of a heuristic shared with metric_skill.py/conversation.py, which had silently drifted apart; strips a leading "to clarify, " discourse marker before its substring checks, since that phrase means "in other words" in a final answer, not a request for one. - run_agentic_kda_skill rejects k < 1 -- a bad env-driven KDA_RUN_K value (0, a typo, negative) previously ran silently once instead of surfacing the bad config, indistinguishable from a deliberate k=1. - stream_ended is now set the moment the response_ended event: line itself is parsed, not its data: line -- an event with no data payload at all previously left the flag unset. - t0 is set before opening the SSE stream, not after -- it was missing the request/connection/server-setup time a caller actually waits through. - Disambiguated-turn latency no longer double-counts: it used to sum every turn's time plus the simulated-reply OpenAI call itself, inflating exactly the cases that needed disambiguation (usually the harder ones) with time that has nothing to do with gen-ai or KDA. - The 6 per-field informational correctness scores (measure/date_attribute/ analyzed_period/reference_period/filters/summary_correct) and their support code (_measure_matches, _filters_match, _within_relative_tolerance, _within_absolute_tolerance, _resolve_summary_tolerance, _normalize_measure, _to_number) are removed -- team confirmed this PR's scope is trigger+ complete only; correctness is QA-28699's job, not a half-implemented, unread-by-anything set of scores here. (One of these, _filters_match(actual, None), silently mismatched an explicit "expect no filters" case -- moot now that the function is gone.) - create/execute tracked independently across turns instead of required to pair from the same turn's tool calls -- a run that calls create in turn 1 and only completes execute in turn 2 was previously scored as if execute never happened. - KdaEvaluation.kda_triggered renamed to triggered, matching the other three core fields (executed/success/turn_completed), none of which carry the kda_ prefix -- the Langfuse score name kda_triggered is unchanged. - generate_simulated_kda_response's OpenAI call now has a 30s timeout -- it previously had none, so a stalled OpenAI request could hang the run indefinitely. JIRA: QA-28800
1 parent a1ccba8 commit 8a7de88

7 files changed

Lines changed: 1526 additions & 7 deletions

File tree

packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@
3030
evaluate_agentic_guardrail,
3131
run_agentic_guardrail,
3232
)
33+
from gooddata_eval.core.agentic.kda_skill import (
34+
AgenticKdaSummary,
35+
KdaEvaluation,
36+
KdaRunResult,
37+
KdaSkillAssertionError,
38+
evaluate_agentic_kda_skill,
39+
run_agentic_kda_skill,
40+
)
3341
from gooddata_eval.core.agentic.metric_skill import (
3442
AgenticMetricSummary,
3543
MetricRunResult,
@@ -56,6 +64,7 @@
5664
"AgenticAlertSummary",
5765
"AgenticGeneralQuestionSummary",
5866
"AgenticGuardrailSummary",
67+
"AgenticKdaSummary",
5968
"AgenticMetricSummary",
6069
"AgenticSearchSummary",
6170
"AgenticRunSummary",
@@ -69,6 +78,9 @@
6978
"GeneralQuestionResult",
7079
"GuardrailAssertionError",
7180
"GuardrailResult",
81+
"KdaEvaluation",
82+
"KdaRunResult",
83+
"KdaSkillAssertionError",
7284
"MetricRunResult",
7385
"MetricSkillAssertionError",
7486
"RunResult",
@@ -81,13 +93,15 @@
8193
"evaluate_agentic_conversation",
8294
"evaluate_agentic_general_question",
8395
"evaluate_agentic_guardrail",
96+
"evaluate_agentic_kda_skill",
8497
"evaluate_agentic_metric_skill",
8598
"evaluate_agentic_search_tool",
8699
"evaluate_agentic_visualization",
87100
"run_agentic_alert_skill",
88101
"run_agentic_conversation",
89102
"run_agentic_general_question",
90103
"run_agentic_guardrail",
104+
"run_agentic_kda_skill",
91105
"run_agentic_metric_skill",
92106
"run_agentic_search_tool",
93107
"run_agentic_visualization",

0 commit comments

Comments
 (0)