Skip to content

Commit 4e6872c

Browse files
committed
fix(gooddata-eval): don't feed fallback-trace latency into KDA value_score
_select_kda_trace falls back to the max-latency candidate only so scores still attach to a real trace instead of being orphaned when KDA never triggered -- that fallback trace's latency/cost are not meaningful KDA numbers (previous comment said so but the caller still passed pt.latency into log_quality_and_value_scores regardless, contaminating value_score with an unrelated turn's duration). Gate on ev.kda_triggered (our own tool-call extraction, not Langfuse) before trusting pt.latency/pt.total_cost. When kda_triggered is False, pass None so log_quality_and_value_scores' own worst-case default (speed=0) applies, which is the correct outcome for a case that never triggered KDA at all. Verified offline: a never-triggered case with a mocked fallback trace (latency=999.0, totalCost=5.0) now logs latency_sec=None/cost_usd=None instead of leaking those numbers into value_score. JIRA: QA-28800 risk: nonprod
1 parent 361590a commit 4e6872c

1 file changed

Lines changed: 7 additions & 4 deletions

File tree

  • packages/gooddata-eval/src/gooddata_eval/core/agentic

packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -156,8 +156,9 @@ def _trace_has_kda_call(langfuse: Any, trace_id: str) -> bool:
156156

157157

158158
def _select_kda_trace(langfuse: Any, candidates: list[Any]) -> Any | None:
159-
"""Pick the candidate trace that made the KDA tool call; fall back to max-latency
160-
if none did (KDA never triggered -- nothing meaningful to time anyway)."""
159+
"""Pick the candidate trace that made the KDA tool call. Falls back to max-latency
160+
only so scores still attach to some real trace instead of being orphaned -- callers
161+
must not treat that fallback trace's latency as a real KDA duration."""
161162
for candidate in candidates:
162163
if _trace_has_kda_call(langfuse, candidate.id):
163164
return candidate
@@ -433,12 +434,14 @@ def evaluate_agentic_kda_skill(
433434
with observe(langfuse, pt.id if pt else None, dataset_item_id, run_name, run_metadata) as tid:
434435
for score_name, value in {**strict_checks, **informational_checks}.items():
435436
score_safe(langfuse, tid, name=score_name, value=float(value), data_type="BOOLEAN")
437+
# pt can be a fallback (non-KDA) trace when kda_triggered is False -- its
438+
# latency/cost aren't real KDA numbers, so don't feed them into value_score.
436439
log_quality_and_value_scores(
437440
langfuse,
438441
tid,
439442
strict_checks=strict_checks,
440-
latency_sec=pt.latency if pt else None,
441-
cost_usd=pt.total_cost if pt else None,
443+
latency_sec=pt.latency if pt and ev.kda_triggered else None,
444+
cost_usd=pt.total_cost if pt and ev.kda_triggered else None,
442445
)
443446

444447
if not summary.pass_at_k:

0 commit comments

Comments
 (0)