feat(examples): add evaluation + optimization closed-loop pipeline - #285
feat(examples): add evaluation + optimization closed-loop pipeline#285coder-mtj wants to merge 2 commits into
Conversation
Reproducible Evaluation + Optimization closed loop for issue trpc-group#91 (examples/optimization/eval_optimize_loop/): baseline eval -> failure attribution -> prompt optimization -> validation regression -> multi-dim gate -> audited report. fake / trace / live modes, 419 tests.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #285 +/- ##
==========================================
Coverage ? 88.44269%
==========================================
Files ? 491
Lines ? 46118
Branches ? 0
==========================================
Hits ? 40788
Misses ? 5330
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
AI Code Review所有发现均已验证。现在让我确认一下 让我整理一下审查结果。关键/最重要的问题是 audit errors 的覆盖问题。其余均为警告/建议级别。 发现的问题🚨 Critical
|
| "improvement_note": improvement_note, | ||
| "baseline_train_pass_rate": baseline_train.pass_rate, | ||
| "candidate_train_pass_rate": candidate_train.pass_rate, | ||
| "errors": errors, |
There was a problem hiding this comment.
审计 errors 字段被覆盖，丢失已记录错误
tracer.to_dict() 已把 tracer 累积的错误（如 holdout 评分失败经 tracer.add_error 记录）写入 audit_dict["errors"]，随后 audit_dict.update({"errors": errors}) 用本地 errors 列表覆盖，导致仅通过 tracer 记录的错误从最终报告中静默丢失。建议改为合并 audit_dict["errors"] = (audit_dict.get("errors") or []) + errors 或统一以 tracer 为单一来源。
…meric match, reuse find_repo_root in tests Address review 1 Critical + 3 Warnings (helloopenworld @ trpc-group#285 5a26df4): - Critical: audit_dict.update overwrote tracer-accumulated errors (holdout etc.) with local optimization/scenario errors, silently dropping audit trail; now merged. - W1: fake-mode score now based on cumulative fixed cases (monotonic), and best_so_far is the running max, not the current round score. - W2: multi-numeric expectation now matches one-to-one (paired actual number is consumed), so '20 and 20' no longer passes against a single '20'. - W3: tests reuse find_repo_root instead of hardcoding 4-level pardir. +3 tests (422 total green).
AI Code ReviewREADME 中写道 "└── sample_output/ # 示例报告输出" —— 暗示报告文件应该被提交。但根目录的 我现在已经有了结论。代码质量很高。让我来完成审查总结。 基于我的深入分析，代码经过了高度加固，并配有完善的测试。我确认了几个值得注意的小项。让我把它们写下来。 发现的问题
|
✅ CI 全绿 — head
|
2 participants
Summary | 概述
This PR adds a reproducible Evaluation + Optimization closed-loop pipeline under
examples/optimization/eval_optimize_loop/(Tencent Rhinoceros Bird issue #91).The pipeline automates the full loop for prompt evaluation and optimization: baseline evaluation → failure attribution → prompt optimization → validation regression (overfit detection) → multi-dimensional gate → audited report, in three modes (
fake/trace/live).交付状态
python -m pytest tests/ -q）bae2544）方案设计（300–500 字）
optimization_report.{json,md}。使用方式
详见
README.md/DESIGN.md。