Rubric judge未获得被评 agent 的执行轨迹——与论文不一致,且「是否使用了某文件」类 rubric 恒被判通过

[ligouhei]

您好，感谢发布 Workspace-Bench。

我们最近在使用 Workspace-Bench 评测多个 Agent 系统。整体上非常认可这个 benchmark 的方向，尤其是大规模 noisy workspace、cross-file dependency reasoning 和 workspace-level grounding。这些设计比传统“小文件包 QA benchmark”更接近真实 Agent 场景。

但是我发现论文明确 Agent-as-a-Judge 评 rubric 时会把被评 agent 的执行轨迹一并给裁判;当前代码(evaluation/src/agent_as_a_judge.py)没给，裁判只能看原始输入 + 候选输出。后果:(1) rubric 通过率无法与论文对标；(2)「agent 是否使用/整合/追溯了某输入文件」这类过程型 rubric 被恒判通过（因为input本质上是模型应该用到哪些文件的gt），抬高分数。

论文规定 — 裁判应拿到 4 样:输出文件 + 输入文件 + rubrics + 执行轨迹。

代码现状 — _build_judge_prompt 的 payload = task/steps/rubrics/data/judgeView{cwd, inputsPath, originalTaskMetadataPath, candidateOutputPath};_prepare_judge_view 只暴露 inputs/ + candidate_output/。无 executionTrace。 max_trace_items 对 rubric 评分是无效参数(轨迹只在 dep_graph_recognition._trace_summary 依赖图那步用)。

影响 — (1) 与论文口径不一致;(2) 只有 inputs/ 可见、目标文件本就在其中 → 「是否用了 X」几乎恒判通过。

[wu-uk]

感谢你指出这个问题。

你指出的情况是正确的：公开版本的代码存在疏漏，在构造 agent-as-a-judge 的输入时，没有将 agent 执行轨迹的 snapshot 提供给评测 agent，导致公开代码与实际用于 leaderboard 评测的版本不一致。这是我们的失误，对此我们表示抱歉。

需要说明的是，leaderboard 上的结果在实际评测时确实使用了 agent 执行轨迹的 snapshot 作为评测依据。我们目前已经更新了公开代码，使其与 leaderboard 评测流程保持一致：评测 agent 会读取执行轨迹的 snapshot，而不是直接读取原始 trace。

再次感谢你发现并报告这个问题。这帮助我们发现并修复了公开代码与实际评测流程之间的不一致。

Thank you for pointing this out.

You are right that there was an oversight in the public version of the code. When constructing the input for agent-as-a-judge, the public implementation did not provide the agent execution-trace snapshot to the evaluating agent. As a result, the public code was inconsistent with the version used for the leaderboard evaluation. This was our mistake, and we apologize for the confusion.

To clarify, the leaderboard results were generated using evaluation runs in which the agent execution-trace snapshot was provided as evaluation evidence. We have now updated the public code to match that evaluation pipeline: the evaluating agent reads the execution-trace snapshot rather than accessing the raw trace directly.

Thank you again for identifying and reporting this issue. It helped us find and fix the inconsistency between the public implementation and the leaderboard evaluation process.

[ligouhei]

感谢你指出这个问题。

你指出的情况是正确的：公开版本的代码存在疏漏，在构造 agent-as-a-judge 的输入时，没有将 agent 执行轨迹的 snapshot 提供给评测 agent，导致公开代码与实际用于 leaderboard 评测的版本不一致。这是我们的失误，对此我们表示抱歉。

需要说明的是，leaderboard 上的结果在实际评测时确实使用了 agent 执行轨迹的 snapshot 作为评测依据。我们目前已经更新了公开代码，使其与 leaderboard 评测流程保持一致：评测 agent 会读取执行轨迹的 snapshot，而不是直接读取原始 trace。

再次感谢你发现并报告这个问题。这帮助我们发现并修复了公开代码与实际评测流程之间的不一致。

Thank you for pointing this out.

You are right that there was an oversight in the public version of the code. When constructing the input for agent-as-a-judge, the public implementation did not provide the agent execution-trace snapshot to the evaluating agent. As a result, the public code was inconsistent with the version used for the leaderboard evaluation. This was our mistake, and we apologize for the confusion.

To clarify, the leaderboard results were generated using evaluation runs in which the agent execution-trace snapshot was provided as evaluation evidence. We have now updated the public code to match that evaluation pipeline: the evaluating agent reads the execution-trace snapshot rather than accessing the raw trace directly.

Thank you again for identifying and reporting this issue. It helped us find and fix the inconsistency between the public implementation and the leaderboard evaluation process.

感谢回复！还有个小问题是关于引入的/input来源是tasks下的data文件夹，这应该是模型需要用到哪些文件的gt，直接丢给judge模型判断有没有使用某个文件的的时候好像都会被评估为使用了，因为模型会误以为input就是模型实际用了的文件。不过增加轨迹的snapshot应该会改善这一点，还没做过消融，请问考虑引入/input是怎么样的考量呀

[wu-uk]

感谢你的补充问题。引入 /input 的初衷是：仅依据最终输出进行评测，在部分任务中可能不足以完整反映任务要求及其上下文；而输入文件包含了任务所依赖的原始材料，能够帮助 judge 更准确地理解应满足的需求，并据此判断输出是否完成了任务。

不过，你指出的风险确实存在：/input 应当被视为任务上下文和需求依据，而不应被视为 agent 实际使用过某个文件的证据。对于“是否使用、整合或追溯了某个文件”这类过程性 rubric，判断应主要依据执行轨迹的 snapshot，而不是仅根据该文件是否出现在 /input 中。否则，确实可能造成 judge 将“任务提供了该文件”误判为“agent 使用了该文件”。

我们加入执行轨迹 snapshot 也是希望将这两类信息区分开：/input 用于理解任务需求，轨迹用于判断 agent 的实际操作过程。我们也会进一步关注 rubric 表述和 judge 提示中是否需要更明确地区分这两种证据，避免产生类似的歧义。感谢你的细致思考和建议。

Thank you for the follow-up question. The motivation for including /input was that evaluating only the final output may not fully capture the task requirements and context in some cases. The input files contain the source materials on which the task depends, helping the judge understand the expected requirements more accurately and assess whether the output fulfills the task.

That said, the concern you raised is valid: /input should be treated as task context and evidence of the requirements, rather than evidence that the agent actually used a particular file. For process-oriented rubrics—such as whether the agent used, integrated, or traced information from a file—the judgment should rely primarily on the execution-trace snapshot, not merely on the file’s presence in /input. Otherwise, the judge may indeed confuse “the task provided this file” with “the agent used this file.”

Adding the execution-trace snapshot is intended to separate these two kinds of evidence: /input helps the judge understand the task requirements, while the trace is used to assess the agent’s actual process. We will also continue to examine whether the rubric wording and judge prompt should make this distinction more explicit, to avoid this kind of ambiguity. Thank you again for the careful analysis and suggestion.