Wildbench Integration by kargibora · Pull Request #72 · OpenEuroLLM/JudgeArena

kargibora · 2026-07-15T09:03:13Z

Summary

This PR closes #68.

wildbench-score: checklist-based individual scoring.
wildbench-reward: pairwise evaluation using one custom baseline or the three
official reference models.

Implementation

Uses the official WildBench V2 dataset, prompts, JSON formats, and metrics.
Supports multi-turn conversations and instance-specific checklists.
Downloads the official baseline outputs for WB-Reward.
Reports overall, per-category, and per-baseline results.
Saves annotations, model outputs, resolved configuration, and run metadata.

Example

export OPENROUTER_API_KEY="..."

uv run judgearena \
  --task wildbench-score \
  --model.name OpenRouter/google/gemma-3-4b-it \
  --judge.model OpenRouter/openai/gpt-4.1-mini \
  --generation.n_instructions 10

Example YAML configurations are included under configs/.

## Validation

- Added dataset, metric, parser, configuration, CLI, and artifact tests.
- Verified loading all 1,024 WildBench V2 tasks and official baseline outputs.

kargibora · 2026-07-15T09:07:18Z

Notes:

Most of the LOC comes from the wildbench.py which is basically provided by the wildbench repo and adapted to our codebase. It uses styles for extending the dict/tuple objects to multiline which in-fact creates more lines. So content-wise PR is not large
Original wildbench does not have system prompt, should we not use You are a helpful assistant.. for models we test?

kargibora · 2026-07-18T11:25:03Z

Although this PR adds Wildbench, it was hard to ingerate it. Thus I have proposed more cleaner solution in #74. We should discuss this possibly toward making it file-based (task=yaml file)

ErlisLushtaku · 2026-07-18T10:33:19Z

+                self.wildbench = WildBenchArgs()
+            if self.model.name is None:
+                raise ValueError("model.name is required for WildBench tasks.")
+            if self.judge.temperature is None:


WildBench’s judge temperature is defaulted to 0 here, but the evaluated model keeps backend defaults. For vLLM that currently means temperature 0.6/top-p 0.95 and the shared 32k output budget, while the official WildBench launch uses temperature 0/top-p 1/max-tokens 4096. As a result the example configs are stochastic and not leaderboard-comparable. Could we set or require explicit WildBench generation defaults and maybe record deviations?

I agree, we should add this to the config file. Thanks for catching it

ErlisLushtaku · 2026-07-18T10:36:33Z

+                    "baseline_output": baseline,
+                    "prompt": prompt,
+                }
+                if not candidate.strip() and not baseline.strip():


The synthetic empty-response choices match the official evaluator, but the official leaderboard does not feed them into metrics this way. WB-Score keeps score=1 in the overall mean but excludes empty rows from category/task-macro values. WB-Reward excludes empties from win/loss counts and categories; because its overall denominator remains the full result count, an empty row contributes 0 rather than ±100.

I will add a filtering as the last step, thanks

ErlisLushtaku · 2026-07-18T10:38:43Z

+        normalized = value.strip().upper().replace(" ", "")
+        if normalized in _CHOICE_TO_A_REWARD:
+            return normalized
+    matches = re.findall(r"A\+\+|B\+\+|A=B|A\+|B\+", strip_thinking_tags(text))


This fallback scans the entire response for verdict tokens. The required JSON schema contains a "reason of A=B" key, so a truncated or malformed response without choice is silently parsed as A=B instead of missing. It would be better to restrict fallback parsing to the choice field or a tightly scoped final-choice pattern.

ErlisLushtaku · 2026-07-18T19:49:30Z

+    pending_ids = []
+    records = []
+    for session_id, example in examples.iterrows():
+        output = candidate_outputs.loc[str(session_id)]


output is passed directly to the score prompt, unlike the reward path. Consequently, cfg.judge.strip_thinking_before_judging has no effect for WB-Score. I think we should apply strip_thinking_tags(output) here.

geoalgo

LGTM overall we can merge once you have addressed @ErlisLushtaku comments.

geoalgo · 2026-07-20T15:14:43Z

@@ -0,0 +1,14 @@
+task: wildbench-reward
+model:
+  name: VLLM/utter-project/EuroLLM-9B


perhaps we can comment this from user POV?

Suggested change

name: VLLM/utter-project/EuroLLM-9B

name: VLLM/utter-project/EuroLLM-9B # model to be evaluated

geoalgo · 2026-07-20T15:15:12Z

+  # Set an integer K to tie slight wins caused by responses longer by >K chars.
+  # Omit this field to disable the length penalty.


Is this the default in the paper? If so:

Suggested change

# Set an integer K to tie slight wins caused by responses longer by >K chars.

# Omit this field to disable the length penalty.

# Set an integer K to tie slight wins caused by responses longer by >K chars.

# Omit this field to disable the length penalty. Default to paper value.

geoalgo · 2026-07-20T15:16:30Z

+model:
+  name: VLLM/utter-project/EuroLLM-9B
+judge:
+  model: OpenRouter/openai/gpt-4o


question: should we rather use gemma4-31B for such defaults?

I see the point to default to the paper but this model is going to be replaced and is pareto dominated by gemma4.

geoalgo · 2026-07-20T15:18:42Z


 def load_instructions(dataset: str, n_instructions: int | None = None) -> pd.DataFrame:
-    if dataset == "mt-bench":
+    if dataset in {"wildbench-score", "wildbench-reward"}:


FYI, I was wondering about this (lists were more natural to me, this choice is PEP compliant):

# are there in coding guideline for membership tests in python? Yes. For membership tests against a literal collection, PEP 8 and common style guidance favor a set (or frozenset) when checking in, because set lookups are O(1) versus O(n) for lists. So {"wildbench-score", "wildbench-reward"} is generally preferred. However, with only two string elements the performance difference is negligible, and some style guides (or linters like Ruff's C0201/SIM rules) actually flag the opposite direction depending on configuration. Ruff's literal-membership rule (PLR6201) specifically recommends converting list/tuple literals to sets in membership tests, which endorses the first form. Practical guidance: use a set literal for membership checks. Use a tuple ("a", "b") if you want to signal immutability without the set overhead and order matters, or a list only if the collection is used elsewhere as an actual list. For your case, the set version is the idiomatic choice.

geoalgo · 2026-07-20T15:22:41Z

 ## ✨ Key Features

-🎯 **Flexible Benchmarking** – Evaluate models on `Alpaca-Eval`, `Arena-Hard`, `m-Arena-Hard` and others
+🎯 **Flexible Benchmarking** – Evaluate models on `Alpaca-Eval`, `Arena-Hard`, `m-Arena-Hard`, `WildBench V2` and others


Perhaps we add a column for wildbench in the table l18?

kargibora added 2 commits July 15, 2026 10:42

Add WildBench V2 score and reward evaluation

9ac2dbb

Streamline WildBench README coverage

2407f29

fix: ruff check

0190d20

kargibora mentioned this pull request Jul 17, 2026

Reorganize the package around independent benchmark integrations #74

Open

kargibora requested review from ErlisLushtaku and geoalgo July 18, 2026 11:24

ErlisLushtaku reviewed Jul 18, 2026

View reviewed changes

geoalgo approved these changes Jul 20, 2026

View reviewed changes

	name: VLLM/utter-project/EuroLLM-9B
	name: VLLM/utter-project/EuroLLM-9B # model to be evaluated

		# Set an integer K to tie slight wins caused by responses longer by >K chars.
		# Omit this field to disable the length penalty.

Uh oh!

Conversation

kargibora commented Jul 15, 2026

Summary

Implementation

Example

Uh oh!

kargibora commented Jul 15, 2026

Uh oh!

kargibora commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geoalgo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kargibora commented Jul 18, 2026 •

edited

Loading