Skip to content

Spring futureeval 2026 updates - #30

Open
CodexVeritas wants to merge 10 commits into
mainfrom
spring-futureeval-2026
Open

Spring futureeval 2026 updates#30
CodexVeritas wants to merge 10 commits into
mainfrom
spring-futureeval-2026

Conversation

@CodexVeritas

@CodexVeritas CodexVeritas commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Align spot peer/baseline scoring with Metaculus: N/(N−1) peer factor, include self in the geometric mean, log-scale via zero_point, discrete question support, and peer-baseline filters for non-primary bots / exclude_from_aggregations.
  • Fix spot forecast selection to match Metaculus (get_spot_scoring_time chain including open_time, and active window start ≤ spot < end).
  • Update dataclip/export SQL for Spring 2026 fields; filter pro exports with is_bot = false and bot exports with is_bot = true.
  • Point run_q1_q2_simulations.py at Spring 2026 FutureEval data and keep the cross-tournament control-bot alias list (skip comparison-team outputs if the pair isn’t present).

Validation

  • Bot top-30 simulated scores match the Metaculus DB leaderboard exactly.
  • Pro scores match within ~0.01.
  • Existing unit/integration tests pass (85 passed, 4 skipped).

@CodexVeritas CodexVeritas changed the title Spring futureeval 2026 Spring futureeval 2026 updates Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant