feat: EvalPort export for meta-benchmark rows (sibling to inspect_export.py)

[adhabnr-ux]

Summary

harness/inspect_export.py already does something worth generalizing: it takes the meta-benchmark's own rows and serializes them into another project's log format, with zero runtime dependency on that project, purely so downstream tooling in that ecosystem can read your results. I think the same move is worth making for EvalPort — a small, Apache-2.0, JSON-only interchange schema for eval data (TestCase / Grader / EvalSuite / Result / ResultSet, spec/SPEC.md). It's not a runner and it doesn't compete with anything here — it's the same "borrow the log format, keep your own runtime" pattern you already used for Inspect, aimed at a wider set of downstream readers (the repo currently ships ~30 adapter packages for tools like DeepEval, Ragas, LangSmith, Braintrust, MLflow, CrewAI, AutoGen and Opik — see adapters/, and e.g. adapters/opik-openeval-adapter for a worked example of one).

Why the fit is unusually direct here

You already have exactly the types this needs, and they don't need to change:

• openadapt_types.BenchmarkTask (task_id, instruction, domain, initial_state_ref, time_limit_steps, raw_config, evaluation_spec) → EvalPort TestCase (id, input, metadata, graders) almost field-for-field.
• openadapt_evals.adapters.base.BenchmarkResult (task_id, success, score, error, reason, error_type, num_steps, total_time_seconds) → EvalPort Result (test_case_id, passed, grader_results, error, duration_ms, metadata).
• openadapt_evals.evaluation.verifier_registry.VerificationResult (success, score, details) → EvalPort GraderResult (score, passed, metadata) — one grader result per env verifier, since Environment.verify() in harness/protocol.py is already the single point every benchmark family (WAA native evaluator, registry verifier, effect verifier) funnels through.

Because your own scoring comes from a real environment verifier rather than an LLM judge or string match, EvalPort's custom/framework-native grader type is the right fit — the spec requires a params.handler string for any non-standard type precisely so a downstream reader can skip gracefully instead of guessing at semantics (see "Type openness" in the spec's Grader section). That maps cleanly onto naming the handler after your own env field ("waa", "mockmed", "parallels", ...).

Sketch

A sibling module to harness/inspect_export.py, same shape, same "no dependency on the target ecosystem" stance (EvalPort's Python SDK, openeval-sdk, only needs to be a dev-dependency if you want validate_suite/validate_result_set in tests — the emitted dicts are plain JSON either way):

# openadapt_evals/harness/evalport_export.py
"""Serialize meta-benchmark tasks/rows to EvalPort (TestCase/EvalSuite, Result/ResultSet).

Mirrors harness/inspect_export.py: this borrows EvalPort's document shape so any
of its ~30 downstream adapters can read a meta-benchmark run, without adopting
EvalPort as a runtime dependency here. No new types -- BenchmarkTask and
BenchmarkResult are the source of truth; this only reshapes them.
"""
from __future__ import annotations

from datetime import datetime, timezone
from typing import Any

from openadapt_types import BenchmarkTask
from openadapt_evals.adapters.base import BenchmarkResult

_SPEC_VERSION = "1.0.0"


def task_to_test_case(task: BenchmarkTask, *, env: str) -> dict[str, Any]:
    grader_id = f"gr_{env}_verify"
    return {
        "id": task.task_id,
        "input": task.instruction,
        "graders": [grader_id],
        "metadata": {
            "domain": task.domain,
            "initial_state_ref": task.initial_state_ref,
            "time_limit_steps": task.time_limit_steps,
            "openadapt_evals": {"env": env, "raw_config": task.raw_config},
        },
    }


def tasks_to_suite(
    tasks: list[BenchmarkTask], *, suite_id: str, env: str, name: str | None = None,
) -> dict[str, Any]:
    grader_id = f"gr_{env}_verify"
    return {
        "version": _SPEC_VERSION,
        "id": suite_id,
        "name": name or f"openadapt-evals: {env}",
        "metadata": {"openeval.profile": "agent"},
        "graders": [{
            "id": grader_id,
            # Framework-native type, not a bare "custom" wrapper -- spec requires
            # params.handler for any non-standard type so an unaware reader can
            # skip it instead of mis-scoring it.
            "type": f"{env}_native",
            "params": {"handler": "openadapt_evals.harness.protocol.Environment.verify"},
        }],
        "test_cases": [task_to_test_case(t, env=env) for t in tasks],
    }


def result_to_evalport(result: BenchmarkResult, *, env: str) -> dict[str, Any]:
    grader_id = f"gr_{env}_verify"
    return {
        "test_case_id": result.task_id,
        "passed": result.success,
        "grader_results": [{
            "grader_id": grader_id,
            "type": f"{env}_native",
            "score": result.score,
            "passed": result.success,
            "reason": result.reason,
        }],
        "duration_ms": int(result.total_time_seconds * 1000),
        "metadata": {"openadapt_evals": {"num_steps": result.num_steps, "error_type": result.error_type}},
        **({"error": {"message": result.error, "type": result.error_type or "runner_error"}} if result.error else {}),
    }


def results_to_result_set(
    results: list[BenchmarkResult], *, suite_id: str, run_id: str, env: str,
    started_at: str | None = None,
) -> dict[str, Any]:
    started_at = started_at or datetime.now(timezone.utc).isoformat()
    rows = [result_to_evalport(r, env=env) for r in results]
    total = len(rows)
    passed = sum(1 for r in rows if r["passed"])
    return {
        "version": _SPEC_VERSION,
        "suite_id": suite_id,
        "run_id": run_id,
        "started_at": started_at,
        "completed_at": datetime.now(timezone.utc).isoformat(),
        "runner": {"name": "openadapt-evals", "version": env},
        "results": rows,
        "summary": {
            "total": total, "passed": passed, "failed": total - passed, "skipped": 0,
            "pass_rate": (passed / total) if total else 0.0,
        },
    }

evaluate_agent_on_benchmark's existing list[BenchmarkResult] output feeds results_to_result_set directly, same as it already feeds compute_metrics. Nothing about the harness, the Environment protocol, or MetaMetricsRow changes.

What this buys

• Anyone already on DeepEval/Ragas/LangSmith/Braintrust/MLflow/CrewAI/AutoGen/Opik gets a path to load WAA/mock/local results into their existing dashboards, without you writing per-tool exporters — one JSON shape, ~30 readers.
• openeval.validate.validate_suite() / validate_result_set() (from the reference SDK) give you a free schema-conformance check in CI if you want it — optional, dev-only.
• It composes with what's already here rather than replacing it: inspect_export.py stays as-is: a reader could go WAA run → EvalPort ResultSet → any EvalPort-aware tool, entirely separately from the existing WAA → Inspect path.

I'd be glad to turn the sketch above into an actual PR (tests included) against main if this is something you'd take — I know from the README that's branches + conventional-commit PR titles only. Happy to adjust the mapping (e.g. whether MetaMetricsRow should be the export source instead of raw BenchmarkResult, or how structural_rung_rate/cost_usd should land in Result.metadata) based on what's actually useful to you; this is a starting proposal, not a finished design.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Precedent adapter (same "standalone, zero-core-changes" shape): https://github.com/adhabnr-ux/evalport/tree/main/adapters/opik-openeval-adapter

— Sahi, independent contributor (not affiliated with this project)