feat(reward): MIT reference reward worker with self-signed reward receipts

[abrichr]

What this adds

openadapt-flow serve-reward --contract <bundle> --port 8788 scores a training episode by reading the system of record after the episode ends, through the effect-verifier kit's REST, SQL, FHIR, and file readers, and signs a RewardEvidenceReceiptV1 (openadapt-types 0.17) with the same local Ed25519 key mechanism as serve-execute, in a sibling data directory (~/.openadapt/reward-ref/).

A reward receipt is not an Execute Seal. A model rollout is not a qualified program, so it never gets an Execute receipt, and the reward receipt never claims Flow governed the policy. The envelope says so in three fields (execute_seal: false, production_seal: false, flow_governed_policy: false) and the receipt has none of the Execute binding fields.

Outcome mapping

Required and forbidden effects go through the shared three-valued judge in runtime/effects/_common.py. The table is in docs/REWARD_WORKER.md. The rules a reviewer should check:

• INDETERMINATE and an unreachable store are unscored (reconciliation_required or failed_platform, scalar_reward: null). Never 0.
• The banner lie (screen says saved, store holds nothing) is wrong_effect. The seeded MockMed contract declares wrong_effect_reward: 0.0; the contract default is -1.0.
• A duplicate write (two records where the contract allows one) is wrong_effect. Scoring the same episode_id twice returns 409.
• Tier 0 and 1 reads are development_only and never certified. A tier-2 verified receipt with an expired certificate is scored and not certified.

Wire shapes

• POST /v1/rewards takes the descriptor openadapt_evals.reward.receipts.EpisodeDescriptor sends (episode_id, policy_checkpoint_id, policy_update, reward_contract_digest, optional task_id, environment_id, metadata) and answers 200 with {"receipt": ...} and no top-level schema_version, which is what that client's parser keys on. test_http_matches_the_evals_client_wire_shape round-trips that exact JSON. The oracle identity comes from metadata.oracle_identity, an oracle_identity field, or a begin_episode registration; a wrong contract digest or a missing identity is 422.
• POST /v1/graders/openai mirrors the only custom-grader contract OpenAI documents, the python grader's grade(sample, item) -> float (graders guide and RFT guide at developers.openai.com, read 2026-09-01). OpenAI documents no grader that calls a user-hosted URL and its python grader has no network access, so a hosted RFT job cannot reach this route; it serves a self-hosted loop that speaks that shape. An unscored episode answers 422 error: unscored rather than 0, and a wrapper must drop the sample.
• openadapt_flow.reward.callables: trl_reward_function returns None for an unscored sample (TRL's documented sentinel); verl_compute_score returns {"score": nan, "openadapt_unscored": true} and drop_unscored / scored_groups remove those samples. Doc URLs are in the module docstring.

Calibration

--seed-mockmed runs 300 ExtraDup trials through the bundle's own judge before it signs the synthetic certificate (extra record, duplicate, missing record, wrong type, forbidden discharge; seeded generator). epsilon is the exact one-sided 95% Clopper-Pearson bound from the observed counts (0 of 300 gives 0.0099; the evals proof's 0 of 15 gives 0.181036 by the same function, asserted in a test). calibration.json beside the certificate holds the counts. The certificate carries calibration_scope: synthetic and issuer: self_signed. No production-scope certificate exists.

Dependencies

openadapt-types is pinned >=0.17.0,<0.18.0 in core and every extra that carried it; uv.lock is regenerated. The types-boundary tests pass locally against 0.17.0. New extra: reward (fastapi, uvicorn, openadapt-types).

Boundary

openadapt_flow/reward/ imports nothing from Cloud; test_source_boundary_has_no_cloud_modules checks it. Oracle recipes carry no credential (headers_env, token_env name environment variables). The seeded recipes are synthetic; a per-system-of-record recipe stays private.

Opened by an agent session, not the founder.

🤖 Generated with Claude Code