fix(reward): trainer adapters live in openadapt_evals.reward; flow keeps the worker and the client

[abrichr]

What changed

openadapt_flow.reward.callables no longer offers a trainer-facing reward function. trl_reward_function, verl_compute_score, UNSCORED_REWARD, is_unscored, drop_unscored and scored_groups are gone. The module keeps HttpRewardClient (now with an injectable httpx transport for tests), episode_from_columns, scalar_of and the RewardScorer protocol: the pieces a trainer node needs to reach a worker over HTTP.

docs/REWARD_WORKER.md and the README reward section now point at openadapt_evals.reward (pip install 'openadapt-evals>=0.96.0') for the TRL and verl adapters, and say why. The "what runs where" table is unchanged. The wording matches https://docs.openadapt.ai/commercial/seal-reward/.

The reward extra in pyproject.toml listed neither trl nor verl, so nothing changed there.

Why

The flow TRL adapter returned None for an unscored episode and the verl adapter returned NaN. TRL's GRPOTrainer turns a None reward into NaN, combines the per-function rewards with nansum, and takes the group mean over the result. With one reward function that row trains as reward 0.0. The reward contract forbids 0.0 for reconciliation_required and failed_platform: an unscored episode is dropped, never paid 0.

openadapt_evals.reward.trl.CertifiedRewardFunction and openadapt_evals.reward.verl.CertifiedRewardManager (openadapt-evals 0.96.0, openadapt-evals#326) drop an unscored episode the one way a per-completion scalar allows. The episode gets the mean reward of its scored group-mates, so its GRPO advantage is exactly zero and the scored mean is unchanged. Those are the canonical adapters.

The dependency runs one way: openadapt-evals depends on openadapt-flow. flow cannot import the adapters, and a second copy here would drift. This PR adds no dependency on openadapt-evals.

What a reviewer checks

• openadapt_flow/reward/callables.py: no trl_reward_function, no compute_score, no UNSCORED_REWARD. The module docstring names the canonical adapters and states the TRL None to 0.0 fact.
• tests/test_reward_worker.py::test_callables_offer_no_trainer_adapter asserts the absence, with the reason in a comment beside it.
• test_episode_from_columns_matches_the_evals_descriptor_shape and test_http_reward_client_roundtrip cover the pieces that stay. The flow payload carries the evals descriptor keys plus schema_version; the worker accepts both.
• docs/REWARD_WORKER.md "Trainer adapters" section and README lines 469 to 475.

Integration check (report only, nothing committed in evals)

Scratch venv with this branch (.[reward]) and openadapt-evals==0.96.0. Started openadapt-flow serve-reward --seed-mockmed --port 8788 and drove openadapt_evals.reward.receipts.HttpRewardEndpoint against it.

• Gold episode (patient-honest-0001): POST /v1/rewards body {"episode_id", "policy_checkpoint_id", "policy_update", "reward_contract_digest", "metadata": {"oracle_identity": {...}}} with Authorization: Bearer. Worker answered 200 with {"receipt": ...} and no top-level schema_version. parse_receipt accepted it. assess_receipt: verified, scalar 1.0, certified true, scope synthetic, tier 2.
• Unscored episode (same patient, metadata.runtime_signal: failed_platform; the seed has no unscored fixture of its own): 200, failed_platform, scalar None, unscored true. No 0.0 anywhere.
• Envelope shapes agree. No field-name mismatch on either direction.

One gap, not a shape mismatch: CertifiedRewardFunction builds its descriptors with no metadata, so it never sends oracle_identity. Against a worker with no prior RewardWorker.begin_episode registration for the episode, the worker answers 422 identity_missing and the adapter raises RewardEndpointError. The worker has no HTTP route for begin_episode, so today the environment must register the identity in-process on the worker before the rollout. Reported here for routing; not changed in this PR.

Opened by an agent session, not the founder.

🤖 Generated with Claude Code