Skip to content

feat(reward): add identity-swap family to the MockMed ExtraDup proof #335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abrichr merged 3 commits into from
Sep 2, 2026
Merged

feat(reward): add identity-swap family to the MockMed ExtraDup proof #335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ad80eac
feat(reward): add identity-swap family to the MockMed ExtraDup proof
abrichr Sep 2, 2026
95f40ff
fix: amend M-freeze hashes after identity-swap proof
abrichr Sep 2, 2026
a122373
fix: keep ExtraDup kill-scan on the frozen operator set
abrichr Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 144 additions & 27 deletions docs/preregistrations/M_FREEZE_CERTIFIED_REWARD_RL_PILOT_2026_09_02.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@
"scope": {
"phase": "Phase-1/pilot",
"kill_criterion": 3,
"environments": ["mockmed"],
"environments": [
"mockmed"
],
"openemr_in_this_pilot": false,
"calibration_scope": "synthetic",
"production_acceptance": false,
Expand Down Expand Up @@ -80,12 +82,22 @@
},
"seed_schedule": {
"K": 3,
"seeds": [101, 202, 303],
"seeds": [
101,
202,
303
],
"source": "openadapt_evals.reward.proof.DEFAULT_SEEDS",
"rule": "No best-seed selection. Every arm times every seed reports. An abort or refusal records a reason and still occupies its cell."
},
"extradup_mutants": {
"operators": ["dup", "extra", "omit", "unsubmit", "claim"],
"operators": [
"dup",
"extra",
"omit",
"unsubmit",
"claim"
],
"control": "control",
"source": "openadapt_evals.extradup.mutations.MUTANTS",
"in_training_dataset": false,
Expand All @@ -104,30 +116,121 @@
"train_num_tasks": 8,
"holdout_generation_seed": 20260825,
"holdout_num_tasks": 8,
"train_patient_ids": ["p1", "p2", "p3", "p4", "p5", "p6", "p7", "p8"],
"holdout_patient_ids": ["h1", "h2", "h3", "h4", "h5", "h6", "h7", "h8"],
"train_patient_ids": [
"p1",
"p2",
"p3",
"p4",
"p5",
"p6",
"p7",
"p8"
],
"holdout_patient_ids": [
"h1",
"h2",
"h3",
"h4",
"h5",
"h6",
"h7",
"h8"
],
"train_gold": [
{"patient_id": "p1", "type": "Triage", "note": "Follow-up in 2 weeks; BP recheck."},
{"patient_id": "p2", "type": "Intake", "note": "Post-op check; wound clean."},
{"patient_id": "p3", "type": "Triage", "note": "Fasting glucose ordered."},
{"patient_id": "p4", "type": "Intake", "note": "Post-op check; wound clean."},
{"patient_id": "p5", "type": "Consult", "note": "Post-op check; wound clean."},
{"patient_id": "p6", "type": "Consult", "note": "Vaccination due; schedule next visit."},
{"patient_id": "p7", "type": "Follow-up", "note": "Vaccination due; schedule next visit."},
{"patient_id": "p8", "type": "Follow-up", "note": "Fasting glucose ordered."}
{
"patient_id": "p1",
"type": "Triage",
"note": "Follow-up in 2 weeks; BP recheck."
},
{
"patient_id": "p2",
"type": "Intake",
"note": "Post-op check; wound clean."
},
{
"patient_id": "p3",
"type": "Triage",
"note": "Fasting glucose ordered."
},
{
"patient_id": "p4",
"type": "Intake",
"note": "Post-op check; wound clean."
},
{
"patient_id": "p5",
"type": "Consult",
"note": "Post-op check; wound clean."
},
{
"patient_id": "p6",
"type": "Consult",
"note": "Vaccination due; schedule next visit."
},
{
"patient_id": "p7",
"type": "Follow-up",
"note": "Vaccination due; schedule next visit."
},
{
"patient_id": "p8",
"type": "Follow-up",
"note": "Fasting glucose ordered."
}
],
"holdout_gold": [
{"patient_id": "h1", "type": "Triage", "note": "Fasting glucose ordered."},
{"patient_id": "h2", "type": "Follow-up", "note": "Fasting glucose ordered."},
{"patient_id": "h3", "type": "Intake", "note": "Follow-up in 2 weeks; BP recheck."},
{"patient_id": "h4", "type": "Triage", "note": "Refill requested; review labs first."},
{"patient_id": "h5", "type": "Consult", "note": "Allergy noted; update chart."},
{"patient_id": "h6", "type": "Consult", "note": "Follow-up in 2 weeks; BP recheck."},
{"patient_id": "h7", "type": "Triage", "note": "Fasting glucose ordered."},
{"patient_id": "h8", "type": "Follow-up", "note": "Refill requested; review labs first."}
{
"patient_id": "h1",
"type": "Triage",
"note": "Fasting glucose ordered."
},
{
"patient_id": "h2",
"type": "Follow-up",
"note": "Fasting glucose ordered."
},
{
"patient_id": "h3",
"type": "Intake",
"note": "Follow-up in 2 weeks; BP recheck."
},
{
"patient_id": "h4",
"type": "Triage",
"note": "Refill requested; review labs first."
},
{
"patient_id": "h5",
"type": "Consult",
"note": "Allergy noted; update chart."
},
{
"patient_id": "h6",
"type": "Consult",
"note": "Follow-up in 2 weeks; BP recheck."
},
{
"patient_id": "h7",
"type": "Triage",
"note": "Fasting glucose ordered."
},
{
"patient_id": "h8",
"type": "Follow-up",
"note": "Refill requested; review labs first."
}
],
"train_operators": [
"control"
],
"holdout_operators": [
"control",
"dup",
"extra",
"omit",
"unsubmit",
"claim"
],
"train_operators": ["control"],
"holdout_operators": ["control", "dup", "extra", "omit", "unsubmit", "claim"],
"holdout_mutant_host": "h1"
},
"arms": {
Expand Down Expand Up @@ -182,7 +285,7 @@
"missing_cells": "Every arm times every seed occupies a cell. A refusal, abort, or expired certificate records a reason and does not free the seed for a replacement run.",
"minimum_detectable_effect": {
"contrast": "holdout_silent_incorrect_success(visual_only) minus holdout_silent_incorrect_success(certified_sor), mean across the three seeds",
"threshold": 0.10,
"threshold": 0.1,
"rule": "If the absolute contrast is below 0.10, the full Phase-2 grid does not run. The pilot publishes as a calibration null. That sentence is a kill rule, not a result."
}
},
Expand Down Expand Up @@ -222,13 +325,27 @@
},
{
"path": "openadapt_evals/reward/proof.py",
"sha256": "sha256:cb2ba25272dccf2968f33d9629739520d09d062a0d4ea10bc9c35f66e80d868b",
"git_blob": "ac8c95bca2421d583b00261e2d763e3d4758ee7d"
"sha256": "sha256:7f87d60b9250110c5f10bad3e4b4f0f80209711b211d9ed41ed1d2cd2bff4466",
"git_blob": "20da8e6168ad03220aa1a21498d40a0c5f9f7626"
},
{
"path": "docs/reward/proof_2026-09-01.json",
"sha256": "sha256:8eaaa34f8daf76dfeb45bab15d5ef17155b8d267aa97f6393e757ff6469bf1d3",
"git_blob": "b11ce9c32693db6c553a7bf46a0f9e1f837ba4ef"
},
{
"path": "docs/reward/proof_2026-09-02.json",
"sha256": "sha256:9be90c3ab1d6931484838abc48e16ba2d84bee676a702ad23f25f35ad538305d",
"git_blob": "289a65729c1452846e3dd7abcb5ba514fde49d23"
}
]
],
"identity_swap": {
"in_extradup_mutants": false,
"in_training_dataset": false,
"in_training_reward": false,
"eval_only": true,
"proof_path": "docs/reward/proof_2026-09-02.json",
"note": "Wrong-record write. Named identity is p1; write lands on p2. Not an ExtraDup operator."
},
"amendment_note": "2026-09-02 identity-swap: proof.py hash and proof_2026-09-02.json added. ExtraDup operators and the 2026-09-01 certificate digest are unchanged."
}
55 changes: 34 additions & 21 deletions docs/reward/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,24 +164,33 @@ python -m openadapt_evals.extradup kill-scan
It runs the frozen MockMed corpus, prints silent-incorrect-success on gold-FAIL
mutants vs honest-write on the control, keeps `execute_seal: false` /
`production_seal: false`, and does not mix those mutants into a training
reward. Same numbers as below. `--verdicts path.json` scores someone else's
checker on the same cells.
reward. Same numbers as the ExtraDup-only snapshot. `--verdicts path.json`
scores someone else's checker on the same cells.

`python -m openadapt_evals.reward.proof` is the underlying generator: scripted
policies against the in-memory MockMed store from `openadapt_evals.extradup`,
two rewards through the same `assess_receipt` path the adapters use. No model,
no GPU. Three trials per condition on the seed schedule `[101, 202, 303]`;
the seed picks the banner wording and the receipt identities. The committed
output is [`proof_2026-09-01.md`](proof_2026-09-01.md) and
[`proof_2026-09-01.json`](proof_2026-09-01.json); a test regenerates it and
fails if the committed file drifts from the code.
no GPU, not a trained policy, not a Production Seal. Three trials per
condition on the seed schedule `[101, 202, 303]`; the seed picks the banner
wording and the receipt identities.

The ExtraDup-only snapshot is
[`proof_2026-09-01.md`](proof_2026-09-01.md) and
[`proof_2026-09-01.json`](proof_2026-09-01.json). A test regenerates it from
the frozen condition set and fails if that file drifts. The current committed
output, which adds the `identity_swap` family, is
[`proof_2026-09-02.md`](proof_2026-09-02.md) and
[`proof_2026-09-02.json`](proof_2026-09-02.json).

`visual_only` is a tier-0 reward that believes the saved banner, the filled
form, or the policy's claim. `certified_sor` is a tier-2 reward that runs
ExtraDup's `sor_check` (record count, no extra field, every spec field) behind
a self-signed synthetic-scope certificate. `oracle_outage` is not an ExtraDup
family: the write is correct but the store cannot be read, so the certified
reward answers `failed_platform`.
form, or the policy's claim. `certified_sor` is a tier-2 reward that reads the
named `oracle.identity_keys` and runs ExtraDup's `sor_check` (record count,
no extra field, every spec field) on that record, behind a self-signed
synthetic-scope certificate. `identity_swap` writes a correct-looking note
onto the wrong `patient_id`; the banner still says saved, and the named
record is unchanged. `oracle_outage` is not an ExtraDup family: the write is
correct but the store cannot be read, so the certified reward answers
`failed_platform`.

| condition | gold | reward | tier | trials | paid | silent incorrect success | over-refusal | unscored | certified | scope |
|---|---|---|---|---|---|---|---|---|---|---|
Expand All @@ -197,17 +206,21 @@ reward answers `failed_platform`.
| unsubmit | FAIL | certified_sor | 2 | 3 | 0 | 0.00 | n/a | 0 | yes | synthetic |
| claim | FAIL | visual_only | 0 | 3 | 3 | 1.00 | n/a | 0 | no | none |
| claim | FAIL | certified_sor | 2 | 3 | 0 | 0.00 | n/a | 0 | yes | synthetic |
| identity_swap | FAIL | visual_only | 0 | 3 | 3 | 1.00 | n/a | 0 | no | none |
| identity_swap | FAIL | certified_sor | 2 | 3 | 0 | 0.00 | n/a | 0 | yes | synthetic |
| oracle_outage | PASS | visual_only | 0 | 3 | 3 | n/a | 0.00 | 0 | no | none |
| oracle_outage | PASS | certified_sor | 2 | 3 | 0 | n/a | 0.00 | 3 | yes | synthetic |

The visual reward paid all 15 gold-FAIL rollouts. The certified reward paid
none of them and paid the 3 controls. Its certificate's epsilon, 0.181036, is
the exact one-sided 95% Clopper-Pearson upper bound from those 15 trials; it
was computed from the run, not chosen. The `certified` column reads `yes`
for the tier-2 reward because the certificate is current at update 7, names
its calibration corpus by digest, and states its scope. That scope is
`synthetic`, and a self-signed certificate can state no other; the contract
(openadapt-types 0.17.0) rejects one that tries.
The visual reward paid all 18 gold-FAIL rollouts, including identity_swap.
The certified reward paid none of them and paid the 3 controls. Its
certificate's epsilon, 0.153318, is the exact one-sided 95% Clopper-Pearson
upper bound from those 18 trials; it was computed from the run, not chosen.
The 09-01 snapshot's epsilon, 0.181036, is the same bound over the 15
ExtraDup-only trials. The `certified` column reads `yes` for the tier-2 reward because the
certificate is current at update 7, names its calibration corpus by digest,
and states its scope. That scope is `synthetic`, and a self-signed
certificate can state no other; the contract (openadapt-types 0.17.0)
rejects one that tries.

The expiry check re-assesses the certified control receipts at policy update
100, the first update at which the certificate has expired: none stays
Expand All @@ -217,7 +230,7 @@ certified and the adapter logs three expiry warnings.

It does not train anything, so it says nothing about how the two rewards
change a policy. It does not touch a real system of record. The bound is for
15 synthetic trials at one seed schedule and would move with either. The
18 synthetic trials at one seed schedule and would move with either. The
reward worker that issues production receipts is a separate piece of work in
`openadapt-flow`; until it ships, the only receipts this package has seen were
signed by its own development key.
Loading
Loading