diff --git a/README.md b/README.md index 4400015..6e24092 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,7 @@ out the pixels. Coordinates are the thing that breaks when a window moves. | `ArtifactRefV1` | A path-free reference to an immutable process artifact | | `CodeCapabilityManifestV1` | Exact Python, locked dependencies, typed I/O, permissions, and verifier bindings | | `ProcessEvidenceReceiptV1` | One signed root over child receipts, human receipts, and the artifact graph | +| `RewardEvidenceReceiptV1` | One verified terminal effect for a training episode. Not an Execute Seal | | `AuthenticationTaskContractV1` | A value-free login requirement bound to an existing attended task | | `AuthoringObserveV1` | PHI-safe authoring observe tree for the hosted MCP wire | | `AuthoringCommandV1` | Mailbox envelope. Hosted click is `node_id` only; compile is `needs_human_admit` | @@ -89,7 +90,9 @@ asynchronous qualified execution, `EffectStrengthV1`, and the contracts may and may not carry is in [docs/CONTRACTS.md](docs/CONTRACTS.md). Oracle tiers and the ten-line adapter are in [docs/ORACLE.md](docs/ORACLE.md). Code capabilities and process -artifacts are in [docs/PROCESS_CAPABILITIES.md](docs/PROCESS_CAPABILITIES.md). +artifacts are in [docs/PROCESS_CAPABILITIES.md](docs/PROCESS_CAPABILITIES.md). Reward +contracts and certificates for training against a verified terminal effect +are in [docs/REWARD.md](docs/REWARD.md). ## JSON Schema for everything else @@ -101,7 +104,7 @@ print(json.dumps(ComputerState.model_json_schema(), indent=2)) ``` The same schemas ship as JSON under `openadapt_types/schemas/` for TypeScript, -Rust, and anything else that isn't Python. Thirty-one files, including +Rust, and anything else that isn't Python. Thirty-four files, including `execute-v1-openapi.json`, the public OpenAdapt Execute contract. ## Converting from the older formats diff --git a/docs/CONTRACTS.md b/docs/CONTRACTS.md index cf45661..3820e31 100644 --- a/docs/CONTRACTS.md +++ b/docs/CONTRACTS.md @@ -102,6 +102,25 @@ secret values, evidence bytes, and oracle recipes stay inside the declared execution boundary. See [PROCESS_CAPABILITIES.md](PROCESS_CAPABILITIES.md) for the package and runtime split. +## Reward contracts + +`RewardContractV1`, `RewardCertificateV1`, and `RewardEvidenceReceiptV1` let a +trainer optimize against a verified terminal effect and know the bound the +checker carries. The contract binds task, environment, required and forbidden +effect contracts, and the oracle recipe by digest. The certificate carries +`epsilon`, `delta`, `threshold`, the calibration corpus digest, and an expiry +denominated in policy updates. The receipt carries the outcome, the component +vector, the scalar, and the certificate state. + +`reconciliation_required` and `failed_platform` are unscored. They carry no +scalar and the contract cannot map them to zero. `certified` is true only at +oracle tier 2 or 3 with a current certificate. Tier 0 and 1 receipts are +`development_only`. + +The reward receipt is not an Execute Seal. It has its own schema id and none +of the Seal's fields. It says OpenAdapt verified one episode's terminal +effect. It does not say Flow governed the policy. See [REWARD.md](REWARD.md). + ## Authentication tasks `AuthenticationTaskContractV1` adds authentication semantics to the existing diff --git a/docs/REWARD.md b/docs/REWARD.md new file mode 100644 index 0000000..626aac2 --- /dev/null +++ b/docs/REWARD.md @@ -0,0 +1,95 @@ +# Reward contracts + +A trainer that optimizes against a checker inherits the checker's errors. The +reward contract says what earns reward, who reads the terminal effect, and +what bound that reader carries. The receipt says one thing: OpenAdapt verified +the terminal effect of one episode. It does not say that Flow governed the +policy's actions. + +This is not an Execute Seal. A model rollout never receives +`ExecuteEvidenceReceiptV1`. A production Flow result requires a qualified +deterministic program with zero model use. The reward receipt has its own +schema id, `openadapt.reward-evidence-receipt/v1`, and none of the Seal's +fields (`execution_id`, `workflow_digest`, `qualification_id`, `contracts`), so +the two cannot be swapped. + +## The three contracts + +`RewardContractV1` binds a task and environment by opaque id and digest, the +required and forbidden effect contracts by digest, the independent oracle +(channel plus identity keys), the reward components and their weights, the +scoring policy, and the certificate policy. Its `digest` is the canonical +SHA-256 over sorted JSON. Components sort by name, so two authors who list them +in a different order get the same digest. + +`RewardCertificateV1` is the bound: `epsilon`, `delta`, `threshold`, the +calibration corpus digest, the checker configuration digest, the policy update +it was issued at, and its expiry in policy updates. It is signed. Expiry +counts updates, not hours, because on-policy training breaks the +exchangeability the bound assumes. `is_current(policy_update)` answers whether +a trainer may still use it. + +`RewardEvidenceReceiptV1` binds the contract digest, the policy checkpoint and +update number, the episode, the oracle tier, the evidence digest, the +component vector, the scalar, the certificate reference and its state, and two +booleans a trainer must read: `certified` and `development_only`. + +## Outcome to scalar + +| `reward_outcome` | Class | Scalar | +| --- | --- | --- | +| `verified` | admitted positive | `verified_reward` (default 1.0) | +| `halted_before_effect` | zero or penalty | `halted_before_effect_reward` (default 0.0) | +| `refused` | zero or penalty | `refused_reward` (default 0.0) | +| `rejected_policy` | zero or penalty | `rejected_policy_reward` (default 0.0) | +| `wrong_effect` | zero or penalty | `wrong_effect_reward` (default -1.0) | +| `reconciliation_required` | unscored | none | +| `failed_platform` | unscored | none | + +An unscored episode has no scalar. `score()` returns `None`, the receipt +refuses a `scalar_reward`, and the contract cannot declare +`uncertain_episodes` or `platform_failures` as anything but `unscored`. A +trainer that folds those into 0.0 is training on platform noise, which is the +failure this contract exists to stop. + +## Tier to certification + +| Oracle tier | `development_only` | `certified` | +| --- | --- | --- | +| 0 (visual, OCR) | yes | never | +| 1 (second session) | yes | never | +| 2 (API, DB, file, ack) | no | with a current certificate | +| 3 (counterparty) | no | with a current certificate | + +The tier comes from the oracle channel, as it does for every Seal. +`refuse_development_certification` raises `RewardCertificationRefused` for +tier 0 and 1, and the receipt validator calls it. A tier-0 reward can drive a +local experiment. It cannot be certified, and a receipt that claims otherwise +does not validate. + +## What a trainer receives + +```python +from openadapt_types import RewardOutcomeV1, score + +scalar, certified, development_only = score( + RewardOutcomeV1.VERIFIED, + tier=2, + certificate=certificate, + policy_update=120, +) +``` + +Per episode, a signed `RewardEvidenceReceiptV1`: ids, digests, the tier, the +outcome, the component vector, the scalar or its absence, and the certificate +state. The trainer checks `certified` before it counts the episode toward a +certified arm and drops the episode when `scalar_reward` is `None`. + +## What stays on the organization node + +The calibration corpus, the tuned adversary parameters, the checker threshold's +derivation, the oracle read recipe, and the evidence bytes. Every one of those +appears in the contract as a digest only. A consumer can prove which corpus +certified a reward. It cannot read the corpus from the receipt. + +Worked contracts are in [`tests/test_reward.py`](../tests/test_reward.py). diff --git a/openadapt_types/__init__.py b/openadapt_types/__init__.py index 3163a9b..06ae8e0 100644 --- a/openadapt_types/__init__.py +++ b/openadapt_types/__init__.py @@ -324,6 +324,31 @@ CodeRuntimeKind, ProcessEvidenceReceiptV1, ) +from openadapt_types.reward import ( + DEFAULT_REWARD_SCORING, + REWARD_CERTIFICATE_SCHEMA, + REWARD_CERTIFIED_MINIMUM_TIER, + REWARD_CONTRACT_SCHEMA, + REWARD_EVIDENCE_RECEIPT_SCHEMA, + REWARD_SCORING_CLASS, + UNSCORED_REWARD_OUTCOMES, + RewardCertificatePolicyV1, + RewardCertificateStateV1, + RewardCertificateV1, + RewardCertificationRefused, + RewardComponentV1, + RewardContractV1, + RewardEvidenceReceiptV1, + RewardOracleV1, + RewardOutcomeV1, + RewardScoreV1, + RewardScoringClassV1, + RewardScoringPolicyV1, + RewardUncertaintyStateV1, + certificate_state, + refuse_development_certification, + score, +) from openadapt_types.runner_capability import ( RUNNER_CAPABILITY_MANIFEST_SCHEMA, EffectVerificationTier, @@ -612,6 +637,30 @@ "CodePermissionContractV1", "CodeRuntimeKind", "ProcessEvidenceReceiptV1", + # reward contracts (not an Execute Seal) + "DEFAULT_REWARD_SCORING", + "REWARD_CERTIFICATE_SCHEMA", + "REWARD_CERTIFIED_MINIMUM_TIER", + "REWARD_CONTRACT_SCHEMA", + "REWARD_EVIDENCE_RECEIPT_SCHEMA", + "REWARD_SCORING_CLASS", + "UNSCORED_REWARD_OUTCOMES", + "RewardCertificatePolicyV1", + "RewardCertificateStateV1", + "RewardCertificateV1", + "RewardCertificationRefused", + "RewardComponentV1", + "RewardContractV1", + "RewardEvidenceReceiptV1", + "RewardOracleV1", + "RewardOutcomeV1", + "RewardScoreV1", + "RewardScoringClassV1", + "RewardScoringPolicyV1", + "RewardUncertaintyStateV1", + "certificate_state", + "refuse_development_certification", + "score", # failure "FailureCategory", "FailureRecord", diff --git a/openadapt_types/reward.py b/openadapt_types/reward.py new file mode 100644 index 0000000..6e2f1fe --- /dev/null +++ b/openadapt_types/reward.py @@ -0,0 +1,562 @@ +"""Versioned reward contracts for training against verified terminal effects. + +A reward receipt reuses the evidence, signature, admission, and revocation +mechanisms of the Execute contracts. It states one thing: OpenAdapt verified +the terminal effect of one episode against one reward contract. It does not +state that Flow governed the policy's actions. It is not an Execute Seal. + +An arbitrary model rollout never receives ``ExecuteEvidenceReceiptV1``. A +production Flow result requires a qualified deterministic program with zero +model use. The reward receipt therefore carries its own schema id, its own +outcome enum, and no ``execution_id``, ``workflow_digest``, +``qualification_id``, or ``contracts`` field, so the two receipts cannot be +exchanged for one another. + +The certificate fields follow the certified-reward RL preregistration: a +distribution-free bound P(false-accept) <= epsilon at confidence 1 - delta, +calibrated on a corpus that is referenced by digest only, with an expiry +denominated in policy updates. The corpus contents, tuned adversary +parameters, and deployment thresholds stay private. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from enum import Enum +from typing import Any, Literal, NamedTuple + +from pydantic import ( + Field, + StrictBool, + StrictFloat, + StrictInt, + StrictStr, + field_validator, + model_validator, +) + +from openadapt_types.execute import OracleTierV1 +from openadapt_types.oracle import ( + PRODUCTION_SEAL_MINIMUM_TIER, + OracleChannel, + OracleTier, + tier_of, +) +from openadapt_types.process_capability import ( + _digest_payload, + _parse_timestamp, + _StrictContract, + _validate_signature, +) + +REWARD_CONTRACT_SCHEMA: Literal["openadapt.reward-contract/v1"] = ( + "openadapt.reward-contract/v1" +) +REWARD_CERTIFICATE_SCHEMA: Literal["openadapt.reward-certificate/v1"] = ( + "openadapt.reward-certificate/v1" +) +REWARD_EVIDENCE_RECEIPT_SCHEMA: Literal["openadapt.reward-evidence-receipt/v1"] = ( + "openadapt.reward-evidence-receipt/v1" +) + +# A certified reward needs the same oracle floor as a production Seal. The +# floor is shared. The receipt is not. +REWARD_CERTIFIED_MINIMUM_TIER = PRODUCTION_SEAL_MINIMUM_TIER + +_OPAQUE_ID_PATTERN = r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$" +_NAME_PATTERN = r"^[A-Za-z][A-Za-z0-9_-]{0,127}$" +_SHA256_PATTERN = r"^sha256:[a-f0-9]{64}$" +_TIMESTAMP_PATTERN = r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(Z|[+-]\d{2}:\d{2})$" +_MAX_REWARD_MAGNITUDE = 1_000_000.0 +_MAX_COMPONENTS = 64 +_MAX_POLICY_UPDATES = 9_007_199_254_740_991 + + +class RewardOutcomeV1(str, Enum): + """Terminal outcome of one episode as the reward oracle saw it. + + Each member has exactly one scoring class, listed in + ``REWARD_SCORING_CLASS``. The class decides whether a scalar exists. + """ + + VERIFIED = "verified" + """The independent oracle read the required terminal effect. Admitted + positive reward.""" + + HALTED_BEFORE_EFFECT = "halted_before_effect" + """The runtime stopped the episode before any consequential effect. + Zero or the declared penalty.""" + + REFUSED = "refused" + """The policy declined the task before any effect. Zero or the declared + penalty. Kept apart from a runtime halt so a trainer can weigh them + differently.""" + + REJECTED_POLICY = "rejected_policy" + """An admission or policy contract refused the episode. Zero or the + declared penalty.""" + + WRONG_EFFECT = "wrong_effect" + """The oracle read a terminal effect that differs from the required one, + or a forbidden effect. Zero or the declared penalty. This is the silent + wrong action the certificate bounds.""" + + RECONCILIATION_REQUIRED = "reconciliation_required" + """Delivery or effect is uncertain. UNSCORED. Never 0.0.""" + + FAILED_PLATFORM = "failed_platform" + """The runner or oracle failed for a reason unrelated to the policy. + UNSCORED. Never 0.0.""" + + +class RewardScoringClassV1(str, Enum): + ADMITTED_POSITIVE = "admitted_positive" + ZERO_OR_PENALTY = "zero_or_penalty" + UNSCORED = "unscored" + + +REWARD_SCORING_CLASS: Mapping[RewardOutcomeV1, RewardScoringClassV1] = { + RewardOutcomeV1.VERIFIED: RewardScoringClassV1.ADMITTED_POSITIVE, + RewardOutcomeV1.HALTED_BEFORE_EFFECT: RewardScoringClassV1.ZERO_OR_PENALTY, + RewardOutcomeV1.REFUSED: RewardScoringClassV1.ZERO_OR_PENALTY, + RewardOutcomeV1.REJECTED_POLICY: RewardScoringClassV1.ZERO_OR_PENALTY, + RewardOutcomeV1.WRONG_EFFECT: RewardScoringClassV1.ZERO_OR_PENALTY, + RewardOutcomeV1.RECONCILIATION_REQUIRED: RewardScoringClassV1.UNSCORED, + RewardOutcomeV1.FAILED_PLATFORM: RewardScoringClassV1.UNSCORED, +} + +UNSCORED_REWARD_OUTCOMES = frozenset( + outcome + for outcome, scoring_class in REWARD_SCORING_CLASS.items() + if scoring_class is RewardScoringClassV1.UNSCORED +) + + +class RewardCertificateStateV1(str, Enum): + ABSENT = "absent" + NOT_YET_VALID = "not_yet_valid" + CURRENT = "current" + EXPIRED = "expired" + + +class RewardUncertaintyStateV1(str, Enum): + NONE = "none" + DELIVERY_UNCERTAIN = "delivery_uncertain" + EFFECT_UNCERTAIN = "effect_uncertain" + ORACLE_UNAVAILABLE = "oracle_unavailable" + + +class RewardCertificationRefused(ValueError): + """Raised when a tier-0 or tier-1 reward receipt is marked certified.""" + + +def refuse_development_certification(tier: OracleTier | int) -> None: + """Raise unless ``tier`` may carry a certified reward.""" + + if int(tier) < REWARD_CERTIFIED_MINIMUM_TIER: + raise RewardCertificationRefused( + "a certified reward requires oracle tier 2 or 3" + ) + + +def _finite(value: float, field_name: str) -> float: + if value != value or value in (float("inf"), float("-inf")): + raise ValueError(f"{field_name} must be finite") + return value + + +class RewardComponentV1(_StrictContract): + """One named term of the reward vector and its weight in the scalar.""" + + name: StrictStr = Field(pattern=_NAME_PATTERN) + weight: StrictFloat = Field(gt=0.0, le=_MAX_REWARD_MAGNITUDE, allow_inf_nan=False) + + +class RewardOracleV1(_StrictContract): + """The independent oracle that reads the terminal effect. + + The channel sets the tier. The identity keys name the record the oracle + reads. The oracle contract digest binds the read recipe without carrying + it. + """ + + channel: OracleChannel + identity_keys: tuple[StrictStr, ...] = Field( + min_length=1, + max_length=32, + json_schema_extra={"uniqueItems": True}, + ) + oracle_contract_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + + @field_validator("identity_keys") + @classmethod + def _canonical_keys(cls, values: tuple[str, ...]) -> tuple[str, ...]: + if any(re.fullmatch(_NAME_PATTERN, value) is None for value in values): + raise ValueError("identity_keys contains an invalid key") + if len(values) != len(set(values)): + raise ValueError("identity_keys must not contain duplicates") + return tuple(sorted(values)) + + @property + def tier(self) -> OracleTier: + return tier_of(self.channel) + + +class RewardScoringPolicyV1(_StrictContract): + """How each scoring class becomes a scalar. + + Uncertain episodes and platform failures are fixed to ``unscored`` in + this version. A contract cannot declare them to be zero. + """ + + verified_reward: StrictFloat = Field( + default=1.0, + gt=0.0, + le=_MAX_REWARD_MAGNITUDE, + allow_inf_nan=False, + ) + halted_before_effect_reward: StrictFloat = Field( + default=0.0, + ge=-_MAX_REWARD_MAGNITUDE, + le=0.0, + allow_inf_nan=False, + ) + refused_reward: StrictFloat = Field( + default=0.0, + ge=-_MAX_REWARD_MAGNITUDE, + le=0.0, + allow_inf_nan=False, + ) + rejected_policy_reward: StrictFloat = Field( + default=0.0, + ge=-_MAX_REWARD_MAGNITUDE, + le=0.0, + allow_inf_nan=False, + ) + wrong_effect_reward: StrictFloat = Field( + default=-1.0, + ge=-_MAX_REWARD_MAGNITUDE, + le=0.0, + allow_inf_nan=False, + ) + uncertain_episodes: Literal["unscored"] = "unscored" + platform_failures: Literal["unscored"] = "unscored" + + def scalar_for(self, outcome: RewardOutcomeV1) -> float | None: + """Return the declared scalar, or ``None`` for an unscored outcome.""" + + declared: Mapping[RewardOutcomeV1, float] = { + RewardOutcomeV1.VERIFIED: self.verified_reward, + RewardOutcomeV1.HALTED_BEFORE_EFFECT: self.halted_before_effect_reward, + RewardOutcomeV1.REFUSED: self.refused_reward, + RewardOutcomeV1.REJECTED_POLICY: self.rejected_policy_reward, + RewardOutcomeV1.WRONG_EFFECT: self.wrong_effect_reward, + } + return declared.get(RewardOutcomeV1(outcome)) + + +DEFAULT_REWARD_SCORING = RewardScoringPolicyV1() + + +class RewardCertificatePolicyV1(_StrictContract): + """The certificate a reward must hold before a trainer may call it certified. + + ``epsilon`` bounds the false-accept probability. ``delta`` is one minus + the confidence. ``threshold`` is the checker decision threshold the bound + was calibrated at. ``calibration_corpus_digest`` names the corpus without + carrying it. ``expiry_policy_updates`` is the number of policy updates a + certificate stays current after issue. + """ + + epsilon: StrictFloat = Field(gt=0.0, lt=1.0, allow_inf_nan=False) + delta: StrictFloat = Field(gt=0.0, lt=1.0, allow_inf_nan=False) + threshold: StrictFloat = Field(allow_inf_nan=False) + calibration_corpus_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + expiry_policy_updates: StrictInt = Field(ge=1, le=_MAX_POLICY_UPDATES) + + +class RewardContractV1(_StrictContract): + """One immutable statement of what earns reward and who checks it.""" + + schema_version: Literal["openadapt.reward-contract/v1"] = REWARD_CONTRACT_SCHEMA + contract_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + contract_version: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + task_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + task_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + environment_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + environment_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + required_effect_contract_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + forbidden_effect_contract_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + oracle: RewardOracleV1 + components: tuple[RewardComponentV1, ...] = Field( + min_length=1, + max_length=_MAX_COMPONENTS, + json_schema_extra={"uniqueItems": True}, + ) + scoring: RewardScoringPolicyV1 = Field(default_factory=RewardScoringPolicyV1) + certificate_policy: RewardCertificatePolicyV1 + + @field_validator("components") + @classmethod + def _canonical_components( + cls, values: tuple[RewardComponentV1, ...] + ) -> tuple[RewardComponentV1, ...]: + names = tuple(item.name for item in values) + if len(names) != len(set(names)): + raise ValueError("reward component names must be unique") + return tuple(sorted(values, key=lambda item: item.name)) + + @property + def component_names(self) -> tuple[str, ...]: + return tuple(item.name for item in self.components) + + @property + def digest(self) -> str: + return _digest_payload(self.model_dump(mode="json")) + + +class RewardCertificateV1(_StrictContract): + """A signed, expiring bound on one reward contract's false-accept rate. + + Expiry counts policy updates, not wall-clock time. A certificate issued + at update ``i`` with expiry ``n`` is current for updates ``i`` through + ``i + n - 1``. Revocation is a separate list keyed by ``certificate_id`` + and is checked by the issuer, as it is for every other admission. + """ + + schema_version: Literal["openadapt.reward-certificate/v1"] = ( + REWARD_CERTIFICATE_SCHEMA + ) + certificate_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + reward_contract_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + checker_configuration_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + epsilon: StrictFloat = Field(gt=0.0, lt=1.0, allow_inf_nan=False) + delta: StrictFloat = Field(gt=0.0, lt=1.0, allow_inf_nan=False) + threshold: StrictFloat = Field(allow_inf_nan=False) + calibration_corpus_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + issued_at_policy_update: StrictInt = Field(ge=0, le=_MAX_POLICY_UPDATES) + expiry_policy_updates: StrictInt = Field(ge=1, le=_MAX_POLICY_UPDATES) + issued_at: StrictStr = Field(pattern=_TIMESTAMP_PATTERN) + issuer_key_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + signature_algorithm: Literal["ed25519"] = "ed25519" + signature: StrictStr = Field(min_length=88, max_length=88) + + @field_validator("signature") + @classmethod + def _signature(cls, value: str) -> str: + return _validate_signature(value, 64, "reward certificate signature") + + @model_validator(mode="after") + def _issue_window(self) -> RewardCertificateV1: + _parse_timestamp(self.issued_at, "issued_at") + if self.issued_at_policy_update + self.expiry_policy_updates > _MAX_POLICY_UPDATES: + raise ValueError("reward certificate expiry overflows the update counter") + return self + + @property + def expires_at_policy_update(self) -> int: + """The first policy update at which this certificate is expired.""" + + return self.issued_at_policy_update + self.expiry_policy_updates + + def state_at(self, policy_update: int) -> RewardCertificateStateV1: + if policy_update < self.issued_at_policy_update: + return RewardCertificateStateV1.NOT_YET_VALID + if policy_update >= self.expires_at_policy_update: + return RewardCertificateStateV1.EXPIRED + return RewardCertificateStateV1.CURRENT + + def is_current(self, policy_update: int) -> bool: + """True when ``policy_update`` falls inside the issue window.""" + + return self.state_at(policy_update) is RewardCertificateStateV1.CURRENT + + def satisfies(self, policy: RewardCertificatePolicyV1) -> bool: + """True when this certificate is at least as strong as the policy asks.""" + + return ( + self.epsilon <= policy.epsilon + and self.delta <= policy.delta + and self.threshold == policy.threshold + and self.calibration_corpus_digest == policy.calibration_corpus_digest + and self.expiry_policy_updates <= policy.expiry_policy_updates + ) + + def unsigned_payload(self) -> dict[str, Any]: + return self.model_dump( + mode="json", + exclude={"signature", "signature_algorithm"}, + ) + + @property + def digest(self) -> str: + return _digest_payload(self.model_dump(mode="json")) + + +def certificate_state( + certificate: RewardCertificateV1 | None, policy_update: int +) -> RewardCertificateStateV1: + """Classify a certificate reference at one policy update.""" + + if certificate is None: + return RewardCertificateStateV1.ABSENT + return certificate.state_at(policy_update) + + +class RewardScoreV1(NamedTuple): + scalar: float | None + certified: bool + development_only: bool + + +def score( + outcome: RewardOutcomeV1, + tier: OracleTier | int, + certificate: RewardCertificateV1 | None, + policy_update: int, + *, + scoring: RewardScoringPolicyV1 = DEFAULT_REWARD_SCORING, +) -> RewardScoreV1: + """Score one episode. Pure. Never turns an unscored outcome into 0.0. + + * ``scalar`` is ``None`` for ``RECONCILIATION_REQUIRED`` and + ``FAILED_PLATFORM``. A trainer must drop or hold those episodes. + * ``certified`` is true only at tier 2 or 3 with a certificate that is + current at ``policy_update``. + * ``development_only`` is true at tier 0 or 1. A tier-0 reward can train + a local experiment. It can never be certified. + """ + + if policy_update < 0: + raise ValueError("policy_update must be non-negative") + development_only = int(tier) < REWARD_CERTIFIED_MINIMUM_TIER + state = certificate_state(certificate, policy_update) + certified = not development_only and state is RewardCertificateStateV1.CURRENT + scalar = scoring.scalar_for(RewardOutcomeV1(outcome)) + return RewardScoreV1(scalar, certified, development_only) + + +class RewardEvidenceReceiptV1(_StrictContract): + """A signed statement that one episode's terminal effect was verified. + + This receipt binds a reward contract, a policy checkpoint, an episode, + and the oracle read. It is not an Execute Seal. It carries no + ``execution_id``, ``workflow_digest``, ``qualification_id``, or Execute + contract block, and it does not claim that Flow governed the policy. + """ + + schema_version: Literal["openadapt.reward-evidence-receipt/v1"] = ( + REWARD_EVIDENCE_RECEIPT_SCHEMA + ) + receipt_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + reward_contract_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + policy_checkpoint_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + policy_update: StrictInt = Field(ge=0, le=_MAX_POLICY_UPDATES) + episode_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + oracle_tier: OracleTierV1 + reward_outcome: RewardOutcomeV1 + evidence_digest: StrictStr = Field(pattern=_SHA256_PATTERN) + reward_components: dict[StrictStr, StrictFloat] = Field( + default_factory=dict, + max_length=_MAX_COMPONENTS, + ) + scalar_reward: StrictFloat | None = Field( + default=None, + ge=-_MAX_REWARD_MAGNITUDE, + le=_MAX_REWARD_MAGNITUDE, + allow_inf_nan=False, + ) + certificate_id: StrictStr | None = Field(default=None, pattern=_OPAQUE_ID_PATTERN) + certificate_digest: StrictStr | None = Field(default=None, pattern=_SHA256_PATTERN) + certificate_state: RewardCertificateStateV1 + uncertainty: RewardUncertaintyStateV1 + certified: StrictBool + development_only: StrictBool + issuer_key_id: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + nonce: StrictStr = Field(pattern=_OPAQUE_ID_PATTERN) + issued_at: StrictStr = Field(pattern=_TIMESTAMP_PATTERN) + signature_algorithm: Literal["ed25519"] = "ed25519" + signature: StrictStr = Field(min_length=88, max_length=88) + + @field_validator("reward_components") + @classmethod + def _canonical_components(cls, values: dict[str, float]) -> dict[str, float]: + for name, value in values.items(): + if re.fullmatch(_NAME_PATTERN, name) is None: + raise ValueError("reward_components contains an invalid name") + _finite(value, f"reward_components[{name}]") + if abs(value) > _MAX_REWARD_MAGNITUDE: + raise ValueError(f"reward_components[{name}] is out of range") + return dict(sorted(values.items())) + + @field_validator("signature") + @classmethod + def _signature(cls, value: str) -> str: + return _validate_signature(value, 64, "reward receipt signature") + + @model_validator(mode="after") + def _scoring_contract(self) -> RewardEvidenceReceiptV1: + _parse_timestamp(self.issued_at, "issued_at") + expected_development = self.oracle_tier < REWARD_CERTIFIED_MINIMUM_TIER + if self.development_only != expected_development: + raise ValueError("development_only must be true exactly at oracle tier 0 or 1") + if self.certified: + refuse_development_certification(self.oracle_tier) + if self.certificate_state is not RewardCertificateStateV1.CURRENT: + raise ValueError("a certified reward requires a current certificate") + has_reference = ( + self.certificate_id is not None or self.certificate_digest is not None + ) + if self.certificate_state is RewardCertificateStateV1.ABSENT: + if has_reference: + raise ValueError("an absent certificate cannot carry a reference") + elif self.certificate_id is None or self.certificate_digest is None: + raise ValueError("a referenced certificate requires id and digest") + + scoring_class = REWARD_SCORING_CLASS[self.reward_outcome] + if scoring_class is RewardScoringClassV1.UNSCORED: + if self.scalar_reward is not None: + raise ValueError( + f"{self.reward_outcome.value} is unscored and cannot carry a scalar" + ) + if self.reward_components: + raise ValueError( + f"{self.reward_outcome.value} is unscored and cannot carry components" + ) + else: + if self.scalar_reward is None: + raise ValueError(f"{self.reward_outcome.value} requires a scalar reward") + if not self.reward_components: + raise ValueError( + f"{self.reward_outcome.value} requires at least one reward component" + ) + if scoring_class is RewardScoringClassV1.ADMITTED_POSITIVE: + if self.scalar_reward <= 0.0: + raise ValueError("a verified reward must be positive") + elif self.scalar_reward > 0.0: + raise ValueError( + f"{self.reward_outcome.value} yields zero or a declared penalty" + ) + + if self.reward_outcome is RewardOutcomeV1.VERIFIED: + if self.uncertainty is not RewardUncertaintyStateV1.NONE: + raise ValueError("a verified reward cannot carry uncertainty") + if self.reward_outcome is RewardOutcomeV1.RECONCILIATION_REQUIRED: + if self.uncertainty is RewardUncertaintyStateV1.NONE: + raise ValueError("reconciliation_required requires an uncertainty state") + return self + + @property + def scoring_class(self) -> RewardScoringClassV1: + return REWARD_SCORING_CLASS[self.reward_outcome] + + def unsigned_payload(self) -> dict[str, Any]: + return self.model_dump( + mode="json", + exclude={"signature", "signature_algorithm"}, + ) + + @property + def digest(self) -> str: + return _digest_payload(self.model_dump(mode="json")) diff --git a/openadapt_types/schemas/reward-certificate-v1.json b/openadapt_types/schemas/reward-certificate-v1.json new file mode 100644 index 0000000..fca4fcc --- /dev/null +++ b/openadapt_types/schemas/reward-certificate-v1.json @@ -0,0 +1,98 @@ +{ + "additionalProperties": false, + "description": "A signed, expiring bound on one reward contract's false-accept rate.\n\nExpiry counts policy updates, not wall-clock time. A certificate issued\nat update ``i`` with expiry ``n`` is current for updates ``i`` through\n``i + n - 1``. Revocation is a separate list keyed by ``certificate_id``\nand is checked by the issuer, as it is for every other admission.", + "properties": { + "calibration_corpus_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Calibration Corpus Digest", + "type": "string" + }, + "certificate_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Certificate Id", + "type": "string" + }, + "checker_configuration_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Checker Configuration Digest", + "type": "string" + }, + "delta": { + "exclusiveMaximum": 1.0, + "exclusiveMinimum": 0.0, + "title": "Delta", + "type": "number" + }, + "epsilon": { + "exclusiveMaximum": 1.0, + "exclusiveMinimum": 0.0, + "title": "Epsilon", + "type": "number" + }, + "expiry_policy_updates": { + "maximum": 9007199254740991, + "minimum": 1, + "title": "Expiry Policy Updates", + "type": "integer" + }, + "issued_at": { + "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(Z|[+-]\\d{2}:\\d{2})$", + "title": "Issued At", + "type": "string" + }, + "issued_at_policy_update": { + "maximum": 9007199254740991, + "minimum": 0, + "title": "Issued At Policy Update", + "type": "integer" + }, + "issuer_key_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Issuer Key Id", + "type": "string" + }, + "reward_contract_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Reward Contract Digest", + "type": "string" + }, + "schema_version": { + "const": "openadapt.reward-certificate/v1", + "default": "openadapt.reward-certificate/v1", + "title": "Schema Version", + "type": "string" + }, + "signature": { + "maxLength": 88, + "minLength": 88, + "title": "Signature", + "type": "string" + }, + "signature_algorithm": { + "const": "ed25519", + "default": "ed25519", + "title": "Signature Algorithm", + "type": "string" + }, + "threshold": { + "title": "Threshold", + "type": "number" + } + }, + "required": [ + "certificate_id", + "reward_contract_digest", + "checker_configuration_digest", + "epsilon", + "delta", + "threshold", + "calibration_corpus_digest", + "issued_at_policy_update", + "expiry_policy_updates", + "issued_at", + "issuer_key_id", + "signature" + ], + "title": "RewardCertificateV1", + "type": "object" +} diff --git a/openadapt_types/schemas/reward-contract-v1.json b/openadapt_types/schemas/reward-contract-v1.json new file mode 100644 index 0000000..015290d --- /dev/null +++ b/openadapt_types/schemas/reward-contract-v1.json @@ -0,0 +1,254 @@ +{ + "$defs": { + "OracleChannel": { + "description": "How the oracle read the effect. The channel, not the payload, sets the tier.", + "enum": [ + "visual", + "ocr", + "second_session", + "api", + "db", + "file", + "ack", + "counterparty" + ], + "title": "OracleChannel", + "type": "string" + }, + "RewardCertificatePolicyV1": { + "additionalProperties": false, + "description": "The certificate a reward must hold before a trainer may call it certified.\n\n``epsilon`` bounds the false-accept probability. ``delta`` is one minus\nthe confidence. ``threshold`` is the checker decision threshold the bound\nwas calibrated at. ``calibration_corpus_digest`` names the corpus without\ncarrying it. ``expiry_policy_updates`` is the number of policy updates a\ncertificate stays current after issue.", + "properties": { + "calibration_corpus_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Calibration Corpus Digest", + "type": "string" + }, + "delta": { + "exclusiveMaximum": 1.0, + "exclusiveMinimum": 0.0, + "title": "Delta", + "type": "number" + }, + "epsilon": { + "exclusiveMaximum": 1.0, + "exclusiveMinimum": 0.0, + "title": "Epsilon", + "type": "number" + }, + "expiry_policy_updates": { + "maximum": 9007199254740991, + "minimum": 1, + "title": "Expiry Policy Updates", + "type": "integer" + }, + "threshold": { + "title": "Threshold", + "type": "number" + } + }, + "required": [ + "epsilon", + "delta", + "threshold", + "calibration_corpus_digest", + "expiry_policy_updates" + ], + "title": "RewardCertificatePolicyV1", + "type": "object" + }, + "RewardComponentV1": { + "additionalProperties": false, + "description": "One named term of the reward vector and its weight in the scalar.", + "properties": { + "name": { + "pattern": "^[A-Za-z][A-Za-z0-9_-]{0,127}$", + "title": "Name", + "type": "string" + }, + "weight": { + "exclusiveMinimum": 0.0, + "maximum": 1000000.0, + "title": "Weight", + "type": "number" + } + }, + "required": [ + "name", + "weight" + ], + "title": "RewardComponentV1", + "type": "object" + }, + "RewardOracleV1": { + "additionalProperties": false, + "description": "The independent oracle that reads the terminal effect.\n\nThe channel sets the tier. The identity keys name the record the oracle\nreads. The oracle contract digest binds the read recipe without carrying\nit.", + "properties": { + "channel": { + "$ref": "#/$defs/OracleChannel" + }, + "identity_keys": { + "items": { + "type": "string" + }, + "maxItems": 32, + "minItems": 1, + "title": "Identity Keys", + "type": "array", + "uniqueItems": true + }, + "oracle_contract_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Oracle Contract Digest", + "type": "string" + } + }, + "required": [ + "channel", + "identity_keys", + "oracle_contract_digest" + ], + "title": "RewardOracleV1", + "type": "object" + }, + "RewardScoringPolicyV1": { + "additionalProperties": false, + "description": "How each scoring class becomes a scalar.\n\nUncertain episodes and platform failures are fixed to ``unscored`` in\nthis version. A contract cannot declare them to be zero.", + "properties": { + "halted_before_effect_reward": { + "default": 0.0, + "maximum": 0.0, + "minimum": -1000000.0, + "title": "Halted Before Effect Reward", + "type": "number" + }, + "platform_failures": { + "const": "unscored", + "default": "unscored", + "title": "Platform Failures", + "type": "string" + }, + "refused_reward": { + "default": 0.0, + "maximum": 0.0, + "minimum": -1000000.0, + "title": "Refused Reward", + "type": "number" + }, + "rejected_policy_reward": { + "default": 0.0, + "maximum": 0.0, + "minimum": -1000000.0, + "title": "Rejected Policy Reward", + "type": "number" + }, + "uncertain_episodes": { + "const": "unscored", + "default": "unscored", + "title": "Uncertain Episodes", + "type": "string" + }, + "verified_reward": { + "default": 1.0, + "exclusiveMinimum": 0.0, + "maximum": 1000000.0, + "title": "Verified Reward", + "type": "number" + }, + "wrong_effect_reward": { + "default": -1.0, + "maximum": 0.0, + "minimum": -1000000.0, + "title": "Wrong Effect Reward", + "type": "number" + } + }, + "title": "RewardScoringPolicyV1", + "type": "object" + } + }, + "additionalProperties": false, + "description": "One immutable statement of what earns reward and who checks it.", + "properties": { + "certificate_policy": { + "$ref": "#/$defs/RewardCertificatePolicyV1" + }, + "components": { + "items": { + "$ref": "#/$defs/RewardComponentV1" + }, + "maxItems": 64, + "minItems": 1, + "title": "Components", + "type": "array", + "uniqueItems": true + }, + "contract_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Contract Id", + "type": "string" + }, + "contract_version": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Contract Version", + "type": "string" + }, + "environment_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Environment Digest", + "type": "string" + }, + "environment_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Environment Id", + "type": "string" + }, + "forbidden_effect_contract_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Forbidden Effect Contract Digest", + "type": "string" + }, + "oracle": { + "$ref": "#/$defs/RewardOracleV1" + }, + "required_effect_contract_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Required Effect Contract Digest", + "type": "string" + }, + "schema_version": { + "const": "openadapt.reward-contract/v1", + "default": "openadapt.reward-contract/v1", + "title": "Schema Version", + "type": "string" + }, + "scoring": { + "$ref": "#/$defs/RewardScoringPolicyV1" + }, + "task_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Task Digest", + "type": "string" + }, + "task_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Task Id", + "type": "string" + } + }, + "required": [ + "contract_id", + "contract_version", + "task_id", + "task_digest", + "environment_id", + "environment_digest", + "required_effect_contract_digest", + "forbidden_effect_contract_digest", + "oracle", + "components", + "certificate_policy" + ], + "title": "RewardContractV1", + "type": "object" +} diff --git a/openadapt_types/schemas/reward-evidence-receipt-v1.json b/openadapt_types/schemas/reward-evidence-receipt-v1.json new file mode 100644 index 0000000..7d88cc2 --- /dev/null +++ b/openadapt_types/schemas/reward-evidence-receipt-v1.json @@ -0,0 +1,201 @@ +{ + "$defs": { + "RewardCertificateStateV1": { + "enum": [ + "absent", + "not_yet_valid", + "current", + "expired" + ], + "title": "RewardCertificateStateV1", + "type": "string" + }, + "RewardOutcomeV1": { + "description": "Terminal outcome of one episode as the reward oracle saw it.\n\nEach member has exactly one scoring class, listed in\n``REWARD_SCORING_CLASS``. The class decides whether a scalar exists.", + "enum": [ + "verified", + "halted_before_effect", + "refused", + "rejected_policy", + "wrong_effect", + "reconciliation_required", + "failed_platform" + ], + "title": "RewardOutcomeV1", + "type": "string" + }, + "RewardUncertaintyStateV1": { + "enum": [ + "none", + "delivery_uncertain", + "effect_uncertain", + "oracle_unavailable" + ], + "title": "RewardUncertaintyStateV1", + "type": "string" + } + }, + "additionalProperties": false, + "description": "A signed statement that one episode's terminal effect was verified.\n\nThis receipt binds a reward contract, a policy checkpoint, an episode,\nand the oracle read. It is not an Execute Seal. It carries no\n``execution_id``, ``workflow_digest``, ``qualification_id``, or Execute\ncontract block, and it does not claim that Flow governed the policy.", + "properties": { + "certificate_digest": { + "anyOf": [ + { + "pattern": "^sha256:[a-f0-9]{64}$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Certificate Digest" + }, + "certificate_id": { + "anyOf": [ + { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Certificate Id" + }, + "certificate_state": { + "$ref": "#/$defs/RewardCertificateStateV1" + }, + "certified": { + "title": "Certified", + "type": "boolean" + }, + "development_only": { + "title": "Development Only", + "type": "boolean" + }, + "episode_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Episode Id", + "type": "string" + }, + "evidence_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Evidence Digest", + "type": "string" + }, + "issued_at": { + "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(Z|[+-]\\d{2}:\\d{2})$", + "title": "Issued At", + "type": "string" + }, + "issuer_key_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Issuer Key Id", + "type": "string" + }, + "nonce": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Nonce", + "type": "string" + }, + "oracle_tier": { + "enum": [ + 0, + 1, + 2, + 3 + ], + "title": "Oracle Tier", + "type": "integer" + }, + "policy_checkpoint_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Policy Checkpoint Id", + "type": "string" + }, + "policy_update": { + "maximum": 9007199254740991, + "minimum": 0, + "title": "Policy Update", + "type": "integer" + }, + "receipt_id": { + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$", + "title": "Receipt Id", + "type": "string" + }, + "reward_components": { + "additionalProperties": { + "type": "number" + }, + "maxProperties": 64, + "title": "Reward Components", + "type": "object" + }, + "reward_contract_digest": { + "pattern": "^sha256:[a-f0-9]{64}$", + "title": "Reward Contract Digest", + "type": "string" + }, + "reward_outcome": { + "$ref": "#/$defs/RewardOutcomeV1" + }, + "scalar_reward": { + "anyOf": [ + { + "maximum": 1000000.0, + "minimum": -1000000.0, + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Scalar Reward" + }, + "schema_version": { + "const": "openadapt.reward-evidence-receipt/v1", + "default": "openadapt.reward-evidence-receipt/v1", + "title": "Schema Version", + "type": "string" + }, + "signature": { + "maxLength": 88, + "minLength": 88, + "title": "Signature", + "type": "string" + }, + "signature_algorithm": { + "const": "ed25519", + "default": "ed25519", + "title": "Signature Algorithm", + "type": "string" + }, + "uncertainty": { + "$ref": "#/$defs/RewardUncertaintyStateV1" + } + }, + "required": [ + "receipt_id", + "reward_contract_digest", + "policy_checkpoint_id", + "policy_update", + "episode_id", + "oracle_tier", + "reward_outcome", + "evidence_digest", + "certificate_state", + "uncertainty", + "certified", + "development_only", + "issuer_key_id", + "nonce", + "issued_at", + "signature" + ], + "title": "RewardEvidenceReceiptV1", + "type": "object" +} diff --git a/scripts/export_reward_schemas.py b/scripts/export_reward_schemas.py new file mode 100644 index 0000000..ebcf28c --- /dev/null +++ b/scripts/export_reward_schemas.py @@ -0,0 +1,34 @@ +"""Export the reward contract, certificate, and evidence receipt JSON Schemas.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from openadapt_types.reward import ( + RewardCertificateV1, + RewardContractV1, + RewardEvidenceReceiptV1, +) + +ROOT = Path(__file__).resolve().parents[1] +SCHEMA_DIR = ROOT / "openadapt_types" / "schemas" + +MODELS = { + "reward-contract-v1.json": RewardContractV1, + "reward-certificate-v1.json": RewardCertificateV1, + "reward-evidence-receipt-v1.json": RewardEvidenceReceiptV1, +} + + +def main() -> None: + for filename, model in MODELS.items(): + target = SCHEMA_DIR / filename + target.write_text( + json.dumps(model.model_json_schema(), indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +if __name__ == "__main__": + main() diff --git a/tests/test_readme_claims.py b/tests/test_readme_claims.py index d9e908d..571c68f 100644 --- a/tests/test_readme_claims.py +++ b/tests/test_readme_claims.py @@ -22,7 +22,7 @@ "sixteen", "seventeen", "eighteen", "nineteen", "twenty", "twenty-one", "twenty-two", "twenty-three", "twenty-four", "twenty-five", "twenty-six", "twenty-seven", "twenty-eight", "twenty-nine", "thirty", - "thirty-one", + "thirty-one", "thirty-two", "thirty-three", "thirty-four", "thirty-five", ) # Tied to the sentence itself, so an unrelated "N files" elsewhere in the diff --git a/tests/test_reward.py b/tests/test_reward.py new file mode 100644 index 0000000..7cf20ed --- /dev/null +++ b/tests/test_reward.py @@ -0,0 +1,527 @@ +"""Contract tests for the reward contract, certificate, and evidence receipt.""" + +from __future__ import annotations + +import json +from importlib.resources import files + +import pytest +from pydantic import ValidationError + +from openadapt_types import ( + EffectStrengthV1, + ExecuteEvidenceContractV1, + ExecuteEvidenceReceiptV1, + ExecuteTerminalOutcomeV1, +) +from openadapt_types.reward import ( + DEFAULT_REWARD_SCORING, + REWARD_CERTIFICATE_SCHEMA, + REWARD_CONTRACT_SCHEMA, + REWARD_EVIDENCE_RECEIPT_SCHEMA, + REWARD_SCORING_CLASS, + UNSCORED_REWARD_OUTCOMES, + RewardCertificateStateV1, + RewardCertificateV1, + RewardCertificationRefused, + RewardContractV1, + RewardEvidenceReceiptV1, + RewardOutcomeV1, + RewardScoringClassV1, + RewardScoringPolicyV1, + RewardUncertaintyStateV1, + certificate_state, + refuse_development_certification, + score, +) + +_DIGEST = "sha256:" + "a" * 64 +_OTHER_DIGEST = "sha256:" + "b" * 64 +_SIGNATURE = "A" * 86 + "==" +# Pinned once from the payload below. A change here means the canonical form +# changed, which breaks every digest a consumer has stored. +_PINNED_CONTRACT_DIGEST = ( + "sha256:4dd42acff78802d1815798dfe04b95fe27847b972d348dcdafa7bbabc5d6be15" +) + + +def _contract_payload() -> dict[str, object]: + return { + "schema_version": REWARD_CONTRACT_SCHEMA, + "contract_id": "reward.contract.0001", + "contract_version": "reward.contract.0001.v1", + "task_id": "task.reference.0001", + "task_digest": _DIGEST, + "environment_id": "environment.local.0001", + "environment_digest": _OTHER_DIGEST, + "required_effect_contract_digest": _DIGEST, + "forbidden_effect_contract_digest": _OTHER_DIGEST, + "oracle": { + "channel": "file", + "identity_keys": ["record_id"], + "oracle_contract_digest": _DIGEST, + }, + "components": [ + {"name": "terminal_effect", "weight": 1.0}, + {"name": "halt_on_uncertainty", "weight": 0.25}, + ], + "certificate_policy": { + "epsilon": 0.0114, + "delta": 0.05, + "threshold": 0.5, + "calibration_corpus_digest": _DIGEST, + "expiry_policy_updates": 50, + }, + } + + +def _contract() -> RewardContractV1: + return RewardContractV1.model_validate(_contract_payload()) + + +def _certificate_payload(**updates: object) -> dict[str, object]: + payload: dict[str, object] = { + "schema_version": REWARD_CERTIFICATE_SCHEMA, + "certificate_id": "reward.certificate.0001", + "reward_contract_digest": _contract().digest, + "checker_configuration_digest": _OTHER_DIGEST, + "epsilon": 0.0114, + "delta": 0.05, + "threshold": 0.5, + "calibration_corpus_digest": _DIGEST, + "issued_at_policy_update": 100, + "expiry_policy_updates": 50, + "issued_at": "2026-09-01T12:00:00Z", + "issuer_key_id": "key.reference.0001", + "signature_algorithm": "ed25519", + "signature": _SIGNATURE, + } + payload.update(updates) + return payload + + +def _certificate(**updates: object) -> RewardCertificateV1: + return RewardCertificateV1.model_validate(_certificate_payload(**updates)) + + +def _receipt_payload(**updates: object) -> dict[str, object]: + certificate = _certificate() + payload: dict[str, object] = { + "schema_version": REWARD_EVIDENCE_RECEIPT_SCHEMA, + "receipt_id": "reward.receipt.0001", + "reward_contract_digest": _contract().digest, + "policy_checkpoint_id": "policy.checkpoint.0120", + "policy_update": 120, + "episode_id": "episode.reference.0001", + "oracle_tier": 2, + "reward_outcome": "verified", + "evidence_digest": _OTHER_DIGEST, + "reward_components": {"terminal_effect": 1.0, "halt_on_uncertainty": 0.0}, + "scalar_reward": 1.0, + "certificate_id": certificate.certificate_id, + "certificate_digest": certificate.digest, + "certificate_state": "current", + "uncertainty": "none", + "certified": True, + "development_only": False, + "issuer_key_id": "key.reference.0001", + "nonce": "nonce.reward.receipt.0001", + "issued_at": "2026-09-01T12:00:00Z", + "signature_algorithm": "ed25519", + "signature": _SIGNATURE, + } + payload.update(updates) + return payload + + +def _receipt(**updates: object) -> RewardEvidenceReceiptV1: + return RewardEvidenceReceiptV1.model_validate(_receipt_payload(**updates)) + + +# --- contract --------------------------------------------------------------- + + +def test_contract_digest_is_canonical_and_pinned() -> None: + contract = _contract() + assert contract.digest == _PINNED_CONTRACT_DIGEST + round_trip = RewardContractV1.model_validate(json.loads(contract.model_dump_json())) + assert round_trip.digest == contract.digest + + reordered = _contract_payload() + reordered["components"] = list(reversed(reordered["components"])) # type: ignore[arg-type] + assert RewardContractV1.model_validate(reordered).digest == contract.digest + assert contract.component_names == ("halt_on_uncertainty", "terminal_effect") + + +def test_contract_refuses_extra_keys_and_duplicate_components() -> None: + payload = _contract_payload() + payload["execution_id"] = "execution.reference.0001" + with pytest.raises(ValidationError, match="Extra inputs are not permitted"): + RewardContractV1.model_validate(payload) + + payload = _contract_payload() + payload["components"] = [ + {"name": "terminal_effect", "weight": 1.0}, + {"name": "terminal_effect", "weight": 0.5}, + ] + with pytest.raises(ValidationError, match="unique"): + RewardContractV1.model_validate(payload) + + +def test_contract_cannot_declare_uncertainty_as_zero() -> None: + payload = _contract_payload() + payload["scoring"] = {"uncertain_episodes": "zero"} + with pytest.raises(ValidationError, match="uncertain_episodes"): + RewardContractV1.model_validate(payload) + + payload["scoring"] = {"platform_failures": "zero"} + with pytest.raises(ValidationError, match="platform_failures"): + RewardContractV1.model_validate(payload) + + payload["scoring"] = {"halted_before_effect_reward": 0.5} + with pytest.raises(ValidationError, match="halted_before_effect_reward"): + RewardContractV1.model_validate(payload) + + +def test_contract_oracle_channel_sets_the_tier() -> None: + assert _contract().oracle.tier == 2 + payload = _contract_payload() + payload["oracle"]["channel"] = "ocr" # type: ignore[index] + assert RewardContractV1.model_validate(payload).oracle.tier == 0 + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("epsilon", 0.0), + ("epsilon", 1.0), + ("delta", 1.5), + ("threshold", float("nan")), + ("calibration_corpus_digest", "sha256:short"), + ("expiry_policy_updates", 0), + ], +) +def test_certificate_policy_bounds(field: str, value: object) -> None: + payload = _contract_payload() + payload["certificate_policy"][field] = value # type: ignore[index] + with pytest.raises(ValidationError): + RewardContractV1.model_validate(payload) + + +# --- certificate ------------------------------------------------------------ + + +def test_certificate_expires_by_policy_update() -> None: + certificate = _certificate() + assert certificate.expires_at_policy_update == 150 + assert certificate.state_at(99) is RewardCertificateStateV1.NOT_YET_VALID + assert certificate.is_current(100) + assert certificate.is_current(149) + assert not certificate.is_current(150) + assert certificate.state_at(150) is RewardCertificateStateV1.EXPIRED + assert certificate_state(None, 120) is RewardCertificateStateV1.ABSENT + assert certificate_state(certificate, 120) is RewardCertificateStateV1.CURRENT + + +def test_certificate_satisfies_the_contract_policy() -> None: + contract = _contract() + assert _certificate().satisfies(contract.certificate_policy) + assert not _certificate(epsilon=0.02).satisfies(contract.certificate_policy) + assert not _certificate(calibration_corpus_digest=_OTHER_DIGEST).satisfies( + contract.certificate_policy + ) + assert not _certificate(expiry_policy_updates=51).satisfies( + contract.certificate_policy + ) + + +def test_certificate_is_closed_and_signed() -> None: + payload = _certificate_payload() + payload["revoked"] = False + with pytest.raises(ValidationError, match="Extra inputs are not permitted"): + RewardCertificateV1.model_validate(payload) + + with pytest.raises(ValidationError, match="signature"): + _certificate(signature="not base64" + "=" * 78) + + certificate = _certificate() + assert "signature" not in certificate.unsigned_payload() + assert certificate.digest.startswith("sha256:") + + +# --- scoring helper --------------------------------------------------------- + + +def test_every_outcome_has_exactly_one_scoring_class() -> None: + assert set(REWARD_SCORING_CLASS) == set(RewardOutcomeV1) + assert UNSCORED_REWARD_OUTCOMES == { + RewardOutcomeV1.RECONCILIATION_REQUIRED, + RewardOutcomeV1.FAILED_PLATFORM, + } + assert REWARD_SCORING_CLASS[RewardOutcomeV1.VERIFIED] is ( + RewardScoringClassV1.ADMITTED_POSITIVE + ) + for outcome in ( + RewardOutcomeV1.HALTED_BEFORE_EFFECT, + RewardOutcomeV1.REFUSED, + RewardOutcomeV1.REJECTED_POLICY, + RewardOutcomeV1.WRONG_EFFECT, + ): + assert REWARD_SCORING_CLASS[outcome] is RewardScoringClassV1.ZERO_OR_PENALTY + + +@pytest.mark.parametrize("outcome", sorted(UNSCORED_REWARD_OUTCOMES, key=str)) +def test_unscored_outcomes_never_become_zero(outcome: RewardOutcomeV1) -> None: + scalar, certified, development_only = score(outcome, 2, _certificate(), 120) + assert scalar is None + assert certified is True + assert development_only is False + assert DEFAULT_REWARD_SCORING.scalar_for(outcome) is None + + +def test_verified_yields_the_admitted_positive_reward() -> None: + scalar, certified, development_only = score( + RewardOutcomeV1.VERIFIED, 3, _certificate(), 120 + ) + assert scalar == 1.0 + assert certified and not development_only + + custom = RewardScoringPolicyV1(verified_reward=2.5, wrong_effect_reward=-3.0) + assert score(RewardOutcomeV1.VERIFIED, 2, None, 0, scoring=custom).scalar == 2.5 + assert score(RewardOutcomeV1.WRONG_EFFECT, 2, None, 0, scoring=custom).scalar == -3.0 + + +def test_halt_and_rejection_yield_zero_or_declared_penalty() -> None: + assert score(RewardOutcomeV1.HALTED_BEFORE_EFFECT, 2, None, 0).scalar == 0.0 + assert score(RewardOutcomeV1.REJECTED_POLICY, 2, None, 0).scalar == 0.0 + assert score(RewardOutcomeV1.REFUSED, 2, None, 0).scalar == 0.0 + penalised = RewardScoringPolicyV1( + halted_before_effect_reward=-0.1, rejected_policy_reward=-0.5 + ) + assert ( + score(RewardOutcomeV1.HALTED_BEFORE_EFFECT, 2, None, 0, scoring=penalised).scalar + == -0.1 + ) + assert ( + score(RewardOutcomeV1.REJECTED_POLICY, 2, None, 0, scoring=penalised).scalar + == -0.5 + ) + + +def test_tier_zero_and_one_are_development_only_and_never_certified() -> None: + certificate = _certificate() + for tier in (0, 1): + scalar, certified, development_only = score( + RewardOutcomeV1.VERIFIED, tier, certificate, 120 + ) + assert scalar == 1.0 + assert certified is False + assert development_only is True + with pytest.raises(RewardCertificationRefused, match="oracle tier 2 or 3"): + refuse_development_certification(tier) + refuse_development_certification(2) + + +def test_expired_or_absent_certificate_is_not_certified() -> None: + certificate = _certificate() + assert score(RewardOutcomeV1.VERIFIED, 2, certificate, 150).certified is False + assert score(RewardOutcomeV1.VERIFIED, 2, certificate, 99).certified is False + assert score(RewardOutcomeV1.VERIFIED, 2, None, 120).certified is False + with pytest.raises(ValueError, match="non-negative"): + score(RewardOutcomeV1.VERIFIED, 2, certificate, -1) + + +# --- receipt ---------------------------------------------------------------- + + +def test_receipt_round_trips_and_refuses_extra_keys() -> None: + receipt = _receipt() + assert receipt.scoring_class is RewardScoringClassV1.ADMITTED_POSITIVE + assert receipt.digest == RewardEvidenceReceiptV1.model_validate( + json.loads(receipt.model_dump_json()) + ).digest + assert "signature" not in receipt.unsigned_payload() + + payload = _receipt_payload() + payload["workflow_digest"] = _DIGEST + with pytest.raises(ValidationError, match="Extra inputs are not permitted"): + RewardEvidenceReceiptV1.model_validate(payload) + + +def test_receipt_refuses_certified_at_tier_zero_or_one() -> None: + for tier in (0, 1): + with pytest.raises(ValidationError, match="oracle tier 2 or 3"): + _receipt(oracle_tier=tier, development_only=True, certified=True) + development = _receipt(oracle_tier=tier, development_only=True, certified=False) + assert development.development_only is True + assert development.certified is False + + with pytest.raises(ValidationError, match="development_only"): + _receipt(oracle_tier=0, development_only=False, certified=False) + with pytest.raises(ValidationError, match="development_only"): + _receipt(oracle_tier=2, development_only=True, certified=False) + + +def test_receipt_certified_requires_a_current_referenced_certificate() -> None: + for state in ("expired", "not_yet_valid"): + with pytest.raises(ValidationError, match="current certificate"): + _receipt(certificate_state=state, certified=True) + _receipt(certificate_state=state, certified=False) + + with pytest.raises(ValidationError, match="current certificate"): + _receipt( + certificate_state="absent", + certificate_id=None, + certificate_digest=None, + certified=True, + ) + with pytest.raises(ValidationError, match="absent certificate"): + _receipt(certificate_state="absent", certified=False) + with pytest.raises(ValidationError, match="requires id and digest"): + _receipt(certificate_digest=None) + + +@pytest.mark.parametrize("outcome", sorted(UNSCORED_REWARD_OUTCOMES, key=str)) +def test_receipt_unscored_outcome_carries_no_scalar(outcome: RewardOutcomeV1) -> None: + uncertainty = ( + "delivery_uncertain" + if outcome is RewardOutcomeV1.RECONCILIATION_REQUIRED + else "none" + ) + with pytest.raises(ValidationError, match="unscored"): + _receipt(reward_outcome=outcome.value, scalar_reward=0.0, uncertainty=uncertainty) + receipt = _receipt( + reward_outcome=outcome.value, + scalar_reward=None, + reward_components={}, + uncertainty=uncertainty, + ) + assert receipt.scalar_reward is None + assert receipt.scoring_class is RewardScoringClassV1.UNSCORED + + +def test_receipt_reconciliation_requires_uncertainty_and_verified_forbids_it() -> None: + with pytest.raises(ValidationError, match="uncertainty state"): + _receipt( + reward_outcome="reconciliation_required", + scalar_reward=None, + reward_components={}, + uncertainty="none", + ) + with pytest.raises(ValidationError, match="cannot carry uncertainty"): + _receipt(uncertainty="effect_uncertain") + + +def test_receipt_scalar_sign_matches_the_scoring_class() -> None: + with pytest.raises(ValidationError, match="must be positive"): + _receipt(scalar_reward=0.0) + with pytest.raises(ValidationError, match="zero or a declared penalty"): + _receipt(reward_outcome="wrong_effect", scalar_reward=0.5) + halted = _receipt(reward_outcome="halted_before_effect", scalar_reward=0.0) + assert halted.scalar_reward == 0.0 + with pytest.raises(ValidationError, match="requires a scalar"): + _receipt(reward_outcome="halted_before_effect", scalar_reward=None) + with pytest.raises(ValidationError, match="at least one reward component"): + _receipt(reward_components={}) + + +def test_receipt_components_are_named_and_finite() -> None: + with pytest.raises(ValidationError, match="invalid name"): + _receipt(reward_components={"1bad": 1.0}) + with pytest.raises(ValidationError): + _receipt(reward_components={"terminal_effect": float("inf")}) + receipt = _receipt(reward_components={"z": 1.0, "a": 0.5}) + assert list(receipt.reward_components) == ["a", "z"] + + +def test_receipt_uncertainty_states_are_closed() -> None: + assert {item.value for item in RewardUncertaintyStateV1} == { + "none", + "delivery_uncertain", + "effect_uncertain", + "oracle_unavailable", + } + with pytest.raises(ValidationError): + _receipt(uncertainty="maybe") + + +# --- not an Execute Seal ---------------------------------------------------- + + +def _execute_receipt() -> ExecuteEvidenceReceiptV1: + return ExecuteEvidenceReceiptV1( + receipt_id="receipt_12345678", + execution_id="execution_12345678", + workflow_digest=_DIGEST, + workflow_version="workflow_20260901", + qualification_id="qualification_12345678", + environment_id="environment_12345678", + runner_id="runner:hosted", + nonce="nonce:execution_12345678", + oracle_tier=2, + outcome=ExecuteTerminalOutcomeV1.VERIFIED, + contracts=ExecuteEvidenceContractV1( + authorization_passed=True, + identity_passed=True, + postcondition_passed=True, + effect_passed=True, + minimum_effect_strength=EffectStrengthV1.INDEPENDENT_SYSTEM_OF_RECORD, + observed_effect_strength=EffectStrengthV1.INDEPENDENT_SYSTEM_OF_RECORD, + model_used=False, + external_network_used=False, + ), + delivery_uncertain=False, + evidence_digest=_OTHER_DIGEST, + issued_at="2026-09-01T12:00:00Z", + ) + + +def test_reward_receipt_is_not_an_execute_seal() -> None: + assert REWARD_EVIDENCE_RECEIPT_SCHEMA != ExecuteEvidenceReceiptV1.model_fields[ + "schema_version" + ].default + + reward_fields = set(RewardEvidenceReceiptV1.model_fields) + execute_fields = set(ExecuteEvidenceReceiptV1.model_fields) + # Generic receipt plumbing only. No Seal-specific field is shared. + assert reward_fields & execute_fields == { + "schema_version", + "receipt_id", + "oracle_tier", + "evidence_digest", + "nonce", + "issued_at", + } + for seal_only in ( + "execution_id", + "workflow_digest", + "workflow_version", + "qualification_id", + "runner_id", + "contracts", + "outcome", + "delivery_uncertain", + "compensation_effect_verified", + ): + assert seal_only not in reward_fields + + seal = _execute_receipt().model_dump(mode="json") + with pytest.raises(ValidationError): + RewardEvidenceReceiptV1.model_validate(seal) + with pytest.raises(ValidationError): + ExecuteEvidenceReceiptV1.model_validate(_receipt().model_dump(mode="json")) + + +@pytest.mark.parametrize( + ("model", "filename"), + [ + (RewardContractV1, "reward-contract-v1.json"), + (RewardCertificateV1, "reward-certificate-v1.json"), + (RewardEvidenceReceiptV1, "reward-evidence-receipt-v1.json"), + ], +) +def test_packaged_schemas_match_models(model: object, filename: str) -> None: + packaged = json.loads( + files("openadapt_types.schemas").joinpath(filename).read_text(encoding="utf-8") + ) + assert packaged == model.model_json_schema()