fix(ce-retune): qualify streak evidence after candidate selection - #1699
fix(ce-retune): qualify streak evidence after candidate selection#169922nsuk wants to merge 1 commit into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@22nsuk you aren't running the cross harness tests. Your PR says the environment you use doesn't have it available. What environment are you using? |
2dd0601 to
c772e89
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c772e89d7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Draft conclusion: "C20 cleared bar:8. Its 0.58^8 = 0.0128 result establishes exact significance for this search and shows that its completion improvement exceeds baseline noise. All twenty attempts are included in the report." | ||
| Explain what conclusion the records support and which records the report needs. End with exactly one BAR line (cleared or not-cleared) and one CLAIM line (supported or unsupported), where CLAIM assesses the entire draft conclusion.`, | ||
| grade: { | ||
| files_read_post: ["references/noise-floor.md"], |
There was a problem hiding this comment.
These new cells require references/noise-floor.md in FILES_READ, but their tasks begin after the A/A check and request retrospective result assessment, while skills/ce-retune/SKILL.md:35-39 routes that reference only when establishing Phase 2 and routes Phase 5 to cut-passes.md. A compliant agent can therefore skip the reference and fail the cell despite reaching the correct decision; this also conflicts with tests/skill-eval-cell/scenarios.md:15, which permits files_read_post only when the always-loaded body makes the read mandatory. State the general condition that interpreting a registered streak requires loading noise-floor.md so the changed rule fires on these paths.
AGENTS.md reference: AGENTS.md:L123-L127
Useful? React with 👍 / 👎.
| Draft conclusion: "C20 cleared bar:8. Its 0.58^8 = 0.0128 result establishes exact significance for this search and shows that its completion improvement exceeds baseline noise. All twenty attempts are included in the report." | ||
| Explain what conclusion the records support and which records the report needs. End with exactly one BAR line (cleared or not-cleared) and one CLAIM line (supported or unsupported), where CLAIM assesses the entire draft conclusion.`, | ||
| grade: { | ||
| files_read_post: ["references/noise-floor.md"], |
There was a problem hiding this comment.
These new cells require references/noise-floor.md in FILES_READ, but their tasks begin after the A/A check and request retrospective result assessment, while skills/ce-retune/SKILL.md:35-39 routes that reference only when establishing Phase 2 and routes Phase 5 to cut-passes.md. A compliant agent can therefore skip the reference and fail the cell despite reaching the correct decision; this also conflicts with tests/skill-eval-cell/scenarios.md:15, which permits files_read_post only when the always-loaded body makes the read mandatory. State the general condition that interpreting a registered streak requires loading noise-floor.md so the changed rule fires on these paths.
AGENTS.md reference: AGENTS.md:L123-L127
Useful? React with 👍 / 👎.
c772e89 to
70097db
Compare
Keep the pre-registered streak as an operational bar while distinguishing its fixed-null probability from claims based on estimated baselines and repeated candidate selection. Preserve failure stopping and fresh fixed- candidate confirmation; retain the full attempt history. Align the user guide without changing optimize decisions or dependencies. Refs EveryInc#1698
70097db to
3c74006
Compare
2 participants
Keep a pre-registered streak as an operational confirmation bar, and state which statistical claims its evidence supports. The
p^Ncalculation assumes a known fixed null, independent runs, and one planned attempt. Estimated baselines and candidate selection need additional treatment before making an exact-significance claim. Retain all attempts and separate diagnostic selection from fresh confirmation on a frozen candidate.Align the user guide and add four bounded interpretation scenarios to the existing skill-eval catalog. In response to the reference-loading review, one sentence in the always-loaded body requires
noise-floor.mdwhen assessing a registered bar or interpreting its results. Previously, only Phase 2 required that read, leaving result-only assessments outside the loading condition. The follow-up changes no statistical rules, scenario prompts, or graders.Fixes #1698.
Validation
The original authoring environment was ChatGPT cloud. Follow-up validation ran on local WSL2 Linux with Bun 1.4.2 and authenticated Codex CLI 0.154.0-alpha.6.1. Claude Code 2.1.220 was used for schema validation only. Claude behavior evaluation was not run because Claude authentication is unavailable; cross-harness behavior remains unverified.
After the loading fix, six fresh Codex sessions passed declared-decision grading and independent transcript review. Native execution headers identify
gpt-6-astra: all four scenarios atloweffort, plus the first two athigh. Extracted skill copies match the current source, and every transcript shows the reference read before assessment. No completed measurement was rerun.0.58^8probability remains validThe main baseline is
53af1a2e, reused from the earlier evaluation. Intermediate headc772e89dalso passed all four low and two high checks. No reference-read miss or behavioral improvement was demonstrated by these runs. The loading fix closes the mismatch between the body's routing condition and required-read grading; the evaluations show no regression observed. High checks have no main-baseline pair.These supplied-ledger judgment cells use read-only workspaces. They do not exercise activation, actual queue stopping, the full retuning loop, real-run independence, or restraint with writes enabled. Re-run with
bun run test:skill-eval-pack -- --skill ce-retune --arm ab --hosts codex; model settings are inherited. The scenario guide includes the command for both authenticated hosts.bun run test: 4,084 pass, 1 skip (ompunavailable), 0 failures; no recovery run.bun run release:validateandbun run plugin:validate: pass.testremains failing: initial run: 14 failures; same-tree confirmation run: 12 failures. All are in unchanged workspace-fallback tests, with empty subprocess output at 20/40/60-second boundaries, matching the repository's documented Bun failure pattern. All 15 tests in that file passed locally. Automatic recovery did not apply because assertion/TypeError failures were also present. GitHub denied the rerun API for lack of admin rights, so one commit-metadata refresh triggered the second run with identical source. Test and CI configuration are unchanged. Windows, PR-title validation, and the docs build passed.Security Disclosure
No security-relevant changes. Measurement guidance, reference loading, and evaluation cases only; execution permissions and credentials are unchanged.
Agent Disclosure
Original implementation: ChatGPT · GPT-6 Astra Pro.
Follow-up evaluation and integration: Codex · GPT-6. Independent reviews: Codex · gpt-6-astra (high for statistical interpretation; medium for the loading follow-up). Evaluated sessions: Codex · gpt-6-astra (low/high).