Skip to content

Correlate Codex turn dispatches with the turn the turn/start response names - #3503

Open
SawyerHood wants to merge 3 commits into
bb/3462-fix-stop-leaving-threads-stuck-thr_vdv2rx6c7wfrom
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w
Open

Correlate Codex turn dispatches with the turn the turn/start response names#3503
SawyerHood wants to merge 3 commits into
bb/3462-fix-stop-leaving-threads-stuck-thr_vdv2rx6c7wfrom
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

Stacked on #3502 (merge that first). This layer removes the trigger behind #3462: the Codex bridge answered turn/start, then 250 ms later, if no Codex turn had opened yet, fabricated a completed zero-work turn (scheduleZeroWorkTurnSettlement). When the real turn/started arrived after that grace period, which happened on every one of the four production threads in #3462 (thread startup took ~40 s under load), the server saw a finished turn and the real work then ran as an unrequested root turn with no turn/input/accepted. That is #2580 (with #2634 folded in). The app-server's turn/start response already names the turn (TurnStartResponse { turn: { id, status } }), and the bridge discarded it.

What changed

This is Ratul Sarna's #2639, cherry-picked onto the #3502 branch with the protocol pin resolved to 205; the approach is unchanged and credited in the commit.

  • plugins/provider-codex/src/bridge/bridge.ts: turn/start and thread/compact/start results are parsed; when the response names a turn, settleAcceptedDispatch opens that turn from the response (or only records input.accepted if the notification already opened it). The timer-based zero-work settlement now runs only for a turn/start result that names no turn.
  • plugins/provider-codex/src/translator.ts: openTurnFromStartResponse emits the lifecycle from the response and dedupes the later turn/started/turn/completed notifications for that turn id; a response whose turn is already settled emits both boundaries.
  • plugins/provider-codex/src/schemas.ts: exports codexTurnSchema.
  • fake-codex-app-server.mjs: late start delay raised past the old grace period; new scripted prompts for respond-then-exit, respond-completed, steer-into-active, interrupt-before-start; compaction runs as a delayed turn, with env-selected modes for an idle report without a turn and an exit before the turn starts; a /late-start-interruptible prompt rejects an interrupt before turn/started like a real app-server.
  • Compaction (added on top of Correlate codex turn dispatches with the turn the turn/start response names #2639): thread/compact/start answers {} and codex then opens the compaction as a turn (192 ms later in the committed real recording; a real app-server opens one even with nothing to compact and ends it failed). A compaction dispatch now waits for its turn/started and settles synthetically only on a signal from codex: thread/status/changed to idle (completed) or systemError (failed) while no turn is open, or the app-server exiting first (failed). No timer decides compaction.
  • openTurnFromStartResponse no longer records a turn id in the already-open branch, where no later notification would remove it.
  • Interrupts: opening a turn from the response exposes it before codex activates it. A real app-server answers turn/start with {turn: inProgress} and emits turn/started a few ms later; a turn/interrupt in that gap fails with no active turn to interrupt, so a stop right after bb saw the turn failed (the off-pipeline @bb/agent-runtime codex integration test "stops an active turn and recovers with a follow-up" hit it under load). The bridge now tracks response-opened turns and, only when an interrupt of one is rejected, waits for codex's own turn/started (or the turn settling or the app-server exiting, within the interrupt settlement timeout) and interrupts once more.
  • HOST_DAEMON_PROTOCOL_VERSION 204 → 205 (on top of Reconcile daemon turns the server lost track of on explicit stop #3502's 204).

How you verified

  • bridge.zero-work-turn.test.ts: the exactly-once late-start test now runs with a 350 ms delay and fails on main (two starts), plus new cases for a response followed by app-server exit, a response already completed, a steer into the active turn, and an interrupt before the notification.
  • pnpm exec turbo run typecheck --filter=bb-plugin-provider-codex --filter=@bb/host-daemon-contract and turbo run test for the same packages: codex plugin 276/276 (27 files), contract 55/55, typecheck green. With bridge.ts, translator.ts and schemas.ts swapped back to the Reconcile daemon turns the server lost track of on explicit stop #3502 base, five of the seven zero-work tests fail (the late-start exactly-once case and the four new ones) and the two pre-existing ones pass.
  • Live dev app with real Codex (codex CLI 0.154.0, device-auth login in an isolated scratch CODEX_HOME) behind a documented delay relay around the real codex app-server (an untracked codex shim holds turn/started and following notifications 400 ms while responses pass): on origin/main and on the Reconcile daemon turns the server lost track of on explicit stop #3502 base, every prompt produced two root turns within the same second, a synthetic one with turn/input/accepted and an instant completed, then the real turn with no accepted input (Codex threads 01a09199-7ab8…, 01a0919a-0f7e…). On this layer, under the same relay, three consecutive real turns (app-server ids 01a0919a-b048…, 01a0919b-7d59…, 01a0919b-b5b4…) each produced exactly one lifecycle with its accepted input, and a follow-up sent right after the first completion steered into the running turn instead of being fabricated or refused.
  • Compaction regressions in bridge.zero-work-turn.test.ts: a turn/started 600 ms after the empty compact result keeps one lifecycle on the real turn; an idle status report without a turn settles it completed; an app-server exit before the turn settles it failed. With bridge.ts and translator.ts swapped back to the previous layer head, the 600 ms and exit cases fail (two lifecycles; a false completed), and the idle case passed there too (the old timer also completed it).
  • On the rebased head: turbo run typecheck lint and turbo run test for bb-plugin-provider-codex (288), @bb/provider-bridge-protocol (238), @bb/provider-parity (56, including the Codex replay), @bb/host-daemon-contract (56); format check clean; EAP codename scan clean for the push range.
  • Real Codex, delayed compaction (same scratch login, relay holding turn/started 800 ms): at the commit before the compaction fix, bb thread compact produced a synthetic turn accepted and completed in the same millisecond, then the real compaction as an unrequested second root turn with the contextCompaction item (real app-server turn 01a09269-fbad…). With the fix, the same request produced one root turn with the accepted input, the contextCompaction item and the real completion (real app-server turn 01a0926a-a9e6…).
  • Interrupt regression in bridge.zero-work-turn.test.ts: a stop sent as soon as the response-opened turn is seen succeeds and settles interrupted; with bridge.ts swapped back to the previous layer head it fails with the -32000 no active turn to interrupt error.
  • Real Codex, interrupt in the activation gap: probing the app-server directly, turn/interrupt sent right after the turn/start response returned -32600 no active turn to interrupt and the same call after turn/started succeeded. Driving the bridge against the real app-server and stopping each turn as soon as bb saw it started: on the previous head 1 of 3 stops failed with that error (Codex turn 01a092a5-9c02…); with the fix all 3 succeeded (01a092ac-269e…, 01a092ad-1391…, 01a092ae-0076…). The probe was a temporary test file and is not committed. With the fix, the @bb/agent-runtime codex integration file passes against real Codex (8 of 8, including the stop-and-recover and steer tests).

Fixes #2580
Closes #2639

🤖 Generated with Claude Code

AGENT GENERATED

@SawyerHood
SawyerHood added this pull request to stack #3504 September 11, 2026 17:30
@SawyerHood
SawyerHood force-pushed the bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w branch 2 times, most recently from 7cc1ee3 to 54cafa4 Compare September 11, 2026 17:34
@SawyerHood
SawyerHood force-pushed the bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w branch 2 times, most recently from 96f54be to 4c2be65 Compare September 11, 2026 17:48
@SawyerHood
SawyerHood force-pushed the bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w branch from 4c2be65 to 9c41096 Compare September 11, 2026 21:41
ratulsarna and others added 3 commits September 11, 2026 23:02
… names

The bridge ignored the turn/start result and guessed at settlement with a
250 ms timer. When codex answered before emitting turn/started and the
notification lagged past the grace window, the timer fabricated a completed
zero-work turn and the real turn arrived later as a second lifecycle, so a
parent could see its child complete with no output while the child was still
working (#2580).

Every supported codex (0.136.0 up) answers turn/start with the turn it
opened. The bridge now opens that turn from the response, attaches the
accepted input to it, and suppresses the provider's own later turn/started.
A response reporting an already-terminal turn settles the full lifecycle at
once, with the later notifications suppressed the same way. A dispatch codex
steers into the already-running turn only acknowledges the input against
that turn. The zero-work timer remains for results that name no turn
(thread/compact/start's empty result and a malformed body). A crash between
the answer and the notification now settles the real turn as failed instead
of fabricating a completed one.

Cherry-picked from #2639 (Ratul Sarna) onto the #3462 fix; the host-daemon
protocol version moves from #3462's 204 to 205.

Co-Authored-By: Claude Code <noreply@anthropic.com>
thread/compact/start answers with an empty result, and codex then opens the
compaction as a turn: the committed real recording shows turn/started 192 ms
after the {} result, and a real app-server opens a turn even when there is
nothing to compact (it ends as failed with a systemError status). The bridge
sent compaction through the 250 ms zero-work timer, so a turn/started that
lagged past the grace window produced a fabricated completed turn followed
by the real compaction as an unrequested turn, the #2580 pattern.

A compaction dispatch now stays unopened until its turn/started claims it.
It settles synthetically only on a signal from codex: a thread/status/changed
to idle or systemError while no turn is open (completed or failed), or the
app-server exiting before the turn starts (failed). The zero-work timer is
kept only for a turn/start result that names no turn.

The fake app-server now runs compaction as a delayed turn, with modes for an
idle report without a turn and an exit before the turn starts, and the
regressions cover a turn/started 600 ms after the empty result, both signal
paths, and the exit. The already-open branch of openTurnFromStartResponse no
longer records a turn id that no later notification would remove.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Opening a turn from the turn/start response exposes it to bb before codex
activates it. A real app-server answers turn/start with {turn: inProgress},
then emits turn/started a few milliseconds later, and a turn/interrupt sent
in that gap fails with "no active turn to interrupt". A stop that arrived
right after bb saw the turn therefore failed, which the agent-runtime codex
integration test "stops an active turn and recovers with a follow-up" hit
under load.

The bridge now remembers which turns it opened from a response and whether
codex's own turn/started has arrived. When an interrupt of such a turn is
rejected, it waits for that notification (or for the turn to settle or the
app-server to exit, within the interrupt settlement timeout) and interrupts
again; it does not retry any other failure. The fake app-server gains a
prompt that rejects an interrupt before turn/started like a real
app-server, and a regression covers the stop.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@SawyerHood
SawyerHood force-pushed the bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7w branch from df67e58 to ebcd0e5 Compare September 11, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex bridge fabricates a completed turn when turn/started is delayed

2 participants