Skip to content

Keep resumed threads in their own provider sessions - #3512

Open
SawyerHood wants to merge 4 commits into
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7wfrom
bb/3461-owned-sessions-stack
Open

Keep resumed threads in their own provider sessions#3512
SawyerHood wants to merge 4 commits into
bb/2580-codex-turn-start-correlation-thr_vdv2rx6c7wfrom
bb/3461-owned-sessions-stack

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

A thread whose stored history named another conversation's provider session could resume into that conversation. The user's message landed in the other rollout, and the rightful owner was then locked out on Codex's writer lock (#3461). Merged #3496 stops the runtime from minting fresh mix-ups. However, the server still picked the newest session id stamped on any stored event, and contaminated history from before #3496 still steered resume after a restart, message edits, and anchored forks. Review of the first version of this PR found two more gaps. When two threads claimed a session in the same millisecond, the owner was chosen by rowid, which is not a durable ordering. A thread whose only recorded session belonged to another thread silently started a new session with no context.

What changed

Third layer of the stack: #3502 (Stop recovery) → #3503 (Codex turn correlation) → this PR. Review this diff against #3503's branch. The superseded runtime and writer-retry commits from #3460 are not included.

  • Resolution (packages/db): sessions come only from thread/identity events recorded since the thread's last completed context clear, newest first. Each announced session is compared with every same-provider, same-host claim.
    • owned: no other thread claimed it earlier. The thread resumes it.
    • foreign: another thread claimed it strictly earlier. Resolution skips it and falls back to the thread's own earlier session, which is the incident repair.
    • ambiguous: another thread's earliest claim falls in the same millisecond. Daemon event batches share one timestamp and rowid order is not durable, so resolution stops instead of guessing.
  • Dispatch (apps/server): sends, queued dispatch, turn submission, and goal clears go through one check. No session since the last clear starts fresh as before. A foreign or ambiguous session returns 409 provider_session_unavailable, and details carries the reason, session id, and claimant threads. Nothing is dispatched and no history is written. A queued row records the same message as its failure reason. Message-edit rewinds and anchored or tip forks refuse a foreign or ambiguous completion or source.
  • Remedy: the refusal says "Clear context (/clear or bb thread clear) for a new session; history is kept." The bb-cli failure-recovery reference documents it.
  • Conformance kit: new checks session/start-identity-announced and session/fork-identity-announced. A bridge now fails if it returns a session but never announces it through thread/identity, because bb would start a new session on every send. Every in-repo and example bridge already passes, and their expected-status maps pin both checks.
  • Session handle contract (docs/provider-bridge-protocol.md): the doc now states what resume already relied on. bb sends a providerThreadId to a new bridge process after a restart, so it must name one session among all of that provider's sessions on the host and never come from a per-process counter. Real bridges mint UUIDs, and the published echo example adds a random per-process nonce. The scripted test bridge keeps prov-N counters for single-process suites. It gains uniqueProviderThreadIds, which the integration suite enables because it runs several bridge processes against one store. The first CI run caught exactly that collision, so this fix is in scope.
  • Storage: empty-input fork starts record the daemon's start result as a thread/identity event. Drizzle generated migration 0118, a partial identity-event index used by the claim lookup. Its SQL uses IF NOT EXISTS, like 0107, so rewound migration tests can replay it.
  • Not changed: no daemon wire change, so the protocol stays at Correlate Codex turn dispatches with the turn the turn/start response names #3503's 205. The SDK version stays at main's unpublished 0.4.84. Existing history is not rewritten.

Known limits:

  • Purged owner: ownership holds for archived and deleted-but-present threads. If every record of the original owner is purged, a stale foreign claim can become claimable again.
  • Archived production pairs: on a copy of bee's store, pairs thr_un673bkbeu/thr_k4xuhczhkg and thr_dg7mhskt57/thr_pnjzgs3pz9 claim a shared session in the same millisecond. All four threads are archived. They now get the explicit refusal until their context is cleared. The other two production pairs have strictly ordered claims and resolve to each thread's own session.
  • Context lost on fallback: when falling back from a later foreign claim, the agent does not remember turns that were written into the other thread's rollout.
  • Counter-minted third-party ids: a third-party bridge that mints per-process counter ids breaks the documented contract. With this PR, its threads in different environments, or started after a bridge restart, can be refused. Every in-repo and example bridge mints unique handles.

How you verified

Real authenticated Codex (codex-cli 0.154.0), isolated fresh stores and ports, shared device-auth scratch CODEX_HOME. One script ran against both heads. It spawns five threads on one environment, runs edit, fork, and clear, and then makes the injections below. After that it restarts the daemon so every send resolves from the store, and it checks which rollout file received each message.

Scenario Before: published #3512 head 226a657864 After: this head
Tied claim, rowid order inverted B resumed into A's session; "beta-tie" appeared in A's rollout. A silently got a new session whose rollout lacks A's original prompt. A and B both answered 409 (ambiguous). No new session; A's rollout received neither message.
Only-foreign identity C silently got a new session whose rollout lacks C's original prompt. 409 (foreign); no new session.
Incident: later foreign identity D resumed its own post-edit session. D resumed its own post-edit session.
Remedy (bb thread clear, then send) New session works. New session works; the message went to the new rollout only.
Edit/rewind, tip fork + send, clear + send All work; each thread gets its own session. All work; each thread gets its own session.

These rows were injected on purpose, because #3496 prevents the original fresh trigger:

  • Tied claim: B got A's session id at A's exact first-claim millisecond (a real 13-digit timestamp) with a lower rowid. That mirrors the archived same-batch pairs, with the storage order rowid doesn't guarantee inverted.
  • Only-foreign: C's identity rows were rewritten to name A's session.
  • Incident: D got a later foreign identity.

The earlier run for the first version of this PR used a timestamp about 100,000× too large, from date +%s%3N printing nanoseconds on this host. These runs take timestamps from the store. The incident's fail-before on main, where A resumed into B's rollout and B hit the active-writer error, is from that earlier run. Main's session selector is unchanged since then.

Tests (fail before, pass after where they cover changed behavior):

  • db: production-shaped same-millisecond pairs, a 140 ms-apart pair, ambiguity stopping at the newest identity, a thread outside a tie, context clear, provider/host scoping, batch resolution, and the query plan using the new index.
  • server: foreign fallback resume; foreign-only refusal followed by /clear starting fresh; ambiguous refusal for both threads with history byte-identical; the queued failure reason; ambiguous edit, anchored fork, and tip fork refusals; Fix provider session identity mixups during concurrent thread starts #3496's checkpoint-identity suite.
  • conformance: a missing announcement, a mismatched announcement, and a passing stub, plus all bridge suites.

Commands on this head, rebased on #3503 ebcd0e5c7a, which carries #3502's SDK bump to 0.4.85:

pnpm exec turbo run typecheck lint --filter=@bb/db --filter=@bb/server --filter=@bb/provider-bridge-protocol --filter=@get-bb/plugin-sdk --filter=bb-plugin-provider-codex --filter=bb-plugin-provider-claude-code --filter=bb-plugin-provider-pi --filter=@bb/provider-bridge-acp --filter=bb-plugin-bb-guide
pnpm exec turbo run test --filter=@bb/db --filter=@bb/server --filter=@bb/provider-bridge-protocol --filter=@get-bb/plugin-sdk --filter=bb-plugin-provider-codex --filter=bb-plugin-provider-claude-code --filter=bb-plugin-provider-pi --filter=@bb/provider-bridge-acp --filter=bb-plugin-scripted-echo-provider --filter=bb-plugin-echo-provider --filter=bb-plugin-bb-guide --force
pnpm exec turbo run test --filter=@bb/integration-tests --filter=@bb/agent-runtime --filter=@bb/host-daemon --force
node packages/plugin-sdk/scripts/check-npm-version-guard.mjs
Suite Tests passing
db 500
server 2585 (1 skipped)
provider-bridge-protocol 241
Codex 288
Claude Code 348
ACP 312
Pi 154 (1 skipped)
plugin-sdk 270
echo examples 15 + 1
bb-guide 5
integration 82
agent-runtime 327
host-daemon 608 (1 skipped)

The SDK npm version guard passes: this layer changes the published conformance kit, and 0.4.85 is not yet on npm. Off-pipeline checks also pass: the server provider-corpus gate, and agent-runtime integration.provider-basic against real Codex (24 tests). The EAP codename scan is clean for the working tree, HEAD, the push range, and this text.

Fixes #3461

🤖 Generated with Claude Code

AGENT GENERATED

@SawyerHood
SawyerHood added this pull request to stack #3504 September 11, 2026 18:50
@SawyerHood
SawyerHood force-pushed the bb/3461-owned-sessions-stack branch 4 times, most recently from d2863af to f9e8804 Compare September 11, 2026 22:59
SawyerHood and others added 3 commits September 11, 2026 23:04
`getLastStoredProviderThreadId` took the newest `provider_thread_id` stamped
on any event of the thread, so one mis-stamped event, or a `thread/identity`
recorded after the server itself resumed into the wrong session, redirected
every later resume, archive forward, and interruption stamp into another
thread's provider session (#3461).

Resolution now reads only `thread/identity` events recorded since the
thread's last completed context clear, newest first, and classifies each
announced session against every same-provider, same-host claim:

- owned: no other thread claimed it, or this thread's earliest claim is
  strictly earlier than every other thread's.
- foreign: another thread claimed it strictly earlier. Resolution skips it
  and falls back to the thread's own earlier session, which repairs the
  persisted contamination the 2026-09-11 incident left behind.
- ambiguous: another thread's earliest claim falls in the same millisecond.
  Daemon event batches share one timestamp and rowid order is not durable,
  so resolution stops instead of picking a winner.

`resolveStoredProviderSessions` returns none, owned, foreign, or ambiguous,
so callers can refuse explicitly instead of treating an unavailable session
like a brand-new thread. `classifyStoredProviderThreadClaim` gives edit and
fork paths the same classification for a completion's stamp. A partial
index on identity rows backs the claim lookup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…er session

Sends, queued dispatch, turn submission, and goal clears resumed whatever
session id the stored history named. With a contaminated stamp that was
another thread's rollout (#3461); with only a foreign or tied identity, a
send silently started a new session without the thread's context.

Dispatch now goes through `requireDispatchableProviderThreadId`. An owned
session resumes and no session since the last context clear starts fresh.
A foreign or ambiguous session answers 409 `provider_session_unavailable`
with the session id and claimant threads in `details`, and the message
"Send /clear to start a new session; history is kept." Nothing is
dispatched and no history is rewritten; a queued row records the same text
as its failure reason. Message-edit rewinds and anchored or tip forks refuse
a completion or source whose session is foreign or ambiguous.

Empty-input fork starts record the daemon's start result as a real
`thread/identity` event, so the resolver sees the session the server itself
established. Test fixtures that established a session through a stamped
`turn/started` alone now announce it through `seedThreadIdentity`, matching
what every bridge emits.

Fixes #3461

Co-Authored-By: Claude Code <noreply@anthropic.com>
bb persists a provider session only from `thread/identity`, resumes only
sessions recorded that way, and now refuses a session another thread
announced first or in the same millisecond. The kit checked that
`thread/start` and `thread/fork` return `providerThreadId` on the result but
never that the session is announced, so a third-party bridge that skipped the
notification would pass conformance and start a new provider session on
every send.

`session/start-identity-announced` and `session/fork-identity-announced`
require a `thread/identity` notification or `thread.identity` delta naming
the returned session for that thread. Every in-repo and example bridge
already passes, and their expected-status maps pin both checks.

The protocol doc now states what resume already relied on: a
`providerThreadId` is a durable handle bb sends to a new bridge process after
a restart, so it must name one session among all of that provider's sessions
on the host and never come from a per-process counter. Real bridges mint
UUIDs, and the published echo example adds a random per-process nonce. The
scripted test bridge keeps readable `prov-N` counters for single-process
suites and gains `uniqueProviderThreadIds`, which the integration suite
enables because it runs several bridge processes against one server store.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@SawyerHood
SawyerHood force-pushed the bb/3461-owned-sessions-stack branch from f9e8804 to 7106c27 Compare September 11, 2026 23:05
Session resolution loaded every thread/identity event since the last
context clear and deduplicated them in JavaScript. Codex announces its
session on every turn, so the cost grew with turn count: 3.9 ms per lookup
on a production thread with 2,464 identity events, against 0.06 ms for the
previous LIMIT 1 query.

Resolution now reads the newest identity event (LIMIT 1), then finds the
session's earliest claim with a single statement whose ORDER BY created_at
LIMIT 1 subquery seeks the partial identity index. It reads further back
only when the newest session belongs to another thread. The ownership
semantics are unchanged: on a copy of the production store the result
matched the previous resolver for all 4,812 threads and 9,620 claim
checks, and a lookup takes about 0.25 ms regardless of history length.

The query-plan test now pins two statements, each using an index, for a
thread with a long identity history.

Co-Authored-By: Claude Code <noreply@anthropic.com>
stack merge was automatically disabled September 12, 2026 00:42

Pull Request is not mergeable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server resumes a thread into whichever provider thread its latest event names, with no cross-thread ownership check

1 participant