fix(sandbox): avoid session mirrors after sandbox release - #3039
fix(sandbox): avoid session mirrors after sandbox release#3039larry-zy wants to merge 6 commits into
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Guards the best-effort session mirror against races with self-managed sandbox release: a new static Sandbox → MirrorGate (ReentrantReadWriteLock) in PinnedSandboxFilesystem blocks release until in-flight uploads finish and rejects new ones afterwards, SandboxLifecycleMiddleware marks the gate before sandboxManager.release(...), SessionTree skips mirrors for a released sandbox, and ObjectStoreTranscriptStore.appendSegment / mirrorToFilesystem now fail loudly instead of ignoring a returned failure response. The problem is real (#3036) and the "stop reporting a mirror as done when the upload actually failed" part is a genuine improvement — compact() in particular was previously free to delete the input segments after a lost upload.
Findings
- [Critical]
PinnedSandboxFilesystem.java:81— the read lock spans the whole remote upload while release takes the write lock, so call teardown now waits on an unbounded network transfer; one stalled upload hangsreleaseForCall(and the lease close) on the agent's completion path. - [Warning]
PinnedSandboxFilesystem.java:44—releasedis a one-way latch keyed on theSandboxinstance with no reset path (resume/reconnect rebinds instances), andWeakHashMapkey equality is not identity-based. - [Warning]
SessionTree.java:745— the new early return also skips the localindex.upsertFromLocalFile, so a failed mirror leaves the workspace index stale for a file that exists on disk. - [Warning]
ObjectStoreTranscriptStore.java:91—appendSegmentnow throws, whichscheduleSegmentMirrorlogs andcompact()propagates; the interface documents no failure contract, and the "released" case becomes a WARN storm instead of a distinguishable skip. - [Warning]
SessionTreeMirrorTest.java:213— both new tests exerciseisRunning() == false(the pre-existing liveness check);markSandboxReleasedand the gate interaction are untested. - [Info]
KubernetesSandbox.java:289— solid addition;exec/persistWorkspace/hydrateWorkspaceremain unguarded andWORKSPACE_STOP_ERRORis reused for a closed connection.
Suggestions
The blocking-teardown item is the one I'd like resolved before merge — a bounded tryLock on the gate keeps the intended ordering for new mirrors without letting an un-timed remote write stall call release. The rest are contract/test hardening: name the "skipped because released" outcome so callers can treat it as normal, state the throws contract on TranscriptStore#appendSegment, keep the local index update independent of the remote copy, and add one test that drives the gate itself.
Also related for the maintainer's eye: #3094 (defer self-managed sandbox shutdown via a retain/release counter) and #3106 (move transcript persistence off the completion path) are attacking the same lifecycle window from three directions with three different mechanisms — worth converging on one owner for "async write vs. sandbox release" before all three land.
Automated review by github-manager-bot
Summary
Testing
Fixes #3036