Skip to content

feat: pluggable snapshot storage broker with per-snapshot signed URLs - #1282

Draft
Anish Ramasekar (aramase) wants to merge 8 commits into
agent-substrate:mainfrom
aramase:aramase/f/uds_s3_broker
Draft

feat: pluggable snapshot storage broker with per-snapshot signed URLs#1282
Anish Ramasekar (aramase) wants to merge 8 commits into
agent-substrate:mainfrom
aramase:aramase/f/uds_s3_broker

Conversation

@aramase

@aramase Anish Ramasekar (aramase) commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Adds a vendor-neutral snapshot storage broker. The control plane mints
short-lived, per-snapshot signed URLs and atelet reads and writes snapshots over
plain HTTP, so the node links no cloud SDK and holds no cloud credential. The
cloud-specific signer runs out of process as a sidecar over a Unix socket, which
keeps the control-plane core free of any cloud SDK as well. Ships an S3 sidecar.
Off by default.

This is the storage-decoupling touchpoint from the Cloud Provider Decoupling
design, and it stands up the same per-snapshot capability #777 needs: the node
stops holding a broad, permanent storage credential. It does not yet make the
authorization decision before minting; that needs the broader control-plane
authz layer.

Design doc: https://docs.google.com/document/d/1I2SlYey0Y2mcGQBZNpfYufWhwLbQDmaqI5T56sz33_4/edit?usp=sharing

What's in it

  • storagebroker interface and registry in ate-api-server. New(backend)
    selects one; empty disables signing and the existing in-atelet path is
    unchanged.
  • ate-api-server pre-mints per-snapshot capabilities at resume (actor plus
    golden) and suspend (write), passed to atelet via a new signed_access map on
    the atelet proto.
  • atelet signedObjectStore satisfies the existing ategcs.ObjectStorage and
    speaks S3 (presigned GET, POST starts-with policy) or Azure (PUT plus token)
    over net/http, with no SDK and no credential.
  • UDS broker client so the signer is a sidecar. ate-api-server links 0 AWS SDK
    packages (was 39 with the in-tree signer).
  • cmd/s3-broker-server: the S3 sidecar (presigned GET per object, one POST
    policy per snapshot).
  • Selected via ATE_SNAPSHOT_BROKER (for example uds), separate from atelet's
    ATE_STORAGE_BACKEND.

Testing

  • Unit tests for the mint wiring and the signed HTTP read/write dispatch. Full
    verify-all is green.
  • Broker-on kind e2e (parking suspend/resume) exercises the sidecar mint and
    atelet presigned GET/POST end to end.
  • Benchmarks show the mint is off the byte path: resume p50 is 244 ms native vs
    253 ms in-tree signer vs 243 ms over the socket, so no measurable latency.

Not in this PR

  • The authorization decision before minting (the decision half of Per-snapshot authorization #777).
  • A GCS broker, and retiring atelet's existing S3 object-store client to take the
    node to zero SDK.
  • Chunked upload for very large snapshots.
  • Wiring the sidecar into the in-repo manifests so CI exercises the broker path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant