Skip to content

Denormalize aggregate callback accounting onto CHASM component state - #12119

Draft
chrsmith wants to merge 1 commit into
chrsmith/cb-2-standalone-componentsfrom
chrsmith/cb-3-denormalize-totals
Draft

chrsmith wants to merge 1 commit into
chrsmith/cb-2-standalone-componentsfrom
chrsmith/cb-3-denormalize-totals

Conversation

@chrsmith

Copy link
Copy Markdown
Contributor

Stack 3/6 — base: chrsmith/cb-2-standalone-components (#12118)

⚠️ This is the only PR in the stack that changes persistence. Landing it early so it bakes.

Why

The aggregate size cap needs to know how many bytes of callbacks an execution already holds, and that cannot be derived from the CHASM tree. Child blobs are only serialized when the transaction closes, so a mid-transaction walk sees stale bytes, and there is no exported API for a node's serialized size in any case.

What

  • total_callbacks_size on ActivityState (24) and OperationState (21).
  • The CHASM Workflow component gets a real state message. It previously embedded google.protobuf.Empty because mutable state owns workflow state; it now carries WorkflowState with the count and size, left nil until first written so a callback-free workflow still persists a zero-byte blob.

Workflows also get a denormalized count, not just a size. totalCallbackCount walked every WorkflowUpdate on each attach, and under a MutableContext that Get also marks each node dirty — so attaching one callback re-serialized every update. That now happens only during the one-time backfill.

Compatibility — the part worth scrutinising

The swap is invisible to the tree, which identifies nodes by registered component type ID rather than proto type, and both encodings agree on the wire.

The direction that could actually bite is the downgrade: an old server decoding WorkflowState into Empty and writing it back must not drop the new fields, or the counters silently reset to zero and look like data predating them. workflow_state_compat_test.go pins all three directions through the codec the tree really uses.

Note chasm/lib/buf.yaml breaking checks are not wired (Makefile:508 TODO), so CI will not catch proto issues here.

Backfill

No migration and no marker field. A stored size of zero alongside a non-empty callback set can only be pre-field data, because a validated callback always serializes to more than zero bytes. The recomputed total is written back even when nothing was inserted, so it happens at most once per execution.

Also

Limits are now charged against what a call would actually persist rather than the request as sent. The insert loops already skipped keys a retry re-derives, so charging the full request could reject a retry for exceeding a cap it does not move. Planning the insertions first also stops a rejected update request from leaving a half-built update component behind.

No limit changes: the size cap is still disabled by default.

🤖 Generated with Claude Code

The aggregate size cap needs to know how many bytes of callbacks an
execution already holds, and that cannot be derived from the CHASM tree.
Child blobs are only serialized when the transaction closes, so a
mid-transaction walk sees stale bytes, and there is no exported API for a
node's serialized size in any case. Denormalize the totals instead.

Add total_callbacks_size to ActivityState and OperationState, and give
the CHASM Workflow component a real state message. It previously embedded
google.protobuf.Empty because mutable state owns workflow state; it now
carries WorkflowState with the count and size, left nil until first
written so a callback-free workflow still persists a zero-byte blob.

The swap is invisible to the tree, which identifies nodes by registered
component type ID rather than proto type, and both encodings agree on the
wire. The rolling-upgrade direction that could actually bite is the
downgrade: an old server decoding WorkflowState into Empty and writing it
back must not drop the new fields, or the counters silently reset to zero
and look like data predating them. Tests pin all three directions through
the codec the tree really uses.

Workflows also get a denormalized count, not just a size.
totalCallbackCount walked every WorkflowUpdate on each attach, and under
a MutableContext that Get also marks each node dirty, so attaching one
callback re-serialized every update. That now happens only during the
one-time backfill.

Backfill needs no migration or marker: a stored size of zero alongside a
non-empty callback set can only be pre-field data, because a validated
callback always serializes to more than zero bytes. The recomputed total
is written back even when nothing was inserted, so it happens at most
once per execution.

Limits are now charged against what a call would actually persist rather
than the request as sent. The insert loops already skipped keys a retry
re-derives, so charging the full request could reject a retry for
exceeding a cap it does not move. Planning the insertions first also
stops a rejected update request from leaving a half-built update
component behind.

No limit changes: the size cap is still disabled by default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@chrsmith
chrsmith force-pushed the chrsmith/cb-2-standalone-components branch from 66a9bda to efe7c20 Compare September 17, 2026 15:50
@chrsmith
chrsmith force-pushed the chrsmith/cb-3-denormalize-totals branch from 8a0f294 to e6bdcf4 Compare September 17, 2026 15:50
@chrsmith
chrsmith removed this pull request from stack #12123 September 18, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant