multi-actor worker API - #1283
Draft
Benjamin Elder (BenTheElder) wants to merge 8 commits into
Draft
Conversation
Collaborator
Author
|
x-ref: #1164 🙈 |
Benjamin Elder (BenTheElder)
force-pushed
the
worker-assignments-as-rows
branch
2 times, most recently
from
August 28, 2026 23:58
96e9c63 to
975778f
Compare
Benjamin Elder (BenTheElder)
requested review from
Julian Gutierrez Oschmann (juli4n) and
Tim Hockin (thockin)
August 29, 2026 02:12
Benjamin Elder (BenTheElder)
force-pushed
the
worker-assignments-as-rows
branch
from
August 29, 2026 03:02
0e6d5f6 to
5ac7f97
Compare
status.assignment held one Actor -- the implementation's limit written into the wire format. Describe a set now, while there is one implementation to keep in step, and leave the implementation admitting one. assignments is repeated and reuses field 2, which is wire-compatible with the singular field it replaces, so an old record reads back as a set of one. How many may be bound is capacity.actors, a dimension alongside cpu_milli and memory_bytes because it bounds the same thing they do -- what one more Actor costs -- for the costs that are not compute. Unset means ONE, so nothing admits more until a Worker says it can. status.allocated is what the assignments took, as a running total: placement reads it for every Worker on every decision, and summing the list there would cost the fleet's whole actor count. ActorAssignment.resources records what each Actor was admitted for, so a release gives back what its bind took. Capacity also stops being immutable -- a pod can be resized, and the actor ceiling is the Worker's own. Clearing it is still refused.
Mostly mechanical: readers of status.assignment go through resources.WorkerAssignmentFor, and bind/release keep status.allocated in step with the list they count. Three that are not. Placement asks what is left rather than whether the Worker is empty, so eligibility splits from room. A caller re-validating a Worker that already holds the Actor must not ask about room: the Actor's own resources are already counted against the Worker, and it would be evicted from a valid placement. ListWorkers leaves the assignments empty and reports occupancy through allocated, so listing the fleet costs its size and not its actor count. GetWorker carries them. kubectl-ate prints ASSIGNED(n/m) instead of naming the Actor, and resolves --atespace by asking the Actors which Worker they are on. The claude-code-multiplex demo loses its template-namespace filter for the same reason: a listing has no template to filter on.
capacity.actors has no writer, so every Worker sits at the unset default of one. The number is the ateom's: it owns the slot allocator, and only the node the Worker runs on can observe it. A WorkerPool cannot state it either, because two Workers in one pool running different ateom builds do not have the same answer. So it arrives by report rather than by assertion, and a fleet may run mixed ateom versions with each Worker admitting what it can actually take.
ReportWorkerCapacity sits on ActorIdentity for the authorization that service already establishes: atelet calls it for the Worker with its own client certificate, as it does for MintCert, and may speak only for the Workers on its node. A Worker on another node is NOT_FOUND, not PERMISSION_DENIED -- a caller learns nothing about Workers elsewhere. A report states only the dimensions its sender can see, so an unset one keeps what is recorded rather than clearing it. The syncer carries the reported ceiling across its own recomputes for the same reason: it owns cpu and memory and would otherwise clear the report on every pod event, forever. Re-sending an unchanged capacity is not an update, so a reporter on a timer costs a read. No caller yet: that belongs with an ateom that has something other than one to say.
A Worker carried its assignments in its own record, so the record, its change event, and every watcher's copy of it all grew with the Actors on it. Give each assignment a row instead. Only status.allocated stays on the Worker: placement reads it for every candidate on every decision, so it has to be there. The store moves it in the same transaction as the assignment it counts, and ClearAssignments drops the list on the way in, so there is one place the truth lives. Binding does not read the assignment it might replace. The ordinary bind is a first bind, and the insert landing is proof nothing was there. Deciding it on the write is also what makes it correct: a read cannot see a claim that commits after it, so two replicas claiming one Actor onto two Workers would each find nothing and each add one, leaving the loser counting an Actor it does not host. The contract pins that, and it fails against a read-first bind.
Benjamin Elder (BenTheElder)
force-pushed
the
worker-assignments-as-rows
branch
from
August 29, 2026 17:34
5ac7f97 to
a7b750e
Compare
Claiming a worker rewrites its whole record. That suits racing ateapi replicas but not racing goroutines, which is what a worker hosting thousands of actors makes them: on an empty worker this refused 21% of activations at 12 in flight and 67% at 24. The store already binds inside one transaction with the worker's row locked, so it is the only place that can decide whether the worker has room without the answer going stale before the write. BindActorToWorker takes an admit callback and asks there. Refusing rolls the transaction back, which is also what takes the speculative assignment row out again. Deciding it there is what removes the race rather than detecting it: no version precondition to lose, no re-read of the worker that is stale by the time it is used, and no lookup asking whether this worker already holds the actor, which the insert answers on its own. The row lock covers every replica, which no in-process lock can. The retry budget still had to grow. A claim that is refused for want of room re-runs scheduling, and that was five steps from 10ms.
Benjamin Elder (BenTheElder)
force-pushed
the
worker-assignments-as-rows
branch
from
August 29, 2026 21:21
a7b750e to
0533447
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1266
This is a draft of the core API + data store changes.
It's still a large PR, apologies.
The "as rows" commit could be split out, but this takes it to ~all of the breaking changes we can't hide behind updating internals.
Same for the claimlock, but in both cases it seems these are worth understanding when considering the API shape.
They're loadbearing for performance once we actually have multi-actor workers.