From 67d43ccc9443c34ef2ddada417582f075945104c Mon Sep 17 00:00:00 2001 From: Timo Derstappen Date: Mon, 14 Sep 2026 21:53:45 +0200 Subject: [PATCH 1/2] =?UTF-8?q?docs(fork):=20ledger=20=E2=80=94=20the=20li?= =?UTF-8?q?ne=20is=20pinned=20at=20kagent-dev/substrate=20v0.0.29=20and=20?= =?UTF-8?q?runs=20the=20agentgateway=20line's=20v1.5.1-gs.4;=20kagent-dev/?= =?UTF-8?q?substrate#33=20and=20the=20chart=20half=20of=20#9=20fell=20away?= =?UTF-8?q?;=20the=20carried=20patches'=20commits=20after=20the=20re-pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Timo Derstappen --- FORK.md | 78 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 32 deletions(-) diff --git a/FORK.md b/FORK.md index 8a934f9980..c3a6d052e9 100644 --- a/FORK.md +++ b/FORK.md @@ -25,36 +25,34 @@ kagent line built the same way [giantswarm/giantswarm#37010](https://github.com/ | | | |---|---| -| Upstream tag | **v0.0.26** (2026-09-06; tag commit `0cb6a535`) | -| Why this one | `giantswarm/kagent-upstream` pins `github.com/kagent-dev/substrate v0.0.26` in `go/go.mod` (the `replace` of `github.com/agent-substrate/substrate`): the ate-api gRPC contract between kagent's client and Substrate's server must match. | -| agentgateway it runs | `AGENTGATEWAY_IMAGE` in `publish.yaml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema): a release of the agentgateway line — **`v1.5.1-gs.2`** = upstream v1.5.0 + agentgateway#3237 (every egress CONNECT authorized against ate-api: the actor's UID, then its state) + the line's patch admitting a `RESUMING` actor (giantswarm/agentgateway-upstream#4). Upstream's chart pins `ghcr.io/kagent-dev/substrate/agentgateway:c0f5597c7cb8` at v0.0.26 (a build of upstream agentgateway from 2026-08-30 with no revision label: a pre-merge build of #3237, `RUNNING` only) and `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.9f9744cf` on `main` (kagent-dev/substrate#28, upstream commit `9f9744cf` — the new substrate ingress header, agentgateway#3409). The line's pin follows: v1.5.0-based releases for this pin, ≥ `9f9744cf` for the first pin containing #28 (the agentgateway line's `FORK.md`, "Convergence with the Substrate line"). | +| Upstream tag | **v0.0.29** (2026-09-12; tag commit `463f483c`, "feat(chart): configure OTLP export per signal (#37)" — upstream `main` of 2026-09-14). Previous pin: v0.0.26 (2026-09-10 → 2026-09-14, releases `v0.0.27-gs.1`–`gs.9`) | +| Why this one | `giantswarm/kagent-upstream` pins `github.com/kagent-dev/substrate v0.0.29` in `go/go.mod` (the `replace` of `github.com/agent-substrate/substrate`) since upstream kagent-dev/kagent#2802 (`2d843e37`, "upgrade Substrate to v0.0.29"): the ate-api gRPC contract between kagent's client and Substrate's server must match, and that kagent addresses an actor by the `ate-target-actor` header, which the router knows from v0.0.28 on — a kagent at or past #2802 on a 0.0.27 Substrate is green in CI and fails every turn on a cluster, so the two lines moved together (2026-09-14). | +| agentgateway it runs | `AGENTGATEWAY_IMAGE` in `publish.yaml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema): a release of the agentgateway line — **`v1.5.1-gs.4`** = upstream agentgateway `main` @ `c1d24607` (2026-09-14, ≥ `9f9744cf`: #3237's CONNECT-time actor authorization, #3318's `substrateEgressActorResolution` frontend policy, #3409's substrate ingress header, #3428) + the line's patch admitting a `RESUMING` actor + its `GRPCRoute` method-match translation. Upstream's chart pins `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.9f9744cf` at v0.0.29 (kagent-dev/substrate#28, a nightly of that commit); the line runs the same protocol from its own scanned and signed build (the agentgateway line's `FORK.md`, "Convergence with the Substrate line"). | | When it moves | only together with kagent's pin, proven in agentlab first (`agentlab configure --defaults --chart-branch poc/kagent-main && agentlab up` and the proofs) — see "Re-pin". Not on a schedule. | | Derived how | `git describe --tags --abbrev=0 --match 'v[0-9]*' --exclude '*-*' giantswarm` with upstream's tags fetched; the line's own tags carry a pre-release suffix and are excluded. The workflows compute it, nothing records it twice. | ## Carried patches -Everything on `giantswarm` that is not in the pin (`git log v0.0.26..giantswarm`): +Everything on `giantswarm` that is not in the pin (`git log v0.0.29..giantswarm`): | Patch | Purpose | Fork commit | Upstream | |---|---|---|---| -| Grant atelet cluster-wide read access to sandbox configs | atelet's sandbox-asset prewarm degraded on the second test cluster without the RBAC ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 10) | `74b45f9e` (`git cherry-pick -x a7505e9c`) | [kagent-dev/substrate#33](https://github.com/kagent-dev/substrate/pull/33), merged 2026-09-08, not in v0.0.26 — falls away at the re-pin onto the first tag that contains it | -| Let an actor's egress through while it resumes (ateom arms tunneled egress before the first container starts, atenet admits `RESUMING` actors, both hops log a refusal) | an actor whose workload fetches what it needs to become ready — kagent's Go ADK and Claude harnesses materialise git skills before readyz — never got its golden snapshot: atunnel dropped the fetch (`Broken pipe`), atenet would have refused a non-`RUNNING` actor, nothing was logged ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) rows 8 and 13; acceptance test `agentlab skills-test`, [agentlab#137](https://github.com/giantswarm/agentlab/issues/137)) | [#4](https://github.com/giantswarm/substrate/pull/4) (`181762747bb2`; first published as `0.0.27-dev.giantswarm.2026-09-10.22-37-39.h1817627`) | to file: the upstream-shaped patch is branch [`upstream/atenet-egress-during-resume`](https://github.com/giantswarm/substrate/tree/upstream/atenet-egress-during-resume) here (`3a95d7cf`, on the mirror `main`); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it. **Complete with the egress dataplane this pull request pins:** the chart's `images.agentgateway` is the check in the request path (the egress config carries the `substrateEgress` policy and no `ext_proc`, so atenet's handler is not consulted); kagent-dev's `c0f5597c7cb8` (a pre-merge build of agentgateway#3237) authorized every CONNECT against ate-api itself — UID, then `RUNNING` — and refused the golden boot (`atunnel failed to open egress tunnel … 403 Forbidden: actor is not running`, agentlab 2026-09-11); upstream agentgateway v1.5.0 has no such check at all (its `substrateEgress` derives the actor from the SPIFFE id and checks nothing else — `agentlab skills-test` green on both halves with it swapped into `atenet-egress`, 2026-09-11, the run that proved the Substrate half). The line now runs the agentgateway line's `v1.5.1-gs.2`, which keeps #3237's UID and state check and admits `RESUMING` (giantswarm/agentgateway-upstream#4; upstream-facing branch [`upstream/substrate-egress-resuming`](https://github.com/giantswarm/agentgateway-upstream/tree/upstream/substrate-egress-resuming), #37742 row 8). Acceptance test of the combined fix: `agentlab skills-test` on the first build of this merge, recorded on agentlab#137 | -| Declare the egress actor authorization as a frontend policy (`frontendPolicies.substrateEgress` in the atenet-egress config, the route-level policy removed) and pin `images.agentgateway` to the agentgateway line's `v1.5.1-gs.2` | the line's dataplane carries agentgateway#3237, which moved the CONNECT-time actor check from a route policy to a frontend policy; with v0.0.26's route-level shape the gs.2 dataplane refuses its config (`unknown field substrateEgress`, atenet-egress CrashLoopBackOff, agentlab 2026-09-11) and the pin moves with the config because `v1.5.1-gs.1` (v1.5.0) rejects the frontend-level field and the pre-merge build `c0f5597c7cb8` the route-level one only | the `chart: authorize the egress actor as a frontend policy at CONNECT time` commit of pull request #9 | [kagent-dev/substrate#28](https://github.com/kagent-dev/substrate/pull/28) (merged 2026-09-10, on `main`) makes the same move for its `v0.0.0-alpha.988ac151` dataplane under the name agentgateway#3318 gave the policy, `substrateEgressActorResolution`; falls away at the re-pin onto the first tag containing #28 once the agentgateway line carries #3318 (until then the field name differs — resolve by keeping the line's). The e2e install manifests (`manifests/ate-install/components/agentgateway`) still run kagent-dev's `c0f5597c7cb8` with the route-level config, self-consistent; #28 moved them too | -| Read ate-api-server's PostgreSQL connection string from a Secret (`postgres.connectionStringSecretRef`; the `ate-api-server-envvars` ConfigMap then carries only the schema) | meta chart 4.0 puts Substrate's control-plane database on the platform's CNPG cluster and hands ate-api-server the DSN through a Secret, never a ConfigMap ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 24) | `c1e4e32d` (`git cherry-pick -x 1872249e`) and `f06f5ef9` (`git cherry-pick -x 41097da7`, the `helm plugin install --verify=false` of the same pull request), [#8](https://github.com/giantswarm/substrate/pull/8) | [kagent-dev/substrate#32](https://github.com/kagent-dev/substrate/pull/32), open (2026-09-04), not ours — falls away at the re-pin onto the first release that carries it | -| The atelet DaemonSet takes `nodeSelector`, `tolerations` and `affinity` (`atelet.{nodeSelector,tolerations,affinity}`, empty by default) | the platform pins atelet to worker nodes / node pools; the chart had no scheduling knob ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 23) | `084d916d`, [#8](https://github.com/giantswarm/substrate/pull/8) | to file: the upstream-shaped patch is branch [`upstream/atelet-scheduling`](https://github.com/giantswarm/substrate/tree/upstream/atelet-scheduling) here (`b34c1690`, on the mirror `main`); [kagent-dev/substrate#16](https://github.com/kagent-dev/substrate/pull/16) touches the same knob (`atelet.nodeSelector`, no tolerations or affinity) inside a fork-wide 92-file pull request that has conflicted since July — align with the maintainers there; a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | -| atelet mounts `/var/lib/kubelet/plugins` with `mountPropagation: HostToContainer` | with the default (private) propagation atelet's mount namespace kept a copy of every CSI globalmount on the node; after a pod moved, the volume's filesystem (and LUKS mapper) stayed open there and Longhorn's `NodeUnstageVolume` failed forever with `luksClose: Device is still in use` — the SPIRE outage of 2026-09-11 on the homelab cluster ([giantswarm/giantswarm#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 30) | `ef7d3e78` | to prepare | -| Keep the `ate.dev` CRDs when the `substrate-crds` release is uninstalled (`helm.sh/resource-policy: keep` on the three CRD templates, set at the generator as a `+kubebuilder:metadata:annotations` marker on the root types, so `hack/verify/crd-chart.sh` keeps the templates a verbatim copy) | the chart ships its CRDs as templates, so an uninstall of that release deleted the CRDs and every `WorkerPool`, `SandboxConfig` and `CSIDriverConfig` with them; a GitOps controller that uninstalls the releases concurrently — a cluster's own Flux finalizing the meta chart's component `HelmRelease`s — can remove the CRDs first, and the `substrate` release's uninstall then fails for good on its `SandboxConfig` (`failed to delete release: substrate`, Helm cannot delete an object whose kind is gone; [giantswarm/agent-platform#385](https://github.com/giantswarm/agent-platform/issues/385)). The kagent line carries the same policy on `kagent-crds` (giantswarm/kagent-upstream#10) ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 28). A kept CRD means its objects survive a Substrate uninstall: the consumer deletes its `WorkerPool`s and `SandboxConfig`s, or the CRDs explicitly | `d37ebf63` (as it is on `giantswarm` after the rebase merge), [#12](https://github.com/giantswarm/substrate/pull/12) | to file: the upstream-shaped patch is branch [`upstream/substrate-crds-resource-policy-keep`](https://github.com/giantswarm/substrate/tree/upstream/substrate-crds-resource-policy-keep) here (`41bd9df0`, on the mirror `main`; the pull-request text is in #12); neither kagent-dev/substrate nor agent-substrate/substrate carries or proposes the policy (searched 2026-09-11); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | -| Fail an actor whose image the registry refuses instead of resuming it forever (the image cache tags a registry's final word — a 4xx other than 408 or 429: manifest or repository unknown, unauthorized, denied — with `ReasonFailedGetExternalObject`; the Run/Restore boundaries claim it; `maybeCrashActor` returns the crash with its cause and the directive; the ActorTemplate reconciler fails the template with that cause when the resume reports the crash) | a Harness whose `workload.image` could not be pulled never booted and was never reported: atelet's pull failed with the registry's answer, the Run RPC returned it untagged, ate-api retried the resume with backoff, `GoldenSnapshotStatus.ErrorMessage` stayed empty, kagent reported `Ready=False ActorTemplatePending` for as long as anyone waited and a worker stayed pinned to the golden actor ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 27; measured in giantswarm/agent's ATS on `v0.0.27-gs.2`: 300 s, no message). Now `Ready=False ActorTemplateFailed` with `GoldenActorCrashed: actor ate-golden/ crashed: … MANIFEST_UNKNOWN: manifest unknown`, in seconds | `bbe92143`, [#14](https://github.com/giantswarm/substrate/pull/14) | to file: the upstream-shaped patch is branch [`upstream/golden-boot-image-pull-terminal`](https://github.com/giantswarm/substrate/tree/upstream/golden-boot-image-pull-terminal) here (`4a56c0af`, the same commit on the mirror `main`; its message is the pull-request text). agent-substrate/substrate#1220 (open since 2026-08-26, review comments unaddressed, needs a rebase, does not apply to v0.0.26) proposes to crash actors on every failure not marked retriable and classifies registry answers with `transport.Error.Temporary()`; its reviewer asked for the status-based classification this patch does for the one class that is a registry's final word — the patch applies on its own and folds into #1220's shape if that lands. A team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | -| Keep `podcertificate-controller-system` across an uninstall (`helm.sh/resource-policy: keep` on the chart's `Namespace`; the kubectl-apply manifest and a chart unit test with it) | the namespace holds the two CA pools the podcertificate-controller signs from — provisioned into it out of band (upstream `kubectl-ate admin make-ca-pool`; the platform's connectivity bootstrap hook, which keeps them) — while the signers' `ClusterTrustBundle`s are cluster-scoped and outlive the release. An uninstall took the pools and left the bundles; the reinstall minted new roots, the controller republished the bundles within seconds, but every pod had already read the surviving bundle when it started (`ateapiauth` loads the ate-api CA file once, at dial time) and failed each handshake against ate-api-server with `x509: certificate signed by unknown authority` — no golden boot (the agent-platform ATS own-Flux scenario on the cluster its smoke had uninstalled from, [giantswarm/agent-platform#384](https://github.com/giantswarm/agent-platform/issues/384); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 29). Kept, a reinstall signs from the roots the bundles already carry, as `ate-system`'s pools do (`createNamespace: false`) | `e73c1dc0`, [#13](https://github.com/giantswarm/substrate/pull/13) (first release v0.0.27-gs.5) | to file: the upstream-shaped patch is branch [`upstream/keep-podcert-namespace`](https://github.com/giantswarm/substrate/tree/upstream/keep-podcert-namespace) here (`38a4e3a1`, on the mirror `main` @ `007eb1ee`); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it. No upstream issue or pull request covers it (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-11: agent-substrate#146 and #1166 touch stale CA material in the kind and e2e setups only) | -| Re-register a worker whose pod IP changed in place (the Syncer's `createOrUpdateWorker` treats an IP disagreement as a new ateom in the same pod: `reconcileDeadWorker` — the DeleteWorker workflow, NOT_FOUND = success — then `registerWorker`; the Worker's `ip` stays immutable, a new server uid replaces the dead record) | a pod's IP changes only when its sandbox is recreated — a node reboot, a kubelet or container-runtime restart, a host resume — which takes the ateom and every actor it hosted down while the pod keeps its UID and so its Worker; the Syncer only logged `registered worker IP disagrees with its pod`, the router kept sending the assigned actors' requests to the dead `worker_pod_ip`, each timed out, and an actor caught mid-resume stayed `RESUMING`, blocking its suspend and delete until the worker pods were rolled by hand (the second half of the lab's hanging turns after a host resume; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 34) | `c3d78657`, [#20](https://github.com/giantswarm/substrate/pull/20) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/workersync-pod-ip-repair`](https://github.com/giantswarm/substrate/tree/upstream/workersync-pod-ip-repair) here (`13e574e0`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). Composes with agent-substrate/substrate#944 (per-family worker addresses for dual-stack — an issue, not a pull request): the Worker's shape and its immutability rule are unchanged. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | -| atelet pins images the cache must always hold (`--image-cache-pinned-images`: every GC pass pulls each pinned reference that is not cached and roots it — record, layers and the multi-arch twin — like a placed actor's; a failed pull is logged and retried next pass without gating the pass; with the pass disabled the pins are pulled once at start; `imagecache.Store.Pin` is in-memory, re-pinned after a restart) | the eviction pass protects only placed actors' images and anything younger than `--image-cache-min-age`; on a node whose cache volume sits above `--image-cache-high-percent` for reasons other than the cache (a shared root volume, foreign disk pressure) every pass evicted every image older than two minutes, the cache ran at a permanent 0 % hit rate and every first actor after an idle spell paid a cold pull and unpack (~5 s for a 1 GiB harness image) on the resume path — the first half of the lab's hanging turns ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `58d24855`, [#21](https://github.com/giantswarm/substrate/pull/21) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/atelet-image-cache-pinned-images`](https://github.com/giantswarm/substrate/tree/upstream/atelet-image-cache-pinned-images) here (`f6757c3a` + `4c2f33ea`, on the mirror `main` @ `007eb1ee`; the messages are the pull-request text). agent-substrate/substrate#1383 (open, 2026-09-02) is this exact failure — a cold actor-image cache wedges a request-driven resume — and #811 the same shape through the sandbox-asset cache; the eviction engine (#735, #836, design #463) pins or spares nothing. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | -| The chart exposes atelet's image-cache flags (`atelet.imageCache.{gcPeriod,highPercent,lowPercent,minAge,maxBytes,pinnedImages}`; every key null/empty leaves atelet's built-in default) | the platform pins its pool's Harness images and raises the watermarks on nodes that share the cache volume without hand-writing `atelet.extraArgs`; the meta chart sets `pinnedImages` through this key | `ffab110f`, [#21](https://github.com/giantswarm/substrate/pull/21) | with the row above on `upstream/atelet-image-cache-pinned-images` (`4c2f33ea`; the scheduling knobs of the carried patch above are not on that branch) | -| A lifecycle workflow outlives its caller's cancellation (Resume and Suspend run under `context.WithoutCancel` of the caller's, bounded by `--actor-workflow-deadline`, 5m, and the lease as before); the chart sets the ingress dataplane's request parking (`atenetRouter.requestParking`: budget 30 s, max 1024, on all four `substrateIngress` blocks) | a Resume ran under the caller's context, and the caller that gave up was the dataplane: agentgateway's `substrateIngress.requestParking.budget` defaults to 5 s and the chart set nothing, so on every first turn after an idle spell on a node whose cache had evicted the image it cancelled the atelet Restore at 5.0 s (`context canceled`, reason UNKNOWN) after the workflow had durably claimed the worker and moved the actor to `RESUMING`; nothing reclaimed either, and every retry fast-forwarded to the same step and died at the same budget (six retries, each in the same layer). `helm-e2e` pins the old 5 s budget so upstream's parking suite keeps its timing ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `24afbd45`, [#22](https://github.com/giantswarm/substrate/pull/22) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/actor-lifecycle-budgets`](https://github.com/giantswarm/substrate/tree/upstream/actor-lifecycle-budgets) here (`40b3d12c` + `861e41d1`, on the mirror `main` @ `007eb1ee`; the messages are the pull-request text). agent-substrate/substrate#646 is the shape (`Canceled` strands actors in RESUMING/SUSPENDING), #1383 the failure. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | -| The atelet restore is bounded and retried, and an actor whose restore keeps exceeding its budget is failed with the new reason `RESTORE_TIMED_OUT` (`--actor-restore-budget`, default 90 s per attempt with a cold pull and unpack included, 0 = the workflow deadline only; three attempts with a doubling pause within `--actor-workflow-deadline`, the layers an interrupted pull fetched staying cached so a retry makes progress; then the crash directive: CRASHED, worker released, deletable, `ate.actor.crashes` labelled) | the Restore step had one attempt and no budget of its own: a restore that ran out of time came back unclassified (reason UNKNOWN), was not retried and left the actor `RESUMING` on a claimed worker that nothing reclaimed, undeletable (FailedPrecondition) until a resume succeeded ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `95ce0c43`, [#22](https://github.com/giantswarm/substrate/pull/22) | with the row above on `upstream/actor-lifecycle-budgets` (`861e41d1`); agent-substrate/substrate#1562 (atenet retries `ResumeActor` on gRPC status only) and #1518 (an actor stays `RESUMING` on a runsc error) are its upstream tracks | -| ateom keeps re-asserting its capacity report (`internal/ateomcapacity.Report` re-sends the accepted report every 10 s through `reassertReport`, each re-assertion retried like the first, 500 ms → 30 s; atelet's per-call `Recorded worker capacity` line at Debug; ate-api's `Worker reported its capacity` stays the line that marks a change) | a Worker record replaced under a running ateom — the Syncer's re-registration after a pod-IP change (#20), a store restored from a backup — is created without capacity (`CreateWorker` leaves it to the ateom) and never received one, so `AssignWorker` answered `no free workers available` on every resume until the pool was rolled (the lab proof of giantswarm/giantswarm#37762 box 3, 2026-09-12; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 38). An unchanged report costs ate-api a read and no write (100 workers ≈ 10 reads/s); the ≤ 10 s gap after a re-registration sits inside the router's 30 s parking budget | `5fccbf08`, [#25](https://github.com/giantswarm/substrate/pull/25) (rebase-merged 2026-09-12; first release v0.0.27-gs.7) | to file: the upstream-shaped patch is branch [`upstream/ateom-reassert-capacity`](https://github.com/giantswarm/substrate/tree/upstream/ateom-reassert-capacity) here (`cedac921`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text; the `syncer.go` comment hunk, which annotates #20's path, is not on it). agent-substrate/substrate#1473 (open since 2026-09-04: a `RecycleWorker` RPC keeps the record of a Worker whose ateom container was replaced *because* an ateom reports its capacity once per process and a re-registered Worker would stay unschedulable) designs around the constraint this patch removes — complementary, not cherry-picked; a recycled record's re-assertion is a no-op read. No upstream issue on the capacity loss itself (searched 2026-09-12). A team member opens the pull request with DCO sign-off once #37742 has reviewed it. Keep at a re-pin onto a release that carries #1473: a recreated sandbox has a new IP and stays a re-registration under #20 | -| The two ServiceAccounts take annotations and ate-api-server takes `extraEnv` (`atelet.serviceAccount.annotations`, `ateApiServer.serviceAccount.annotations`, `ateApiServer.extraEnv`; all empty by default, the rendered manifests unchanged) | an installation whose snapshot store is not the bundled RustFS binds a pod identity to the pods that reach it (IRSA `eks.amazonaws.com/role-arn`, GKE Workload Identity `iam.gke.io/gcp-service-account`) and hands ate-api-server the store's endpoint; atelet took `extraEnv`, ate-api-server only the `ate-api-server-envvars` ConfigMap the chart itself creates, and neither ServiceAccount an annotation. The meta chart's snapshot-store block sets all three ([giantswarm/agent-platform#411](https://github.com/giantswarm/agent-platform/issues/411); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 40) | `8fbf9e60`, [#27](https://github.com/giantswarm/substrate/pull/27) (rebase-merged 2026-09-13; first release v0.0.27-gs.8) | to file: the upstream-shaped patch is branch [`upstream/serviceaccount-annotations-extraenv`](https://github.com/giantswarm/substrate/tree/upstream/serviceaccount-annotations-extraenv) here (`c5f2a3d4`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). No upstream issue or pull request covers it (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-13: agent-substrate/substrate#1277 documents the GKE Workload Identity grants `setup-gcp` creates out of band, kagent-dev/substrate#31 makes the ServiceAccount names configurable). A team member opens the pull request with DCO sign-off once #37742 has reviewed it | -| Terminate a workload whose runsc containers are already gone (`cleanupContainers` lists the containers runsc knows under the actor's state root — `runsc list -quiet`, nothing for a root without state — and checks and deletes only those; a container that is gone is logged and skipped, and a `runsc delete` that fails once its container is gone counts as done; application containers first, the pause container last, as before) | `runsc state` is fatal on a container runsc does not know (`loading container: file does not exist`, exit 128), so a terminate whose first attempt had removed the container but failed after that could never succeed: when a golden boot crashes, the Run-failure cleanup detaches the bundle's rootfs overlay (its own `runsc delete -force pause` is killed by the 30 s cleanup budget), the first terminate's `runsc delete -force pause` destroys the container state and then fails on the filestore file that lived on the overlay (`destroying container: failed to delete filestore file … no such file or directory`), and every retry fails at `runsc state pause` — atelet never reset the actor's directories, ateapi never released the worker, the actor stayed `DELETING` with its worker assigned until the pod was replaced. On gazelle (2026-09-13, control plane `0.0.27-gs.8`, worker `0.0.27-gs.7`) two crashed golden actors of superseded `AgentTemplate` revisions pinned two of the pool's four workers while kagent's controller logged `failed to collect runtime revision … runsc state: exit status 128` every minute per template (giantswarm/giantswarm#37773; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 43). Reproduced and proven in agentlab on 2026-09-13: on `0.0.27-gs.7` the superseded revision's template was never collected (56 min, the same three errors), on this patch's dev build the next sweep after the template change collected it and freed the worker | `1dd5d1fb`, [#30](https://github.com/giantswarm/substrate/pull/30) (rebase-merged 2026-09-13; first release v0.0.27-gs.9) | to file: the upstream-shaped patch is branch [`upstream/ateom-terminate-gone-containers`](https://github.com/giantswarm/substrate/tree/upstream/ateom-terminate-gone-containers) here (`b043db2e`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). No upstream issue or pull request on the terminate path (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-13); agent-substrate/substrate#1518 (open) is the sibling symptom — a `runsc restore` error leaves the actor `RESUMING` with its worker assigned — and agent-substrate/substrate#641 the node-directory leak of a delete that never reaches the node. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | -| Fork infrastructure: this file, the README pointer, `CODEOWNERS`, `.github/workflows/publish.yaml`, `.github/workflows/sync-upstream.yaml`, `.trivyignore`, and the branch triggers of `pr-workflow.yaml`, `helm-e2e.yaml`, `govulncheck.yaml` (`main` → `giantswarm`, govulncheck also on pull requests) | the line's CI, publishing and sync | the `giantswarm` branch history | not for upstream | +| Let an actor's egress through while it resumes (ateom arms tunneled egress before the first container starts, atenet admits `RESUMING` actors, both hops log a refusal) | an actor whose workload fetches what it needs to become ready — kagent's Go ADK and Claude harnesses materialise git skills before readyz — never got its golden snapshot: atunnel dropped the fetch (`Broken pipe`), atenet would have refused a non-`RUNNING` actor, nothing was logged ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) rows 8 and 13; acceptance test `agentlab skills-test`, [agentlab#137](https://github.com/giantswarm/agentlab/issues/137)) | `fb1b0980` ([#4](https://github.com/giantswarm/substrate/pull/4), squash `181762747bb2` on the v0.0.26 base; first published as `0.0.27-dev.giantswarm.2026-09-10.22-37-39.h1817627`) | to file: the upstream-shaped patch is branch [`upstream/atenet-egress-during-resume`](https://github.com/giantswarm/substrate/tree/upstream/atenet-egress-during-resume) here (`3a95d7cf`, on the mirror `main`); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it. **Complete with the egress dataplane this pull request pins:** the chart's `images.agentgateway` is the check in the request path (the egress config carries the `substrateEgress` policy and no `ext_proc`, so atenet's handler is not consulted); kagent-dev's `c0f5597c7cb8` (a pre-merge build of agentgateway#3237) authorized every CONNECT against ate-api itself — UID, then `RUNNING` — and refused the golden boot (`atunnel failed to open egress tunnel … 403 Forbidden: actor is not running`, agentlab 2026-09-11); upstream agentgateway v1.5.0 has no such check at all (its `substrateEgress` derives the actor from the SPIFFE id and checks nothing else — `agentlab skills-test` green on both halves with it swapped into `atenet-egress`, 2026-09-11, the run that proved the Substrate half). The line runs the agentgateway line's release (`v1.5.1-gs.2` on the v0.0.26 pin, `v1.5.1-gs.4` since the v0.0.29 pin), which keeps #3237's UID and state check and admits `RESUMING` (giantswarm/agentgateway-upstream#4; upstream-facing branch [`upstream/substrate-egress-resuming`](https://github.com/giantswarm/agentgateway-upstream/tree/upstream/substrate-egress-resuming), #37742 row 8). Acceptance test of the combined fix: `agentlab skills-test` on the first build of this merge, recorded on agentlab#137 | +| Read ate-api-server's PostgreSQL connection string from a Secret (`postgres.connectionStringSecretRef`; the `ate-api-server-envvars` ConfigMap then carries only the schema) | meta chart 4.0 puts Substrate's control-plane database on the platform's CNPG cluster and hands ate-api-server the DSN through a Secret, never a ConfigMap ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 24) | `bea1c85f` (`git cherry-pick -x 1872249e`) and `851e62ec` (`git cherry-pick -x 41097da7`, the `helm plugin install --verify=false` of the same pull request), [#8](https://github.com/giantswarm/substrate/pull/8) | [kagent-dev/substrate#32](https://github.com/kagent-dev/substrate/pull/32), open (2026-09-04), not ours — falls away at the re-pin onto the first release that carries it | +| The atelet DaemonSet takes `nodeSelector`, `tolerations` and `affinity` (`atelet.{nodeSelector,tolerations,affinity}`, empty by default) | the platform pins atelet to worker nodes / node pools; the chart had no scheduling knob ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 23) | `30ed59e9`, [#8](https://github.com/giantswarm/substrate/pull/8) | to file: the upstream-shaped patch is branch [`upstream/atelet-scheduling`](https://github.com/giantswarm/substrate/tree/upstream/atelet-scheduling) here (`b34c1690`, on the mirror `main`); [kagent-dev/substrate#16](https://github.com/kagent-dev/substrate/pull/16) touches the same knob (`atelet.nodeSelector`, no tolerations or affinity) inside a fork-wide 92-file pull request that has conflicted since July — align with the maintainers there; a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | +| atelet mounts `/var/lib/kubelet/plugins` with `mountPropagation: HostToContainer` | with the default (private) propagation atelet's mount namespace kept a copy of every CSI globalmount on the node; after a pod moved, the volume's filesystem (and LUKS mapper) stayed open there and Longhorn's `NodeUnstageVolume` failed forever with `luksClose: Device is still in use` — the SPIRE outage of 2026-09-11 on the homelab cluster ([giantswarm/giantswarm#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 30) | `2c78c0aa` | to prepare | +| Keep the `ate.dev` CRDs when the `substrate-crds` release is uninstalled (`helm.sh/resource-policy: keep` on the three CRD templates, set at the generator as a `+kubebuilder:metadata:annotations` marker on the root types, so `hack/verify/crd-chart.sh` keeps the templates a verbatim copy) | the chart ships its CRDs as templates, so an uninstall of that release deleted the CRDs and every `WorkerPool`, `SandboxConfig` and `CSIDriverConfig` with them; a GitOps controller that uninstalls the releases concurrently — a cluster's own Flux finalizing the meta chart's component `HelmRelease`s — can remove the CRDs first, and the `substrate` release's uninstall then fails for good on its `SandboxConfig` (`failed to delete release: substrate`, Helm cannot delete an object whose kind is gone; [giantswarm/agent-platform#385](https://github.com/giantswarm/agent-platform/issues/385)). The kagent line carries the same policy on `kagent-crds` (giantswarm/kagent-upstream#10) ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 28). A kept CRD means its objects survive a Substrate uninstall: the consumer deletes its `WorkerPool`s and `SandboxConfig`s, or the CRDs explicitly | `8dd8e936` (as it is on `giantswarm` after the v0.0.29 re-pin), [#12](https://github.com/giantswarm/substrate/pull/12) | to file: the upstream-shaped patch is branch [`upstream/substrate-crds-resource-policy-keep`](https://github.com/giantswarm/substrate/tree/upstream/substrate-crds-resource-policy-keep) here (`41bd9df0`, on the mirror `main`; the pull-request text is in #12); neither kagent-dev/substrate nor agent-substrate/substrate carries or proposes the policy (searched 2026-09-11); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | +| Fail an actor whose image the registry refuses instead of resuming it forever (the image cache tags a registry's final word — a 4xx other than 408 or 429: manifest or repository unknown, unauthorized, denied — with `ReasonFailedGetExternalObject`; the Run/Restore boundaries claim it; `maybeCrashActor` returns the crash with its cause and the directive; the ActorTemplate reconciler fails the template with that cause when the resume reports the crash) | a Harness whose `workload.image` could not be pulled never booted and was never reported: atelet's pull failed with the registry's answer, the Run RPC returned it untagged, ate-api retried the resume with backoff, `GoldenSnapshotStatus.ErrorMessage` stayed empty, kagent reported `Ready=False ActorTemplatePending` for as long as anyone waited and a worker stayed pinned to the golden actor ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 27; measured in giantswarm/agent's ATS on `v0.0.27-gs.2`: 300 s, no message). Now `Ready=False ActorTemplateFailed` with `GoldenActorCrashed: actor ate-golden/ crashed: … MANIFEST_UNKNOWN: manifest unknown`, in seconds | `00a39294`, [#14](https://github.com/giantswarm/substrate/pull/14) | to file: the upstream-shaped patch is branch [`upstream/golden-boot-image-pull-terminal`](https://github.com/giantswarm/substrate/tree/upstream/golden-boot-image-pull-terminal) here (`4a56c0af`, the same commit on the mirror `main`; its message is the pull-request text). agent-substrate/substrate#1220 (open since 2026-08-26, review comments unaddressed, needs a rebase, does not apply to v0.0.26) proposes to crash actors on every failure not marked retriable and classifies registry answers with `transport.Error.Temporary()`; its reviewer asked for the status-based classification this patch does for the one class that is a registry's final word — the patch applies on its own and folds into #1220's shape if that lands. A team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | +| Keep `podcertificate-controller-system` across an uninstall (`helm.sh/resource-policy: keep` on the chart's `Namespace`; the kubectl-apply manifest and a chart unit test with it) | the namespace holds the two CA pools the podcertificate-controller signs from — provisioned into it out of band (upstream `kubectl-ate admin make-ca-pool`; the platform's connectivity bootstrap hook, which keeps them) — while the signers' `ClusterTrustBundle`s are cluster-scoped and outlive the release. An uninstall took the pools and left the bundles; the reinstall minted new roots, the controller republished the bundles within seconds, but every pod had already read the surviving bundle when it started (`ateapiauth` loads the ate-api CA file once, at dial time) and failed each handshake against ate-api-server with `x509: certificate signed by unknown authority` — no golden boot (the agent-platform ATS own-Flux scenario on the cluster its smoke had uninstalled from, [giantswarm/agent-platform#384](https://github.com/giantswarm/agent-platform/issues/384); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 29). Kept, a reinstall signs from the roots the bundles already carry, as `ate-system`'s pools do (`createNamespace: false`) | `a5518699`, [#13](https://github.com/giantswarm/substrate/pull/13) (first release v0.0.27-gs.5) | to file: the upstream-shaped patch is branch [`upstream/keep-podcert-namespace`](https://github.com/giantswarm/substrate/tree/upstream/keep-podcert-namespace) here (`38a4e3a1`, on the mirror `main` @ `007eb1ee`); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it. No upstream issue or pull request covers it (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-11: agent-substrate#146 and #1166 touch stale CA material in the kind and e2e setups only) | +| Re-register a worker whose pod IP changed in place (the Syncer's `createOrUpdateWorker` treats an IP disagreement as a new ateom in the same pod: `reconcileDeadWorker` — the DeleteWorker workflow, NOT_FOUND = success — then `registerWorker`; the Worker's `ip` stays immutable, a new server uid replaces the dead record) | a pod's IP changes only when its sandbox is recreated — a node reboot, a kubelet or container-runtime restart, a host resume — which takes the ateom and every actor it hosted down while the pod keeps its UID and so its Worker; the Syncer only logged `registered worker IP disagrees with its pod`, the router kept sending the assigned actors' requests to the dead `worker_pod_ip`, each timed out, and an actor caught mid-resume stayed `RESUMING`, blocking its suspend and delete until the worker pods were rolled by hand (the second half of the lab's hanging turns after a host resume; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 34) | `a0b0575b`, [#20](https://github.com/giantswarm/substrate/pull/20) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/workersync-pod-ip-repair`](https://github.com/giantswarm/substrate/tree/upstream/workersync-pod-ip-repair) here (`13e574e0`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). Composes with agent-substrate/substrate#944 (per-family worker addresses for dual-stack — an issue, not a pull request): the Worker's shape and its immutability rule are unchanged. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | +| atelet pins images the cache must always hold (`--image-cache-pinned-images`: every GC pass pulls each pinned reference that is not cached and roots it — record, layers and the multi-arch twin — like a placed actor's; a failed pull is logged and retried next pass without gating the pass; with the pass disabled the pins are pulled once at start; `imagecache.Store.Pin` is in-memory, re-pinned after a restart) | the eviction pass protects only placed actors' images and anything younger than `--image-cache-min-age`; on a node whose cache volume sits above `--image-cache-high-percent` for reasons other than the cache (a shared root volume, foreign disk pressure) every pass evicted every image older than two minutes, the cache ran at a permanent 0 % hit rate and every first actor after an idle spell paid a cold pull and unpack (~5 s for a 1 GiB harness image) on the resume path — the first half of the lab's hanging turns ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `b1f25e97`, [#21](https://github.com/giantswarm/substrate/pull/21) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/atelet-image-cache-pinned-images`](https://github.com/giantswarm/substrate/tree/upstream/atelet-image-cache-pinned-images) here (`f6757c3a` + `4c2f33ea`, on the mirror `main` @ `007eb1ee`; the messages are the pull-request text). agent-substrate/substrate#1383 (open, 2026-09-02) is this exact failure — a cold actor-image cache wedges a request-driven resume — and #811 the same shape through the sandbox-asset cache; the eviction engine (#735, #836, design #463) pins or spares nothing. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | +| The chart exposes atelet's image-cache flags (`atelet.imageCache.{gcPeriod,highPercent,lowPercent,minAge,maxBytes,pinnedImages}`; every key null/empty leaves atelet's built-in default) | the platform pins its pool's Harness images and raises the watermarks on nodes that share the cache volume without hand-writing `atelet.extraArgs`; the meta chart sets `pinnedImages` through this key | `53453e1d`, [#21](https://github.com/giantswarm/substrate/pull/21) | with the row above on `upstream/atelet-image-cache-pinned-images` (`4c2f33ea`; the scheduling knobs of the carried patch above are not on that branch) | +| A lifecycle workflow outlives its caller's cancellation (Resume and Suspend run under `context.WithoutCancel` of the caller's, bounded by `--actor-workflow-deadline`, 5m, and the lease as before); the chart sets the ingress dataplane's request parking (`atenetRouter.requestParking`: budget 30 s, max 1024, on all four `substrateIngress` blocks) | a Resume ran under the caller's context, and the caller that gave up was the dataplane: agentgateway's `substrateIngress.requestParking.budget` defaults to 5 s and the chart set nothing, so on every first turn after an idle spell on a node whose cache had evicted the image it cancelled the atelet Restore at 5.0 s (`context canceled`, reason UNKNOWN) after the workflow had durably claimed the worker and moved the actor to `RESUMING`; nothing reclaimed either, and every retry fast-forwarded to the same step and died at the same budget (six retries, each in the same layer). `helm-e2e` pins the old 5 s budget so upstream's parking suite keeps its timing ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `764b74a4`, [#22](https://github.com/giantswarm/substrate/pull/22) (rebase-merged 2026-09-12) | to file: the upstream-shaped patch is branch [`upstream/actor-lifecycle-budgets`](https://github.com/giantswarm/substrate/tree/upstream/actor-lifecycle-budgets) here (`40b3d12c` + `861e41d1`, on the mirror `main` @ `007eb1ee`; the messages are the pull-request text). agent-substrate/substrate#646 is the shape (`Canceled` strands actors in RESUMING/SUSPENDING), #1383 the failure. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | +| The atelet restore is bounded and retried, and an actor whose restore keeps exceeding its budget is failed with the new reason `RESTORE_TIMED_OUT` (`--actor-restore-budget`, default 90 s per attempt with a cold pull and unpack included, 0 = the workflow deadline only; three attempts with a doubling pause within `--actor-workflow-deadline`, the layers an interrupted pull fetched staying cached so a retry makes progress; then the crash directive: CRASHED, worker released, deletable, `ate.actor.crashes` labelled) | the Restore step had one attempt and no budget of its own: a restore that ran out of time came back unclassified (reason UNKNOWN), was not retried and left the actor `RESUMING` on a claimed worker that nothing reclaimed, undeletable (FailedPrecondition) until a resume succeeded ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 35) | `23856fe5`, [#22](https://github.com/giantswarm/substrate/pull/22); at the v0.0.29 re-pin the patch also passes the budget in upstream's new `workflow_delete_test.go` (#1541) | with the row above on `upstream/actor-lifecycle-budgets` (`861e41d1`); agent-substrate/substrate#1562 (atenet retries `ResumeActor` on gRPC status only) and #1518 (an actor stays `RESUMING` on a runsc error) are its upstream tracks | +| ateom keeps re-asserting its capacity report (`internal/ateomcapacity.Report` re-sends the accepted report every 10 s through `reassertReport`, each re-assertion retried like the first, 500 ms → 30 s; atelet's per-call `Recorded worker capacity` line at Debug; ate-api's `Worker reported its capacity` stays the line that marks a change) | a Worker record replaced under a running ateom — the Syncer's re-registration after a pod-IP change (#20), a store restored from a backup — is created without capacity (`CreateWorker` leaves it to the ateom) and never received one, so `AssignWorker` answered `no free workers available` on every resume until the pool was rolled (the lab proof of giantswarm/giantswarm#37762 box 3, 2026-09-12; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 38). An unchanged report costs ate-api a read and no write (100 workers ≈ 10 reads/s); the ≤ 10 s gap after a re-registration sits inside the router's 30 s parking budget | `b3791094`, [#25](https://github.com/giantswarm/substrate/pull/25) (rebase-merged 2026-09-12; first release v0.0.27-gs.7) | to file: the upstream-shaped patch is branch [`upstream/ateom-reassert-capacity`](https://github.com/giantswarm/substrate/tree/upstream/ateom-reassert-capacity) here (`cedac921`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text; the `syncer.go` comment hunk, which annotates #20's path, is not on it). agent-substrate/substrate#1473 (open since 2026-09-04: a `RecycleWorker` RPC keeps the record of a Worker whose ateom container was replaced *because* an ateom reports its capacity once per process and a re-registered Worker would stay unschedulable) designs around the constraint this patch removes — complementary, not cherry-picked; a recycled record's re-assertion is a no-op read. No upstream issue on the capacity loss itself (searched 2026-09-12). A team member opens the pull request with DCO sign-off once #37742 has reviewed it. Keep at a re-pin onto a release that carries #1473: a recreated sandbox has a new IP and stays a re-registration under #20 | +| The two ServiceAccounts take annotations and ate-api-server takes `extraEnv` (`atelet.serviceAccount.annotations`, `ateApiServer.serviceAccount.annotations`, `ateApiServer.extraEnv`; all empty by default, the rendered manifests unchanged) | an installation whose snapshot store is not the bundled RustFS binds a pod identity to the pods that reach it (IRSA `eks.amazonaws.com/role-arn`, GKE Workload Identity `iam.gke.io/gcp-service-account`) and hands ate-api-server the store's endpoint; atelet took `extraEnv`, ate-api-server only the `ate-api-server-envvars` ConfigMap the chart itself creates, and neither ServiceAccount an annotation. The meta chart's snapshot-store block sets all three ([giantswarm/agent-platform#411](https://github.com/giantswarm/agent-platform/issues/411); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 40) | `062bb9b8`, [#27](https://github.com/giantswarm/substrate/pull/27) (rebase-merged 2026-09-13; first release v0.0.27-gs.8) | to file: the upstream-shaped patch is branch [`upstream/serviceaccount-annotations-extraenv`](https://github.com/giantswarm/substrate/tree/upstream/serviceaccount-annotations-extraenv) here (`c5f2a3d4`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). No upstream issue or pull request covers it (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-13: agent-substrate/substrate#1277 documents the GKE Workload Identity grants `setup-gcp` creates out of band, kagent-dev/substrate#31 makes the ServiceAccount names configurable). A team member opens the pull request with DCO sign-off once #37742 has reviewed it | +| Terminate a workload whose runsc containers are already gone (`cleanupContainers` lists the containers runsc knows under the actor's state root — `runsc list -quiet`, nothing for a root without state — and checks and deletes only those; a container that is gone is logged and skipped, and a `runsc delete` that fails once its container is gone counts as done; application containers first, the pause container last, as before) | `runsc state` is fatal on a container runsc does not know (`loading container: file does not exist`, exit 128), so a terminate whose first attempt had removed the container but failed after that could never succeed: when a golden boot crashes, the Run-failure cleanup detaches the bundle's rootfs overlay (its own `runsc delete -force pause` is killed by the 30 s cleanup budget), the first terminate's `runsc delete -force pause` destroys the container state and then fails on the filestore file that lived on the overlay (`destroying container: failed to delete filestore file … no such file or directory`), and every retry fails at `runsc state pause` — atelet never reset the actor's directories, ateapi never released the worker, the actor stayed `DELETING` with its worker assigned until the pod was replaced. On gazelle (2026-09-13, control plane `0.0.27-gs.8`, worker `0.0.27-gs.7`) two crashed golden actors of superseded `AgentTemplate` revisions pinned two of the pool's four workers while kagent's controller logged `failed to collect runtime revision … runsc state: exit status 128` every minute per template (giantswarm/giantswarm#37773; [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 43). Reproduced and proven in agentlab on 2026-09-13: on `0.0.27-gs.7` the superseded revision's template was never collected (56 min, the same three errors), on this patch's dev build the next sweep after the template change collected it and freed the worker | `e1e2ebe0`, [#30](https://github.com/giantswarm/substrate/pull/30) (rebase-merged 2026-09-13; first release v0.0.27-gs.9); at the v0.0.29 re-pin hand-merged with upstream's `_pause` root container (#1496) and the restore without `-direct` (#1549) — `cleanupContainers` keeps both behaviours, the tests stayed | to file: the upstream-shaped patch is branch [`upstream/ateom-terminate-gone-containers`](https://github.com/giantswarm/substrate/tree/upstream/ateom-terminate-gone-containers) here (`b043db2e`, on the mirror `main` @ `007eb1ee`; its message is the pull-request text). No upstream issue or pull request on the terminate path (searched kagent-dev/substrate and agent-substrate/substrate, 2026-09-13); agent-substrate/substrate#1518 (open) is the sibling symptom — a `runsc restore` error leaves the actor `RESUMING` with its worker assigned — and agent-substrate/substrate#641 the node-directory leak of a delete that never reaches the node. A team member opens the pull request with DCO sign-off once #37742 has reviewed it | +| Fork infrastructure: this file, the README pointer, `CODEOWNERS`, `.github/workflows/publish.yaml` (with the `AGENTGATEWAY_IMAGE` pin and the drift check against the chart default), `.github/workflows/sync-upstream.yaml`, `.trivyignore`, and the branch triggers of `pr-workflow.yaml`, `helm-e2e.yaml`, `govulncheck.yaml` (`main` → `giantswarm` and `sync/**` — a re-pin candidate is a rebased branch, so a pull request from it runs no `pull_request` workflow; the push runs the suites — govulncheck also on pull requests); the chart default `images.agentgateway` and its comment (`b726035c`, the rest of #9 after upstream #28 made the same frontend-policy move) | the line's CI, publishing and sync | the `giantswarm` branch history | not for upstream | Ten patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no skill-carrying agent of the platform boots, the atelet scheduling knobs, the keep policy on the CRD chart's @@ -73,6 +71,15 @@ and by meta chart 4.0; the `WorkerPool` the platform's Harnesses run on comes wi (`kagent.substrateWorkerPool`); feature gates, Kyverno exceptions and network policies are cluster configuration. +Dropped at the re-pin onto v0.0.29 (2026-09-14): the `cherry-pick -x` of kagent-dev/substrate#33 (atelet's +cluster-wide read of sandbox configs, `74b45f9e` → upstream `af7d1539`, in v0.0.29 — [#37742](https://github.com/giantswarm/giantswarm/issues/37742) +row 10 closes) and the chart half of #9 (the egress actor check as a frontend policy, which kagent-dev/substrate#28 +made under agentgateway#3318's name `substrateEgressActorResolution`; the line's dataplane `v1.5.1-gs.4` carries +#3318). The rebase conflicted where expected: the agentgateway pin (upstream's alpha image vs the line's release — +the line's), `cmd/ateom-gvisor/main.go` (the terminate patch vs #1496/#1549 — hand-merged), and README/values +hunks of the chart patches beside upstream's OTLP keys (both kept); upstream's new `workflow_delete_test.go` +(#1541) constructs the workflow without the bounded restore's budget and was given it inside that patch. + Dropped at the bootstrap of the line (2026-09-10): the three June commits of the old fork `main` — `6564754e` enable websockets (upstream has it: `cmd/atenet/internal/router/xds.go` `UpgradeConfigs`), `5169cdd4` ActorTemplate env refs (upstream kagent-dev/substrate#20, merged), `63ea2b0e` a dispatch-only release workflow @@ -147,16 +154,22 @@ The re-pin moves the line onto a new upstream release tag and replays the carrie merged falls away by itself (`git rebase` drops already-applied patches). It is the one sanctioned rewrite of `giantswarm`. -1. kagent first: the new pin is whatever `giantswarm/kagent-upstream`'s `go/go.mod` `replace` names after its own - re-pin. Do not move Substrate ahead of kagent — the ate-api contract is versioned by that pin. +1. The pin is whatever upstream kagent `main`'s `go/go.mod` `replace` names — the ate-api contract is versioned by + it, and since kagent-dev/kagent#2802 the router protocol too (`ate-target-actor`). The three lines move in one + order: the agentgateway line first (this chart pins its release for the router — a `main` commit ≥ the one + upstream's chart pins), then this line, then `giantswarm/kagent-upstream` onto the upstream head whose `go.mod` + names the new pin. Never move Substrate ahead of the agentgateway release it needs, and never kagent ahead of + Substrate: a kagent past #2802 on a 0.0.27 Substrate is green in CI and fails every turn on a cluster. 2. Run **Actions → sync-upstream → Run workflow** with `pin` = the tag (for example `v0.0.27`). The workflow mirrors `main`, rebases the carried patches onto the tag, runs `go build ./... && go test ./...`, and force-pushes `giantswarm`. The push runs upstream's suites (`pr-workflow`, `helm-e2e`, `govulncheck`) and `publish` builds the dev build. - On a conflict it pushes `sync/-` (the new tag + the patches that applied before the conflict) and opens a pull request that names the conflicting patch and the ones behind it. Finish it by hand: check the - branch out, `git cherry-pick -x` the rest, resolve, test, `git push --force-with-lease origin HEAD:giantswarm`, - close the pull request. **Do not merge it** — the line is a rebased branch; a merge would fold the old pin back in. + branch out, `git cherry-pick -x` the rest, resolve, test, push the branch (`pr-workflow`, `helm-e2e` and + `govulncheck` run on a `sync/**` push — the pull request itself runs nothing, a rebased branch has no merge + commit), then `git push --force-with-lease=refs/heads/giantswarm origin HEAD:giantswarm` and close the pull + request. **Do not merge it** — the line is a rebased branch; a merge would fold the old pin back in. - `dry_run: true` does everything except the pushes; the run summary shows the outcome. 3. Update this file (pin, carried patches) in a pull request, and the Substrate rows of #37742. 4. Move the consumers to the new dev version (see "Consumers"), prove it in agentlab, then let the meta chart's @@ -184,9 +197,10 @@ Manual equivalent (a workstation, upstream as a remote): ```sh git fetch upstream main --tags git checkout giantswarm -git rebase --onto v0.0.27 v0.0.26 # new pin, old pin -go build ./... && go test ./... -git push --force-with-lease origin giantswarm +git rebase --onto v0.0.30 v0.0.29 # new pin, old pin +go build ./... && go test ./... && helm unittest charts/substrate +git push origin HEAD:sync/$(date -u +%Y%m%d)-v0.0.30 # the suites on the candidate +git push --force-with-lease=refs/heads/giantswarm origin HEAD:giantswarm ``` ## Publishing @@ -211,13 +225,13 @@ Not published from here: `ateom-microvm` and the demo images (the platform runs `v` on its image tags (`v1.5.1-gs.1`) because its consumers and the retagger rules carry it — two deliberate choices, do not "fix" one to match the other. - Dev build, on every push to `giantswarm`: `-dev.giantswarm...h` - (for the pin v0.0.26: `0.0.27-dev.giantswarm.…`), the schema the kagent line uses — base = the pin's patch + 1, + (for the pin v0.0.29: `0.0.30-dev.giantswarm.…`), the schema the kagent line uses — base = the pin's patch + 1, branch lowercased to `[a-z0-9-]`, committer date in UTC, so a rebuild of the same commit yields the same version and versions sort chronologically within the branch. Consumers that follow the channel use a Flux - `OCIRepository` with `semver: ">=0.0.27-0 <0.1.0-0"` and `semverFilter: ".*-dev\.giantswarm\..*"`; exact pins + `OCIRepository` with `semver: ">=0.0.30-0 <0.1.0-0"` and `semverFilter: ".*-dev\.giantswarm\..*"`; exact pins name the full string. - Release, on a tag `vX.Y.Z-gs.N` where `X.Y.Z` is upstream's **next** version (the dev base) and `N` counts the - line's releases of that pin: `v0.0.27-gs.1`. Ordering by semver: `0.0.27-dev.… < 0.0.27-gs.1 < 0.0.27`, so a dev + line's releases of that pin: `v0.0.30-gs.1`. Ordering by semver: `0.0.30-dev.… < 0.0.30-gs.1 < 0.0.30`, so a dev build never outranks a release, a fork release never outranks the upstream version it anticipates, and the switch to an upstream tag one day is a range change, not a rename. A fleet consumer follows `semverFilter: ".*-gs\..*"`. From ee5be4422e86f712e4c592d7eb96232b9d6aa10c Mon Sep 17 00:00:00 2001 From: Timo Derstappen Date: Mon, 14 Sep 2026 22:54:26 +0200 Subject: [PATCH 2/2] =?UTF-8?q?docs(fork):=20release=20table=20=E2=80=94?= =?UTF-8?q?=20v0.0.30-gs.1=20with=20its=20image=20and=20chart=20digests=20?= =?UTF-8?q?(run=2034895330411)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Timo Derstappen --- FORK.md | 1 + 1 file changed, 1 insertion(+) diff --git a/FORK.md b/FORK.md index c3a6d052e9..8052e9ab30 100644 --- a/FORK.md +++ b/FORK.md @@ -251,6 +251,7 @@ digests are recorded here: | **v0.0.27-gs.6** (2026-09-12, annotated tag on `29825f84` = gs.5 + #20: a worker whose pod IP changed in place is re-registered (`c3d78657`) + #21: atelet's pinned image set and the chart's `atelet.imageCache.*` (`58d24855`, `ffab110f`) + #22: lifecycle workflows outlive their caller and the router parks a request 30 s, the restore bounded and retried with `RESTORE_TIMED_OUT` (`24afbd45`, `95ce0c43`), with their ledger rows) | v0.0.26 | images `ateapi` `sha256:516d6c84…`, `atecontroller` `sha256:86b56a9c…`, `atelet` `sha256:984988f4…`, `atenet` `sha256:c7dd582d…`, `podcertcontroller` `sha256:2df12254…`, `ateom-gvisor` `sha256:395c3428…` (linux/amd64 + arm64); charts `substrate` `sha256:db1f2c7f…`, `substrate-crds` `sha256:d3e7b20c…`; dataplane `ghcr.io/giantswarm/agentgateway-upstream/agentgateway:v1.5.1-gs.2`; [run 34690425122](https://github.com/giantswarm/substrate/actions/runs/34690425122), every scan clean. The `substrate` chart carries `atelet.imageCache.*` and `atenetRouter.requestParking` (budget 30 s) from this release on; the agent-platform meta chart's range `>=0.0.27-gs.5 <0.0.28-0` admits it without a re-pin | | **v0.0.27-gs.7** (2026-09-12, annotated tag on `5fccbf08` = gs.6 + #25: an ateom keeps re-asserting its capacity report every 10 s, so a Worker record replaced under a running ateom receives its capacity within one interval (`5fccbf08`); the ledger rows follow the tag) | v0.0.26 | images `ateapi` `sha256:aa159b9e…`, `atecontroller` `sha256:8ed3e650…`, `atelet` `sha256:433fe02c…`, `atenet` `sha256:45dd771b…`, `podcertcontroller` `sha256:3a1b684c…`, `ateom-gvisor` `sha256:54679d01…` (linux/amd64 + arm64); charts `substrate` `sha256:30ce1165…`, `substrate-crds` `sha256:f7bb5954…`; dataplane `ghcr.io/giantswarm/agentgateway-upstream/agentgateway:v1.5.1-gs.2`; [run 34698001170](https://github.com/giantswarm/substrate/actions/runs/34698001170), every scan clean. The re-assertion runs in the worker image: a `WorkerPool` takes effect once its `workerImage` is this release's `ateom-gvisor`; the agent-platform meta chart's range `>=0.0.27-gs.6 <0.0.28-0` (4.7.18 and `main`) admits it without a re-pin | | **v0.0.27-gs.8** (2026-09-13, annotated tag on `8fbf9e60` = gs.7 + #27: the chart takes ServiceAccount annotations for atelet and ate-api-server and `extraEnv` for ate-api-server (`atelet.serviceAccount.annotations`, `ateApiServer.serviceAccount.annotations`, `ateApiServer.extraEnv`; `8fbf9e60`); the ledger rows follow the tag) | v0.0.26 | images `ateapi` `sha256:790fb55b…`, `atecontroller` `sha256:e6e3b3f8…`, `atelet` `sha256:5c0228ec…`, `atenet` `sha256:5955a99e…`, `podcertcontroller` `sha256:6a16045d…`, `ateom-gvisor` `sha256:6d8f57f0…` (linux/amd64 + arm64); charts `substrate` `sha256:cff433f5…`, `substrate-crds` `sha256:12ff400f…`; dataplane `ghcr.io/giantswarm/agentgateway-upstream/agentgateway:v1.5.1-gs.2`; [run 34723438405](https://github.com/giantswarm/substrate/actions/runs/34723438405), every scan clean. Defaults empty: the rendered manifests are those of gs.7; the agent-platform meta chart's snapshot-store block (giantswarm/agent-platform#411) sets the three keys, its `components.substrate` range `>=0.0.27-gs.6 <0.0.28-0` admits the release without a re-pin | +| **v0.0.30-gs.1** (2026-09-14, tag on `21027aa1` = the re-pin onto kagent-dev/substrate **v0.0.29** with the ten carried patches replayed, the agentgateway line's `v1.5.1-gs.4` pinned for the router and egress gateway, the suites on `sync/**` pushes; the first release of the new base — cut after the agentlab proof of the three lines together on the same tree's dev build `0.0.30-dev.giantswarm.2026-09-14.19-48-17.h21027aa`: `platform-test` 7/7, `agents-test`, three `skills-test` fixtures (public, `frontmatter-fields`, private with `credentialRef`), `a2a-test` 5/5, a unary A2A turn outliving its caller's deadline, `toolsets-test` 11/11, as an upgrade from `0.0.27-gs.9`) | v0.0.29 | images `ateapi` `sha256:79816517…`, `atecontroller` `sha256:baaffc43…`, `atelet` `sha256:a254fdad…`, `atenet` `sha256:aa0a7f7c…`, `podcertcontroller` `sha256:02ccb3e3…`, `ateom-gvisor` `sha256:cab2bb94…` (linux/amd64 + arm64); charts `substrate` `sha256:abb39761…`, `substrate-crds` `sha256:8beaea23…`; dataplane `ghcr.io/giantswarm/agentgateway-upstream/agentgateway:v1.5.1-gs.4`; [run 34895330411](https://github.com/giantswarm/substrate/actions/runs/34895330411), every scan clean. Consumers: kagent-upstream `Makefile` `SUBSTRATE_VERSION` (giantswarm/kagent-upstream#37 → `v0.11.0-gs.14`), the agent-platform meta chart's ranges `>=0.0.30-gs.1 <0.0.31-0` | | **v0.0.27-gs.9** (2026-09-13, tag on `36d00f80` = gs.8 + #30: the gVisor worker's terminate collects a workload whose runsc containers are already gone (`1dd5d1fb` — `cleanupContainers` lists the containers runsc knows and checks and deletes only those; a delete that fails after removing its container counts as done), with its ledger row; giantswarm/giantswarm#37773) | v0.0.26 | images `ateapi` `sha256:cb267526…`, `atecontroller` `sha256:f2c845ab…`, `atelet` `sha256:0882ecb1…`, `atenet` `sha256:72939cda…`, `podcertcontroller` `sha256:d3eab8cc…`, `ateom-gvisor` `sha256:2240d996…` (linux/amd64 + arm64); charts `substrate` `sha256:bb6ba693…`, `substrate-crds` `sha256:09e363c6…`; dataplane `ghcr.io/giantswarm/agentgateway-upstream/agentgateway:v1.5.1-gs.2`; [run 34783226348](https://github.com/giantswarm/substrate/actions/runs/34783226348), every scan clean | **Scans.** Every own image is scanned with Trivy (HIGH and CRITICAL, fixable only) after the push and before the charts that reference it are published. A fixable finding fails the publish: bump the module (upstream first) or,