From b7f138ebcb51c83187427cfde96257a729bf4159 Mon Sep 17 00:00:00 2001 From: Pavel Tcholakov Date: Thu, 3 Sep 2026 16:57:35 +0200 Subject: [PATCH 1/2] Avoid terminal classification of transient Admin failures --- docs/03-deploying-services.md | 27 +++++++++++++++------------ scripts/wait-restatedeployment.sh | 2 +- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/03-deploying-services.md b/docs/03-deploying-services.md index 3fe02dd..acb0f7d 100644 --- a/docs/03-deploying-services.md +++ b/docs/03-deploying-services.md @@ -312,7 +312,7 @@ Restate reason. The optional `terraform/03-services` root avoids this race with `scripts/wait-restatedeployment.sh`. The script waits for -`status.observedGeneration` to reach `metadata.generation`, then requires +`status.observedGeneration` to equal `metadata.generation`, then requires `Ready=True`; on timeout it prints the last reason and message. If you manage a `RestateDeployment` in another Terraform root, use the same generation-aware gate rather than a positive condition wait alone: @@ -348,8 +348,8 @@ data: if c.status == "True" then hs.status = "Healthy" hs.message = c.message or "Deployed" - elseif c.status == "Unknown" or c.reason == "AdminCallRejected" - or c.reason == "ForeignDeployment" or c.reason == "NotLatest" + elseif c.status == "Unknown" or c.reason == "ForeignDeployment" + or c.reason == "NotLatest" or c.reason == "FailedReconcile" then hs.status = "Degraded" hs.message = c.message @@ -363,11 +363,14 @@ data: return hs ``` -With this in place, a sync of a rejected revision fails its health check within -one reconcile instead of waiting on a timeout, and the previous revision keeps -serving because the operator never replaced it. Teams that deploy the cluster -stages with Terraform and the applications with Argo CD get the boundary this -guide recommends without giving up automated health gating. +With this in place, a controller failure or a foreign-deployment conflict shows +as Degraded within one reconcile. Do not classify `AdminCallRejected` as +terminal: the operator uses it for transient 5xx responses as well as +incompatible registrations, so the check leaves it Progressing and its message +explains the timeout. The previous revision keeps serving because the operator +never replaced it. Teams that deploy the cluster stages with Terraform and the +applications with Argo CD get the boundary this guide recommends without giving +up automated health gating. ### Flux @@ -397,15 +400,15 @@ spec: status.conditions.exists(c, c.type == 'Ready' && (c.status == 'Unknown' || - c.reason == 'AdminCallRejected' || c.reason == 'ForeignDeployment' || c.reason == 'NotLatest' || c.reason == 'FailedReconcile')) ``` -When neither expression is true, Flux continues waiting. Keep a bounded timeout -for transient scaling and admin-connectivity failures, and inspect the `Ready` -condition if it expires. +When neither expression is true, Flux continues waiting. Do not include +`AdminCallRejected` in `failed`: the operator also uses that reason for +transient 5xx responses. Keep a bounded timeout for transient scaling and +admin-connectivity failures, and inspect the `Ready` condition if it expires. ## Useful fields diff --git a/scripts/wait-restatedeployment.sh b/scripts/wait-restatedeployment.sh index 4ae0632..f20542c 100755 --- a/scripts/wait-restatedeployment.sh +++ b/scripts/wait-restatedeployment.sh @@ -60,7 +60,7 @@ while ((SECONDS < deadline)); do if [[ "$observed_generation" =~ ^[0-9]+$ ]] \ && [[ "$generation" =~ ^[0-9]+$ ]] \ - && ((observed_generation >= generation)) \ + && ((observed_generation == generation)) \ && [[ "$ready_status" == "True" ]]; then echo "RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME} is Ready at generation ${generation}." exit 0 From f3132c3bd4cee0c6470158b5ecdf7f932a64a6c3 Mon Sep 17 00:00:00 2001 From: Pavel Tcholakov Date: Thu, 3 Sep 2026 22:30:12 +0200 Subject: [PATCH 2/2] Reduce initial Restate volume allocation --- README.md | 4 ++-- docs/00-architecture.md | 8 ++++++-- docs/04-profile-fidelity.md | 6 +++++- resources/04-restate-cluster.yaml | 16 +++++++++------- 4 files changed, 22 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 61f6f04..bf9862d 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ The sizing and runtime tuning come from Restate Cloud's |---|---| | Restate nodes | 3, with hard host anti-affinity | | Per-node request | 24 vCPU, 50 GiB memory | -| Data volume | 1 TiB encrypted `restate-gp3` EBS volume | +| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; increase after measuring usage | | Partitions | 48 | | Node replication | 2 | | Restate image | `docker.restate.dev/restatedev/restate:1.7.7` | @@ -126,7 +126,7 @@ and draining. See [Deploying services](docs/03-deploying-services.md). namespace: restate-apps RestateDeployment revisions - each Restate pod ──► 1 TiB EBS PV cluster ──IRSA──► dedicated S3 bucket + each Restate pod ──► 256 GiB EBS PV cluster ──IRSA──► dedicated S3 bucket ``` The operator owns the `restate` namespace and materializes the StatefulSet, diff --git a/docs/00-architecture.md b/docs/00-architecture.md index adbebfa..a9533cc 100644 --- a/docs/00-architecture.md +++ b/docs/00-architecture.md @@ -17,7 +17,7 @@ version `3.0.1`. - A `RestateCluster` named `restate` produces an operator-owned namespace of the same name. - Three Restate pods run as one StatefulSet with one pod per Kubernetes node. -- Each pod has a 1 TiB EBS volume; partition snapshots are written to a +- Each pod starts with a 256 GiB EBS volume; partition snapshots are written to a cluster-dedicated S3 bucket. - The operator, not an individual Restate node, provisions the cluster exactly once. @@ -313,7 +313,7 @@ later tuning step. ## Storage and snapshots -Each Restate pod receives a 1 TiB PVC using the repository-owned +Each Restate pod starts with a 256 GiB PVC using the repository-owned `restate-gp3` StorageClass: - EBS CSI provisioner; @@ -322,6 +322,10 @@ Each Restate pod receives a 1 TiB PVC using the repository-owned - `WaitForFirstConsumer`, so the volume is provisioned in the pod's zone; - `Retain`, so deleting the PVC does not delete the EBS PV. +The PVC can grow after observing actual log and snapshot behavior, but it +cannot shrink. Choose a larger initial value before the first apply when the +expected workload requires it. + `Retain` is a safety net, not an automatic restore process. A Released PV keeps its former claim reference and must be handled explicitly during recovery. diff --git a/docs/04-profile-fidelity.md b/docs/04-profile-fidelity.md index 936c132..5997c6e 100644 --- a/docs/04-profile-fidelity.md +++ b/docs/04-profile-fidelity.md @@ -32,7 +32,6 @@ intentional. A customer performing a standard installation can skip it. - **Scheduling**: required hostname anti-affinity + preferred zone spread; the `cloud.restate.dev/interruptible` toleration (inert unless you taint nodes with it). -- **Storage**: 1 TiB gp3 per node. ## Moved, same effect @@ -68,6 +67,11 @@ intentional. A customer performing a standard installation can skip it. than cloud's `gp3`: on a shared cluster a generic `gp3` class often already exists, StorageClass parameters are immutable (applying over it fails), and a scoped name keeps other workloads off a class this stack owns. +- Storage capacity: the source profile uses 1 TiB per node; this reference + starts with 256 GiB per node. Kubernetes PVCs can grow but not shrink, so the + lower initial allocation avoids provisioning unused EBS storage while + preserving a path to increase capacity as log and snapshot usage becomes + clear. - NetworkPolicy: cloud exposes ingress **and admin** only to its own authenticating gateway namespace; this repo opens ingress to `restate-apps` and keeps the unauthenticated admin API closed to workloads entirely diff --git a/resources/04-restate-cluster.yaml b/resources/04-restate-cluster.yaml index 7f33b06..ef8bf8a 100644 --- a/resources/04-restate-cluster.yaml +++ b/resources/04-restate-cluster.yaml @@ -1,10 +1,12 @@ # RestateCluster: 3-node replicated Restate on EKS, managed by the # restate-operator (https://github.com/restatedev/restate-operator). # -# Sizing/tuning is a faithful translation of the Restate Cloud profile +# Sizing/tuning derives from the Restate Cloud profile # `3-node.xlarge-vqueues` (restate-cloud/config/profiles.json): -# 3 replicas x 24 CPU / 50 GiB, 48 partitions, 1 TiB gp3 volume each, -# high-throughput production cell tuning with vqueues enabled. +# 3 replicas x 24 CPU / 50 GiB, 48 partitions, and high-throughput +# production cell tuning with vqueues enabled. The profile uses 1 TiB gp3 +# volumes. This reference initially allocates 256 GiB per node, allowing a +# new installation to increase storage after observing workload usage. # # Deliberately dropped from the cloud rendering (cloud-control-plane machinery # that doesn't apply to a standalone deployment): @@ -388,8 +390,8 @@ spec: storage: # restate-gp3 StorageClass from resources/03-gp3-storageclass.yaml (EBS - # CSI driver required; scoped name — see that file's header). 1 TiB per - # node, from the profile; can be grown later (storageRequestBytes can - # only ever increase). + # CSI driver required; scoped name — see that file's header). The PVC + # starts at 256 GiB per node and can grow later, but + # storageRequestBytes cannot be decreased. storageClassName: restate-gp3 - storageRequestBytes: 1099511627776 + storageRequestBytes: 274877906944