Complete storage and delivery guidance

README.md

@@ -22,14 +22,16 @@ Keep these three layers separate while reading the guide:
 | Restate cluster | Three stateful Restate server pods installed into EKS |
 | SDK service | Customer application code that uses a Restate SDK; deployed through an independent application workflow |
 
-The sizing and runtime tuning come from Restate Cloud's
-`3-node.xlarge-vqueues` profile:
+The cluster shape and runtime tuning are derived from Restate Cloud's
+`3-node.xlarge-vqueues` profile. This standalone reference adapts selected
+infrastructure settings, including its initial storage capacity; the
+[profile-fidelity record](docs/04-profile-fidelity.md) explains each difference.
 
 | Setting | Value |
 |---|---|
 | Restate nodes | 3, with hard host anti-affinity |
 | Per-node request | 24 vCPU, 50 GiB memory |
-| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; increase after measuring usage |
+| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; monitor utilization and expand with headroom |
 | Partitions | 48 |
 | Node replication | 2 |
 | Restate image | `docker.restate.dev/restatedev/restate:1.7.7` |

docs/00-architecture.md

@@ -322,9 +322,10 @@ Each Restate pod starts with a 256 GiB PVC using the repository-owned
 - `WaitForFirstConsumer`, so the volume is provisioned in the pod's zone;
 - `Retain`, so deleting the PVC does not delete the EBS PV.
 
-The PVC can grow after observing actual log and snapshot behavior, but it
-cannot shrink. Choose a larger initial value before the first apply when the
-expected workload requires it.
+Monitor `/restate-data` utilization and growth on every pod, then expand PVC
+capacity while enough operational headroom remains. Choose a larger initial
+value before the first apply when the expected workload requires it. See
+[Monitor and expand storage](05-operations.md#monitor-and-expand-storage).
 
 `Retain` is a safety net, not an automatic restore process. A Released PV keeps
 its former claim reference and must be handled explicitly during recovery.

docs/03-deploying-services.md

@@ -364,13 +364,28 @@ data:
 ```
 
 With this in place, a controller failure or a foreign-deployment conflict shows
-as Degraded within one reconcile. Do not classify `AdminCallRejected` as
-terminal: the operator uses it for transient 5xx responses as well as
-incompatible registrations, so the check leaves it Progressing and its message
-explains the timeout. The previous revision keeps serving because the operator
-never replaced it. Teams that deploy the cluster stages with Terraform and the
-applications with Argo CD get the boundary this guide recommends without giving
-up automated health gating.
+as Degraded within one reconcile. `AdminCallRejected` is not reliably terminal:
+the operator uses it for transient 5xx responses as well as incompatible
+registrations, so the check leaves it Progressing and preserves the response
+details in the message. The previous revision keeps serving because the
+operator never replaced it.
+
+The health check itself has no elapsed-time rule. Bound an interactive or CI
+wait explicitly so a permanent rejection cannot leave the caller waiting
+indefinitely:
+
+```bash
+argocd app sync <application> --timeout 600
+# For a sync that another actor started:
+argocd app wait <application> --health --timeout 600
+```
+
+A CLI timeout does not repair or reclassify the revision. Inspect the
+`RestateDeployment` condition, correct an incompatible specification, and
+configure an alert on prolonged Progressing health for automated syncs. Teams
+that deploy the cluster stages with Terraform and applications with Argo CD
+retain automated health gating without treating a transient server response as
+a failed release.
 
 ### Flux
 

docs/04-profile-fidelity.md

@@ -68,10 +68,9 @@ intentional. A customer performing a standard installation can skip it.
   exists, StorageClass parameters are immutable (applying over it fails), and
   a scoped name keeps other workloads off a class this stack owns.
 - Storage capacity: the source profile uses 1 TiB per node; this reference
-  starts with 256 GiB per node. Kubernetes PVCs can grow but not shrink, so the
-  lower initial allocation avoids provisioning unused EBS storage while
-  preserving a path to increase capacity as log and snapshot usage becomes
-  clear.
+  starts with 256 GiB per node. The lower initial allocation avoids
+  provisioning unused EBS storage while preserving a path to expand capacity
+  as local log and partition-store utilization grows.
 - NetworkPolicy: cloud exposes ingress **and admin** only to its own
   authenticating gateway namespace; this repo opens ingress to `restate-apps`
   and keeps the unauthenticated admin API closed to workloads entirely

docs/05-operations.md

@@ -184,6 +184,20 @@ Restate needs no agent or sidecar; the work is on the platform side.
   policy blocks, so add its address under `spec.security.networkEgressRules`
   as described under
   [Private AWS endpoints](00-architecture.md#private-aws-endpoints).
+- **Storage.** The 256 GiB initial PVC size is a starting point rather than a
+  capacity forecast. Monitor the filesystem mounted at `/restate-data` on all
+  three pods, alert on percentage used and recent growth, and expand with
+  enough headroom for the time your change process takes. This repository does
+  not install an observability stack, so connect these signals to the
+  platform's existing monitoring before handing the cluster over.
+
+For a direct point-in-time check:
+
+```bash
+kubectl -n restate exec restate-0 -- df -h /restate-data
+kubectl -n restate exec restate-1 -- df -h /restate-data
+kubectl -n restate exec restate-2 -- df -h /restate-data
+```
 
 ## Verify the snapshot path
 
@@ -468,12 +482,22 @@ own the example service in separate application state. Whichever delivery tool
 owns the service should apply the revision and its rollback. See
 [Deploying services](03-deploying-services.md#roll-out-a-new-version).
 
-### Increase storage
+### Monitor and expand storage
 
-`spec.storage.storageRequestBytes` may only increase. Increasing it updates the
-PVC request; actual expansion depends on the EBS CSI driver and the
-StorageClass. Keep the value unchanged or increase it; retained PVs do not
-shrink.
+Use the `/restate-data` utilization and growth rate described under
+[Observability](#observability) to choose a new capacity with operational
+headroom. Increase `spec.storage.storageRequestBytes` in
+`resources/04-restate-cluster.yaml`, review the manifest diff or Terraform
+plan, and apply it through the workflow that owns the cluster. The operator
+updates the PVC request; completed expansion depends on the EBS CSI driver and
+the `restate-gp3` StorageClass. Confirm the new capacity on every pod:
+
+```bash
+kubectl -n restate get pvc
+kubectl -n restate exec restate-0 -- df -h /restate-data
+kubectl -n restate exec restate-1 -- df -h /restate-data
+kubectl -n restate exec restate-2 -- df -h /restate-data
+```
 
 ### Change runtime sizing or configuration
 

misc/pdf/build_manual_reference.py

@@ -21,9 +21,9 @@
 
 ROOT = Path(__file__).resolve().parents[2]
 DEFAULT_OUTPUT = "output/pdf/restate-eks-manual-deployment-reference.pdf"
-DEFAULT_SOURCE_COMMIT = "388fbec8fdb93b9e9218efaae8fc7fb63120a50b"
-DEFAULT_SOURCE_DATE = "2026-09-01"
-DEFAULT_PREPARED = "1 September 2026"
+DEFAULT_SOURCE_COMMIT = "b31a2e8274bf56fc7493188b903e55106ac42d08"
+DEFAULT_SOURCE_DATE = "2026-09-04"
+DEFAULT_PREPARED = "4 September 2026"
 
 parser = argparse.ArgumentParser(description=__doc__)
 parser.add_argument("--output", default=DEFAULT_OUTPUT)
@@ -466,7 +466,7 @@ def draw_later(canvas, doc):
         ["Component", "Reference value", "Operational meaning"],
         [
             ("Restate", "3 replicas; 24 vCPU and 50 GiB each", "Requires three different eligible nodes"),
-            ("Data", "1 TiB encrypted gp3 per pod; Retain", "EBS survives PVC deletion; reattachment is manual"),
+            ("Data", "256 GiB encrypted gp3 per pod; Retain", "Monitor /restate-data and expand with operational headroom"),
             ("Snapshots", "Dedicated S3 bucket through IRSA", "One bucket or unique prefix per Restate cluster"),
             ("Network", "Default-deny policies where enforced", "Admin 9070 stays private; SDK port 9080 is isolated"),
             ("Exposure", "ClusterIP only", "No ingress, DNS, public endpoint, or auth gateway is created"),
@@ -817,7 +817,7 @@ def draw_later(canvas, doc):
 story.append(
     callout(
         "EXPECTED STORAGE",
-        "EBS CSI, encrypted XFS, 6000 IOPS, 500 MiB/s, WaitForFirstConsumer, allowVolumeExpansion, and reclaimPolicy Retain. Retain preserves EBS after PVC deletion; it does not perform automatic recovery.",
+        "Each pod starts with 256 GiB. The StorageClass uses EBS CSI, encrypted XFS, 6000 IOPS, 500 MiB/s, WaitForFirstConsumer, allowVolumeExpansion, and reclaimPolicy Retain. Monitor /restate-data on every pod and expand with operational headroom. Retain preserves EBS after PVC deletion; it does not perform automatic recovery.",
         "blue",
     )
 )
@@ -1071,8 +1071,8 @@ def draw_later(canvas, doc):
 story.append(P("Safe-change pattern", "H2Custom"))
 story.append(bullet("Verify cluster health and a recent snapshot before changing runtime sizing, storage, image, chart, or experimental settings."))
 story.append(bullet("Review one change at a time. Pod-template changes can roll all three stateful pods and move partition leadership."))
-story.append(bullet("Keep requested storage at its current size or increase it. Existing volume expansion depends on the EBS CSI driver and StorageClass."))
-story.append(bullet("For upgrades, validate release compatibility and every experimental/profile-derived setting; changing only the image is not a complete upgrade plan."))
+story.append(bullet("Monitor /restate-data on every pod. Expand with operational headroom, then confirm capacity."))
+story.append(bullet("For upgrades, revalidate release compatibility and profile-derived settings; changing only the image is incomplete."))
 story.append(PageBreak())
 
 # 12

resources/04-restate-cluster.yaml

@@ -391,7 +391,7 @@ spec:
   storage:
     # restate-gp3 StorageClass from resources/03-gp3-storageclass.yaml (EBS
     # CSI driver required; scoped name — see that file's header). The PVC
-    # starts at 256 GiB per node and can grow later, but
-    # storageRequestBytes cannot be decreased.
+    # starts at 256 GiB per node; monitor /restate-data and increase
+    # storageRequestBytes with enough operational headroom.
     storageClassName: restate-gp3
     storageRequestBytes: 274877906944