From 35cc6d946710f338d56228b197c002a752f3467d Mon Sep 17 00:00:00 2001 From: igalshilman Date: Thu, 3 Sep 2026 16:39:44 +0200 Subject: [PATCH] Align deployment docs and lifecycle gates --- README.md | 45 ++++--- docs/00-architecture.md | 50 ++++---- docs/01-prerequisites.md | 14 +- docs/02-runbook.md | 7 + docs/03-deploying-services.md | 93 ++++++++++---- docs/05-operations.md | 49 +++++-- resources/05-restate-compute.yaml | 6 +- scripts/validate.sh | 2 + scripts/wait-restatedeployment.sh | 80 ++++++++++++ terraform/01-foundation/variables.tf | 4 +- terraform/02-restate/cluster.tf | 6 +- terraform/02-restate/variables.tf | 4 +- terraform/03-services/outputs.tf | 5 +- terraform/03-services/service.tf | 30 ++++- terraform/03-services/variables.tf | 4 +- terraform/README.md | 183 ++++++++++++++++++--------- terraform/terraform.tfvars.example | 2 +- 17 files changed, 417 insertions(+), 167 deletions(-) create mode 100755 scripts/wait-restatedeployment.sh diff --git a/README.md b/README.md index ce35891..61f6f04 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ Keep these three layers separate while reading the guide: |---|---| | EKS cluster | Existing AWS/Kubernetes infrastructure; this repository leaves it unchanged | | Restate cluster | Three stateful Restate server pods installed into EKS | -| SDK service | Customer application code that uses a Restate SDK; deployed separately | +| SDK service | Customer application code that uses a Restate SDK; deployed through an independent application workflow | The sizing and runtime tuning come from Restate Cloud's `3-node.xlarge-vqueues` profile: @@ -53,12 +53,14 @@ The sizing and runtime tuning come from Restate Cloud's roll out and drain SDK service revisions safely. - Two deployment paths that consume the same manifests: - a transparent, command-by-command `kubectl`/Helm runbook; - - a two-stage Terraform or OpenTofu workflow. + - a two-stage Terraform or OpenTofu cluster workflow, with an optional third + application stage kept in separate state. -Completing either deployment path gives you a healthy Restate cluster, its -operator, persistent storage, and snapshot access. It does **not** deploy a -customer application or expose a public endpoint. Those are separate handoffs -to the application and networking owners. +Completing the cluster portion of either deployment path gives you a healthy +Restate cluster, its operator, persistent storage, and snapshot access. It does +**not** expose a public endpoint. A customer application is a separate handoff +to its application owner; the Terraform path includes an optional, separately +state-managed example for teams that want it. ## Before you deploy @@ -75,10 +77,15 @@ provides the commands that verify each one. Service-CIDR egress policy is derived from `serviceIpv4Cidr`. 4. **Snapshots:** use an S3 bucket dedicated to this Restate cluster. The snapshot prefix is not unique across installations. -5. **Persistent data:** deleting the `RestateCluster` removes its namespace and +5. **Metadata durability:** choose the replicated metadata store shipped in the + example or an S3 metadata store before the first cluster apply. The S3 + option removes metadata quorum from the Restate volumes, while adding an + external dependency and latency consideration. See + [Data durability](docs/00-architecture.md#data-durability-model). +6. **Persistent data:** deleting the `RestateCluster` removes its namespace and PVCs. The StorageClass retains the underlying PVs, but recovery is a manual operation; retained volumes do not reattach automatically. -6. **Existing infrastructure:** the EBS CSI driver, sufficient EKS access, and +7. **Existing infrastructure:** the EBS CSI driver, sufficient EKS access, and an IAM OIDC provider for IRSA must already exist unless the Terraform path is explicitly told to create the OIDC provider. @@ -95,11 +102,11 @@ The complete checklist and verification commands are in Please use one path per installation. If you move an existing installation to Terraform, first import its AWS and Kubernetes resources into Terraform state. -Both paths finish with the Restate cluster installed. Deploy your SDK services -separately, with `kubectl` or your existing application pipeline — they are -still operator-managed: the operator reconciles them as `RestateDeployment` -resources, handling revisioning, registration, and draining. See -[Deploying services](docs/03-deploying-services.md). +Both paths finish with the Restate cluster installed. Deploy SDK services from +an application-owned workflow: use `kubectl`, your existing delivery system, +or the optional `terraform/03-services` example in separate state. The operator +then reconciles each `RestateDeployment`, handling revisioning, registration, +and draining. See [Deploying services](docs/03-deploying-services.md). ## Architecture at a glance @@ -152,6 +159,7 @@ resources/ canonical Kubernetes YAML, Helm values, and IAM policy terraform/01-foundation S3, IAM/IRSA, namespaces, StorageClass, operator terraform/02-restate RestateCluster and its Service-CIDR egress policy +terraform/03-services optional SDK service example in independent state docs/ architecture, deployment, operations, and design notes misc/pdf/ source and LLM-oriented build guide for the PDF companion output/pdf/ committed customer-facing PDF artifacts @@ -175,9 +183,9 @@ deployment paths: The manual path requires every active `REPLACE_ME_*` value in a file being applied to be replaced first. The commented, non-automated Pod Identity adaptation may stay unset, and the compute image may stay unset while compute is -skipped. The Terraform path performs the substitutions it needs in memory, from -its variables and from the EKS cluster itself; the service image is not among -them, because Terraform does not deploy services. +skipped. The Terraform path performs substitutions in memory from its variables +and the EKS cluster. Stages 01 and 02 install the cluster; optional stage 03 +substitutes `service_image` into the SDK service example. ## Validation @@ -201,7 +209,8 @@ cluster or AWS account. | Dedicated snapshot bucket | A snapshot repository belongs to one Restate cluster | | `Retain` EBS reclaim policy | Preserves volumes after accidental CR/namespace deletion | | Restate-specific StorageClass name | Avoids colliding with a shared cluster's generic `gp3` class | -| Two Terraform stages | Restate CRDs must exist in the live cluster before custom resources can be planned | +| Two ordered Terraform cluster stages | Restate CRDs must exist in the live cluster before custom resources can be planned | +| Separate optional service state | Application releases keep their cadence and blast radius separate from cluster infrastructure | ## Known boundaries @@ -210,6 +219,8 @@ cluster or AWS account. not expose port 9070 through an unauthenticated LoadBalancer or Ingress. - S3 snapshots and retained EBS volumes reduce recovery risk, but this repository does not define a complete disaster-recovery procedure. +- The validated cluster manifest uses replicated metadata. Decide whether to + adopt the documented S3 metadata option before the first cluster apply. - Runtime upgrades require re-validating the experimental vqueues settings and should not be performed by changing the image alone. - The example SDK service has placeholder image and sizing values; treat it as diff --git a/docs/00-architecture.md b/docs/00-architecture.md index 819730a..adbebfa 100644 --- a/docs/00-architecture.md +++ b/docs/00-architecture.md @@ -265,30 +265,28 @@ nothing; losing two volumes at once loses log records that had not yet been covered by a snapshot, and can lose the metadata Raft majority. Partition snapshots in S3 exist to speed up that rebuild and to let the log be trimmed; they are not a backup of the cluster. There is no supported backup and restore -procedure for a Restate cluster today. Protecting the EBS volumes is therefore -the operator's first duty: the `Retain` reclaim policy, encryption, and a -deliberate teardown order are what this repository provides toward it. +procedure for a Restate cluster today. Please include the EBS volumes in the +deployment's data-protection plan. This repository supports that with a +`Retain` reclaim policy, encryption, and a deliberate teardown order. -### Recommendation: keep metadata out of the volumes +### Choose the metadata provider before deployment Restate can store cluster metadata in Amazon S3 instead of the built-in Raft -`metadata-server` role, and also supports DynamoDB (Restate 1.5.4 and later) -and etcd; see the +`metadata-server` role; see the [metadata storage documentation](https://docs.restate.dev/server/metadata). -For a deployment whose data matters, we strongly recommend the object-store -provider on AWS: - -- it removes the one piece of irreplaceable state that would otherwise share a - volume with the log, and the object store's durability replaces the Raft - majority as the thing that has to survive; -- it makes the object store a day-one dependency instead of something that can - be deferred. A cluster with the replicated metadata store starts and serves - traffic with no object store and no snapshots configured at all, but its log - is then never trimmed, and the volumes fill up later with no warning that - anything was missing; -- the provider is chosen at initial deployment. Migrating from replicated to - an external store later is supported, but it stops invocation processing for - the duration of the migration. +For a production AWS deployment, please consider the S3 object-store provider +alongside the replicated provider shipped in this example: + +- **S3 metadata** removes the metadata Raft majority from the Restate volumes + and places that state in AWS's object-store durability model. It also makes + S3 availability and latency part of cluster operations. +- **Replicated metadata** keeps the deployment self-contained and matches the + profile validated by this repository. Its metadata remains dependent on a + majority of the three node volumes. + +Choose before the initial deployment when practical. A later migration from +replicated to external metadata is supported, but invocation processing stops +for the duration of that migration. The configuration change in `resources/04-restate-cluster.yaml` is to remove `metadata-server` from `roles` and add, next to the snapshot destination: @@ -302,13 +300,11 @@ aws-region = "" The IAM policy in `resources/01-restate-snapshots-iam-policy.json` grants bucket-wide object read, write, and delete, which is what the provider uses. -This repository's validation covers the replicated store only; test the -object-store configuration before adopting it. Only Amazon S3 is -supported for metadata; S3-compatible stores such as MinIO are supported for -snapshots but not for metadata, and the bucket must be in the same region as -the cluster because metadata latency affects cluster operations directly. -Outside AWS, the equivalent is etcd; GCS and Azure Blob are snapshot -destinations only. +This repository's validation covers the replicated store only, so test the +object-store configuration before adopting it. Only Amazon S3 is supported for +object-store metadata; S3-compatible stores such as MinIO are supported for +snapshots but not for metadata. Keep the bucket in the same region as the +cluster because metadata latency affects cluster operations directly. This repository still ships the replicated metadata store because it is what the source profile runs and what was validated end to end here. Treat the diff --git a/docs/01-prerequisites.md b/docs/01-prerequisites.md index 0eb166f..1ce2134 100644 --- a/docs/01-prerequisites.md +++ b/docs/01-prerequisites.md @@ -41,6 +41,9 @@ checklist has passed. enforced; for Terraform, a globally unique name has been chosen. - [ ] The EKS cluster name is at most 46 characters so the derived snapshot role name stays within IAM's 64-character limit. +- [ ] The replicated or S3 metadata provider has been chosen for the initial + deployment after reviewing the + [durability trade-off](00-architecture.md#data-durability-model). - [ ] The required tools for one deployment path are installed. ## Set deployment context @@ -321,11 +324,11 @@ Choose one path; you do not need every tool in both columns. | Tool | Manual path | Terraform path | Purpose | |---|:---:|:---:|---| | AWS CLI v2 | ✓ | ✓ | Identity, EKS lookup, IAM, S3, exec auth | -| `kubectl` | ✓ | recommended | Apply and diagnose Kubernetes resources | +| `kubectl` | ✓ | recommended; required for optional stage 03 | Apply, diagnose, and run the service readiness gate | | `eksctl` | ✓ | — | OIDC provider and IRSA role plumbing | | Helm | ✓ | optional* | Install the operator manually; verify or authenticate the OCI chart pull | -| Terraform ≥1.5 or OpenTofu | — | ✓ | Apply the two Terraform stages | -| `jq` | ✓ | recommended | Format API responses | +| Terraform ≥1.5 or OpenTofu | — | ✓ | Apply the two cluster stages and optional service stage | +| `jq` | ✓ | recommended; required for optional stage 03 | Format API responses and parse service readiness | | `restatectl` | via pod | via pod | Cluster status and snapshots; provisioning remains operator-managed | | `restate` CLI | optional | optional | Service/deployment administration | @@ -365,8 +368,9 @@ grep -RIn 'REPLACE_ME' resources | `REPLACE_ME_SERVICE_CIDR` | `06-restate-service-cidr-egress.yaml` | Cluster Service IPv4 CIDR; needed where the CNI enforces NetworkPolicy | | `REPLACE_ME_EKS_CLUSTER_NAME` | `02-restate-operator.values.yaml` (commented) | Only when adapting the repository for EKS Pod Identity; the supplied IAM paths implement IRSA only | -The Terraform path does not modify the files. It replaces the required values -in memory from `terraform.tfvars`. +The Terraform path does not modify the files. It replaces values in memory from +`terraform.tfvars`, the EKS API, and—for optional stage 03—the +`TF_VAR_service_image` environment variable. ## Appendix: an example cluster, for illustration only diff --git a/docs/02-runbook.md b/docs/02-runbook.md index 103ec66..a7e7cbd 100644 --- a/docs/02-runbook.md +++ b/docs/02-runbook.md @@ -246,6 +246,13 @@ kubectl get storageclass restate-gp3 -o yaml The expected class uses encrypted XFS, 6000 IOPS, 500 MiB/s, `WaitForFirstConsumer`, and `reclaimPolicy: Retain`. +Before the first cluster apply, choose the metadata provider. The canonical +manifest uses the replicated provider validated by this repository. An S3 +metadata provider reduces dependence on a majority of the node volumes but +adds an external availability and latency dependency. Review +[Data durability](00-architecture.md#data-durability-model) and, if you choose +S3, make and validate the documented manifest change before continuing. + Confirm that no active placeholder remains in the cluster manifest, then apply it: diff --git a/docs/03-deploying-services.md b/docs/03-deploying-services.md index 5c8dda2..3fe02dd 100644 --- a/docs/03-deploying-services.md +++ b/docs/03-deploying-services.md @@ -17,11 +17,11 @@ no service image. Build one from the your team uses; any image whose SDK endpoint listens on port 9080 fits the manifest unchanged. -Both deployment paths finish with the Restate cluster installed. Neither the -runbook nor the Terraform modules deploy your services: a service changes at -your application's cadence, from your application's pipeline, and does not -belong in the state that owns the cluster. Apply the manifest below with -`kubectl`, or fold it into whatever already ships your applications. +Both deployment paths finish with the Restate cluster installed. A service +changes at your application's cadence and should not share the state that owns +the cluster. Apply the manifest below with `kubectl`, fold it into your existing +delivery system, or use the optional `terraform/03-services` example in its own +state. That is a boundary of the deployment paths, not of the operator. The operator installed in stage 01 reconciles `RestateDeployment` resources the same way it @@ -284,10 +284,14 @@ refused to register them. | `Ready` | Reason | Meaning | |---|---|---| | `True` | `Deployed` | Latest revision registered and serving new invocations | -| `False` | `ReplicaSetScaling`, `ReplicaSetPodNotReady`, `ReplicaSetPodNotAvailable`, `ReplicaSetNoStatus` | Pods still starting; normal during a rollout | -| `False` | `AdminCallFailed` | Admin API unreachable or returned a server error; the operator retries | -| `False` | `AdminCallRejected` | Restate refused the registration; the message carries Restate's error and the operator retries every 30 s but will not succeed until the template changes | -| `False` | `HashCollision`, `FailedReconcile` | Operator-side error; inspect the operator logs | +| `False` | `ReplicaSetScaling`, `ReplicaSetPodNotReady`, `ReplicaSetPodNotAvailable` | Pods still starting; normal during a rollout | +| `False` | initial ReplicaSet status absent | Operator 3.0.1 may put `ReplicaSetNoStatus` in the message rather than the reason; continue waiting for pod status | +| `False` | `AdminCallFailed` | The admin request failed in transport or response decoding; the operator retries | +| `False` | `AdminCallRejected` | The admin API returned a non-success response, including a 5xx; inspect the response in the message to distinguish a transient server problem from an incompatible registration | +| `False` | `NotLatest`, `ForeignDeployment` | The revision conflicts with the deployment Restate considers current; inspect the resource, Restate deployments, and operator logs before retrying | +| `False` | `HashCollision` | The generated revision name collided; the operator retries with a new collision count | +| `False` | `RouteNotReady`, `ConfigurationNotReady` | A Knative-backed service is still reconciling | +| `Unknown` | `FailedReconcile` | The controller hit an unexpected reconciliation error; inspect the condition message and operator logs | The operator also publishes a Warning Event with the same message for `AdminCallFailed` and `AdminCallRejected`, so `kubectl describe @@ -300,24 +304,24 @@ run unregistered until the spec is corrected. ### Terraform -The `kubernetes_manifest` `wait` block matches only positive states: a -condition reaching a value, a field matching a regex, or a rollout completing -for the built-in workload kinds. It cannot fail on `Ready=False`, so a rejected -revision makes `terraform apply` block until its timeout and then report -`context deadline exceeded` without Restate's reason. +The `kubernetes_manifest` condition waiter compares condition type and status, +but not `status.observedGeneration`. During an update it can therefore accept +`Ready=True` from the previous generation before the operator observes the new +template. It also cannot stop early on `Ready=False` or `Unknown` with a useful +Restate reason. -If a `RestateDeployment` is applied from Terraform anyway, set an update -timeout well under the default, print the condition on failure, and read the -reason from the resource rather than from Terraform: +The optional `terraform/03-services` root avoids this race with +`scripts/wait-restatedeployment.sh`. The script waits for +`status.observedGeneration` to reach `metadata.generation`, then requires +`Ready=True`; on timeout it prints the last reason and message. If you manage a +`RestateDeployment` in another Terraform root, use the same generation-aware +gate rather than a positive condition wait alone: ```bash kubectl -n restate-apps get restatedeployment \ -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason}: {.message}{"\n"}{end}' ``` -State is not left inconsistent: the provider keeps the previous manifest in -state when the wait fails, so the next plan already proposes the rollback. - ### Argo CD Argo CD has no built-in health assessment for `restate.dev` kinds and reports @@ -332,8 +336,9 @@ data: if obj.status == nil then return hs end - if obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil - and obj.status.observedGeneration < obj.metadata.generation then + if obj.metadata.generation ~= nil and + (obj.status.observedGeneration == nil or + obj.status.observedGeneration < obj.metadata.generation) then hs.message = "Waiting for the operator to observe the latest generation" return hs end @@ -343,7 +348,9 @@ data: if c.status == "True" then hs.status = "Healthy" hs.message = c.message or "Deployed" - elseif c.reason == "AdminCallRejected" then + elseif c.status == "Unknown" or c.reason == "AdminCallRejected" + or c.reason == "ForeignDeployment" or c.reason == "NotLatest" + or c.reason == "FailedReconcile" then hs.status = "Degraded" hs.message = c.message else @@ -364,11 +371,41 @@ guide recommends without giving up automated health gating. ### Flux -Flux's health checks use kstatus, which treats a `Ready=False` condition as -still reconciling and reports failure only on a `Stalled=True` condition. The -operator does not set `Stalled`, so a rejected revision keeps a Flux -`Kustomization` in progress until its `timeout`. Set that timeout to a few -minutes and read the `Ready` reason as above to see why. +Without a custom expression, Flux's kstatus handling leaves these +`Ready=False` failures in progress until the `Kustomization` timeout because +the operator does not set `Stalled=True`. Current Flux releases support +[`healthCheckExprs`](https://fluxcd.io/flux/components/kustomize/kustomizations/#health-check-expressions), +so add a generation-aware check to the application `Kustomization`: + +```yaml +spec: + wait: true + timeout: 10m + healthCheckExprs: + - apiVersion: restate.dev/v1beta1 + kind: RestateDeployment + current: >- + has(status.observedGeneration) && + status.observedGeneration == metadata.generation && + has(status.conditions) && + status.conditions.exists(c, + c.type == 'Ready' && c.status == 'True') + failed: >- + has(status.observedGeneration) && + status.observedGeneration == metadata.generation && + has(status.conditions) && + status.conditions.exists(c, + c.type == 'Ready' && + (c.status == 'Unknown' || + c.reason == 'AdminCallRejected' || + c.reason == 'ForeignDeployment' || + c.reason == 'NotLatest' || + c.reason == 'FailedReconcile')) +``` + +When neither expression is true, Flux continues waiting. Keep a bounded timeout +for transient scaling and admin-connectivity failures, and inspect the `Ready` +condition if it expires. ## Useful fields diff --git a/docs/05-operations.md b/docs/05-operations.md index 9c55b62..5157cb0 100644 --- a/docs/05-operations.md +++ b/docs/05-operations.md @@ -463,8 +463,9 @@ Update the pod template—usually the image—in `resources/05-restate-compute.yaml`, review the diff, and apply it. The operator creates a new immutable revision and drains the old one. -This is the same on both deployment paths: the Terraform modules deploy the -cluster, not your services. See +The required Terraform stages deploy the cluster, while optional stage 03 can +own the example service in separate application state. Whichever delivery tool +owns the service should apply the revision and its rollback. See [Deploying services](03-deploying-services.md#roll-out-a-new-version). ### Increase storage @@ -490,9 +491,11 @@ The operator creates a PodDisruptionBudget on the Restate pods with `maxUnavailable: 1`, so a node drain or a managed node-group upgrade evicts one Restate pod at a time and waits for it to be Ready elsewhere before the next. With three nodes and required host anti-affinity, an evicted pod has nowhere to -go until a replacement node exists, so drain with a surge node available or -expect the pod to sit Pending until the drained node returns. Check -`restatectl status` between nodes, as for any other roll. +go until a replacement node exists. Its EBS volume is also zonal, so the +replacement must be eligible in the volume's Availability Zone and satisfy the +pod's selectors, taints, and host anti-affinity. Provide that capacity before +draining, or expect the pod to remain Pending until an eligible node is +available. Check `restatectl status` between nodes, as for any other roll. ### Upgrade Restate or the operator @@ -509,17 +512,41 @@ The image and chart are intentionally pinned. Before upgrading: Changing only the container image is not a complete upgrade review. -On the Terraform path, `terraform apply` does not wait for the roll. The -`RestateCluster` is already `Ready=True` when the change is submitted, so the -stage-02 wait is satisfied immediately and Terraform returns while the -StatefulSet is still replacing pods one at a time, highest ordinal first. -Gate the next pipeline step on the StatefulSet instead: +On the Terraform path, `terraform apply` does not provide a generation-aware +rollout gate for `RestateCluster`. The condition can still be `Ready=True` from +the previous generation when the change is submitted. For a stage-02 change +that should alter the StatefulSet pod template, record its generation before +the apply, wait until the operator updates it, and only then wait for rollout +completion: ```bash +PREVIOUS_STS_GENERATION="$( + kubectl -n restate get statefulset/restate \ + -o jsonpath='{.metadata.generation}' +)" + +terraform -chdir=terraform/02-restate apply restate.tfplan + +DEADLINE=$((SECONDS + 300)) +until CURRENT_STS_GENERATION="$( + kubectl -n restate get statefulset/restate \ + -o jsonpath='{.metadata.generation}' +)" && ((CURRENT_STS_GENERATION > PREVIOUS_STS_GENERATION)); do + if ((SECONDS >= DEADLINE)); then + echo "The operator did not update statefulset/restate within 5 minutes" >&2 + exit 1 + fi + sleep 3 +done + kubectl -n restate rollout status statefulset/restate --timeout=15m kubectl -n restate exec restate-0 -- restatectl status ``` +Run the initial health checks and capture the old generation before applying +the saved plan. A bare `kubectl rollout status` issued immediately after +Terraform can otherwise report the already-completed previous revision. + Expect the roll itself to take a few minutes for three pods. Clients may see a connection reset at the moment a pod terminates; invocations are retried by Restate, but a client holding an open connection to that pod is not. @@ -545,7 +572,7 @@ Neither mechanism is a complete, automatic disaster-recovery workflow, and snapshots are not a backup: the replicated log and the cluster metadata exist only on the volumes, and partition state is what snapshots let you rebuild. See [Data durability model](00-architecture.md#data-durability-model) for which -data is irreplaceable and for the recommendation to keep metadata in S3. +data is irreplaceable and for the replicated-versus-S3 metadata trade-off. Released PVs retain their old claim references and do not bind to replacement PVCs automatically. Before removal or another data-affecting change, record the PV, PVC, Availability Zone, and EBS volume-id mapping: diff --git a/resources/05-restate-compute.yaml b/resources/05-restate-compute.yaml index 8bf9c83..557865e 100644 --- a/resources/05-restate-compute.yaml +++ b/resources/05-restate-compute.yaml @@ -43,9 +43,9 @@ spec: # REPLACE_ME: your SDK service image, pushed to a registry the nodes # can pull from. This repository ships no service; start from one of # the SDK examples at https://github.com/restatedev/examples in the - # language of your choice. Any image listening on 9080 works. Keep - # the port name "restate" because the operator uses it for the - # registration URL. + # language of your choice. Use a Restate SDK service whose endpoint + # listens on 9080. Keep the port name "restate" because the operator + # uses it for the registration URL. image: REPLACE_ME_SERVICE_IMAGE ports: - name: restate diff --git a/scripts/validate.sh b/scripts/validate.sh index 860c8b7..a7d01b6 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -6,6 +6,8 @@ cd "$REPOSITORY_ROOT" export TF_IN_AUTOMATION=1 +bash -n scripts/validate.sh scripts/wait-restatedeployment.sh + tofu fmt -check -recursive terraform for TOFU_STAGE in terraform/01-foundation terraform/02-restate terraform/03-services; do diff --git a/scripts/wait-restatedeployment.sh b/scripts/wait-restatedeployment.sh new file mode 100755 index 0000000..4ae0632 --- /dev/null +++ b/scripts/wait-restatedeployment.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${CLUSTER_NAME:?Set CLUSTER_NAME to the EKS cluster name}" +: "${AWS_REGION:?Set AWS_REGION to the EKS cluster region}" +: "${RSD_NAMESPACE:?Set RSD_NAMESPACE to the RestateDeployment namespace}" +: "${RSD_NAME:?Set RSD_NAME to the RestateDeployment name}" + +RSD_TIMEOUT_SECONDS="${RSD_TIMEOUT_SECONDS:-600}" + +for command in aws kubectl jq; do + if ! command -v "$command" >/dev/null 2>&1; then + echo "Required command not found: $command" >&2 + exit 1 + fi +done + +TEMP_KUBECONFIG="$(mktemp "${TMPDIR:-/tmp}/restate-kubeconfig.XXXXXX")" +trap 'rm -f -- "$TEMP_KUBECONFIG"' EXIT + +aws eks update-kubeconfig \ + --dry-run \ + --name "$CLUSTER_NAME" \ + --region "$AWS_REGION" >"$TEMP_KUBECONFIG" + +deadline=$((SECONDS + RSD_TIMEOUT_SECONDS)) +last_summary="" + +echo "Waiting for RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME} to become Ready at its current generation..." + +while ((SECONDS < deadline)); do + if deployment_json="$( + kubectl --kubeconfig "$TEMP_KUBECONFIG" \ + --namespace "$RSD_NAMESPACE" \ + get restatedeployment "$RSD_NAME" \ + --output json 2>/dev/null + )"; then + summary="$( + jq -j ' + (.status.conditions // [] | map(select(.type == "Ready")) | last) as $ready + | [ + (.metadata.generation // 0), + (.status.observedGeneration // 0), + ($ready.status // "Missing"), + ($ready.reason // "NoReason"), + ($ready.message // "No Ready condition reported") + ] + | map(tostring) + | join("\u001f") + ' <<<"$deployment_json" + )" + + IFS=$'\x1f' read -r generation observed_generation ready_status ready_reason ready_message <<<"$summary" + + current_summary="generation=${generation}, observedGeneration=${observed_generation}, Ready=${ready_status}, reason=${ready_reason}: ${ready_message}" + if [[ "$current_summary" != "$last_summary" ]]; then + echo "$current_summary" + last_summary="$current_summary" + fi + + if [[ "$observed_generation" =~ ^[0-9]+$ ]] \ + && [[ "$generation" =~ ^[0-9]+$ ]] \ + && ((observed_generation >= generation)) \ + && [[ "$ready_status" == "True" ]]; then + echo "RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME} is Ready at generation ${generation}." + exit 0 + fi + elif [[ "$last_summary" != "resource not found" ]]; then + echo "RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME} is not readable yet; continuing to wait." + last_summary="resource not found" + fi + + sleep 3 +done + +echo "Timed out after ${RSD_TIMEOUT_SECONDS}s waiting for RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME}." >&2 +if [[ -n "$last_summary" ]]; then + echo "Last status: ${last_summary}" >&2 +fi +exit 1 diff --git a/terraform/01-foundation/variables.tf b/terraform/01-foundation/variables.tf index bb8c720..0f624c6 100644 --- a/terraform/01-foundation/variables.tf +++ b/terraform/01-foundation/variables.tf @@ -1,5 +1,5 @@ -# Both stages read the same ../terraform.tfvars (see terraform/README.md), so -# the variable set is declared identically in both stages — each simply +# Both cluster stages read the same ../terraform.tfvars (see terraform/README.md), +# so the variable set is declared identically in both stages — each simply # ignores what it doesn't use. Keep the two files in sync. variable "cluster_name" { diff --git a/terraform/02-restate/cluster.tf b/terraform/02-restate/cluster.tf index cd2ac84..d2628f3 100644 --- a/terraform/02-restate/cluster.tf +++ b/terraform/02-restate/cluster.tf @@ -25,9 +25,9 @@ locals { resource "kubernetes_manifest" "restate_cluster" { manifest = local.restate_cluster_manifest - # Runbook step 4's "watch until provisioned and Ready" as code: the - # operator provisions the cluster once restate-0 is Running, the pods then - # turn Ready, and the operator sets the Ready condition on the CR. + # Gate initial provisioning on the cluster's Ready condition. The CR has no + # observedGeneration, so updates that roll the StatefulSet use the explicit + # generation-sensitive procedure in docs/05-operations.md instead. wait { condition { type = "Ready" diff --git a/terraform/02-restate/variables.tf b/terraform/02-restate/variables.tf index bb8c720..0f624c6 100644 --- a/terraform/02-restate/variables.tf +++ b/terraform/02-restate/variables.tf @@ -1,5 +1,5 @@ -# Both stages read the same ../terraform.tfvars (see terraform/README.md), so -# the variable set is declared identically in both stages — each simply +# Both cluster stages read the same ../terraform.tfvars (see terraform/README.md), +# so the variable set is declared identically in both stages — each simply # ignores what it doesn't use. Keep the two files in sync. variable "cluster_name" { diff --git a/terraform/03-services/outputs.tf b/terraform/03-services/outputs.tf index 5ed2cbe..d192cb2 100644 --- a/terraform/03-services/outputs.tf +++ b/terraform/03-services/outputs.tf @@ -2,6 +2,7 @@ output "service_name" { value = local.service_manifest.metadata.name } -output "invoke_hint" { - value = "kubectl -n restate port-forward svc/restate 8080:8080 & curl localhost:8080/Greeter/greet --json '\"Restate\"'" +output "ingress_port_forward_hint" { + description = "Open a local connection to Restate ingress; the invocation path depends on the deployed SDK service." + value = "kubectl -n restate port-forward svc/restate 8080:8080" } diff --git a/terraform/03-services/service.tf b/terraform/03-services/service.tf index fc34e60..2695f0b 100644 --- a/terraform/03-services/service.tf +++ b/terraform/03-services/service.tf @@ -15,13 +15,6 @@ locals { resource "kubernetes_manifest" "service" { manifest = local.service_manifest - wait { - condition { - type = "Ready" - status = "True" - } - } - timeouts { create = "10m" update = "10m" @@ -29,3 +22,26 @@ resource "kubernetes_manifest" "service" { delete = "30m" } } + +# The provider's condition waiter does not compare status.observedGeneration +# with metadata.generation. On an update it can therefore accept Ready=True +# from the previous revision before the operator has observed the new pod +# template. Use the repository's generation-aware gate after every manifest +# change instead. +resource "terraform_data" "wait_for_service" { + triggers_replace = sha256(jsonencode(local.service_manifest)) + + depends_on = [kubernetes_manifest.service] + + provisioner "local-exec" { + command = "'${path.module}/../../scripts/wait-restatedeployment.sh'" + + environment = { + AWS_REGION = var.region + CLUSTER_NAME = var.cluster_name + RSD_NAME = local.service_manifest.metadata.name + RSD_NAMESPACE = local.service_manifest.metadata.namespace + RSD_TIMEOUT_SECONDS = "600" + } + } +} diff --git a/terraform/03-services/variables.tf b/terraform/03-services/variables.tf index 7f6810a..522a8f8 100644 --- a/terraform/03-services/variables.tf +++ b/terraform/03-services/variables.tf @@ -10,7 +10,9 @@ variable "region" { } variable "snapshots_bucket" { - type = string + type = string + default = null + nullable = true } variable "create_oidc_provider" { diff --git a/terraform/README.md b/terraform/README.md index b23204b..de618ea 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -1,9 +1,10 @@ # Terraform and OpenTofu deployment This guide is for a cloud engineer installing Restate on an **existing EKS -cluster** with Terraform or OpenTofu. The modules create the AWS and Kubernetes -resources needed by the Restate operator and a three-node Restate cluster; they -do not create EKS infrastructure, public ingress, or customer applications. +cluster** with Terraform or OpenTofu. The two required stages create the AWS +and Kubernetes resources needed by the Restate operator and a three-node +Restate cluster. An optional third root demonstrates application delivery in +separate state. The modules do not create EKS infrastructure or public ingress. If Restate itself is new to you, start with the [top-level overview](../README.md) for the product and ownership model, then @@ -12,9 +13,10 @@ return here. The apply is split into two ordered stages because the operator must install Restate's Kubernetes resource definitions before Terraform can plan a `RestateCluster`. Stage 01 creates the foundation and operator. Stage 02 creates -the three-node Restate cluster. If the prerequisites are already complete, go -directly to [Quick start](#quick-start); the intervening sections explain the -plan and ownership boundaries. +the three-node Restate cluster. The independent stage 03 example deploys one +SDK service when an application team chooses Terraform for delivery. If the +prerequisites are already complete, go directly to [Quick start](#quick-start); +the intervening sections explain the plan and ownership boundaries. The examples use Terraform 1.5+. OpenTofu is also supported; replace `terraform` with `tofu` in the commands. @@ -40,9 +42,14 @@ fail planning with a clear error when the target reports another IP family. 02-restate ├─ RestateCluster/restate └─ Service-CIDR egress NetworkPolicy (opt-out) + +03-services (optional, independent application state) + └─ RestateDeployment/service and its readiness gate ``` -The two stages have separate state and must be applied in order. +The two cluster stages have separate state and must be applied in order. The +optional service root also has separate state and is applied only after the +cluster is healthy. The chart installs and upgrades its CRDs, but annotates them with `helm.sh/resource-policy: keep`. Destroying the Helm release therefore leaves @@ -87,7 +94,8 @@ stage 01 HCL: the bucket controls, OIDC lookup/creation, and IRSA trust role. ## Inputs -Both stages read the same `terraform.tfvars` and declare the same input set. +Stages 01 and 02 read the same `terraform.tfvars` and declare the same input +set. | Variable | Required | Default | Meaning | |---|:---:|---|---| @@ -97,10 +105,18 @@ Both stages read the same `terraform.tfvars` and declare the same input set. | `create_oidc_provider` | — | `false` | Create the cluster IAM OIDC provider instead of looking it up | | `create_service_cidr_egress_policy` | — | `true` | Apply the Service-CIDR egress policy; required where the CNI enforces NetworkPolicy | +Optional stage 03 also reads `cluster_name` and `region`. It accepts the shared +variable file, but its only additional required input is `service_image`, which +should be an immutable image digest or release tag supplied by the application +pipeline. + ## Authentication and authorization The AWS provider uses ambient credentials. The Kubernetes and Helm providers -call `aws eks get-token`, so no kubeconfig file is required by Terraform. +call `aws eks get-token`, so no existing kubeconfig file is required by +Terraform. Stage 03's readiness script additionally requires `aws`, `kubectl`, +and `jq`; it creates a temporary kubeconfig with `aws eks update-kubeconfig +--dry-run` and removes it when the check finishes. The identity still needs access on both planes: @@ -162,7 +178,8 @@ DNAT (Calico, Cilium) the policy is unnecessary rather than harmful, and This is also the reason IPv4 is an explicit support boundary. An IPv6 EKS cluster reports `serviceIpv6Cidr` instead, and this reference has not validated the operator policies, Pod Identity agent path, or VPC CNI IPv6 behavior as one -system. Both stages reject a non-IPv4 target rather than partially applying it. +system. Both cluster stages reject a non-IPv4 target rather than partially +applying it. The policy lives in the operator-owned namespace, so destroying the `RestateCluster` removes it too. Its full rationale and the exact scope of the @@ -177,7 +194,7 @@ cp terraform/terraform.tfvars.example terraform/terraform.tfvars ``` Edit the file and set `cluster_name`, `region`, and `snapshots_bucket`. Those -three are the whole required input set. +three are the complete required input set for the cluster stages. ### 1. Initialize and plan stage 01 @@ -217,6 +234,13 @@ kubectl get crd \ ### 2. Initialize and plan stage 02 +Before the first cluster plan, decide whether to keep the validated replicated +metadata store or configure the documented S3 metadata provider. This choice +affects the cluster's durability model and is best made before it stores data; +see [Data durability](../docs/00-architecture.md#data-durability-model). If you +choose S3, update the canonical cluster manifest and validate that configuration +before continuing. + ```bash terraform -chdir=terraform/02-restate init terraform -chdir=terraform/02-restate validate @@ -235,8 +259,11 @@ Apply the reviewed plan: terraform -chdir=terraform/02-restate apply restate.tfplan ``` -The apply waits up to 15 minutes for `RestateCluster/restate` to report -`Ready=True`. +The initial apply waits up to 15 minutes for `RestateCluster/restate` to report +`Ready=True`. The resource does not expose `observedGeneration`, so this +condition alone is not an update-rollout gate; use the generation-sensitive +procedure in [Operations](../docs/05-operations.md#upgrade-restate-or-the-operator) +for later pod-template changes. ### 3. Verify the deployment @@ -254,47 +281,53 @@ Use the [manual runbook completion checklist](../docs/02-runbook.md#completion-c and [Operations guide](../docs/05-operations.md) for the rest of the health checks. -## SDK services: operator-managed, not Terraform-managed - -Stages 01 and 02 deploy the Restate cluster. They do not deploy your services, -and they have no `service_image` variable. A separate, optional -`terraform/03-services` example shows what applying the RestateDeployment from -Terraform looks like if that is where your delivery pipeline already lives; its -limitations are described under -[Health signals for delivery tools](../docs/03-deploying-services.md#health-signals-for-delivery-tools). - -Your services are still managed, just not from here. The operator reconciles -them through its own `RestateDeployment` custom resource, which gives each -revision an immutable ReplicaSet and Service, registers it with the cluster's -admin API automatically, and drains superseded revisions only once no invocation -is still pinned to them. That is a better lifecycle than Terraform could offer -for this object, and it is already running in the cluster stage 01 installed. - -Keeping it out of this state is deliberate. A `RestateDeployment` changes on -every image build, at your application's cadence, from your application's -pipeline. Holding it here would make each image bump an infrastructure change, -report plan drift whenever anything else rolls out a revision, and make -`terraform destroy` of the cluster block on the operator's drain finalizer -waiting for in-flight invocations. Cluster state and application state also have -very different blast radii, and one state file gives them the same one. - -Apply `resources/05-restate-compute.yaml` with `kubectl`, or fold it into -whatever already ships your applications: +## Optional stage 03: deploy the SDK service example + +Stages 01 and 02 deliberately stop at a healthy Restate cluster. SDK services +change at an application's cadence, so keep them in an application-owned +workflow and state. Use `kubectl`, your existing delivery system, or the +independent `terraform/03-services` example. + +To use the example, supply an immutable Restate SDK service image and review a +saved plan: + +```bash +export TF_VAR_service_image='registry.example/service@sha256:' + +terraform -chdir=terraform/03-services init +terraform -chdir=terraform/03-services validate +terraform -chdir=terraform/03-services plan \ + -var-file=../terraform.tfvars \ + -out=service.tfplan +terraform -chdir=terraform/03-services apply service.tfplan +``` + +The root substitutes the image into `resources/05-restate-compute.yaml`. After +each manifest change, its script waits until `status.observedGeneration` has +caught up with `metadata.generation` and the current `Ready` condition is +`True`. This prevents a previous revision's `Ready=True` from satisfying an +update. On a timeout, the last condition reason and message are printed; use +the delivery-health guide below to interpret them. + +The operator gives each revision an immutable ReplicaSet and Service, registers +it with Restate, and drains superseded revisions after pinned invocations +finish. Keeping application state separate prevents an image release from +sharing the cluster infrastructure's state and blast radius. + +Further application-delivery guidance: - [Deploying SDK services](../docs/03-deploying-services.md) — the lifecycle contract, rollout, drain, rollback, and per-symptom troubleshooting; - [Health signals for delivery tools](../docs/03-deploying-services.md#health-signals-for-delivery-tools) - — how a rejected revision surfaces in Terraform, Argo CD, and Flux, with an - Argo CD health check; the common split is Terraform for these two stages and - Argo CD for the applications; + — how a rejected revision surfaces in Terraform, Argo CD, and Flux; - [operator service examples](https://github.com/restatedev/restate-operator/tree/main/examples/services/greeter) — upstream `RestateDeployment` manifests, including a Knative variant; - [Restate on Kubernetes](https://docs.restate.dev/deploy/services/kubernetes) — the product documentation for the same model. -What stage 02 does provide is everything the cluster side of registration needs, -including the Service-CIDR egress policy described above — so a service deployed -by any means can register. +Stage 02 provides the cluster side of registration, including the Service-CIDR +egress policy described above, so a service deployed by any supported workflow +can register. ## Outputs @@ -313,11 +346,18 @@ Stage 02 exposes: | `restate_cluster_name` | Restate custom-resource and generated namespace name | | `port_forward_hint` | Command for local ingress and admin access | +Optional stage 03 exposes: + +| Output | Meaning | +|---|---| +| `service_name` | Name of the example `RestateDeployment` | +| `ingress_port_forward_hint` | Command for local Restate ingress access; the service invocation path is application-specific | + Inspect them with `terraform -chdir= output`. ## State and reproducibility -Both roots default to local state. That is suitable only for a scratch +Each root defaults to local state. That is suitable only for a scratch environment. For shared or durable environments, configure an encrypted, locking remote backend separately in each `versions.tf` before the first apply. @@ -373,32 +413,59 @@ the Restate cluster, PVs, snapshot bucket, or cluster OIDC provider at risk. ## Destroy -Teardown has three separate decisions: drain applications, destroy the Restate -cluster, then decide which stage-01 AWS resources transfer to another owner and -which are actually deleted. Begin with those ownership decisions and reviewed -plans rather than an unconditional stage-01 destroy. +Teardown has three separate decisions: drain applications through their owning +delivery system, destroy the Restate cluster, then decide which stage-01 AWS +resources transfer to another owner and which are actually deleted. Begin with +those ownership decisions and reviewed plans rather than an unconditional +stage-01 destroy. -Before planning either stage: +Before planning a destroy: - pause new traffic and create and verify a current snapshot; -- delete SDK services and let every revision drain—Terraform does not manage - `RestateDeployment`s or wait for their finalizers; +- identify which delivery system owns each SDK service; - record the PV/PVC/AZ/EBS volume mapping before deleting the cluster; - capture the bucket name while the stage-01 output still exists. ```bash -kubectl -n restate-apps delete restatedeployment --all \ - --wait=true --timeout=15m kubectl -n restate get pvc -o wide kubectl get pv \ -o custom-columns='PV:.metadata.name,CLAIM-NS:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name,VOLUME:.spec.csi.volumeHandle,ZONE:.metadata.labels.topology\.kubernetes\.io/zone' BUCKET="$(terraform -chdir=terraform/01-foundation output -raw snapshots_bucket)" ``` +### 1. Destroy optional stage 03, if used + +If the example service is in stage-03 state, destroy it from that state first: + +```bash +export TF_VAR_service_image='registry.example/service@sha256:' + +terraform -chdir=terraform/03-services plan \ + -destroy -var-file=../terraform.tfvars \ + -out=service-destroy.tfplan +terraform -chdir=terraform/03-services apply service-destroy.tfplan +``` + +The operator's finalizer can keep this apply running while pinned invocations +drain. If another tool owns the services, remove them through that tool. For a +manual installation, use: + +```bash +kubectl -n restate-apps delete restatedeployment --all \ + --wait=true --timeout=15m +``` + If service deletion times out, inspect the pinned invocations and continue waiting; allow the finalizer to complete before destroying the cluster. -### 1. Destroy stage 02 +Confirm that no application-owned `RestateDeployment` remains before +continuing. An empty result is expected: + +```bash +kubectl get restatedeployments.restate.dev --all-namespaces +``` + +### 2. Destroy stage 02 Create and apply one saved, reviewed destroy plan: @@ -417,7 +484,7 @@ kubectl get pv \ -o custom-columns='PV:.metadata.name,STATUS:.status.phase,VOLUME:.spec.csi.volumeHandle,SIZE:.spec.capacity.storage' ``` -### 2. Decide stage-01 ownership before planning its destroy +### 3. Decide stage-01 ownership before planning its destroy Stage 01 may own a cluster-wide OIDC provider. If `create_oidc_provider = true` and the EKS cluster is staying, transfer it out of @@ -472,7 +539,7 @@ terraform -chdir=terraform/01-foundation state rm \ Removing an object from state transfers responsibility; it does not delete or continue managing the AWS object. Record the new owner for every transfer. -### 3. Destroy stage 01 +### 4. Destroy stage 01 Only after those decisions, create and apply the saved stage-01 destroy plan: diff --git a/terraform/terraform.tfvars.example b/terraform/terraform.tfvars.example index de8b14c..4d0e945 100644 --- a/terraform/terraform.tfvars.example +++ b/terraform/terraform.tfvars.example @@ -1,5 +1,5 @@ # Copy to terraform.tfvars (git-ignored) next to this file and fill in. -# Both stages read it: +# Both cluster stages read it; optional 03-services accepts it too: # # terraform -chdir=01-foundation apply -var-file=../terraform.tfvars # terraform -chdir=02-restate apply -var-file=../terraform.tfvars