Align deployment docs and lifecycle gates

README.md

@@ -20,7 +20,7 @@ Keep these three layers separate while reading the guide:
 |---|---|
 | EKS cluster | Existing AWS/Kubernetes infrastructure; this repository leaves it unchanged |
 | Restate cluster | Three stateful Restate server pods installed into EKS |
-| SDK service | Customer application code that uses a Restate SDK; deployed separately |
+| SDK service | Customer application code that uses a Restate SDK; deployed through an independent application workflow |
 
 The sizing and runtime tuning come from Restate Cloud's
 `3-node.xlarge-vqueues` profile:
@@ -53,12 +53,14 @@ The sizing and runtime tuning come from Restate Cloud's
   roll out and drain SDK service revisions safely.
 - Two deployment paths that consume the same manifests:
   - a transparent, command-by-command `kubectl`/Helm runbook;
-  - a two-stage Terraform or OpenTofu workflow.
+  - a two-stage Terraform or OpenTofu cluster workflow, with an optional third
+    application stage kept in separate state.
 
-Completing either deployment path gives you a healthy Restate cluster, its
-operator, persistent storage, and snapshot access. It does **not** deploy a
-customer application or expose a public endpoint. Those are separate handoffs
-to the application and networking owners.
+Completing the cluster portion of either deployment path gives you a healthy
+Restate cluster, its operator, persistent storage, and snapshot access. It does
+**not** expose a public endpoint. A customer application is a separate handoff
+to its application owner; the Terraform path includes an optional, separately
+state-managed example for teams that want it.
 
 ## Before you deploy
 
@@ -75,10 +77,15 @@ provides the commands that verify each one.
    Service-CIDR egress policy is derived from `serviceIpv4Cidr`.
 4. **Snapshots:** use an S3 bucket dedicated to this Restate cluster. The
    snapshot prefix is not unique across installations.
-5. **Persistent data:** deleting the `RestateCluster` removes its namespace and
+5. **Metadata durability:** choose the replicated metadata store shipped in the
+   example or an S3 metadata store before the first cluster apply. The S3
+   option removes metadata quorum from the Restate volumes, while adding an
+   external dependency and latency consideration. See
+   [Data durability](docs/00-architecture.md#data-durability-model).
+6. **Persistent data:** deleting the `RestateCluster` removes its namespace and
    PVCs. The StorageClass retains the underlying PVs, but recovery is a manual
    operation; retained volumes do not reattach automatically.
-6. **Existing infrastructure:** the EBS CSI driver, sufficient EKS access, and
+7. **Existing infrastructure:** the EBS CSI driver, sufficient EKS access, and
    an IAM OIDC provider for IRSA must already exist unless the Terraform path
    is explicitly told to create the OIDC provider.
 
@@ -95,11 +102,11 @@ The complete checklist and verification commands are in
 Please use one path per installation. If you move an existing installation to
 Terraform, first import its AWS and Kubernetes resources into Terraform state.
 
-Both paths finish with the Restate cluster installed. Deploy your SDK services
-separately, with `kubectl` or your existing application pipeline — they are
-still operator-managed: the operator reconciles them as `RestateDeployment`
-resources, handling revisioning, registration, and draining. See
-[Deploying services](docs/03-deploying-services.md).
+Both paths finish with the Restate cluster installed. Deploy SDK services from
+an application-owned workflow: use `kubectl`, your existing delivery system,
+or the optional `terraform/03-services` example in separate state. The operator
+then reconciles each `RestateDeployment`, handling revisioning, registration,
+and draining. See [Deploying services](docs/03-deploying-services.md).
 
 ## Architecture at a glance
 
@@ -152,6 +159,7 @@ resources/             canonical Kubernetes YAML, Helm values, and IAM policy
 terraform/01-foundation
                        S3, IAM/IRSA, namespaces, StorageClass, operator
 terraform/02-restate   RestateCluster and its Service-CIDR egress policy
+terraform/03-services  optional SDK service example in independent state
 docs/                  architecture, deployment, operations, and design notes
 misc/pdf/              source and LLM-oriented build guide for the PDF companion
 output/pdf/            committed customer-facing PDF artifacts
@@ -175,9 +183,9 @@ deployment paths:
 The manual path requires every active `REPLACE_ME_*` value in a file being
 applied to be replaced first. The commented, non-automated Pod Identity
 adaptation may stay unset, and the compute image may stay unset while compute is
-skipped. The Terraform path performs the substitutions it needs in memory, from
-its variables and from the EKS cluster itself; the service image is not among
-them, because Terraform does not deploy services.
+skipped. The Terraform path performs substitutions in memory from its variables
+and the EKS cluster. Stages 01 and 02 install the cluster; optional stage 03
+substitutes `service_image` into the SDK service example.
 
 ## Validation
 
@@ -201,7 +209,8 @@ cluster or AWS account.
 | Dedicated snapshot bucket | A snapshot repository belongs to one Restate cluster |
 | `Retain` EBS reclaim policy | Preserves volumes after accidental CR/namespace deletion |
 | Restate-specific StorageClass name | Avoids colliding with a shared cluster's generic `gp3` class |
-| Two Terraform stages | Restate CRDs must exist in the live cluster before custom resources can be planned |
+| Two ordered Terraform cluster stages | Restate CRDs must exist in the live cluster before custom resources can be planned |
+| Separate optional service state | Application releases keep their cadence and blast radius separate from cluster infrastructure |
 
 ## Known boundaries
 
@@ -210,6 +219,8 @@ cluster or AWS account.
   not expose port 9070 through an unauthenticated LoadBalancer or Ingress.
 - S3 snapshots and retained EBS volumes reduce recovery risk, but this
   repository does not define a complete disaster-recovery procedure.
+- The validated cluster manifest uses replicated metadata. Decide whether to
+  adopt the documented S3 metadata option before the first cluster apply.
 - Runtime upgrades require re-validating the experimental vqueues settings and
   should not be performed by changing the image alone.
 - The example SDK service has placeholder image and sizing values; treat it as

docs/00-architecture.md

@@ -265,30 +265,28 @@ nothing; losing two volumes at once loses log records that had not yet been
 covered by a snapshot, and can lose the metadata Raft majority. Partition
 snapshots in S3 exist to speed up that rebuild and to let the log be trimmed;
 they are not a backup of the cluster. There is no supported backup and restore
-procedure for a Restate cluster today. Protecting the EBS volumes is therefore
-the operator's first duty: the `Retain` reclaim policy, encryption, and a
-deliberate teardown order are what this repository provides toward it.
+procedure for a Restate cluster today. Please include the EBS volumes in the
+deployment's data-protection plan. This repository supports that with a
+`Retain` reclaim policy, encryption, and a deliberate teardown order.
 
-### Recommendation: keep metadata out of the volumes
+### Choose the metadata provider before deployment
 
 Restate can store cluster metadata in Amazon S3 instead of the built-in Raft
-`metadata-server` role, and also supports DynamoDB (Restate 1.5.4 and later)
-and etcd; see the
+`metadata-server` role; see the
 [metadata storage documentation](https://docs.restate.dev/server/metadata).
-For a deployment whose data matters, we strongly recommend the object-store
-provider on AWS:
-
-- it removes the one piece of irreplaceable state that would otherwise share a
-  volume with the log, and the object store's durability replaces the Raft
-  majority as the thing that has to survive;
-- it makes the object store a day-one dependency instead of something that can
-  be deferred. A cluster with the replicated metadata store starts and serves
-  traffic with no object store and no snapshots configured at all, but its log
-  is then never trimmed, and the volumes fill up later with no warning that
-  anything was missing;
-- the provider is chosen at initial deployment. Migrating from replicated to
-  an external store later is supported, but it stops invocation processing for
-  the duration of the migration.
+For a production AWS deployment, please consider the S3 object-store provider
+alongside the replicated provider shipped in this example:
+
+- **S3 metadata** removes the metadata Raft majority from the Restate volumes
+  and places that state in AWS's object-store durability model. It also makes
+  S3 availability and latency part of cluster operations.
+- **Replicated metadata** keeps the deployment self-contained and matches the
+  profile validated by this repository. Its metadata remains dependent on a
+  majority of the three node volumes.
+
+Choose before the initial deployment when practical. A later migration from
+replicated to external metadata is supported, but invocation processing stops
+for the duration of that migration.
 
 The configuration change in `resources/04-restate-cluster.yaml` is to remove
 `metadata-server` from `roles` and add, next to the snapshot destination:
@@ -302,13 +300,11 @@ aws-region = "<region>"
 
 The IAM policy in `resources/01-restate-snapshots-iam-policy.json` grants
 bucket-wide object read, write, and delete, which is what the provider uses.
-This repository's validation covers the replicated store only; test the
-object-store configuration before adopting it. Only Amazon S3 is
-supported for metadata; S3-compatible stores such as MinIO are supported for
-snapshots but not for metadata, and the bucket must be in the same region as
-the cluster because metadata latency affects cluster operations directly.
-Outside AWS, the equivalent is etcd; GCS and Azure Blob are snapshot
-destinations only.
+This repository's validation covers the replicated store only, so test the
+object-store configuration before adopting it. Only Amazon S3 is supported for
+object-store metadata; S3-compatible stores such as MinIO are supported for
+snapshots but not for metadata. Keep the bucket in the same region as the
+cluster because metadata latency affects cluster operations directly.
 
 This repository still ships the replicated metadata store because it is what
 the source profile runs and what was validated end to end here. Treat the

docs/01-prerequisites.md

@@ -41,6 +41,9 @@ checklist has passed.
       enforced; for Terraform, a globally unique name has been chosen.
 - [ ] The EKS cluster name is at most 46 characters so the derived snapshot
       role name stays within IAM's 64-character limit.
+- [ ] The replicated or S3 metadata provider has been chosen for the initial
+      deployment after reviewing the
+      [durability trade-off](00-architecture.md#data-durability-model).
 - [ ] The required tools for one deployment path are installed.
 
 ## Set deployment context
@@ -321,11 +324,11 @@ Choose one path; you do not need every tool in both columns.
 | Tool | Manual path | Terraform path | Purpose |
 |---|:---:|:---:|---|
 | AWS CLI v2 | ✓ | ✓ | Identity, EKS lookup, IAM, S3, exec auth |
-| `kubectl` | ✓ | recommended | Apply and diagnose Kubernetes resources |
+| `kubectl` | ✓ | recommended; required for optional stage 03 | Apply, diagnose, and run the service readiness gate |
 | `eksctl` | ✓ | — | OIDC provider and IRSA role plumbing |
 | Helm | ✓ | optional* | Install the operator manually; verify or authenticate the OCI chart pull |
-| Terraform ≥1.5 or OpenTofu | — | ✓ | Apply the two Terraform stages |
-| `jq` | ✓ | recommended | Format API responses |
+| Terraform ≥1.5 or OpenTofu | — | ✓ | Apply the two cluster stages and optional service stage |
+| `jq` | ✓ | recommended; required for optional stage 03 | Format API responses and parse service readiness |
 | `restatectl` | via pod | via pod | Cluster status and snapshots; provisioning remains operator-managed |
 | `restate` CLI | optional | optional | Service/deployment administration |
 
@@ -365,8 +368,9 @@ grep -RIn 'REPLACE_ME' resources
 | `REPLACE_ME_SERVICE_CIDR` | `06-restate-service-cidr-egress.yaml` | Cluster Service IPv4 CIDR; needed where the CNI enforces NetworkPolicy |
 | `REPLACE_ME_EKS_CLUSTER_NAME` | `02-restate-operator.values.yaml` (commented) | Only when adapting the repository for EKS Pod Identity; the supplied IAM paths implement IRSA only |
 
-The Terraform path does not modify the files. It replaces the required values
-in memory from `terraform.tfvars`.
+The Terraform path does not modify the files. It replaces values in memory from
+`terraform.tfvars`, the EKS API, and—for optional stage 03—the
+`TF_VAR_service_image` environment variable.
 
 ## Appendix: an example cluster, for illustration only
 

docs/02-runbook.md

@@ -246,6 +246,13 @@ kubectl get storageclass restate-gp3 -o yaml
 The expected class uses encrypted XFS, 6000 IOPS, 500 MiB/s,
 `WaitForFirstConsumer`, and `reclaimPolicy: Retain`.
 
+Before the first cluster apply, choose the metadata provider. The canonical
+manifest uses the replicated provider validated by this repository. An S3
+metadata provider reduces dependence on a majority of the node volumes but
+adds an external availability and latency dependency. Review
+[Data durability](00-architecture.md#data-durability-model) and, if you choose
+S3, make and validate the documented manifest change before continuing.
+
 Confirm that no active placeholder remains in the cluster manifest, then apply
 it: