Document durability model, delivery health signals, and team isolation

docs/00-architecture.md

@@ -110,6 +110,11 @@ directions:
    selecting every pod in `restate-apps` and allowing only the `restate`
    namespace. This prevents direct calls that bypass Restate.
 
+These boundaries govern network reachability, not authorization between
+services. Any service registered with the cluster can invoke any other through
+Restate, so one `RestateCluster` is one trust domain; see
+[Team isolation](03-deploying-services.md#team-isolation).
+
 ### Admin API
 
 The admin API on port 9070 has no authentication and grants full cluster
@@ -244,6 +249,72 @@ Kubernetes `$(VAR)` expansion in environment values is order-sensitive, so
 entries supplied in `spec.compute.env` override operator defaults with the
 same name.
 
+## Data durability model
+
+Each Restate pod's volume holds three kinds of data under `/restate-data`, and
+they are not equally replaceable:
+
+| Data | Role that writes it | If lost beyond the replication factor |
+|---|---|---|
+| Replicated log segments | `log-server` | **Data loss.** The log is the record of every invocation; nothing else reconstructs it |
+| Cluster metadata (node, log, and partition configuration; service schemas) | `metadata-server`, Raft, majority of nodes | **Cluster loss.** Nodes cannot agree on cluster membership or log configuration without it |
+| Partition stores (RocksDB state per partition) | `worker` | **Recoverable.** Rebuilt from the latest snapshot in the bucket plus replay of the log after it |
+
+With node replication `2` on three nodes, losing one node's volume loses
+nothing; losing two volumes at once loses log records that had not yet been
+covered by a snapshot, and can lose the metadata Raft majority. Partition
+snapshots in S3 exist to speed up that rebuild and to let the log be trimmed;
+they are not a backup of the cluster. There is no supported backup and restore
+procedure for a Restate cluster today. Protecting the EBS volumes is therefore
+the operator's first duty: the `Retain` reclaim policy, encryption, and a
+deliberate teardown order are what this repository provides toward it.
+
+### Recommendation: keep metadata out of the volumes
+
+Restate can store cluster metadata in Amazon S3 instead of the built-in Raft
+`metadata-server` role, and also supports DynamoDB (Restate 1.5.4 and later)
+and etcd; see the
+[metadata storage documentation](https://docs.restate.dev/server/metadata).
+For a deployment whose data matters, we strongly recommend the object-store
+provider on AWS:
+
+- it removes the one piece of irreplaceable state that would otherwise share a
+  volume with the log, and the object store's durability replaces the Raft
+  majority as the thing that has to survive;
+- it makes the object store a day-one dependency instead of something that can
+  be deferred. A cluster with the replicated metadata store starts and serves
+  traffic with no object store and no snapshots configured at all, but its log
+  is then never trimmed, and the volumes fill up later with no warning that
+  anything was missing;
+- the provider is chosen at initial deployment. Migrating from replicated to
+  an external store later is supported, but it stops invocation processing for
+  the duration of the migration.
+
+The configuration change in `resources/04-restate-cluster.yaml` is to remove
+`metadata-server` from `roles` and add, next to the snapshot destination:
+
+```toml
+[metadata-client]
+type = "object-store"
+path = "s3://<snapshots-bucket>/restate/metadata"
+aws-region = "<region>"
+```
+
+The IAM policy in `resources/01-restate-snapshots-iam-policy.json` grants
+bucket-wide object read, write, and delete, which is what the provider uses.
+This repository's validation covers the replicated store only; test the
+object-store configuration before adopting it. Only Amazon S3 is
+supported for metadata; S3-compatible stores such as MinIO are supported for
+snapshots but not for metadata, and the bucket must be in the same region as
+the cluster because metadata latency affects cluster operations directly.
+Outside AWS, the equivalent is etcd; GCS and Azure Blob are snapshot
+destinations only.
+
+This repository still ships the replicated metadata store because it is what
+the source profile runs and what was validated end to end here. Treat the
+switch as a decision to make before the first `RestateCluster` apply, not a
+later tuning step.
+
 ## Storage and snapshots
 
 Each Restate pod receives a 1 TiB PVC using the repository-owned

docs/03-deploying-services.md

@@ -11,7 +11,11 @@ image, configuration, scaling, and rollout. Apply the example after replacing
 its image placeholder.
 
 Start from `resources/05-restate-compute.yaml`, but treat it as a lifecycle
-example—not a complete production application template.
+example—not a complete production application template. This repository ships
+no service image. Build one from the
+[Restate SDK examples](https://github.com/restatedev/examples) in the language
+your team uses; any image whose SDK endpoint listens on port 9080 fits the
+manifest unchanged.
 
 Both deployment paths finish with the Restate cluster installed. Neither the
 runbook nor the Terraform modules deploy your services: a service changes at
@@ -237,11 +241,135 @@ invocations: the finalizer drained the revision and the operator removed the
 ReplicaSet, Services, and pods, leaving the namespace empty without manual
 intervention.
 
+Deletion always takes at least `drainDelaySeconds` (300 seconds by default)
+even when no invocation is in flight, because the latest revision only becomes
+inactive when it is deregistered and then waits out its drain delay like any
+other. Budget for that in pipelines that delete and recreate services.
+
 An object that remains `Terminating` is usually waiting for pinned invocations,
 not stuck. Inspect the Restate deployment before considering finalizer changes.
 If the target `RestateCluster` no longer exists, the operator permits immediate
 cleanup because it cannot query or drain through that cluster.
 
+## Team isolation
+
+A Restate cluster is one trust domain. Every service registered with it can
+invoke every other service through the cluster, regardless of which namespace
+the services run in or who deployed them: the NetworkPolicies in this
+repository control which pods may reach Restate's ingress and which may reach
+SDK pods directly, but service-to-service calls travel through Restate itself,
+which does not authorize them. Service names are also cluster-global, so two
+teams cannot both register a service called `Greeter`.
+
+Sharing one Restate cluster is therefore appropriate for teams that already
+trust each other's code. Teams or applications that must not be able to call
+into each other belong on separate `RestateCluster`s, and should talk to each
+other the way any external client does: through the other cluster's ingress,
+behind the authenticating layer described under
+[Making the playground work for a team](05-operations.md#making-the-playground-work-for-a-team).
+
+The manifests in this repository assume a single cluster named `restate`; the
+[invariants list](00-architecture.md#invariants-to-preserve) enumerates what
+that name is wired into. A second cluster needs its own name and generated
+namespace, its own snapshot bucket and IAM role, and a copy of the
+`restate-apps` isolation policy for its own service namespace.
+
+## Health signals for delivery tools
+
+A `RestateDeployment` reports one `Ready` condition. Delivery tools need to
+read it, because a rejected revision looks healthy at the pod level: the new
+pods run and pass their probes, and only the condition says that Restate
+refused to register them.
+
+| `Ready` | Reason | Meaning |
+|---|---|---|
+| `True` | `Deployed` | Latest revision registered and serving new invocations |
+| `False` | `ReplicaSetScaling`, `ReplicaSetPodNotReady`, `ReplicaSetPodNotAvailable`, `ReplicaSetNoStatus` | Pods still starting; normal during a rollout |
+| `False` | `AdminCallFailed` | Admin API unreachable or returned a server error; the operator retries |
+| `False` | `AdminCallRejected` | Restate refused the registration; the message carries Restate's error and the operator retries every 30 s but will not succeed until the template changes |
+| `False` | `HashCollision`, `FailedReconcile` | Operator-side error; inspect the operator logs |
+
+The operator also publishes a Warning Event with the same message for
+`AdminCallFailed` and `AdminCallRejected`, so `kubectl describe
+restatedeployment <name>` shows the reason without reading logs.
+
+For example, a revision that changes a service's type from Service to Virtual
+Object is rejected by Restate with `META0006`. The condition and Event carry
+that message, the previous revision keeps serving traffic, and the new pods
+run unregistered until the spec is corrected.
+
+### Terraform
+
+The `kubernetes_manifest` `wait` block matches only positive states: a
+condition reaching a value, a field matching a regex, or a rollout completing
+for the built-in workload kinds. It cannot fail on `Ready=False`, so a rejected
+revision makes `terraform apply` block until its timeout and then report
+`context deadline exceeded` without Restate's reason.
+
+If a `RestateDeployment` is applied from Terraform anyway, set an update
+timeout well under the default, print the condition on failure, and read the
+reason from the resource rather than from Terraform:
+
+```bash
+kubectl -n restate-apps get restatedeployment <name> \
+  -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason}: {.message}{"\n"}{end}'
+```
+
+State is not left inconsistent: the provider keeps the previous manifest in
+state when the wait fails, so the next plan already proposes the rollback.
+
+### Argo CD
+
+Argo CD has no built-in health assessment for `restate.dev` kinds and reports
+unknown custom resources as Healthy. Add a health check to `argocd-cm` so a
+rejected revision shows as Degraded with Restate's message, and a rollout in
+progress shows as Progressing:
+
+```yaml
+data:
+  resource.customizations.health.restate.dev_RestateDeployment: |
+    hs = { status = "Progressing", message = "Waiting for RestateDeployment status" }
+    if obj.status == nil then
+      return hs
+    end
+    if obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
+       and obj.status.observedGeneration < obj.metadata.generation then
+      hs.message = "Waiting for the operator to observe the latest generation"
+      return hs
+    end
+    if obj.status.conditions ~= nil then
+      for _, c in ipairs(obj.status.conditions) do
+        if c.type == "Ready" then
+          if c.status == "True" then
+            hs.status = "Healthy"
+            hs.message = c.message or "Deployed"
+          elseif c.reason == "AdminCallRejected" then
+            hs.status = "Degraded"
+            hs.message = c.message
+          else
+            hs.message = (c.reason or "") .. ": " .. (c.message or "")
+          end
+          return hs
+        end
+      end
+    end
+    return hs
+```
+
+With this in place, a sync of a rejected revision fails its health check within
+one reconcile instead of waiting on a timeout, and the previous revision keeps
+serving because the operator never replaced it. Teams that deploy the cluster
+stages with Terraform and the applications with Argo CD get the boundary this
+guide recommends without giving up automated health gating.
+
+### Flux
+
+Flux's health checks use kstatus, which treats a `Ready=False` condition as
+still reconciling and reports failure only on a `Stalled=True` condition. The
+operator does not set `Stalled`, so a rejected revision keeps a Flux
+`Kustomization` in progress until its `timeout`. Set that timeout to a few
+minutes and read the `Ready` reason as above to see why.
+
 ## Useful fields
 
 | Field | Default | Meaning |

docs/05-operations.md

@@ -161,6 +161,30 @@ does not expose anything by itself, and it does not change what the server
 binds to. Equally, exposing ingress without setting it leaves the playground
 pointing somewhere your users cannot reach.
 
+## Observability
+
+Restate needs no agent or sidecar; the work is on the platform side.
+
+- **Metrics.** Every Restate pod serves Prometheus metrics on port 5122 at
+  `/metrics`, the same port as node traffic. The operator's NetworkPolicy
+  admits that port only from the `restate` and `restate-operator` namespaces,
+  so a scraper elsewhere must be added under `security.networkPeers.node` in
+  `resources/04-restate-cluster.yaml` (the older `metrics` key is deprecated).
+  Scrape each pod through `svc/restate-cluster`, the headless Service, so
+  per-node series keep their identity. Restate publishes two Grafana dashboards
+  to import; see [Metrics](https://docs.restate.dev/server/monitoring/metrics).
+- **Logs.** The operator sets `RESTATE_LOG_FORMAT=json` on the pods, so the
+  platform's usual log collector picks them up as structured events. Keep the
+  default `info` level in production; see
+  [Logging](https://docs.restate.dev/server/monitoring/logging).
+- **Traces.** Restate exports OTLP traces of invocations when
+  `tracing-endpoint` points at a collector; see
+  [Tracing](https://docs.restate.dev/server/monitoring/tracing). A collector
+  inside the cluster has a private address, which the operator's default egress
+  policy blocks, so add its address under `spec.security.networkEgressRules`
+  as described under
+  [Private AWS endpoints](00-architecture.md#private-aws-endpoints).
+
 ## Verify the snapshot path
 
 Automatic snapshots require both the configured record threshold and interval,
@@ -221,6 +245,25 @@ replacement time, or log growth — then either schedule manual snapshots or low
 from the profile and is left at its profile value here; see
 [Profile fidelity](04-profile-fidelity.md).
 
+The two settings combine in three ways, per the
+[Restate snapshot documentation](https://docs.restate.dev/server/snapshots#configuring-automatic-snapshotting):
+
+| Configured | Trigger |
+|---|---|
+| Both | Interval elapsed **and** record threshold reached, per partition (the shipped configuration) |
+| `SNAPSHOT_INTERVAL` only | Every partition snapshots on the interval, regardless of traffic |
+| `SNAPSHOT_INTERVAL_NUM_RECORDS` only | Record count only, no time component |
+
+The record gate matters beyond node replacement: the log can only be trimmed
+up to the oldest retained snapshot of each partition, so a low-traffic
+partition that never reaches the threshold also never lets its log segment be
+trimmed. If a bounded time between snapshots is the goal, drop the record
+threshold and set the interval alone; Restate's own documentation example uses
+`60m`. The shipped values are the Restate Cloud profile's and are left as they
+are here so the manifest stays a faithful translation. `NUM_RETAINED` is `2`
+for the same reason; Restate's documentation recommends `1` for most
+deployments, because trimming follows the oldest retained snapshot.
+
 Check where partitions actually stand before concluding anything is broken:
 
 ```bash
@@ -441,6 +484,16 @@ affect a live replicated system. Before applying:
 3. change one dimension at a time;
 4. watch pods and `restatectl status` until the cluster is healthy again.
 
+### Node maintenance
+
+The operator creates a PodDisruptionBudget on the Restate pods with
+`maxUnavailable: 1`, so a node drain or a managed node-group upgrade evicts one
+Restate pod at a time and waits for it to be Ready elsewhere before the next.
+With three nodes and required host anti-affinity, an evicted pod has nowhere to
+go until a replacement node exists, so drain with a surge node available or
+expect the pod to sit Pending until the drained node returns. Check
+`restatectl status` between nodes, as for any other roll.
+
 ### Upgrade Restate or the operator
 
 The image and chart are intentionally pinned. Before upgrading:
@@ -456,6 +509,26 @@ The image and chart are intentionally pinned. Before upgrading:
 
 Changing only the container image is not a complete upgrade review.
 
+On the Terraform path, `terraform apply` does not wait for the roll. The
+`RestateCluster` is already `Ready=True` when the change is submitted, so the
+stage-02 wait is satisfied immediately and Terraform returns while the
+StatefulSet is still replacing pods one at a time, highest ordinal first.
+Gate the next pipeline step on the StatefulSet instead:
+
+```bash
+kubectl -n restate rollout status statefulset/restate --timeout=15m
+kubectl -n restate exec restate-0 -- restatectl status
+```
+
+Expect the roll itself to take a few minutes for three pods. Clients may see a
+connection reset at the moment a pod terminates; invocations are retried by
+Restate, but a client holding an open connection to that pod is not.
+
+A `kubectl port-forward` to `svc/restate` is pinned to one pod and dies when
+that pod is replaced, so a client tunnelled through it sees connection errors
+during a roll that in-cluster clients do not. Re-establish the forward after
+the roll rather than reading those errors as cluster unavailability.
+
 ## Data safety and recovery boundaries
 
 Two independent mechanisms protect different failure modes:
@@ -468,7 +541,11 @@ Two independent mechanisms protect different failure modes:
 - **S3 partition snapshots** allow nodes to bootstrap without replaying the
   entire retained log and provide recovery material outside the EBS volumes.
 
-Neither mechanism is a complete, automatic disaster-recovery workflow.
+Neither mechanism is a complete, automatic disaster-recovery workflow, and
+snapshots are not a backup: the replicated log and the cluster metadata exist
+only on the volumes, and partition state is what snapshots let you rebuild. See
+[Data durability model](00-architecture.md#data-durability-model) for which
+data is irreplaceable and for the recommendation to keep metadata in S3.
 Released PVs retain their old claim references and do not bind to replacement
 PVCs automatically. Before removal or another data-affecting change, record
 the PV, PVC, Availability Zone, and EBS volume-id mapping:

resources/05-restate-compute.yaml

@@ -40,9 +40,12 @@ spec:
     spec:
       containers:
         - name: service
-          # REPLACE_ME: the SDK service image (e.g. an ECR image built from
-          # this repo's node service). Keep the port name "restate" because the
-          # operator uses it for the registration URL.
+          # REPLACE_ME: your SDK service image, pushed to a registry the nodes
+          # can pull from. This repository ships no service; start from one of
+          # the SDK examples at https://github.com/restatedev/examples in the
+          # language of your choice. Any image listening on 9080 works. Keep
+          # the port name "restate" because the operator uses it for the
+          # registration URL.
           image: REPLACE_ME_SERVICE_IMAGE
           ports:
             - name: restate

scripts/validate.sh

@@ -8,7 +8,7 @@ export TF_IN_AUTOMATION=1
 
 tofu fmt -check -recursive terraform
 
-for TOFU_STAGE in terraform/01-foundation terraform/02-restate; do
+for TOFU_STAGE in terraform/01-foundation terraform/02-restate terraform/03-services; do
   tofu -chdir="$TOFU_STAGE" init -backend=false -input=false -no-color >/dev/null
   tofu -chdir="$TOFU_STAGE" validate -no-color
 done

terraform/.gitignore

@@ -14,3 +14,6 @@ crash.log
 # Generate your own with `terraform init` and commit it in your fork if you
 # standardize on one tool.
 .terraform.lock.hcl
+
+# apply/timeline logs written during validation runs
+*.log

terraform/03-services/outputs.tf

@@ -0,0 +1,7 @@
+output "service_name" {
+  value = local.service_manifest.metadata.name
+}
+
+output "invoke_hint" {
+  value = "kubectl -n restate port-forward svc/restate 8080:8080 & curl localhost:8080/Greeter/greet --json '\"Restate\"'"
+}