diff --git a/docs/00-architecture.md b/docs/00-architecture.md index 708de23..819730a 100644 --- a/docs/00-architecture.md +++ b/docs/00-architecture.md @@ -110,6 +110,11 @@ directions: selecting every pod in `restate-apps` and allowing only the `restate` namespace. This prevents direct calls that bypass Restate. +These boundaries govern network reachability, not authorization between +services. Any service registered with the cluster can invoke any other through +Restate, so one `RestateCluster` is one trust domain; see +[Team isolation](03-deploying-services.md#team-isolation). + ### Admin API The admin API on port 9070 has no authentication and grants full cluster @@ -244,6 +249,72 @@ Kubernetes `$(VAR)` expansion in environment values is order-sensitive, so entries supplied in `spec.compute.env` override operator defaults with the same name. +## Data durability model + +Each Restate pod's volume holds three kinds of data under `/restate-data`, and +they are not equally replaceable: + +| Data | Role that writes it | If lost beyond the replication factor | +|---|---|---| +| Replicated log segments | `log-server` | **Data loss.** The log is the record of every invocation; nothing else reconstructs it | +| Cluster metadata (node, log, and partition configuration; service schemas) | `metadata-server`, Raft, majority of nodes | **Cluster loss.** Nodes cannot agree on cluster membership or log configuration without it | +| Partition stores (RocksDB state per partition) | `worker` | **Recoverable.** Rebuilt from the latest snapshot in the bucket plus replay of the log after it | + +With node replication `2` on three nodes, losing one node's volume loses +nothing; losing two volumes at once loses log records that had not yet been +covered by a snapshot, and can lose the metadata Raft majority. Partition +snapshots in S3 exist to speed up that rebuild and to let the log be trimmed; +they are not a backup of the cluster. There is no supported backup and restore +procedure for a Restate cluster today. Protecting the EBS volumes is therefore +the operator's first duty: the `Retain` reclaim policy, encryption, and a +deliberate teardown order are what this repository provides toward it. + +### Recommendation: keep metadata out of the volumes + +Restate can store cluster metadata in Amazon S3 instead of the built-in Raft +`metadata-server` role, and also supports DynamoDB (Restate 1.5.4 and later) +and etcd; see the +[metadata storage documentation](https://docs.restate.dev/server/metadata). +For a deployment whose data matters, we strongly recommend the object-store +provider on AWS: + +- it removes the one piece of irreplaceable state that would otherwise share a + volume with the log, and the object store's durability replaces the Raft + majority as the thing that has to survive; +- it makes the object store a day-one dependency instead of something that can + be deferred. A cluster with the replicated metadata store starts and serves + traffic with no object store and no snapshots configured at all, but its log + is then never trimmed, and the volumes fill up later with no warning that + anything was missing; +- the provider is chosen at initial deployment. Migrating from replicated to + an external store later is supported, but it stops invocation processing for + the duration of the migration. + +The configuration change in `resources/04-restate-cluster.yaml` is to remove +`metadata-server` from `roles` and add, next to the snapshot destination: + +```toml +[metadata-client] +type = "object-store" +path = "s3:///restate/metadata" +aws-region = "" +``` + +The IAM policy in `resources/01-restate-snapshots-iam-policy.json` grants +bucket-wide object read, write, and delete, which is what the provider uses. +This repository's validation covers the replicated store only; test the +object-store configuration before adopting it. Only Amazon S3 is +supported for metadata; S3-compatible stores such as MinIO are supported for +snapshots but not for metadata, and the bucket must be in the same region as +the cluster because metadata latency affects cluster operations directly. +Outside AWS, the equivalent is etcd; GCS and Azure Blob are snapshot +destinations only. + +This repository still ships the replicated metadata store because it is what +the source profile runs and what was validated end to end here. Treat the +switch as a decision to make before the first `RestateCluster` apply, not a +later tuning step. + ## Storage and snapshots Each Restate pod receives a 1 TiB PVC using the repository-owned diff --git a/docs/03-deploying-services.md b/docs/03-deploying-services.md index 8413dc4..5c8dda2 100644 --- a/docs/03-deploying-services.md +++ b/docs/03-deploying-services.md @@ -11,7 +11,11 @@ image, configuration, scaling, and rollout. Apply the example after replacing its image placeholder. Start from `resources/05-restate-compute.yaml`, but treat it as a lifecycle -example—not a complete production application template. +example—not a complete production application template. This repository ships +no service image. Build one from the +[Restate SDK examples](https://github.com/restatedev/examples) in the language +your team uses; any image whose SDK endpoint listens on port 9080 fits the +manifest unchanged. Both deployment paths finish with the Restate cluster installed. Neither the runbook nor the Terraform modules deploy your services: a service changes at @@ -237,11 +241,135 @@ invocations: the finalizer drained the revision and the operator removed the ReplicaSet, Services, and pods, leaving the namespace empty without manual intervention. +Deletion always takes at least `drainDelaySeconds` (300 seconds by default) +even when no invocation is in flight, because the latest revision only becomes +inactive when it is deregistered and then waits out its drain delay like any +other. Budget for that in pipelines that delete and recreate services. + An object that remains `Terminating` is usually waiting for pinned invocations, not stuck. Inspect the Restate deployment before considering finalizer changes. If the target `RestateCluster` no longer exists, the operator permits immediate cleanup because it cannot query or drain through that cluster. +## Team isolation + +A Restate cluster is one trust domain. Every service registered with it can +invoke every other service through the cluster, regardless of which namespace +the services run in or who deployed them: the NetworkPolicies in this +repository control which pods may reach Restate's ingress and which may reach +SDK pods directly, but service-to-service calls travel through Restate itself, +which does not authorize them. Service names are also cluster-global, so two +teams cannot both register a service called `Greeter`. + +Sharing one Restate cluster is therefore appropriate for teams that already +trust each other's code. Teams or applications that must not be able to call +into each other belong on separate `RestateCluster`s, and should talk to each +other the way any external client does: through the other cluster's ingress, +behind the authenticating layer described under +[Making the playground work for a team](05-operations.md#making-the-playground-work-for-a-team). + +The manifests in this repository assume a single cluster named `restate`; the +[invariants list](00-architecture.md#invariants-to-preserve) enumerates what +that name is wired into. A second cluster needs its own name and generated +namespace, its own snapshot bucket and IAM role, and a copy of the +`restate-apps` isolation policy for its own service namespace. + +## Health signals for delivery tools + +A `RestateDeployment` reports one `Ready` condition. Delivery tools need to +read it, because a rejected revision looks healthy at the pod level: the new +pods run and pass their probes, and only the condition says that Restate +refused to register them. + +| `Ready` | Reason | Meaning | +|---|---|---| +| `True` | `Deployed` | Latest revision registered and serving new invocations | +| `False` | `ReplicaSetScaling`, `ReplicaSetPodNotReady`, `ReplicaSetPodNotAvailable`, `ReplicaSetNoStatus` | Pods still starting; normal during a rollout | +| `False` | `AdminCallFailed` | Admin API unreachable or returned a server error; the operator retries | +| `False` | `AdminCallRejected` | Restate refused the registration; the message carries Restate's error and the operator retries every 30 s but will not succeed until the template changes | +| `False` | `HashCollision`, `FailedReconcile` | Operator-side error; inspect the operator logs | + +The operator also publishes a Warning Event with the same message for +`AdminCallFailed` and `AdminCallRejected`, so `kubectl describe +restatedeployment ` shows the reason without reading logs. + +For example, a revision that changes a service's type from Service to Virtual +Object is rejected by Restate with `META0006`. The condition and Event carry +that message, the previous revision keeps serving traffic, and the new pods +run unregistered until the spec is corrected. + +### Terraform + +The `kubernetes_manifest` `wait` block matches only positive states: a +condition reaching a value, a field matching a regex, or a rollout completing +for the built-in workload kinds. It cannot fail on `Ready=False`, so a rejected +revision makes `terraform apply` block until its timeout and then report +`context deadline exceeded` without Restate's reason. + +If a `RestateDeployment` is applied from Terraform anyway, set an update +timeout well under the default, print the condition on failure, and read the +reason from the resource rather than from Terraform: + +```bash +kubectl -n restate-apps get restatedeployment \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason}: {.message}{"\n"}{end}' +``` + +State is not left inconsistent: the provider keeps the previous manifest in +state when the wait fails, so the next plan already proposes the rollback. + +### Argo CD + +Argo CD has no built-in health assessment for `restate.dev` kinds and reports +unknown custom resources as Healthy. Add a health check to `argocd-cm` so a +rejected revision shows as Degraded with Restate's message, and a rollout in +progress shows as Progressing: + +```yaml +data: + resource.customizations.health.restate.dev_RestateDeployment: | + hs = { status = "Progressing", message = "Waiting for RestateDeployment status" } + if obj.status == nil then + return hs + end + if obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil + and obj.status.observedGeneration < obj.metadata.generation then + hs.message = "Waiting for the operator to observe the latest generation" + return hs + end + if obj.status.conditions ~= nil then + for _, c in ipairs(obj.status.conditions) do + if c.type == "Ready" then + if c.status == "True" then + hs.status = "Healthy" + hs.message = c.message or "Deployed" + elseif c.reason == "AdminCallRejected" then + hs.status = "Degraded" + hs.message = c.message + else + hs.message = (c.reason or "") .. ": " .. (c.message or "") + end + return hs + end + end + end + return hs +``` + +With this in place, a sync of a rejected revision fails its health check within +one reconcile instead of waiting on a timeout, and the previous revision keeps +serving because the operator never replaced it. Teams that deploy the cluster +stages with Terraform and the applications with Argo CD get the boundary this +guide recommends without giving up automated health gating. + +### Flux + +Flux's health checks use kstatus, which treats a `Ready=False` condition as +still reconciling and reports failure only on a `Stalled=True` condition. The +operator does not set `Stalled`, so a rejected revision keeps a Flux +`Kustomization` in progress until its `timeout`. Set that timeout to a few +minutes and read the `Ready` reason as above to see why. + ## Useful fields | Field | Default | Meaning | diff --git a/docs/05-operations.md b/docs/05-operations.md index 318e1bf..9c55b62 100644 --- a/docs/05-operations.md +++ b/docs/05-operations.md @@ -161,6 +161,30 @@ does not expose anything by itself, and it does not change what the server binds to. Equally, exposing ingress without setting it leaves the playground pointing somewhere your users cannot reach. +## Observability + +Restate needs no agent or sidecar; the work is on the platform side. + +- **Metrics.** Every Restate pod serves Prometheus metrics on port 5122 at + `/metrics`, the same port as node traffic. The operator's NetworkPolicy + admits that port only from the `restate` and `restate-operator` namespaces, + so a scraper elsewhere must be added under `security.networkPeers.node` in + `resources/04-restate-cluster.yaml` (the older `metrics` key is deprecated). + Scrape each pod through `svc/restate-cluster`, the headless Service, so + per-node series keep their identity. Restate publishes two Grafana dashboards + to import; see [Metrics](https://docs.restate.dev/server/monitoring/metrics). +- **Logs.** The operator sets `RESTATE_LOG_FORMAT=json` on the pods, so the + platform's usual log collector picks them up as structured events. Keep the + default `info` level in production; see + [Logging](https://docs.restate.dev/server/monitoring/logging). +- **Traces.** Restate exports OTLP traces of invocations when + `tracing-endpoint` points at a collector; see + [Tracing](https://docs.restate.dev/server/monitoring/tracing). A collector + inside the cluster has a private address, which the operator's default egress + policy blocks, so add its address under `spec.security.networkEgressRules` + as described under + [Private AWS endpoints](00-architecture.md#private-aws-endpoints). + ## Verify the snapshot path Automatic snapshots require both the configured record threshold and interval, @@ -221,6 +245,25 @@ replacement time, or log growth — then either schedule manual snapshots or low from the profile and is left at its profile value here; see [Profile fidelity](04-profile-fidelity.md). +The two settings combine in three ways, per the +[Restate snapshot documentation](https://docs.restate.dev/server/snapshots#configuring-automatic-snapshotting): + +| Configured | Trigger | +|---|---| +| Both | Interval elapsed **and** record threshold reached, per partition (the shipped configuration) | +| `SNAPSHOT_INTERVAL` only | Every partition snapshots on the interval, regardless of traffic | +| `SNAPSHOT_INTERVAL_NUM_RECORDS` only | Record count only, no time component | + +The record gate matters beyond node replacement: the log can only be trimmed +up to the oldest retained snapshot of each partition, so a low-traffic +partition that never reaches the threshold also never lets its log segment be +trimmed. If a bounded time between snapshots is the goal, drop the record +threshold and set the interval alone; Restate's own documentation example uses +`60m`. The shipped values are the Restate Cloud profile's and are left as they +are here so the manifest stays a faithful translation. `NUM_RETAINED` is `2` +for the same reason; Restate's documentation recommends `1` for most +deployments, because trimming follows the oldest retained snapshot. + Check where partitions actually stand before concluding anything is broken: ```bash @@ -441,6 +484,16 @@ affect a live replicated system. Before applying: 3. change one dimension at a time; 4. watch pods and `restatectl status` until the cluster is healthy again. +### Node maintenance + +The operator creates a PodDisruptionBudget on the Restate pods with +`maxUnavailable: 1`, so a node drain or a managed node-group upgrade evicts one +Restate pod at a time and waits for it to be Ready elsewhere before the next. +With three nodes and required host anti-affinity, an evicted pod has nowhere to +go until a replacement node exists, so drain with a surge node available or +expect the pod to sit Pending until the drained node returns. Check +`restatectl status` between nodes, as for any other roll. + ### Upgrade Restate or the operator The image and chart are intentionally pinned. Before upgrading: @@ -456,6 +509,26 @@ The image and chart are intentionally pinned. Before upgrading: Changing only the container image is not a complete upgrade review. +On the Terraform path, `terraform apply` does not wait for the roll. The +`RestateCluster` is already `Ready=True` when the change is submitted, so the +stage-02 wait is satisfied immediately and Terraform returns while the +StatefulSet is still replacing pods one at a time, highest ordinal first. +Gate the next pipeline step on the StatefulSet instead: + +```bash +kubectl -n restate rollout status statefulset/restate --timeout=15m +kubectl -n restate exec restate-0 -- restatectl status +``` + +Expect the roll itself to take a few minutes for three pods. Clients may see a +connection reset at the moment a pod terminates; invocations are retried by +Restate, but a client holding an open connection to that pod is not. + +A `kubectl port-forward` to `svc/restate` is pinned to one pod and dies when +that pod is replaced, so a client tunnelled through it sees connection errors +during a roll that in-cluster clients do not. Re-establish the forward after +the roll rather than reading those errors as cluster unavailability. + ## Data safety and recovery boundaries Two independent mechanisms protect different failure modes: @@ -468,7 +541,11 @@ Two independent mechanisms protect different failure modes: - **S3 partition snapshots** allow nodes to bootstrap without replaying the entire retained log and provide recovery material outside the EBS volumes. -Neither mechanism is a complete, automatic disaster-recovery workflow. +Neither mechanism is a complete, automatic disaster-recovery workflow, and +snapshots are not a backup: the replicated log and the cluster metadata exist +only on the volumes, and partition state is what snapshots let you rebuild. See +[Data durability model](00-architecture.md#data-durability-model) for which +data is irreplaceable and for the recommendation to keep metadata in S3. Released PVs retain their old claim references and do not bind to replacement PVCs automatically. Before removal or another data-affecting change, record the PV, PVC, Availability Zone, and EBS volume-id mapping: diff --git a/resources/05-restate-compute.yaml b/resources/05-restate-compute.yaml index 27e9367..8bf9c83 100644 --- a/resources/05-restate-compute.yaml +++ b/resources/05-restate-compute.yaml @@ -40,9 +40,12 @@ spec: spec: containers: - name: service - # REPLACE_ME: the SDK service image (e.g. an ECR image built from - # this repo's node service). Keep the port name "restate" because the - # operator uses it for the registration URL. + # REPLACE_ME: your SDK service image, pushed to a registry the nodes + # can pull from. This repository ships no service; start from one of + # the SDK examples at https://github.com/restatedev/examples in the + # language of your choice. Any image listening on 9080 works. Keep + # the port name "restate" because the operator uses it for the + # registration URL. image: REPLACE_ME_SERVICE_IMAGE ports: - name: restate diff --git a/scripts/validate.sh b/scripts/validate.sh index b3d0a67..860c8b7 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -8,7 +8,7 @@ export TF_IN_AUTOMATION=1 tofu fmt -check -recursive terraform -for TOFU_STAGE in terraform/01-foundation terraform/02-restate; do +for TOFU_STAGE in terraform/01-foundation terraform/02-restate terraform/03-services; do tofu -chdir="$TOFU_STAGE" init -backend=false -input=false -no-color >/dev/null tofu -chdir="$TOFU_STAGE" validate -no-color done diff --git a/terraform/.gitignore b/terraform/.gitignore index 151928b..39ad666 100644 --- a/terraform/.gitignore +++ b/terraform/.gitignore @@ -14,3 +14,6 @@ crash.log # Generate your own with `terraform init` and commit it in your fork if you # standardize on one tool. .terraform.lock.hcl + +# apply/timeline logs written during validation runs +*.log diff --git a/terraform/03-services/outputs.tf b/terraform/03-services/outputs.tf new file mode 100644 index 0000000..5ed2cbe --- /dev/null +++ b/terraform/03-services/outputs.tf @@ -0,0 +1,7 @@ +output "service_name" { + value = local.service_manifest.metadata.name +} + +output "invoke_hint" { + value = "kubectl -n restate port-forward svc/restate 8080:8080 & curl localhost:8080/Greeter/greet --json '\"Restate\"'" +} diff --git a/terraform/03-services/providers.tf b/terraform/03-services/providers.tf new file mode 100644 index 0000000..c848312 --- /dev/null +++ b/terraform/03-services/providers.tf @@ -0,0 +1,18 @@ +provider "aws" { + region = var.region +} + +data "aws_eks_cluster" "this" { + name = var.cluster_name +} + +provider "kubernetes" { + host = data.aws_eks_cluster.this.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + args = ["eks", "get-token", "--cluster-name", var.cluster_name, "--region", var.region] + } +} diff --git a/terraform/03-services/service.tf b/terraform/03-services/service.tf new file mode 100644 index 0000000..fc34e60 --- /dev/null +++ b/terraform/03-services/service.tf @@ -0,0 +1,31 @@ +# The example RestateDeployment from resources/05-restate-compute.yaml with its +# image placeholder filled in. Kept in its own state, separate from the cluster +# stages, so an image bump is an application change rather than an +# infrastructure one. +locals { + service_manifest = yamldecode( + replace( + file("${path.module}/../../resources/05-restate-compute.yaml"), + "REPLACE_ME_SERVICE_IMAGE", + var.service_image, + ) + ) +} + +resource "kubernetes_manifest" "service" { + manifest = local.service_manifest + + wait { + condition { + type = "Ready" + status = "True" + } + } + + timeouts { + create = "10m" + update = "10m" + # deletion waits on the operator's drain finalizer + delete = "30m" + } +} diff --git a/terraform/03-services/variables.tf b/terraform/03-services/variables.tf new file mode 100644 index 0000000..7f6810a --- /dev/null +++ b/terraform/03-services/variables.tf @@ -0,0 +1,30 @@ +# Shared with the other stages via ../terraform.tfvars; the unused ones are +# declared so the shared file loads cleanly. + +variable "cluster_name" { + type = string +} + +variable "region" { + type = string +} + +variable "snapshots_bucket" { + type = string +} + +variable "create_oidc_provider" { + type = bool + default = false +} + +variable "create_service_cidr_egress_policy" { + type = bool + default = true +} + +# Immutable image reference for the SDK service; passed per deployment from +# the application pipeline rather than kept in this file. +variable "service_image" { + type = string +} diff --git a/terraform/03-services/versions.tf b/terraform/03-services/versions.tf new file mode 100644 index 0000000..8ed0fde --- /dev/null +++ b/terraform/03-services/versions.tf @@ -0,0 +1,14 @@ +terraform { + required_version = ">= 1.5" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.35" + } + } +} diff --git a/terraform/README.md b/terraform/README.md index baa60f5..b23204b 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -256,8 +256,12 @@ checks. ## SDK services: operator-managed, not Terraform-managed -These modules deploy the Restate cluster. They do not deploy your services, and -there is no `service_image` variable. +Stages 01 and 02 deploy the Restate cluster. They do not deploy your services, +and they have no `service_image` variable. A separate, optional +`terraform/03-services` example shows what applying the RestateDeployment from +Terraform looks like if that is where your delivery pipeline already lives; its +limitations are described under +[Health signals for delivery tools](../docs/03-deploying-services.md#health-signals-for-delivery-tools). Your services are still managed, just not from here. The operator reconciles them through its own `RestateDeployment` custom resource, which gives each @@ -279,6 +283,10 @@ whatever already ships your applications: - [Deploying SDK services](../docs/03-deploying-services.md) — the lifecycle contract, rollout, drain, rollback, and per-symptom troubleshooting; +- [Health signals for delivery tools](../docs/03-deploying-services.md#health-signals-for-delivery-tools) + — how a rejected revision surfaces in Terraform, Argo CD, and Flux, with an + Argo CD health check; the common split is Terraform for these two stages and + Argo CD for the applications; - [operator service examples](https://github.com/restatedev/restate-operator/tree/main/examples/services/greeter) — upstream `RestateDeployment` manifests, including a Knative variant; - [Restate on Kubernetes](https://docs.restate.dev/deploy/services/kubernetes)