From 0288c610a903ce8dbd7ac985ce7a532276fec21f Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Thu, 27 Aug 2026 11:30:13 -0400 Subject: [PATCH 01/12] docs: document Kubernetes Operator 0.2.1 --- .../configuration.md | 232 ++++++++++++++---- .../getting-started/aws.md | 57 +++-- .../getting-started/azure.md | 49 ++-- .../high-availability.md | 208 +++++++++++----- .../enterprise-kubernetes-operator/index.md | 44 ++-- .../installation.md | 85 ++++--- .../known-limitations.md | 117 +++++++-- .../operations/backup-restore.md | 136 +++++++--- .../operations/database.md | 215 ++++++++++++---- .../operations/operator.md | 171 +++++++++---- .../troubleshooting.md | 108 +++++--- 11 files changed, 1039 insertions(+), 383 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/configuration.md b/documentation/enterprise-kubernetes-operator/configuration.md index 6e4b2f052..dcbfae50a 100644 --- a/documentation/enterprise-kubernetes-operator/configuration.md +++ b/documentation/enterprise-kubernetes-operator/configuration.md @@ -1,13 +1,15 @@ --- title: Configure the Kubernetes Operator -description: Configure QuestDB clusters, object storage, backup, replication, storage, and scheduling. +description: + Configure QuestDB clusters, object storage, backup, replication, storage, and + scheduling. --- # Configuration Use this page for the common choices. The generated -[API Reference](/docs/enterprise-kubernetes-operator/reference/api/) is the source for every field, default, -validation, and status property. +[API Reference](/docs/enterprise-kubernetes-operator/reference/api/) is the +source for every field, default, validation, and status property. ## Object storage @@ -47,10 +49,10 @@ spec: Static credential Secrets use these keys: -| Provider | Keys | -| --- | --- | -| S3 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` | -| Azure | `AZURE_STORAGE_KEY` | +| Provider | Keys | +| -------- | -------------------------------------------------------------------------- | +| S3 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` | +| Azure | `AZURE_STORAGE_KEY` | The store, credentials Secret, and consuming cluster must be in the same namespace. Prefer one cloud identity and least-privilege policy per tenant or @@ -59,8 +61,8 @@ ServiceAccount; it is for QuestDB pod S3 access, not image pulls. ### Prefix isolation -QuestDB has no instance-name key in object storage, so prefixes are its isolation -boundary: +QuestDB has no instance-name key in object storage, so prefixes are its +isolation boundary: - `spec.backup.root` defaults to `backup/`. Give each cluster a distinct backup root when clusters share a bucket or container. @@ -107,10 +109,11 @@ spec: ``` The operator creates `-rw` for the current primary. For store-backed -replication it also creates `-ro` for read-eligible replicas; when none are -eligible, `-ro` falls back to the primary. Use the [database -operations](/docs/enterprise-kubernetes-operator/operations/database/#services-and-ports) and -[high-availability](/docs/enterprise-kubernetes-operator/high-availability/) runbooks for routing and failover. +replication it also creates `-ro` for read-eligible replicas; when none +are eligible, `-ro` falls back to the primary. Use the +[database operations](/docs/enterprise-kubernetes-operator/operations/database/#services-and-ports) +and [high-availability](/docs/enterprise-kubernetes-operator/high-availability/) +runbooks for routing and failover. The WAL cleaner is enabled by default. Do not disable it unless another process owns replication-WAL retention; otherwise WAL grows without bound. @@ -139,7 +142,7 @@ Set a firm memory budget with equal request and limit. A CPU limit is optional: ```yaml spec: - image: registry.distribution.questdb.io/questdb:3.3.4-enterprise + image: registry.distribution.questdb.io/questdb:4.0.0-enterprise imagePullSecrets: - name: questdb-registry resources: @@ -151,29 +154,74 @@ spec: ``` `spec.imagePullSecrets` names Secrets in the tenant namespace for database pods. -It is separate from the Helm value -`controllerManager.imagePullSecrets`, which applies only to the operator image -in `questdb-operator-system`. On EKS, worker-node IAM can provide ambient ECR -pull access; IRSA cannot. +It is separate from the Helm value `controllerManager.imagePullSecrets`, which +applies only to the operator image in `questdb-operator-system`. On EKS, +worker-node IAM can provide ambient ECR pull access; IRSA cannot. ## Scheduling and disruption budgets -`spec.scheduling` supports node selectors, affinity, tolerations, topology spread, -priority class, and an operator-managed PodDisruptionBudget (PDB). Without an -override, pods get soft hostname anti-affinity, soft zone spread, and a PDB with -`minAvailable: 1` for one instance or `instances-1` for HA. +`spec.scheduling` supports node selectors, affinity, tolerations, topology +spread, priority class, and an operator-managed PodDisruptionBudget (PDB). +Without an override, pods get soft hostname anti-affinity, soft zone spread, and +a PDB with `minAvailable: 1` for one instance or `instances-1` for HA. The single-instance default intentionally blocks voluntary eviction. Before a node drain, scale out, lower `spec.scheduling.podDisruptionBudget.minAvailable`, -or disable the PDB after accepting the availability risk. Follow the [node -maintenance runbook](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). +or disable the PDB after accepting the availability risk. Follow the +[node maintenance runbook](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). Custom affinity or topology spread replaces the corresponding default rather than merging with it. ## Wire protocols -`spec.protocols` opts a cluster into the wire protocols that are not served by -default. Omit it and the defaults apply. +`spec.protocols` configures optional wire-protocol behavior. Omit it and HTTP, +PGWire, and ILP stay on their defaults: HTTP/Web Console/QWP WebSocket on 9000, +plaintext PGWire on 8812, and ILP over TCP on 9009. + +### PGWire TLS + +PGWire TLS is selected at cluster creation. If `spec.protocols.pgwire.tls` is +absent when the cluster is created, PGWire stays plaintext for that cluster's +lifetime. If the block is present at creation, the same setting enables TLS on +QuestDB and on every operator SQL connection for the cluster's lifetime. Adding +or removing the TLS block later is rejected in v0.2.1; the operator never +retries over plaintext. + +```yaml +spec: + protocols: + pgwire: + tls: + certificateSecret: + name: prod-pgwire-tls + # Defaults to -rw..svc + # serverName: database.example.com + # Development only: encryption without certificate/hostname verification. + # insecureSkipVerify: true +``` + +The `certificateSecret` must be in the same namespace as the `QuestDBCluster`. +It may be type `kubernetes.io/tls` or `Opaque` and must contain `tls.crt` and +`tls.key`; optional `ca.crt` supplies additional trust roots for the operator. +`tls.crt` should include the leaf followed by intermediates. Cert-manager is +optional: the operator does not issue, renew, or rotate these certificates. + +By default, verified operator SQL uses `-rw..svc` as both +the certificate DNS identity and SNI. Strict mode checks trust, hostname, +validity period, and server-auth usage. `insecureSkipVerify` keeps traffic +encrypted but disables certificate and hostname authentication; use it only for +development and expect a Warning event. + +Changing the Secret name or server certificate material safely rolls the +database Pods. Changing only `ca.crt`, `serverName`, or `insecureSkipVerify` +changes operator-client verification without rolling Pods. For CA rollover, make +old and new roots trusted at the same time until the Secret update and any Pod +rollout have converged. + +PGWire TLS does not enable TLS on HTTP, minimal HTTP, ILP, QWP, or the Web +Console, and it does not add client-certificate/mTLS authentication. + +### QWP UDP ```yaml spec: @@ -183,8 +231,8 @@ spec: enabled: true ``` -| Field | Default | Effect | -| --- | --- | --- | +| Field | Default | Effect | +| -------------------------------- | ------- | --------------------------------------------------------------------------------- | | `spec.protocols.qwp.udp.enabled` | `false` | Serves the [QWP UDP receiver](/docs/configuration/qwp/#qwpudpbindto) on 9007/UDP. | Enabling it opens 9007/UDP on the pod and publishes it on `` and @@ -201,15 +249,76 @@ Two properties are worth knowing before you enable it: backpressure, and is intended for metrics workloads where occasional message loss is acceptable. Use the WebSocket transport for reliable ingestion. -It requires an engine that ships the QWP UDP receiver; QuestDB Enterprise 3.3.4 -and later do. The operator writes the `qwp.udp.*` keys only while the receiver is -enabled, so a cluster that leaves it off carries no trace of it. +QuestDB Enterprise 4.0.0 ships the QWP UDP receiver. The operator writes the +`qwp.udp.*` keys only while the receiver is enabled, so a cluster that leaves it +off carries no trace of it. + +There is no setting for QWP over WebSocket. Ingestion (`/write/v4`) and +streaming query results (`/read/v1`) are served by the HTTP server on port 9000 +and share its network settings, so they are available on every cluster. + +Changing `spec.protocols` within the same transport mode can roll affected pods. -There is no setting for QWP over WebSocket. Ingestion (`/write/v4`) and streaming -query results (`/read/v1`) are served by the HTTP server on port 9000 and share -its network settings, so they are available on every cluster. +### Database ingress isolation -Changing `spec.protocols` rolls the affected pods. +The operator does not install a tenant/database NetworkPolicy automatically. If +your CNI enforces NetworkPolicy, add a reviewed policy before exposing tenants. +Direct Pod IP access can otherwise reach the unauthenticated minimal HTTP server +on 9003 even though that port is not published on Services. + +This example shows the intended shape. Adjust namespace and application labels +for your cluster and test kubelet probes with your CNI before relying on it: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: questdb-database-ingress + namespace: +spec: + podSelector: + matchLabels: + questdb.io/cluster: + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: questdb-operator-system + podSelector: + matchLabels: + control-plane: controller-manager + ports: + - protocol: TCP + port: 8812 + - protocol: TCP + port: 9003 + - from: + - namespaceSelector: + matchLabels: + metrics: enabled + ports: + - protocol: TCP + port: 9003 + - from: + - namespaceSelector: + matchLabels: + questdb-client: enabled + ports: + - protocol: TCP + port: 9000 + - protocol: TCP + port: 8812 + - protocol: TCP + port: 9009 + # Include only when spec.protocols.qwp.udp.enabled is true. + - protocol: UDP + port: 9007 +``` + +Customer-managed Ingresses, Gateways, LoadBalancers, and service meshes can +source-NAT traffic; review those source addresses separately. QWP UDP remains +unauthenticated when enabled. ## Extra engine options @@ -226,32 +335,53 @@ Additional backup destinations (`backup.object.store.1` through `.9`) and ambient-identity settings until a Secret-backed mechanism is available. The `qwp.udp.*` receiver keys are also operator-owned: `qwp.udp.enabled` and -`qwp.udp.bind.to` are set through -[`spec.protocols.qwp.udp`](#wire-protocols), and `qwp.udp.unicast` and -`qwp.udp.join` are rejected because multicast cannot be reached through the -unicast `ClusterIP` the operator publishes. The remaining `qwp.udp.*` tuning -keys — commit interval, buffer sizes, thread affinity — stay available. - -Use `spec.replication.config` for supported replication tuning. Values cannot -contain line separators. It rejects the same operator-owned QWP keys as -`spec.config`: both maps are merged into one `server.conf`, so a key owned in -only one of them would not be owned at all. +`qwp.udp.bind.to` are set through [`spec.protocols.qwp.udp`](#qwp-udp), and +`qwp.udp.unicast` and `qwp.udp.join` are rejected because multicast cannot be +reached through the unicast `ClusterIP` the operator publishes. The remaining +`qwp.udp.*` tuning keys — commit interval, buffer sizes, thread affinity — stay +available. + +Keys in both `spec.config` and `spec.replication.config` must match +`^[A-Za-z0-9._-]+$`. Values cannot contain line separators. Use +`spec.replication.config` for supported replication tuning. It rejects the same +operator-owned QWP keys as `spec.config`: both maps are merged into one +`server.conf`, so a key owned in only one of them would not be owned at all. ## Changes and immutable fields -Changes to the image, resources, engine configuration, wire protocols, image -pull Secrets, or pod scheduling roll affected pods. A primary roll remains single-writer-safe but -briefly interrupts writes. Plan these as disruptive changes; do not combine an -unrelated credential rotation with routine reconciliation. +Changes to the image, image pull Secrets, resources, engine configuration, +protocol fields, or pod scheduling can roll affected Pods. Ordinary rollouts are +serialized: replicas roll in serial order before the primary, and at most one +ordinary drift delete is attempted per instanceset reconcile. The controller may +hold a rollout while topology, PVC usability, peer readiness, node health, or +read-route safety is not proven. Before deleting the only `ro-ready` replica, it +first adds the primary as an overlapping Service selector candidate; this is not +an EndpointSlice acknowledgement, connection-draining protocol, or zero-gap +guarantee. A singleton still has read/write downtime while its only Pod +restarts. + +### Rotate static object-store credentials safely + +1. Make the old and new provider credentials valid at the same time. +2. Update the same-namespace Kubernetes Secret. +3. Allow the manager's roughly two-minute resync plus the serialized Pod rollout + to converge. +4. Verify expected Pod UIDs/restarts and `BackupHealthy`/`ReplicationHealthy`. +5. Revoke the old credential only after convergence. + +Immediate provider-side revocation can interrupt in-flight engine object-store +I/O even though the operator preserves the single-writer gate. Important immutable choices include: - `spec.objectStoreRef` once set; - `spec.storage.storageClassName`; - `spec.bootstrap` in presence and value; +- `spec.protocols.pgwire.tls` in presence; - `spec.replication.root` in presence and value; and - `QuestDBObjectStore.spec.provider`. Storage size is expand-only. For exact transition rules and less common fields, -use the [generated API Reference](/docs/enterprise-kubernetes-operator/reference/api/) rather than copying the full -schema from this guide. +use the +[generated API Reference](/docs/enterprise-kubernetes-operator/reference/api/) +rather than copying the full schema from this guide. diff --git a/documentation/enterprise-kubernetes-operator/getting-started/aws.md b/documentation/enterprise-kubernetes-operator/getting-started/aws.md index 579ce02e6..b39d59e52 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/aws.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/aws.md @@ -1,12 +1,13 @@ --- title: Get started on Amazon EKS -description: Prepare Amazon EKS and deploy QuestDB Enterprise with the Kubernetes Operator. +description: + Prepare Amazon EKS and deploy QuestDB Enterprise with the Kubernetes Operator. --- # Get Started on Amazon EKS -By the end of this guide, you'll have QuestDB running on EKS and sending -backups to Amazon S3. +By the end of this guide, you'll have QuestDB running on EKS and sending backups +to Amazon S3. You'll use an EKS cluster you already have. QuestDB will grant your AWS account access to the private operator and database images. @@ -26,20 +27,22 @@ Before you start, make sure your EKS cluster has: - at least 1 CPU and 2 GiB of available memory for the QuestDB pod; and - capacity for one 20 GiB EBS volume. -This guide covers managed Linux node groups. It does not cover Windows, -Fargate, Hybrid Nodes, EKS Auto Mode, Karpenter, or self-managed node groups. +This guide covers managed Linux node groups. It does not cover Windows, Fargate, +Hybrid Nodes, EKS Auto Mode, Karpenter, or self-managed node groups. You'll start with one QuestDB instance. It needs 1 CPU, 2 GiB of memory, and one 20 GiB EBS volume. Each additional instance needs the same resources. -For mitigation of a partial regional outage, place worker nodes in more than one Availability Zone. Cross-AZ connectivity is not required, since all database instances communicate over a shared S3 Bucket. +For mitigation of a partial regional outage, place worker nodes in more than one +Availability Zone. Cross-AZ connectivity is not required, since all database +instances communicate over a shared S3 Bucket. QuestDB also needs these network paths: - the Kubernetes API server to reach the operator webhook on TCP 9443; -- the operator to reach QuestDB pods on TCP 9000, 8812, and 9003; +- the operator to reach QuestDB pods on TCP 8812 and 9003; - worker nodes to pull images from QuestDB ECR in `eu-west-1`; and - QuestDB pods to reach S3 and AWS STS over HTTPS. @@ -47,14 +50,18 @@ Those are the paths the operator itself needs. Your ingestion and query clients reach QuestDB on a different set of ports, which the operator publishes on the cluster Services: 9000 for HTTP, the Web Console, and [QWP](/docs/configuration/qwp/) over WebSocket; 8812 for PostgreSQL wire; and -9009 for [InfluxDB Line Protocol](/docs/connect/compatibility/ilp/overview/) over -TCP. Allow whichever your clients use. The +9009 for [InfluxDB Line Protocol](/docs/connect/compatibility/ilp/overview/) +over TCP. Allow whichever your clients use. The [QWP UDP receiver](/docs/configuration/qwp/#qwpudpbindto) on 9007/UDP is off unless you enable -[`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#wire-protocols), +[`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#qwp-udp), and it is unauthenticated when you do, so restrict it with a NetworkPolicy. See [Services and ports](/docs/enterprise-kubernetes-operator/operations/database/#services-and-ports) -for the full table. +for the full table, and +[Configuration](/docs/enterprise-kubernetes-operator/configuration/#database-ingress-isolation) +for optional network isolation. PGWire TLS must be chosen at cluster creation; +see +[PGWire TLS](/docs/enterprise-kubernetes-operator/configuration/#pgwire-tls). The computer running Helm also needs HTTPS access to `ghcr.io`. For a private cluster, provide NAT or VPC endpoints for the AWS services above. @@ -73,8 +80,8 @@ Before you continue, make sure you have: - Helm 3.10 or later; and - the PostgreSQL `psql` client. -Keep one Bash shell open and run the steps in order. Values you set early in -the guide are reused later. +Keep one Bash shell open and run the steps in order. Values you set early in the +guide are reused later. ### Information from QuestDB @@ -141,8 +148,8 @@ aws eks describe-addon \ kubectl get csidriver ebs.csi.aws.com ``` -Continue if the Kubernetes version is between 1.31 and 1.36. You should also -see `ACTIVE` and the `ebs.csi.aws.com` driver. +Continue if the Kubernetes version is between 1.31 and 1.36. You should also see +`ACTIVE` and the `ebs.csi.aws.com` driver. The driver still needs permission to create volumes. Confirm that its controller role has `AmazonEBSCSIDriverPolicyV2`. @@ -176,6 +183,7 @@ Now make sure the driver can create a volume. This quick check creates a temporary PVC and removes it afterward. ```sh +( kubectl create namespace questdb-storage-check cat <<'EOF' | kubectl apply -f - @@ -220,6 +228,7 @@ kubectl wait pvc/questdb-storage-check \ } kubectl delete namespace questdb-storage-check --wait=true +) ``` When the PVC reaches `Bound`, storage is ready. If it does not, fix the EBS CSI @@ -327,8 +336,8 @@ aws s3 mb "s3://$S3_BUCKET" --region "$AWS_REGION" S3 bucket names are global. If that name is unavailable, add a short suffix to `S3_BUCKET` and try again. -Now create a policy that limits QuestDB to its backup and replication paths. -IAM policy names are account-wide, so change this name if it is already in use. +Now create a policy that limits QuestDB to its backup and replication paths. IAM +policy names are account-wide, so change this name if it is already in use. ```sh S3_POLICY_NAME='QuestDBS3' @@ -482,8 +491,8 @@ kubectl annotate serviceaccount default \ The manifest below connects QuestDB to your S3 bucket and starts one instance with a backup every five minutes. -Keep the QuestDB ECR account ID unchanged. The tested image tag is -`3.3.4-enterprise`; change it only when QuestDB provides another one. +Keep the QuestDB ECR account ID unchanged. The tested image tag for v0.2.1 is +`4.0.0-enterprise`; change it only when QuestDB provides another one. ```sh cat <` value; an unreplaced placeholder can be interpreted as shell redirection. +Before running any command, replace every `` value; an unreplaced +placeholder can be interpreted as shell redirection. -`status.replication.replicas[].caughtUp` is a sticky "caught up at least once" latch used for read routing. `caughtUpNow` is the current observation used by lossless promotion: `true` means reachable, unsuspended, and at zero observed lag; `false` means behind; absent means the operator could not determine freshness. An absent value fails closed. +`status.replication.replicas[].caughtUp` is a sticky "caught up at least once" +latch used for read routing. On an ordinary cluster, `caughtUpNow=true` means +reachable, unsuspended, self-consistent, and either exactly zero lag or bounded +streaming: the replica has applied everything the primary committed within the +operator's short lookback window. On a primaryless follower, only the zero-lag +proof is available. `false` means behind or stale; absent means freshness was +not determined and fails closed. ## Failure behavior There is no automatic promotion or failover decision. -| Failure | Operator behavior | -| --- | --- | -| Primary pod is lost, PVC survives, and fencing is safe | Recreate the same primary identity on its PVC. | -| Primary is stranded on an unreachable node | Report `InstanceUnreachable`; do not risk a second writer. | -| Established primary PVC is missing or Terminating | Fence the primary Pod, set `PromotionRequired=True`, remove ready RW endpoints, and do not create an empty replacement primary. | -| Replica is lost | Recreate/re-seed it from backup and WAL when prerequisites are available. | +| Failure | Operator behavior | +| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | +| Primary pod is lost, PVC survives, and fencing is safe | Recreate the same primary identity on its PVC. | +| Primary is stranded on an unreachable node | Report `InstanceUnreachable`; do not risk a second writer. | +| Established primary PVC is missing or Terminating | Fence the primary Pod, set `PromotionRequired=True`, remove ready RW endpoints, and do not create an empty replacement primary. | +| Replica is lost | Recreate/re-seed it from backup and WAL when prerequisites are available. | -A `QuestDBPromotion` records one explicit, one-shot cutover and remains as its audit object. If a same-name primary Pod was recreated before its PVC deletion began, do not wait forever for the mounted Terminating claim: current operator versions automatically fence that established Pod so `pvc-protection` can release the PVC. `status.currentPrimary` remains the non-empty primary of record until an explicit promotion; fencing is not automatic failover. +A `QuestDBPromotion` records one explicit, one-shot cutover and remains as its +audit object. If a same-name primary Pod was recreated before its PVC deletion +began, do not wait forever for the mounted Terminating claim: current operator +versions automatically fence that established Pod so `pvc-protection` can +release the PVC. `status.currentPrimary` remains the non-empty primary of record +until an explicit promotion; fencing is not automatic failover. ## Promotion and failover -| Mode | Use | Data effect | -| --- | --- | --- | -| `Planned` | Healthy primary and live replica | Drains the old primary and verifies the target before promotion. Fails rather than becoming lossy. | +| Mode | Use | Data effect | +| ----------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Planned` | Healthy primary and live replica | Drains the old primary and verifies the target before promotion. Fails rather than becoming lossy. | | `Emergency` | Primary is lost/wedged and service restoration outweighs data loss | Skips the drain. Unreplicated writes may be lost, but QWP clients using [durable acknowledgements](/docs/high-availability/store-and-forward/when-to-use/#durable-ack-when-to-opt-in) retain and retry writes that were not durably acknowledged. | ### Planned prechecks -Start only when the ordinary writer-health contract is current (`Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`), the target is a live replica, and its `caughtUpNow` is explicitly `true`. `WriteHealthy=True` is an observation, not a synthetic write or disk-capacity guarantee: +Start only when the ordinary writer-health contract is current +(`Available=True/PrimaryReady`, `Progressing=False/Settled`, and +`WriteHealthy=True/Healthy`), the target is a live replica, and its +`caughtUpNow` is explicitly `true`. `WriteHealthy=True` is an observation, not a +synthetic write or disk-capacity guarantee: ```sh kubectl get questdbcluster -n \ @@ -44,7 +65,8 @@ kubectl get pods -n -l questdb.io/cluster= \ -L questdb.io/instance,questdb.io/role -o wide ``` -Choose the integer serial from the target instance name (`-2` has target `2`). Confirm no active cutover: +Choose the integer serial from the target instance name (`-2` has target +`2`). Confirm no active cutover: ```sh kubectl get questdbcluster -n \ @@ -53,9 +75,12 @@ kubectl get questdbcluster -n \ The defaults reflect two different costs: -- `catchUpTimeoutSeconds: 900` bounds pre-drain catch-up while the old primary continues serving writes. -- `primaryGracePeriodSeconds: 120` bounds loss-of-primary and post-drain tail verification while writes are unavailable. -- `0` means wait indefinitely, not "do not wait". Use it only after accepting an unbounded outage/wait. +- `catchUpTimeoutSeconds: 900` bounds pre-drain catch-up while the old primary + continues serving writes. +- `primaryGracePeriodSeconds: 120` bounds loss-of-primary and post-drain tail + verification while writes are unavailable. +- `0` means wait indefinitely, not "do not wait". Use it only after accepting an + unbounded outage/wait. ### Create and watch @@ -76,7 +101,8 @@ spec: EOF ``` -Use a bounded progress loop that stops on both success and failure without terminating your interactive shell: +Use a bounded progress loop that stops on both success and failure without +terminating your interactive shell: ```sh PHASE="" @@ -94,7 +120,9 @@ kubectl get questdbpromotion -n \ [ "$PHASE" = "Completed" ] ``` -The final test succeeds only for `Completed`; `Failed` and the 30-minute timeout return a non-zero command status after printing the latest reason/message, without calling `exit`. +The final test succeeds only for `Completed`; `Failed` and the 30-minute timeout +return a non-zero command status after printing the latest reason/message, +without calling `exit`. ### Verify @@ -106,15 +134,29 @@ kubectl get endpointslice -n \ -l kubernetes.io/service-name=-rw ``` -Confirm `status.currentPrimary` names the target and require current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. Test `-rw` connectivity, critical table timestamps/row counts, and application writes before closing the change. Confirm replicas return healthy after the demoted primary is re-seeded. Finally inspect PVC labels: the target PVC must have `questdb.io/role=primary`, while every demoted/re-seeded replica PVC must have `questdb.io/role=replica`. The separate `questdb.io/bootstrap` label records ancestry/state and does not identify the current role. +Confirm `status.currentPrimary` names the target and require current-generation +`Available=True/PrimaryReady`, `Progressing=False/Settled`, and +`WriteHealthy=True/Healthy`. Test `-rw` connectivity, critical table +timestamps/row counts, and application writes before closing the change. Confirm +replicas return healthy after the demoted primary is re-seeded. Finally inspect +PVC labels: the target PVC must have `questdb.io/role=primary`, while every +demoted/re-seeded replica PVC must have `questdb.io/role=replica`. The separate +`questdb.io/bootstrap` label records ancestry/state and does not identify the +current role. ## Emergency promotion -Use the same object shape with `mode: Emergency` only after selecting a live target and accepting data loss. +Use the same object shape with `mode: Emergency` only after selecting a live +target and accepting data loss. -:::danger -Emergency promotion may lose writes that the old primary had not replicated. QWP clients configured with [durable acknowledgements](/docs/high-availability/store-and-forward/when-to-use/#durable-ack-when-to-opt-in) retain writes until object-store durability is confirmed and retry unacknowledged writes after connecting to the new primary. Clients connected directly to the old pod can briefly receive acknowledgements for writes that will be discarded while it detects loss of store ownership. Stop/repoint writers and use the stable `-rw` Service, not pod addresses. -::: +:::danger Emergency promotion may lose writes that the old primary had not +replicated. QWP clients configured with +[durable acknowledgements](/docs/high-availability/store-and-forward/when-to-use/#durable-ack-when-to-opt-in) +retain writes until object-store durability is confirmed and retry +unacknowledged writes after connecting to the new primary. Clients connected +directly to the old pod can briefly receive acknowledgements for writes that +will be discarded while it detects loss of store ownership. Stop/repoint writers +and use the stable `-rw` Service, not pod addresses. ::: ```sh kubectl apply -f - <-rw` EndpointSlice endpoints; afterward the selected target is the only live primary. Record the accepted recovery point and any expected lost-write window. +Use the bounded terminal-phase watcher above and perform the same primary, full +writer-health, connectivity, data, and PVC current-role checks. Before +promotion, the fenced loss state must have zero ready `-rw` EndpointSlice +endpoints; afterward the selected target is the only live primary. Record the +accepted recovery point and any expected lost-write window. ## If promotion stalls or fails @@ -139,28 +185,45 @@ Read the promotion's status first: ```sh kubectl get questdbpromotion -n \ - -o jsonpath='phase={.status.phase}{" mode="}{.status.mode}{" reason="}{.status.reason}{" message="}{.status.message}{"\n"}' + -o jsonpath='phase={.status.phase}{" mode="}{.status.mode}{" reason="}{.status.reason}{" message="}{.status.message}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" message="}{.message}{"\n"}{end}' ``` -- A running `Planned` cutover may be escalated against the **same immutable target**. Do not create a second promotion while one is active; it is refused. -- A terminal `Failed` object is one-shot. Retry with a new name/object after correcting the cause. +- `Stalled=True/TargetNotPrimary` means the shaped target has not become the + serving primary. +- `Stalled=True/FenceNotEffective` means the target reports that the old owner + still holds the stream. If the old Pod remains on a NotReady node, first power + off or otherwise isolate that machine, then apply Kubernetes' out-of-service + procedure. Do not remove the finalizer or force ownership while the old + process may run. `Stalled` is an alarm, not a terminal failure or relaxation + of fencing. +- A running `Planned` cutover may be escalated against the **same immutable + target**. Do not create a second promotion while one is active; it is refused. +- A terminal `Failed` object is one-shot. Retry with a new name/object after + correcting the cause. - Deleting in `Pending` or `Validating` cancels because nothing has been shaped. -- Deleting in `Draining` or `Promoting` does not abort. The finalizer holds the object while the cutover continues. +- Deleting in `Draining` or `Promoting` does not abort. The finalizer holds the + object while the cutover continues. -:::danger -Escalating an in-flight cutover to `Emergency` abandons the lossless drain and accepts loss of unreplicated writes. -::: +:::danger Escalating an in-flight cutover to `Emergency` abandons the lossless +drain and accepts loss of unreplicated writes. ::: ```sh kubectl patch questdbpromotion -n --type merge \ -p '{"spec":{"mode":"Emergency"}}' ``` -A live but hung drain can remain in `Draining` because the operator cannot distinguish a slow final upload from a wedged one; escalation is the human decision. `Promoting` is also unbounded while the target fails to serve. Diagnose the target pod, image pull, scheduling, PVC, Secret, network, and object-store access. Do not start by removing the promotion finalizer: that deletes the audit/control object but neither stops nor rolls back the shaped cutover. Contact support before considering it. +A live but hung drain can remain in `Draining` because the operator cannot +distinguish a slow final upload from a wedged one; escalation is the human +decision. `Promoting` is also unbounded while the target fails to serve. +Diagnose the target pod, image pull, scheduling, PVC, Secret, network, and +object-store access. Do not start by removing the promotion finalizer: that +deletes the audit/control object but neither stops nor rolls back the shaped +cutover. Contact support before considering it. ## Migrate an existing QuestDB onto the operator -A follower reduces migration downtime by restoring the source's backup and consuming its WAL while the external source continues serving. +A follower reduces migration downtime by restoring the source's backup and +consuming its WAL while the external source continues serving. ### Source prerequisites @@ -170,14 +233,23 @@ The source must: - create completed backups in a known object-store backup root; - upload replication WAL to a known root in the same store; - retain WAL back to the seed backup; -- expose its exact `SELECT backup_instance_name();` value; and +- expose its exact `SELECT backup_instance_name();` value, matching + `^[a-z0-9]+(-[a-z0-9]+)*$`; and - support a controlled stop and one final `primary-catchup-uploads` run. -The operator never connects to, configures, stops, fences, or lists storage for the source. +The operator never connects to, configures, stops, fences, or lists storage for +the source. ### Create and verify the follower -All source selectors are immutable. Confirm `sourceInstanceName`, backup root, and WAL root before creating the cluster. Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`: set `` to that private image and `` to the pull Secret in this namespace. Remove the entire `imagePullSecrets` block only when every node has ambient pull access, such as an authorized EKS worker-node role. +All source selectors are immutable. Confirm `sourceInstanceName`, backup root, +and WAL root before creating the cluster. The source instance name must be +copied exactly and match the lowercase hyphen-separated engine identity format. +Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`: set +`` to that private image and +`` to the pull Secret in this namespace. Remove the +entire `imagePullSecrets` block only when every node has ambient pull access, +such as an authorized EKS worker-node role. ```yaml apiVersion: questdb.io/v1alpha1 @@ -206,7 +278,12 @@ spec: sourceInstanceName: ``` -A healthy follower has `.status.replication.following=true`, no current primary, an empty `-rw`, and reads through `-ro`. It intentionally omits `WriteHealthy` because it has no primary; do not wait for that condition. Verify all instances and `ReplicationHealthy`; a quiet source can make progress indeterminate, so also confirm the immutable source identity and roots directly against the source configuration. +A healthy follower has `.status.replication.following=true`, no current primary, +an empty `-rw`, and reads through `-ro`. It intentionally omits +`WriteHealthy` because it has no primary; do not wait for that condition. Verify +all instances and `ReplicationHealthy`; a quiet source can make progress +indeterminate, so also confirm the immutable source identity and roots directly +against the source configuration. ```sh kubectl get questdbcluster -n \ @@ -217,7 +294,8 @@ kubectl get endpointslice -n \ ### Cut over -Immediately before stopping the source, record when cutover preparation began in the same Bash shell you will use for the post-cutover check: +Immediately before stopping the source, record when cutover preparation began in +the same Bash shell you will use for the post-cutover check: ```bash CUTOVER_TIME_CAPTURED=false @@ -232,18 +310,35 @@ fi 1. Stop application writes to the source, then stop the source database. 2. Restart the source once with `replication.role=primary-catchup-uploads`. -3. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS` in the **source** logs, then stop it again. This final upload may itself be unbounded; supervise it at the source. A normal shutdown does not prove the tail reached object storage. -4. Create a `Planned` `QuestDBPromotion` for a healthy follower instance and use the bounded watcher above. - -The planned follower gate requires the source stream to remain quiet for at least 60 seconds and the target to consume the published WAL before promotion. It still cannot prove an idle source process is stopped. If the engine reports `SourceStillOwnsStore`, stop the source fully and retry with a new promotion object. - -:::danger -Emergency follower promotion accepts losing source WAL that was not uploaded or not consumed. Use it only when the source cannot be drained and that loss is explicitly accepted. -::: - -5. After a completed cutover, verify `-rw`, data, writes, and `currentPrimary`. -6. Permanently decommission the source so it cannot restart and contend for the adopted WAL root. -7. Wait for this cluster's own first post-cutover backup. The hourly deadline must exceed one schedule interval plus the roughly two-minute observation delay, so this check allows about 75 minutes. Do not accept an old `completed` status: require a non-empty `endTime` later than the captured cutover-start time. Both values are RFC3339 UTC timestamps, so the Bash string comparison proves the observed backup completed after cutover preparation began. The WAL cleaner remains off until this backup completes; its release rolls the primary once. Verify the roll and writer readiness afterward. +3. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS` in the **source** logs, then + stop it again. This final upload may itself be unbounded; supervise it at the + source. A normal shutdown does not prove the tail reached object storage. +4. Create a `Planned` `QuestDBPromotion` for a healthy follower instance and use + the bounded watcher above. + +The planned follower gate requires the source stream to remain quiet for at +least 60 seconds and the target to consume the published WAL before promotion. +It still cannot prove an idle source process is stopped. If the engine reports +`SourceStillOwnsStore`, stop the source fully and retry with a new promotion +object. + +:::danger Emergency follower promotion accepts losing source WAL that was not +uploaded or not consumed. Use it only when the source cannot be drained and that +loss is explicitly accepted. ::: + +5. After a completed cutover, verify `-rw`, data, writes, and + `currentPrimary`. +6. Permanently decommission the source so it cannot restart and contend for the + adopted WAL root. +7. Wait for this cluster's own first post-cutover backup. The hourly deadline + must exceed one schedule interval plus the roughly two-minute observation + delay, so this check allows about 75 minutes. Do not accept an old + `completed` status: require a non-empty `endTime` later than the captured + cutover-start time. Both values are RFC3339 UTC timestamps, so the Bash + string comparison proves the observed backup completed after cutover + preparation began. The WAL cleaner remains off until this backup completes; + its release rolls the primary once. Verify the roll and writer readiness + afterward. ```bash BACKUP_VERIFIED=false @@ -268,4 +363,7 @@ done -o jsonpath='{range .status.conditions[?(@.type=="Available")]}Available={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="Progressing")]}Progressing={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="WriteHealthy")]}WriteHealthy={.status}{"/"}{.reason}{"\n"}{end}' ``` -After the roll, require `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy` at the current generation before declaring the cluster writer-ready and the WAL cleaner released. +After the roll, require `Available=True/PrimaryReady`, +`Progressing=False/Settled`, and `WriteHealthy=True/Healthy` at the current +generation before declaring the cluster writer-ready and the WAL cleaner +released. diff --git a/documentation/enterprise-kubernetes-operator/index.md b/documentation/enterprise-kubernetes-operator/index.md index e77b3c009..2951361d8 100644 --- a/documentation/enterprise-kubernetes-operator/index.md +++ b/documentation/enterprise-kubernetes-operator/index.md @@ -9,14 +9,12 @@ import { EnterpriseNote } from "@site/src/components/EnterpriseNote" The Kubernetes Operator manages QuestDB Enterprise clusters. For open source - QuestDB on Kubernetes, use the - [QuestDB Helm chart](/docs/deployment/kubernetes/). + QuestDB on Kubernetes, use the [QuestDB Helm + chart](/docs/deployment/kubernetes/). -:::warning[Supported platforms] -The API is `questdb.io/v1alpha1` and may change between releases. -Amazon EKS and Azure AKS are supported on the tested versions in the -[support matrix](/docs/enterprise-kubernetes-operator/known-limitations/#supported-platforms-and-versions). +:::warning[Supported platforms] The API is `questdb.io/v1alpha1` and may change between +releases. Amazon EKS and Azure AKS are supported on the tested versions in the [support matrix](/docs/enterprise-kubernetes-operator/known-limitations/#supported-platforms-and-versions). See [Support](/docs/enterprise-kubernetes-operator/support/) for support expectations. ::: @@ -33,14 +31,26 @@ static Secret. QuestDB database pods perform all object-store I/O. ## Choose your path -- **Amazon EKS:** follow the [EKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/aws/). -- **Azure AKS:** follow the [AKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/azure/). -- **Shared install requirements:** see [Installation](/docs/enterprise-kubernetes-operator/installation/). -- **Operate the operator:** use the [operator runbook](/docs/enterprise-kubernetes-operator/operations/operator/). -- **Operate a database:** use the [database](/docs/enterprise-kubernetes-operator/operations/database/), - [backup and restore](/docs/enterprise-kubernetes-operator/operations/backup-restore/), or - [high-availability](/docs/enterprise-kubernetes-operator/high-availability/) runbook. -- **Diagnose a problem:** use [Troubleshooting](/docs/enterprise-kubernetes-operator/troubleshooting/). - -Use [Configuration](/docs/enterprise-kubernetes-operator/configuration/) for common resource shapes and the -[API Reference](/docs/enterprise-kubernetes-operator/reference/api/) for every field, default, and validation rule. +- **Amazon EKS:** follow the + [EKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/aws/). +- **Azure AKS:** follow the + [AKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/azure/). +- **Shared install requirements:** see + [Installation](/docs/enterprise-kubernetes-operator/installation/). +- **PGWire TLS and network isolation:** plan them before creation with + [Configuration](/docs/enterprise-kubernetes-operator/configuration/#pgwire-tls). +- **Operate the operator:** use the + [operator runbook](/docs/enterprise-kubernetes-operator/operations/operator/). +- **Operate a database:** use the + [database](/docs/enterprise-kubernetes-operator/operations/database/), + [backup and restore](/docs/enterprise-kubernetes-operator/operations/backup-restore/), + or + [high-availability](/docs/enterprise-kubernetes-operator/high-availability/) + runbook. +- **Diagnose a problem:** use + [Troubleshooting](/docs/enterprise-kubernetes-operator/troubleshooting/). + +Use [Configuration](/docs/enterprise-kubernetes-operator/configuration/) for +common resource shapes and the +[API Reference](/docs/enterprise-kubernetes-operator/reference/api/) for every +field, default, and validation rule. diff --git a/documentation/enterprise-kubernetes-operator/installation.md b/documentation/enterprise-kubernetes-operator/installation.md index 0bbe874f9..6aa31f548 100644 --- a/documentation/enterprise-kubernetes-operator/installation.md +++ b/documentation/enterprise-kubernetes-operator/installation.md @@ -1,6 +1,8 @@ --- title: Install the Kubernetes Operator -description: Install the QuestDB Enterprise Kubernetes Operator from its supported OCI Helm chart. +description: + Install the QuestDB Enterprise Kubernetes Operator from its supported OCI Helm + chart. --- # Installation @@ -8,14 +10,14 @@ description: Install the QuestDB Enterprise Kubernetes Operator from its support The QuestDB Enterprise Operator is available to named design partners on these tested combinations: -| Platform | Kubernetes | QuestDB Enterprise | -| --- | --- | --- | -| Amazon EKS | 1.31–1.36 | 3.3.4 | -| Azure AKS | 1.33–1.36 | 3.3.4 | +| Platform | Kubernetes | QuestDB Enterprise | +| ---------- | ---------- | ------------------ | +| Amazon EKS | 1.31–1.36 | 4.0.0 | +| Azure AKS | 1.33–1.36 | 4.0.0 | Other Kubernetes distributions and versions are untested. Only the latest -release receives fixes; obtain its `` and private registry credentials -or AWS account grant from your QuestDB design-partner contact. +release receives fixes; obtain its `` and private registry +credentials or AWS account grant from your QuestDB design-partner contact. ## Before you install @@ -35,11 +37,11 @@ Check these prerequisites: - Enough schedulable CPU, memory, and zonal disk capacity for every database instance. QuestDB memory request and limit must be equal. - Cluster DNS and network paths for the Kubernetes API server to reach the - operator webhook on TCP 9443; the operator to reach tenant pods on TCP 9000, - 8812, and 9003; and nodes/pods to reach the private image registry, object - store, and cloud identity endpoints over HTTPS. -- Pull access for two private images. The **operator** pull credential belongs in - `questdb-operator-system`; each **database** pull credential belongs in its + operator webhook on TCP 9443; the operator to reach tenant pods on TCP 8812 + and 9003; and nodes/pods to reach the private image registry, object store, + and cloud identity endpoints over HTTPS. +- Pull access for two private images. The **operator** pull credential belongs + in `questdb-operator-system`; each **database** pull credential belongs in its tenant namespace and is named by `spec.imagePullSecrets`. Do not reuse a short-lived ECR login token as a long-lived Secret. @@ -51,8 +53,8 @@ Follow the complete cloud checklist before installing: ## Canonical Helm install The OCI chart on GHCR is public; its operator image is private. Replace the -placeholders first. Your QuestDB contact supplies the current operator version and, -off AWS, static credentials for `registry.distribution.questdb.io`. +placeholders first. Your QuestDB contact supplies the current operator version +and, off AWS, static credentials for `registry.distribution.questdb.io`. ```bash export OPERATOR_VERSION='' @@ -75,8 +77,8 @@ helm install questdb-operator oci://ghcr.io/questdb/charts/questdb-operator \ ``` On EKS with QuestDB's cross-account ECR repository grant, kubelets pull through -the worker-node IAM role. Omit the Secret and both `--set` flags; do not use IRSA -for image pulls. +the worker-node IAM role. Omit the Secret and both `--set` flags; do not use +IRSA for image pulls. Verify the deployment and APIs: @@ -92,18 +94,43 @@ CRDs are listed. ## Common safe values -| Value | Default | Guidance | -| --- | --- | --- | -| `controllerManager.replicas` | `1` | Keep at `1` unless QuestDB advises otherwise. | -| `crd.enable` / `crd.keep` | `true` / `true` | Install CRDs and retain them on Helm uninstall. | -| `webhook.enable` | `true` | Keep admission validation enabled. Reconcile-time validation remains a backstop. | -| `webhook.certMode` | `self-signed` | The operator creates and rotates its serving certificate; cert-manager is not required. | -| `webhook.failurePolicy` | `Ignore` | Fail-open avoids blocking cluster writes during a webhook outage. Use `Fail` only after accepting that availability trade-off. | -| `metrics.enable` | `true` | Exposes authenticated HTTPS metrics on 8443. | -| `prometheus.enable` | `false` | Requires Prometheus Operator `ServiceMonitor` CRDs. Bind the scraper identity to `questdb-operator-metrics-reader`. | -| `certmanager.enable` | `false` | Does not provide working verified controller metrics TLS. Leave disabled unless QuestDB has reviewed a separate integration; see [Known Limitations](/docs/enterprise-kubernetes-operator/known-limitations/#chart-managed-verified-controller-metrics-tls-is-not-operational). | -| `networkPolicy.enable` | `false` | Enable only after allowing the control-plane, operator, tenant, registry, identity, and object-store paths above. | +| Value | Default | Guidance | +| ---------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `controllerManager.replicas` | `1` | Keep at `1` unless QuestDB advises otherwise. | +| `crd.enable` / `crd.keep` | `true` / `true` | Install CRDs and retain them on Helm uninstall. | +| `webhook.enable` | `true` | Keep admission validation enabled. Reconcile-time validation remains a backstop. | +| `webhook.certMode` | `self-signed` | The operator creates and rotates its serving certificate; cert-manager is not required. | +| `webhook.failurePolicy` | `Ignore` | Fail-open avoids blocking cluster writes during a webhook outage. Use `Fail` only after accepting that availability trade-off. | +| `metrics.enable` | `true` | Exposes authenticated HTTPS metrics on 8443. | +| `prometheus.enable` | `false` | Requires Prometheus Operator `ServiceMonitor` CRDs. Bind the scraper identity to `questdb-operator-metrics-reader`. | +| `certmanager.enable` | `false` | When combined with `prometheus.enable=true`, the chart issues and mounts a metrics serving certificate and renders a ServiceMonitor with certificate verification. Install cert-manager first. | +| `networkPolicy.enable` | `false` | Selects the manager only. It restricts metrics ingress to namespaces labeled `metrics=enabled` and leaves webhook 9443 open for managed API servers. It does not create database/tenant ingress policy. | + +## Operator scheduling and availability + +The chart can place and annotate the manager independently of database +`spec.scheduling`. Use `controllerManager.nodeSelector`, `tolerations`, +`affinity`, `topologySpreadConstraints`, and `priorityClassName` for the manager +Pod. Use `controllerManager.pod.annotations`, `controllerManager.pod.labels`, +`controllerManager.serviceAccount.annotations`, and +`controllerManager.container.env` for integration metadata and environment. + +The default manager has one replica and leader election enabled, so only one +reconciler is active. The manager PDB is disabled by default because +`replicas=1`; enable it only after running multiple replicas across failure +domains, for example: + +```yaml +controllerManager: + replicas: 2 + podDisruptionBudget: + enabled: true + minAvailable: 1 +``` + +The manager PDB accepts integer and percentage `minAvailable` values. The +database CRD PDB uses an integer only. Treat upgrades, rollback, and removal as separate lifecycle procedures; use the -[operator operations runbook](/docs/enterprise-kubernetes-operator/operations/operator/) rather than inferring them -from installation commands. +[operator operations runbook](/docs/enterprise-kubernetes-operator/operations/operator/) +rather than inferring them from installation commands. diff --git a/documentation/enterprise-kubernetes-operator/known-limitations.md b/documentation/enterprise-kubernetes-operator/known-limitations.md index 1e4cbe911..d1442207b 100644 --- a/documentation/enterprise-kubernetes-operator/known-limitations.md +++ b/documentation/enterprise-kubernetes-operator/known-limitations.md @@ -1,81 +1,148 @@ --- title: Kubernetes Operator known limitations -description: Review supported platforms, versions, lifecycle constraints, and current Kubernetes Operator limitations. +description: + Review supported platforms, versions, lifecycle constraints, and current + Kubernetes Operator limitations. --- # Known limitations -Support is limited to a fixed platform matrix. These behaviors require operational planning. +Support is limited to a fixed platform matrix. These behaviors require +operational planning. ## Availability and promotion ### A singleton is not highly available -A one-instance cluster has no replica to promote. On an unreachable node, the operator does not create a second pod while the old writer might still run. Recovery waits for safe node/volume recovery or human action. Use store-backed replication before an incident; see [Scale out](/docs/enterprise-kubernetes-operator/operations/database/#scale-out). +A one-instance cluster has no replica to promote. On an unreachable node, the +operator does not create a second pod while the old writer might still run. +Recovery waits for safe node/volume recovery or human action. Use store-backed +replication before an incident; see +[Scale out](/docs/enterprise-kubernetes-operator/operations/database/#scale-out). ### Singleton drain is blocked by default -The default PodDisruptionBudget has `minAvailable: 1`, so voluntary drain of the singleton waits indefinitely. Before maintenance, scale out, lower `minAvailable`, or disable the PDB after accepting downtime. See [Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). +The default PodDisruptionBudget has `minAvailable: 1`, so voluntary drain of the +singleton waits indefinitely. Before maintenance, scale out, lower +`minAvailable`, or disable the PDB after accepting downtime. See +[Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). ### Failover is never automatic -For an established replicated/object-store-backed cluster, the operator recreates a lost primary only when it can preserve the same PVC identity safely. Primary PVC loss fences that primary and sets `PromotionRequired`; a human must recover the storage before a lossless `Planned` cutover, or select a replica and explicitly accept `Emergency` loss when the old primary cannot be drained. A standalone cluster has no replicated-primary loss guard and no promotion target: PVC loss can recreate it on fresh empty storage, so recovery is a restore from backup rather than failover. See [Promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover). +For an established replicated/object-store-backed cluster, the operator +recreates a lost primary only when it can preserve the same PVC identity safely. +Primary PVC loss fences that primary and sets `PromotionRequired`; a human must +recover the storage before a lossless `Planned` cutover, or select a replica and +explicitly accept `Emergency` loss when the old primary cannot be drained. A +standalone cluster has no replicated-primary loss guard and no promotion target: +PVC loss can recreate it on fresh empty storage, so recovery is a restore from +backup rather than failover. See +[Promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover). ### Emergency promotion loses unreplicated writes -`Emergency` skips the old-primary drain. Any write not uploaded and replayed is lost. A client connected directly to the old pod can briefly receive successful acknowledgements while that pod discovers it no longer owns the stream; those writes are also discarded. Use `-rw`, stop/repoint writers, and treat Emergency as explicit data-loss acceptance. +`Emergency` skips the old-primary drain. Any write not uploaded and replayed is +lost. A client connected directly to the old pod can briefly receive successful +acknowledgements while that pod discovers it no longer owns the stream; those +writes are also discarded. Use `-rw`, stop/repoint writers, and treat +Emergency as explicit data-loss acceptance. -Increasing `replication.primary.keepalive.interval` lengthens this stale-direct-client window. Leave its 10-second default unless QuestDB advises otherwise. +Increasing `replication.primary.keepalive.interval` lengthens this +stale-direct-client window. Leave its 10-second default unless QuestDB advises +otherwise. ### Promotion can be unbounded -A live but hung final upload can hold `Draining`: the operator cannot distinguish slow progress from a wedged upload. After the target is shaped, `Promoting` has no timeout and waits until it serves. Deleting during either phase does not abort; a finalizer retains the `QuestDBPromotion` while work continues. Diagnose the database pod and contact support rather than force-removing the audit/control object. See [If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). +A live but hung final upload can hold `Draining`: the operator cannot +distinguish slow progress from a wedged upload. After the target is shaped, +`Promoting` has no timeout and waits until it serves. Deleting during either +phase does not abort; a finalizer retains the `QuestDBPromotion` while work +continues. Diagnose the database pod and contact support rather than +force-removing the audit/control object. See +[If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). ## Backup, restore, and migration ### There is no on-demand backup API -QuestDB Enterprise runs backups from its in-engine schedule. The operator configures and observes it; there is no Kubernetes CronJob or force-backup request. Status can lag the engine observation. See [Backup and restore](/docs/enterprise-kubernetes-operator/operations/backup-restore/). +QuestDB Enterprise runs backups from its in-engine schedule. The operator +configures and observes it; there is no Kubernetes CronJob or force-backup +request. Status can lag the engine observation. See +[Backup and restore](/docs/enterprise-kubernetes-operator/operations/backup-restore/). -### An old PITR target silently restores the earliest backup +### The operator cannot pre-validate the PITR window -A target older than the retained window does not fail. QuestDB restores the earliest available backup and reports success. Confirm the retention window before restore and validate actual data afterward. See [Point-in-time recovery](/docs/enterprise-kubernetes-operator/operations/backup-restore/#point-in-time-recovery-pitr). +The operator does not read the object store, so it cannot pre-check whether a +PITR target is still retained. On supported QuestDB Enterprise 4.0.0, an +older-than-retained target fails startup and surfaces as +`RecoveryFailed=True/RestoreError`. Because `spec.bootstrap` is immutable, +delete the failed destination safely and create a fresh cluster with a valid +target. See +[Point-in-time recovery](/docs/enterprise-kubernetes-operator/operations/backup-restore/#point-in-time-recovery-pitr). ### Follower cutover cannot see WAL that was never uploaded -Migration gates observe the object store, not the source disk. A normal source shutdown can leave an invisible local tail. Run the required source `primary-catchup-uploads` completion step or accept that tail's loss with Emergency mode. +Migration gates observe the object store, not the source disk. A normal source +shutdown can leave an invisible local tail. Run the required source +`primary-catchup-uploads` completion step or accept that tail's loss with +Emergency mode. ### A quiet source is not proof of a stopped source -The follower gate can observe that transactions stopped advancing, but an idle source process can look the same. The engine safely refuses takeover while the source still owns the stream (`SourceStillOwnsStore`). Fully stop and decommission the source as described in [Migration](/docs/enterprise-kubernetes-operator/high-availability/#migrate-an-existing-questdb-onto-the-operator). +The follower gate can observe that transactions stopped advancing, but an idle +source process can look the same. The engine safely refuses takeover while the +source still owns the stream (`SourceStillOwnsStore`). Fully stop and +decommission the source as described in +[Migration](/docs/enterprise-kubernetes-operator/high-availability/#migrate-an-existing-questdb-onto-the-operator). ## Storage and networking -### Chart-managed verified controller metrics TLS is not operational +### PGWire TLS protects only PGWire -The default `ServiceMonitor` uses authenticated HTTPS with `insecureSkipVerify: true`, so it does not verify the controller's serving certificate. Setting `certmanager.enable=true` is not a working verified-metrics path: the generated certificate names do not match the ServiceMonitor server name, and the manager is not configured to serve metrics with the mounted certificate. +`spec.protocols.pgwire.tls` enables TLS for PGWire and the operator's own SQL +connections when selected at cluster creation. It does not add TLS for HTTP/Web +Console, minimal HTTP metrics, ILP, QWP, or client-certificate/mTLS +authentication. Existing plaintext clusters cannot enable PGWire TLS in place in +v0.2.1; create a new TLS-enabled cluster when that transport change is required. -Do not enable that value as a verified-metrics solution. Customers requiring certificate verification need a separately reviewed scraper/certificate integration with QuestDB support. See [Secure controller metrics](/docs/enterprise-kubernetes-operator/operations/operator/#secure-controller-metrics). +### Database Pod network isolation is opt-in + +The operator publishes ClusterIP Services for client ports, but direct Pod IP +access can still reach Pod-only 9003 unless your CNI policy blocks it. No tenant +database NetworkPolicy is installed automatically. Test any custom policy with +your CNI and kubelet probe behavior. ### The operator never cleans object storage -Deleting clusters, PVCs, or the operator never deletes backup or replication objects. Inventory and remove cloud objects separately under customer policy. The operator does not create, list, read, or delete the bucket/container. +Deleting clusters, PVCs, or the operator never deletes backup or replication +objects. Inventory and remove cloud objects separately under customer policy. +The operator does not create, list, read, or delete the bucket/container. ### Database Services are ClusterIP only -The operator reconciles `-rw` and `-ro` as ClusterIP and `` as headless. Use temporary port-forwarding or a separate customer-managed Ingress, Gateway, or LoadBalancer. Do not mutate operator-owned Service types. +The operator reconciles `-rw` and `-ro` as ClusterIP and `` as +headless. Use temporary port-forwarding or a separate customer-managed Ingress, +Gateway, or LoadBalancer. Do not mutate operator-owned Service types. ## Supported platforms and versions -| Platform | Tested Kubernetes | Tested QuestDB Enterprise | -| --- | --- | --- | -| Amazon EKS | 1.31–1.36 | 3.3.4 | -| Azure AKS | 1.33–1.36 | 3.3.4 | +| Platform | Tested Kubernetes | Tested QuestDB Enterprise | +| ---------- | ----------------- | ------------------------- | +| Amazon EKS | 1.31–1.36 | 4.0.0 | +| Azure AKS | 1.33–1.36 | 4.0.0 | -Other Kubernetes distributions, versions, CSI/fsGroup behavior, and QuestDB versions are untested. They are not blocked by admission. +Other Kubernetes distributions, versions, CSI/fsGroup behavior, and QuestDB +versions are untested. They are not blocked by admission. -Google Cloud Storage is rejected in this release. `QuestDBObjectStore.spec.provider` supports the schema value `GCS`, but admission rejects it because only S3 on EKS and Azure Blob on AKS have been validated. +Google Cloud Storage is rejected in this release. +`QuestDBObjectStore.spec.provider` supports the schema value `GCS`, but +admission rejects it because only S3 on EKS and Azure Blob on AKS have been +validated. ## API lifecycle -The API is `questdb.io/v1alpha1` and can change incompatibly between releases. Read release notes and migration requirements before upgrades or rollback. Only the latest release receives fixes; there are no backports. See [Operator upgrades](/docs/enterprise-kubernetes-operator/operations/operator/#upgrade-the-operator) and [Support](/docs/enterprise-kubernetes-operator/support/). +The API is `questdb.io/v1alpha1` and can change incompatibly between releases. +Read release notes and migration requirements before upgrades or rollback. Only +the latest release receives fixes; there are no backports. See +[Operator upgrades](/docs/enterprise-kubernetes-operator/operations/operator/#upgrade-the-operator) +and [Support](/docs/enterprise-kubernetes-operator/support/). diff --git a/documentation/enterprise-kubernetes-operator/operations/backup-restore.md b/documentation/enterprise-kubernetes-operator/operations/backup-restore.md index f3bcb194a..c0a4262e2 100644 --- a/documentation/enterprise-kubernetes-operator/operations/backup-restore.md +++ b/documentation/enterprise-kubernetes-operator/operations/backup-restore.md @@ -1,21 +1,32 @@ --- title: Backup and restore -description: Configure backups and restore Operator-managed QuestDB clusters, including point-in-time recovery. +description: + Configure backups and restore Operator-managed QuestDB clusters, including + point-in-time recovery. --- # Backup and restore -QuestDB Enterprise schedules and performs backups on the primary. The operator configures the engine and observes `backups()`; it does not upload data, create a Kubernetes CronJob, expose an on-demand backup API, or read the object store. Status can lag the database by roughly the manager's resync/observation interval (about two minutes). +QuestDB Enterprise schedules and performs backups on the primary. The operator +configures the engine and observes `backups()`; it does not upload data, create +a Kubernetes CronJob, expose an on-demand backup API, or read the object store. +Status can lag the database by roughly the manager's resync/observation interval +(about two minutes). -For more information about backup and restore mechanics, please refer to the [official documentation](/docs/operations/backup). +For more information about backup and restore mechanics, please refer to the +[official documentation](/docs/operations/backup). -Before running any command, replace every `` value; an unreplaced placeholder can be interpreted as shell redirection. +Before running any command, replace every `` value; an unreplaced +placeholder can be interpreted as shell redirection. ## Configure and verify backup ### Before you start -Create and validate a same-namespace `QuestDBObjectStore` as described in [Configuration](/docs/enterprise-kubernetes-operator/configuration/#object-storage). The object-store CR has no readiness status; proof comes from a consuming database. +Create and validate a same-namespace `QuestDBObjectStore` as described in +[Configuration](/docs/enterprise-kubernetes-operator/configuration/#object-storage). +The object-store CR has no readiness status; proof comes from a consuming +database. Use a prefix reserved for this cluster: @@ -32,11 +43,15 @@ spec: stalledAfterSeconds: 3600 ``` -Apply the cluster change. It can recreate the primary to install the new configuration, briefly interrupting writes. +Apply the cluster change. It can recreate the primary to install the new +configuration, briefly interrupting writes. ### Watch -Before the first run, `BackupHealthy` may be `Unknown` with reason `NoBackupYet`. The bounded deadline must exceed the schedule interval plus the roughly two-minute observation delay; this hourly example allows about 75 minutes: +Before the first run, `BackupHealthy` may be `Unknown` with reason +`NoBackupYet`. The bounded deadline must exceed the schedule interval plus the +roughly two-minute observation delay; this hourly example allows about 75 +minutes: ```sh for _ in $(seq 1 450); do @@ -54,26 +69,49 @@ kubectl get questdbcluster -n \ [ "$STATUS" = "completed" ] ``` -Verify both the first successful run and continued recent successful runs. A failed engine run has `lastBackup.status=failed` and `BackupHealthy=False/Failed`; inspect `.status.backup.lastBackup.error`, database logs, Secret metadata, cloud IAM, DNS, and network access. - -A stalled run is deliberately different: the engine status remains `in_progress`, while `BackupHealthy=False/Stalled` reports that `progressPercent` has not changed for the configured threshold. `.status.backup.lastProgressAt` is when the operator first observed the current run or most recently observed its percentage change. `stalledAfterSeconds` defaults to 3600 seconds. Setting it explicitly to `0` disables stall detection; it does not cancel the backup, mark it failed, or bound how long `in_progress` can remain. Diagnose and remediate the engine/store path rather than editing status. - -There is no safe "force backup" command. Temporarily shortening `spec.backup.schedule` is a schedule/configuration change, **not** an on-demand backup. It can recreate pods and briefly interrupt writes; if used for a controlled test, restore the production schedule afterward. +Verify both the first successful run and continued recent successful runs. A +failed engine run has `lastBackup.status=failed` and +`BackupHealthy=False/Failed`; inspect `.status.backup.lastBackup.error`, +database logs, Secret metadata, cloud IAM, DNS, and network access. + +A stalled run is deliberately different: the engine status remains +`in_progress`, while `BackupHealthy=False/Stalled` reports that +`progressPercent` has not changed for the configured threshold. +`.status.backup.lastProgressAt` is when the operator first observed the current +run or most recently observed its percentage change. `stalledAfterSeconds` +defaults to 3600 seconds. Setting it explicitly to `0` disables stall detection; +it does not cancel the backup, mark it failed, or bound how long `in_progress` +can remain. Diagnose and remediate the engine/store path rather than editing +status. + +There is no safe "force backup" command. Temporarily shortening +`spec.backup.schedule` is a schedule/configuration change, **not** an on-demand +backup. It can recreate pods and briefly interrupt writes; if used for a +controlled test, restore the production schedule afterward. ## Restore into a new cluster -Restore always creates a **new** `QuestDBCluster`. Its immutable `spec.bootstrap.recovery` makes the genesis primary start from backup or not start at all. +Restore always creates a **new** `QuestDBCluster`. Its immutable +`spec.bootstrap.recovery` makes the genesis primary start from backup or not +start at all. -The source `QuestDBObjectStore` must exist in the destination cluster's namespace. A missing source CR safely withholds genesis and retries. Invalid store configuration or an engine/runtime restore problem fails recovery instead of initializing an empty database. +The source `QuestDBObjectStore` must exist in the destination cluster's +namespace. A missing source CR safely withholds genesis and retries. Invalid +store configuration or an engine/runtime restore problem fails recovery instead +of initializing an empty database. ### Before you start - identify the source store and backup root; -- choose a destination name, PVCs, and backup/WAL prefixes that cannot collide with the source or another live cluster; -- for a multi-instance source, determine `sourceInstanceName`; +- choose a destination name, PVCs, and backup/WAL prefixes that cannot collide + with the source or another live cluster; +- for a multi-instance source, determine `sourceInstanceName` and copy it + exactly; it must match the lowercase hyphen-separated pattern + `^[a-z0-9]+(-[a-z0-9]+)*$`; - for PITR, confirm the retained time window before choosing a target. -For an operator-managed source, the current seed identity is normally published here: +For an operator-managed source, the current seed identity is normally published +here: ```sh kubectl get questdbcluster -n \ @@ -86,11 +124,18 @@ Confirm it against the source database when possible: SELECT backup_instance_name(); ``` -Set `spec.bootstrap.recovery.sourceInstanceName` when the source store holds backups from more than one instance. If omitted, the engine selects the only instance in a single-instance source and fails when selection is ambiguous. The operator cannot list the store for you. +Set `spec.bootstrap.recovery.sourceInstanceName` when the source store holds +backups from more than one instance. If omitted, the engine selects the only +instance in a single-instance source and fails when selection is ambiguous. The +operator cannot list the store for you. ### Change -Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`. Set `` to that private image and `` to the pull Secret in the destination namespace. Remove the entire `imagePullSecrets` block only when every destination node has ambient pull access, such as an authorized EKS worker-node role. +Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`. Set +`` to that private image and +`` to the pull Secret in the destination namespace. +Remove the entire `imagePullSecrets` block only when every destination node has +ambient pull access, such as an authorized EKS worker-node role. ```yaml apiVersion: questdb.io/v1alpha1 @@ -134,7 +179,10 @@ Apply the reviewed manifest: kubectl apply -f ``` -The destination backup and replication prefixes must be distinct from the source/live cluster's prefixes. When omitted, the destination replication root defaults to the identity-scoped `db///`; confirm that identity is unique. +The destination backup and replication prefixes must be distinct from the +source/live cluster's prefixes. When omitted, the destination replication root +defaults to the identity-scoped `db///`; confirm that +identity is unique. ### Watch and verify @@ -173,11 +221,17 @@ kubectl get questdbcluster -n \ [ "$WRITE_HEALTHY" = "True Healthy" ] ``` -`Recovered=True` means the engine reported restore completion; the other gates establish current ordinary writer readiness, including observed WAL write health. They still do not prove that you selected the intended data or guarantee free disk capacity. Before sending traffic, query the restored database and validate critical tables, minimum/maximum timestamps, expected row counts, application invariants, and storage headroom. +`Recovered=True` means the engine reported restore completion; the other gates +establish current ordinary writer readiness, including observed WAL write +health. They still do not prove that you selected the intended data or guarantee +free disk capacity. Before sending traffic, query the restored database and +validate critical tables, minimum/maximum timestamps, expected row counts, +application invariants, and storage headroom. ### If it fails -Read `Recovered`/`RecoveryFailed`, then inspect the genesis pod and its recovery init container: +Read `Recovered`/`RecoveryFailed`, then inspect the genesis pod and its recovery +init container: ```sh kubectl get pods -n -l questdb.io/cluster= -o wide @@ -186,9 +240,13 @@ kubectl logs -n -1 -c recovery-bootstrap --tail=500 kubectl logs -n -1 -c questdb --tail=500 ``` -Container names can be confirmed with `kubectl get pod -1 -n -o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'`. +Container names can be confirmed with +`kubectl get pod -1 -n -o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'`. -Because `spec.bootstrap` is immutable, correct a wrong source, identity, root, or target in a new cluster. A live failed destination CR continues reconciling and can race cleanup or recreate its pods, so remove it before touching retained storage: +Because `spec.bootstrap` is immutable, correct a wrong source, identity, root, +or target in a new cluster. A live failed destination CR continues reconciling +and can race cleanup or recreate its pods, so remove it before touching retained +storage: ```sh CLEANUP_READY=false @@ -206,10 +264,11 @@ fi kubectl get pvc -n -l questdb.io/cluster= -o wide ``` -Proceed only if the PVC inventory command ran successfully. Confirm the listed PVC belongs only to the failed destination and contains no needed data. +Proceed only if the PVC inventory command ran successfully. Confirm the listed +PVC belongs only to the failed destination and contains no needed data. -:::danger -Deleting the retained destination PVC permanently destroys its incomplete restore data. Delete only the PVC you have confirmed is disposable. +:::danger Deleting the retained destination PVC permanently destroys its +incomplete restore data. Delete only the PVC you have confirmed is disposable. ::: In the same shell, delete only the confirmed PVC: @@ -219,11 +278,18 @@ In the same shell, delete only the confirmed PVC: kubectl delete pvc -n --timeout=5m ``` -Apply the corrected restore with a **fresh cluster name**, which creates a distinct fresh PVC. Do not reuse `` or its old PVC. +Apply the corrected restore with a **fresh cluster name**, which creates a +distinct fresh PVC. Do not reuse `` or its old PVC. ## Point-in-time recovery (PITR) -PITR selects the newest retained backup at or before an RFC3339 target. Its granularity is the backup cadence, not continuous WAL time. +PITR selects the newest retained backup at or before an RFC3339 target. Its +granularity is the backup cadence, not continuous WAL time. The operator cannot +inspect the object store to pre-validate the recoverable window. + +Kubernetes accepts RFC3339 offsets and fractional seconds; the operator +normalizes the target to UTC `Z` and truncates fractional seconds to six digits +rather than rounding, preserving the at-or-before boundary. Add this immutable block when creating the restored cluster: @@ -240,8 +306,12 @@ spec: timestamp: "2026-06-30T14:00:00Z" ``` -:::danger -If the target is **older than the earliest retained backup**, QuestDB restores the earliest backup and reports success. Before applying the restore, confirm the source retention window and that the target is inside it. Afterward, verify actual table timestamps and row counts; `Recovered=True` alone cannot detect this mistake. -::: +:::warning If the target is older than the retained window, supported QuestDB +Enterprise 4.0.0 fails startup with no backup timestamp at or before the target, +and the operator reports `RecoveryFailed=True/RestoreError`. Because +`spec.bootstrap` is immutable, delete the failed destination safely and create a +fresh cluster with a valid target. ::: -Follow the same bounded watch, failure checks, and data validation as a normal restore. +Confirm the source retention window before creating the immutable destination, +then follow the same bounded watch, failure checks, and data validation as a +normal restore. diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index e227f856b..6de848695 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -1,15 +1,19 @@ --- title: Database operations -description: Operate, scale, resize, maintain, and delete Operator-managed QuestDB clusters. +description: + Operate, scale, resize, maintain, and delete Operator-managed QuestDB + clusters. --- # Database operations -Examples use `` and ``. Status is trustworthy only after the controller has observed the current spec. +Examples use `` and ``. Status is trustworthy only after the +controller has observed the current spec. ## Check database health -Check generation freshness first, then conditions. `status.phase` is only a human-readable summary. +Check generation freshness first, then conditions. `status.phase` is only a +human-readable summary. ```sh kubectl get questdbcluster -n \ @@ -18,9 +22,25 @@ kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{" message="}{.message}{"\n"}{end}' ``` -Do not act on old conditions while `.metadata.generation` differs from `.status.observedGeneration`. For an ordinary writable cluster, writer readiness requires all four signals: current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone can mean only that a replica or a read-serving primary is available. `WriteHealthy=False/PrimarySuspended` names the impaired WAL table or tables in its message; reads and writes to other tables can remain available. `WriteHealthy=True` reports the latest `wal_tables()` observation. It does not perform a synthetic write and does not guarantee free disk capacity, so retain application write probes and storage monitoring where those guarantees are required. - -An intentional replica-only follower is the exception: no primary is correct, so `WriteHealthy` is omitted rather than reported healthy. Require current generation, `phase=Following`, `status.replication.following=true`, the expected `readyInstances`, and an appropriate `ReplicationHealthy` follower result. `True/FollowingExternalSource` reports observed progress; a quiet source may report `Unknown/StreamNotDetermined`, which is acceptable only after confirming the source identity and roots. Treat `ReplicationHealthy=False` as unhealthy. +Do not act on old conditions while `.metadata.generation` differs from +`.status.observedGeneration`. For an ordinary writable cluster, writer readiness +requires all four signals: current generation, `Available=True/PrimaryReady`, +`Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` +alone can mean only that a replica or a read-serving primary is available. +`WriteHealthy=False/PrimarySuspended` names the impaired WAL table or tables in +its message; reads and writes to other tables can remain available. +`WriteHealthy=True` reports the latest `wal_tables()` observation. It does not +perform a synthetic write and does not guarantee free disk capacity, so retain +application write probes and storage monitoring where those guarantees are +required. + +An intentional replica-only follower is the exception: no primary is correct, so +`WriteHealthy` is omitted rather than reported healthy. Require current +generation, `phase=Following`, `status.replication.following=true`, the expected +`readyInstances`, and an appropriate `ReplicationHealthy` follower result. +`True/FollowingExternalSource` reports observed progress; a quiet source may +report `Unknown/StreamNotDetermined`, which is acceptable only after confirming +the source identity and roots. Treat `ReplicationHealthy=False` as unhealthy. ```sh kubectl get questdbcluster -n -o wide @@ -32,30 +52,38 @@ kubectl get pvc -n -l questdb.io/cluster= \ ## Services and ports -| Service | Purpose | -| --- | --- | -| `-rw` | Current primary. Use for writes and administration. | +| Service | Purpose | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-rw` | Current primary. Use for writes and administration. | | `-ro` | Created for store-backed replication. Routes to qualified replicas, but falls back to the primary when no replica qualifies. It is **not** strict replica isolation. | -| `` | Headless, internal identity/DNS service. It publishes unready addresses and must not be used as a client availability endpoint. | +| `` | Headless, internal identity/DNS service. It publishes unready addresses and must not be used as a client availability endpoint. | -| Port | Protocol | Published | -| --- | --- | --- | -| 9000 | HTTP and Web Console, including [QWP](/docs/configuration/qwp/) over WebSocket | Always | -| 8812 | PostgreSQL wire protocol | Always | -| 9009 | [InfluxDB Line Protocol](/docs/connect/compatibility/ilp/overview/) over TCP | Always | -| 9003 | Minimal HTTP server for health and database metrics | Always | -| 9007/UDP | [QWP UDP receiver](/docs/configuration/qwp/#qwpudpbindto) | Only when enabled | +| Port | Scope | +| -------- | ----------------------------------------------------------------------- | +| 9000/TCP | Published on database Services | +| 8812/TCP | Published on database Services; optional TLS chosen at cluster creation | +| 9009/TCP | Published on database Services | +| 9003/TCP | Always on Pods for kubelet/operator health and metrics; not on Services | +| 9007/UDP | Published on headless and `-rw` only when enabled | QWP over WebSocket has no port of its own. Ingestion (`/write/v4`) and streaming query results (`/read/v1`) are served by the HTTP server on 9000 and share its network settings, so they are available wherever 9000 is. +Existing Service-based database metrics scrapers must move to Pod discovery or a +`PodMonitor` that targets the named `metrics` port on QuestDB Pods. A Service +port-forward to 9003 no longer works. + The QWP UDP receiver is off by default. Enable it with -[`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#wire-protocols); +[`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#qwp-udp); until then, port 9007 is neither opened on the pod nor published on any Service. -It is the only port here that is not published on `-ro`. +It is not published on `-ro`. -The operator owns these Services and reconciles them as `ClusterIP` (the identity Service is headless). For temporary access, use port-forwarding. For durable external access, create a separate customer-managed Ingress, Gateway, or LoadBalancer that targets the operator Service. Do not mutate the operator-owned Service type. +The operator owns these Services and reconciles them as `ClusterIP` (the +identity Service is headless). For temporary access, use port-forwarding. For +durable external access, create a separate customer-managed Ingress, Gateway, or +LoadBalancer that targets the operator Service. Do not mutate the operator-owned +Service type. ## Connect @@ -68,7 +96,8 @@ kubectl get secret "$ADMIN_SECRET" -n \ -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp' ``` -When an administrator explicitly needs the credential, load it into the environment without echoing it and clear it after use: +When an administrator explicitly needs the credential, load it into the +environment without echoing it and clear it after use: ```sh ADMIN_PASSWORD="$(kubectl get secret "$ADMIN_SECRET" -n \ @@ -85,16 +114,45 @@ unset ADMIN_PASSWORD trap - EXIT ``` -Use `service/-ro` only for read-only traffic that can tolerate primary fallback. To access the Web Console temporarily, forward `service/-rw` from local port 9000 and stop the process after the session. +For a TLS-enabled cluster whose certificate uses the default Service identity, +keep the same port-forward but give `psql` both the certificate identity and the +local address: + +```bash +PGPASSWORD="$ADMIN_PASSWORD" psql \ + "host=-rw..svc hostaddr=127.0.0.1 port=8812 user=admin dbname=qdb sslmode=verify-full sslrootcert=/secure/path/ca.crt" +``` + +`host` supplies the certificate identity and SNI; `hostaddr` directs the +connection to the local tunnel. + +Use `service/-ro` only for read-only traffic that can tolerate primary +fallback. To access the Web Console temporarily, forward `service/-rw` +from local port 9000 and stop the process after the session. ## Make mutable changes safely -Changes to `spec.image`, `spec.imagePullSecrets`, `spec.resources`, `spec.config`, replication tuning, or scheduling recreate affected pods. A rolling change remains single-writer-safe, but recreating the primary can briefly interrupt writes. +Changes to `spec.image`, `spec.imagePullSecrets`, `spec.resources`, +`spec.config`, replication tuning, protocol fields within their allowed mode, or +scheduling recreate affected pods. Ordinary rollout is serialized: replicas roll +in serial order before the primary, and at most one ordinary delete is attempted +per instanceset reconcile. The operator may hold a rollout for exact topology, +usable PVCs, healthy peers, node readiness, or read-route safety. A rolling +change remains single-writer-safe, but recreating the primary can briefly +interrupt writes; a singleton has read/write downtime while its only Pod +restarts. Before deleting the only `ro-ready` replica, the operator first adds +the primary as an overlapping Service-selector candidate. That overlap is not an +EndpointSlice acknowledgement, connection drain, or zero-gap guarantee. ### Before you start -1. For an ordinary writable cluster, confirm current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone is not writer readiness. -2. For an intentional follower, instead confirm current generation, `phase=Following`, `status.replication.following=true`, the expected ready count, and an appropriate non-failing follower `ReplicationHealthy` result as described above. +1. For an ordinary writable cluster, confirm current generation, + `Available=True/PrimaryReady`, `Progressing=False/Settled`, and + `WriteHealthy=True/Healthy`. `Available=True` alone is not writer readiness. +2. For an intentional follower, instead confirm current generation, + `phase=Following`, `status.replication.following=true`, the expected ready + count, and an appropriate non-failing follower `ReplicationHealthy` result as + described above. 3. Save the current spec. 4. Make one logical change at a time. @@ -105,7 +163,8 @@ kubectl get questdbcluster -n -o yaml \ ### Change and watch -Apply a reviewed manifest or a narrow patch. This example changes an engine setting: +Apply a reviewed manifest or a narrow patch. This example changes an engine +setting: ```sh kubectl patch questdbcluster -n --type merge \ @@ -124,11 +183,34 @@ kubectl wait questdbcluster/ -n \ --for=condition=Available --timeout=5m ``` -The `kubectl wait` above is only an availability gate. Before declaring the writer ready, re-read the current-generation conditions and require `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. Then verify pod UIDs/restarts, connectivity, and the intended setting. If the change fails, inspect `ConfigRejected`, events, and pod logs. Recover by reverting the mutable spec to the saved value and repeat the bounded generation and full writer-health checks. Do not try to revert immutable fields; create a new cluster when required by the [API Reference](/docs/enterprise-kubernetes-operator/reference/api/). +The `kubectl wait` above is only an availability gate. Before declaring the +writer ready, re-read the current-generation conditions and require +`Available=True/PrimaryReady`, `Progressing=False/Settled`, and +`WriteHealthy=True/Healthy`. Then verify pod UIDs/restarts, connectivity, and +the intended setting. `Progressing=True/RollingUpdate` can mean the controller +is safely waiting on topology, PVC, peer, Node, or read-routing conditions. If +the change fails, inspect `ConfigRejected`, events, and pod logs. Recover by +reverting the mutable spec to the saved value and repeat the bounded generation +and full writer-health checks. Do not try to revert immutable fields; create a +new cluster when required by the +[API Reference](/docs/enterprise-kubernetes-operator/reference/api/). + +### Rotate static object-store credentials safely + +1. Make old and new provider credentials valid concurrently. +2. Update the same-namespace Secret referenced by `QuestDBObjectStore`. +3. Allow the manager's roughly two-minute resync plus serialized Pod restart to + converge. +4. Verify expected Pod UID changes and `BackupHealthy`/`ReplicationHealthy`. +5. Revoke the old credentials only after convergence. + +Immediate provider-side revocation can interrupt in-flight QuestDB object-store +I/O. ## Scale out -A replica is born from a completed backup and then consumes object-store WAL. Before increasing `spec.instances`, the cluster needs: +A replica is born from a completed backup and then consumes object-store WAL. +Before increasing `spec.instances`, the cluster needs: - `spec.objectStoreRef`; - enabled scheduled backup; and @@ -162,20 +244,29 @@ kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}' ``` -Verify `readyInstances` equals the requested count, `ReplicationHealthy=True`, and each current replica is healthy. `caughtUp` means it has caught up at least once; it is a latch. `caughtUpNow` is the live freshness reading. An absent `caughtUpNow` means freshness was not determined and must fail closed. +Verify `readyInstances` equals the requested count, `ReplicationHealthy=True`, +and each current replica is healthy. `caughtUp` means it has caught up at least +once; it is a latch. `caughtUpNow` is the live freshness reading. An absent +`caughtUpNow` means freshness was not determined and must fail closed. ## Scale in -Lowering `spec.instances` removes replicas. Instance serials are monotonic and are never reused, so names may not remain contiguous after scale or promotion. +Lowering `spec.instances` removes replicas. Instance serials are monotonic and +are never reused, so names may not remain contiguous after scale or promotion. -| `spec.pvcRetentionPolicy` | Removed replica PVC | -| --- | --- | -| `Retain` (default) | Kept for deliberate inspection or cleanup. | -| `Delete` | Deleted during replica scale-in. A later replica gets a new serial and seeds again. | +| `spec.pvcRetentionPolicy` | Removed replica PVC | +| ------------------------- | ----------------------------------------------------------------------------------- | +| `Retain` (default) | Kept for deliberate inspection or cleanup. | +| `Delete` | Deleted during replica scale-in. A later replica gets a new serial and seeds again. | -The policy applies only to replicas removed by scale-in. It never deletes the current primary PVC and does not govern whole-cluster deletion. On every managed PVC, `questdb.io/role` is the instance's **current** role and changes across promotion; `questdb.io/bootstrap` records how the volume was born and remains ancestry/state. Do not infer the current primary from `bootstrap`. +The policy applies only to replicas removed by scale-in. It never deletes the +current primary PVC and does not govern whole-cluster deletion. On every managed +PVC, `questdb.io/role` is the instance's **current** role and changes across +promotion; `questdb.io/bootstrap` records how the volume was born and remains +ancestry/state. Do not infer the current primary from `bootstrap`. -Before scaling in, identify the primary and replicas, choose the retention policy, and inventory PVCs: +Before scaling in, identify the primary and replicas, choose the retention +policy, and inventory PVCs: ```sh kubectl get questdbcluster -n -o wide @@ -208,7 +299,10 @@ else fi ``` -Verify the managed and ready counts, current primary, the full ordinary writer-health or separate follower contract, and the expected retained or deleted replica PVCs. Confirm each current primary/replica PVC's `questdb.io/role` label matches that role. +Verify the managed and ready counts, current primary, the full ordinary +writer-health or separate follower contract, and the expected retained or +deleted replica PVCs. Confirm each current primary/replica PVC's +`questdb.io/role` label matches that role. ## Grow storage @@ -224,7 +318,8 @@ kubectl get storageclass "$STORAGE_CLASS" \ kubectl get pvc -n -l questdb.io/cluster= ``` -Proceed only when `allowVolumeExpansion=true` and the CSI driver supports the requested expansion. +Proceed only when `allowVolumeExpansion=true` and the CSI driver supports the +requested expansion. ### Change and verify @@ -258,17 +353,25 @@ else fi ``` -The availability wait is not proof of writer health. Require the current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy` conditions before closing the change. If `StorageResizeBlocked=True`, read its reason/message and fix the StorageClass or CSI limitation. A size reduction is rejected. To change StorageClass, restore into a new cluster. +The availability wait is not proof of writer health. Require the +current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`, +and `WriteHealthy=True/Healthy` conditions before closing the change. If +`StorageResizeBlocked=True`, read its reason/message and fix the StorageClass or +CSI limitation. A size reduction is rejected. To change StorageClass, restore +into a new cluster. ## Node maintenance and disruption budgets -The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and `instances-1` for a replicated cluster. A singleton therefore blocks voluntary eviction and can make `kubectl drain` wait indefinitely. +The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and +`instances-1` for a replicated cluster. A singleton therefore blocks voluntary +eviction and can make `kubectl drain` wait indefinitely. Before planned maintenance, choose one safe option: 1. scale out and wait for a healthy, current replica; 2. lower `spec.scheduling.podDisruptionBudget.minAvailable`; or -3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting database downtime. +3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting + database downtime. ```yaml spec: @@ -278,17 +381,28 @@ spec: minAvailable: 0 ``` -Restore the normal PDB after maintenance and verify the current-generation writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. +Restore the normal PDB after maintenance and verify the current-generation +writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and +`WriteHealthy=True/Healthy`. -A PDB protects only voluntary disruption. On node loss, the operator reports `InstanceUnreachable` and does not automatically promote a replica. It avoids recreating a pod while the old pod may still run on an unreachable node. Restore the node/volume or follow the explicit [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) procedure. +A PDB protects only voluntary disruption. On node loss, the operator reports +`InstanceUnreachable` and does not automatically promote a replica. It avoids +recreating a pod while the old pod may still run on an unreachable node. Restore +the node/volume or follow the explicit +[promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) +procedure. ## Delete a database cluster -Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, and related resources. Its data PVCs are deliberately unowned and remain, and object-store backup/WAL data also remains. `pvcRetentionPolicy` does not change this whole-cluster behavior. +Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, +and related resources. Its data PVCs are deliberately unowned and remain, and +object-store backup/WAL data also remains. `pvcRetentionPolicy` does not change +this whole-cluster behavior. ### Before you start -Inventory and export the CR, PVCs, PV reclaim policies, store reference, and effective object prefixes: +Inventory and export the CR, PVCs, PV reclaim policies, store reference, and +effective object prefixes: ```sh kubectl get questdbcluster -n -o yaml \ @@ -297,13 +411,16 @@ kubectl get pvc -n -l questdb.io/cluster= -o wide kubectl get questdbobjectstore -n ``` -:::danger -Deleting the custom resource stops the database and removes its managed runtime objects. Confirm applications are stopped and that the CR export and data-retention inventory are complete. -::: +:::danger Deleting the custom resource stops the database and removes its +managed runtime objects. Confirm applications are stopped and that the CR export +and data-retention inventory are complete. ::: ```sh kubectl delete questdbcluster -n --timeout=5m kubectl get pvc -n -l questdb.io/cluster= ``` -Decide separately whether to retain or delete each PVC/PV according to its StorageClass reclaim policy. Decide separately whether to retain or delete cloud objects using customer-owned cloud tooling and policy. The operator never cleans the object store. +Decide separately whether to retain or delete each PVC/PV according to its +StorageClass reclaim policy. Decide separately whether to retain or delete cloud +objects using customer-owned cloud tooling and policy. The operator never cleans +the object store. diff --git a/documentation/enterprise-kubernetes-operator/operations/operator.md b/documentation/enterprise-kubernetes-operator/operations/operator.md index c0d31ec70..da50d84f2 100644 --- a/documentation/enterprise-kubernetes-operator/operations/operator.md +++ b/documentation/enterprise-kubernetes-operator/operations/operator.md @@ -1,13 +1,17 @@ --- title: Operator operations -description: Monitor, upgrade, roll back, and safely remove the Kubernetes Operator. +description: + Monitor, upgrade, roll back, and safely remove the Kubernetes Operator. --- # Operator operations -The operator is cluster-scoped and normally runs in `questdb-operator-system`. An operator outage does not stop existing QuestDB pods, but it stops reconciliation and failover workflows. +The operator is cluster-scoped and normally runs in `questdb-operator-system`. +An operator outage does not stop existing QuestDB pods, but it stops +reconciliation and failover workflows. -Before running any command, replace every `` value; an unreplaced placeholder can be interpreted as shell redirection. +Before running any command, replace every `` value; an unreplaced +placeholder can be interpreted as shell redirection. ## Check the manager @@ -23,7 +27,8 @@ helm list -n questdb-operator-system helm status questdb-operator -n questdb-operator-system ``` -The manager exposes `/healthz` for liveness and `/readyz` for readiness on port 8081 inside its pod. Confirm the configured probes and recent results: +The manager exposes `/healthz` for liveness and `/readyz` for readiness on port +8081 inside its pod. Confirm the configured probes and recent results: ```sh kubectl describe deployment/questdb-operator-controller-manager \ @@ -44,14 +49,19 @@ kubectl get events -n questdb-operator-system \ ## Secure controller metrics -The chart's metrics endpoint is authenticated HTTPS on TCP 8443. It is not an unauthenticated HTTP endpoint. A scraper needs a Kubernetes service-account token and permission to read `/metrics` through the chart-created `questdb-operator-metrics-reader` ClusterRole. +The chart's metrics endpoint is authenticated HTTPS on TCP 8443. It is not an +unauthenticated HTTP endpoint. A scraper needs a Kubernetes service-account +token and permission to read `/metrics` through the chart-created +`questdb-operator-metrics-reader` ClusterRole. Before enabling the chart's `ServiceMonitor`: 1. Install a compatible Prometheus Operator and its `ServiceMonitor` CRD. 2. Bind the scraper ServiceAccount to `questdb-operator-metrics-reader`. -3. Accept the default lack of certificate verification, or have QuestDB review a separate certificate/scraper integration. -4. If chart NetworkPolicies are enabled, label the **scraper's namespace** `metrics=enabled`. +3. If chart NetworkPolicies are enabled, label the **scraper's namespace** + `metrics=enabled`. +4. For verified metrics TLS, install cert-manager first and enable both + `certmanager.enable=true` and `prometheus.enable=true`. Example RBAC and namespace preparation: @@ -62,19 +72,35 @@ kubectl create clusterrolebinding questdb-operator-prometheus-metrics \ kubectl label namespace metrics=enabled --overwrite ``` -With `prometheus.enable=true`, the default ServiceMonitor uses `insecureSkipVerify: true`. Traffic is encrypted and authenticated, but the scraper does not verify the serving certificate. +With `prometheus.enable=true` alone, metrics traffic is encrypted and +authenticated but the generated ServiceMonitor uses `insecureSkipVerify: true`. +With both `certmanager.enable=true` and `prometheus.enable=true`, the chart +issues and mounts a metrics serving certificate and renders a ServiceMonitor +that verifies the Service DNS name. -Chart-managed verified metrics TLS is [not operational](/docs/enterprise-kubernetes-operator/known-limitations/#chart-managed-verified-controller-metrics-tls-is-not-operational). Do **not** enable `certmanager.enable=true` as a verified-metrics solution. Customers requiring certificate verification should use a separately reviewed scraper/certificate integration with QuestDB support. - -Do not enable `prometheus.enable` until the ServiceMonitor CRD exists. The chart does not create Prometheus, a scraper ServiceAccount, dashboards, or alerts. +Do not enable `prometheus.enable` until the ServiceMonitor CRD exists. The chart +does not create Prometheus, a scraper ServiceAccount, dashboards, or alerts, and +Prometheus must select the chart's ServiceMonitor labels. ## Upgrade the operator -Only the latest release is supported. Read its release notes before changing the controller or CRDs; `questdb.io/v1alpha1` may have breaking changes. +Only the latest release is supported. Read its release notes before changing the +controller or CRDs; `questdb.io/v1alpha1` may have breaking changes. + +For v0.2.0 to v0.2.1, inventory any database metrics consumers that scrape +Service port 9003 and migrate them to Pod discovery. Validate custom +`spec.config` and `spec.replication.config` key names against +`^[A-Za-z0-9._-]+$`, and validate recovery/follower source instance names +against `^[a-z0-9]+(-[a-z0-9]+)*$`. Review the newly functional metrics TLS +rendering if `certmanager.enable=true` is used. Existing plaintext clusters +cannot enable PGWire TLS in place in v0.2.1; TLS must be selected when creating +a new cluster. Ensure QuestDB Enterprise 4.0.0 image entitlement and pull access +are ready. ### Before you start -Check every cluster has fresh status and inspect the condition status and reason: +Check every cluster has fresh status and inspect the condition status and +reason: ```sh kubectl get questdbclusters -A \ @@ -83,21 +109,43 @@ kubectl get pvc -A -l questdb.io/cluster \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,ROLE:.metadata.labels.questdb\.io/role,BOOTSTRAP:.metadata.labels.questdb\.io/bootstrap,DELETING:.metadata.deletionTimestamp' ``` -For an ordinary writable cluster, do not proceed until `generation` equals `observed`, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` does not by itself prove that the writer or every WAL table accepts writes. `WriteHealthy=True` is an engine observation, not a synthetic write or a free-disk guarantee. - -An intentional replica-only follower is the exception: it correctly has no primary and omits `WriteHealthy`. Require current generation, `phase=Following`, `following=true`, the expected `readyInstances`, and an appropriate follower `ReplicationHealthy` result. This is normally `True/FollowingExternalSource` when lag is observable; a quiet source can report `Unknown/StreamNotDetermined`, which is acceptable only after confirming the source identity and roots. Do not proceed on `ReplicationHealthy=False`. - -Do not skip the PVC deletion-timestamp inventory. On the established replicated/object-store-backed path, a pre-existing Terminating primary PVC can be held by a same-name primary Pod recreated by an older controller. Do not wait forever for that claim: current versions fence that established primary Pod, release `pvc-protection`, keep the RW Service without ready endpoints, and require explicit storage recovery or Emergency promotion instead of creating blank primary storage. Record the affected cluster, stop/repoint writers, preserve events and PVC/PV identity, and plan that outage/failover before upgrading the controller. A standalone cluster does not have this replicated-primary loss guard or a replica to promote; PVC loss can recreate it on fresh empty storage, so treat standalone storage loss as data loss/recovery and restore from backup rather than waiting for `PromotionRequired`. +For an ordinary writable cluster, do not proceed until `generation` equals +`observed`, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and +`WriteHealthy=True/Healthy`. `Available=True` does not by itself prove that the +writer or every WAL table accepts writes. `WriteHealthy=True` is an engine +observation, not a synthetic write or a free-disk guarantee. + +An intentional replica-only follower is the exception: it correctly has no +primary and omits `WriteHealthy`. Require current generation, `phase=Following`, +`following=true`, the expected `readyInstances`, and an appropriate follower +`ReplicationHealthy` result. This is normally `True/FollowingExternalSource` +when lag is observable; a quiet source can report `Unknown/StreamNotDetermined`, +which is acceptable only after confirming the source identity and roots. Do not +proceed on `ReplicationHealthy=False`. + +Do not skip the PVC deletion-timestamp inventory. On the established +replicated/object-store-backed path, a pre-existing Terminating primary PVC can +be held by a same-name primary Pod recreated by an older controller. Do not wait +forever for that claim: current versions fence that established primary Pod, +release `pvc-protection`, keep the RW Service without ready endpoints, and +require explicit storage recovery or Emergency promotion instead of creating +blank primary storage. Record the affected cluster, stop/repoint writers, +preserve events and PVC/PV identity, and plan that outage/failover before +upgrading the controller. A standalone cluster does not have this +replicated-primary loss guard or a replica to promote; PVC loss can recreate it +on fresh empty storage, so treat standalone storage loss as data loss/recovery +and restore from backup rather than waiting for `PromotionRequired`. Also: -- confirm a recent successful backup for every protected cluster; -- read release notes and API compatibility/migration instructions; -- export the current Helm values and namespaced custom resources to a protected location; -- record database pod UIDs and restart counts; -- render and review the new chart before applying it. +- confirm a recent successful backup for every protected cluster +- read release notes and API compatibility/migration instructions +- save Helm values and manifests +- record database pod UIDs and restart counts +- render and review the new chart before applying it -Run the remaining upgrade commands in the same shell so they share the protected workspace: +Run the remaining upgrade commands in the same shell so they share the protected +workspace: ```sh UPGRADE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/questdb-operator-upgrade.XXXXXX")" @@ -121,11 +169,15 @@ helm template questdb-operator oci://ghcr.io/questdb/charts/questdb-operator \ diff -u "$UPGRADE_DIR/current.yaml" "$UPGRADE_DIR/proposed.yaml" || true ``` -Review the diff, especially CRDs, manager arguments, RBAC, webhook configuration, image repository, and image-pull Secret names. +Review the diff, especially CRDs, manager arguments, RBAC, webhook +configuration, image repository, and image-pull Secret names. ### Change -Use the same saved user-values file that produced the reviewed render. The new chart supplies its new defaults, while this file reapplies the customer's overrides, including the operator image repository and `controllerManager.imagePullSecrets`. +Use the same saved user-values file that produced the reviewed render. The new +chart supplies its new defaults, while this file reapplies the customer's +overrides, including the operator image repository and +`controllerManager.imagePullSecrets`. ```sh helm upgrade questdb-operator oci://ghcr.io/questdb/charts/questdb-operator \ @@ -149,7 +201,13 @@ diff -u "$UPGRADE_DIR/database-pods-before.txt" \ "$UPGRADE_DIR/database-pods-after.txt" ``` -An operator-only upgrade should not roll database pods except where the new controller must fence an established replicated/object-store-backed primary already found on a missing or Terminating PVC. Investigate every changed UID or restart count, re-run the PVC deletion-timestamp inventory, and require the full writer-health or separate follower contract before declaring success. After the upgrade is verified and any required evidence is transferred according to policy, remove the local workspace: +An operator-only upgrade should not roll database pods except where the new +controller must fence an established replicated/object-store-backed primary +already found on a missing or Terminating PVC. Investigate every changed UID or +restart count, re-run the PVC deletion-timestamp inventory, and require the full +writer-health or separate follower contract before declaring success. After the +upgrade is verified and any required evidence is transferred according to +policy, remove the local workspace: ```sh rm -rf -- "$UPGRADE_DIR" @@ -167,12 +225,12 @@ kubectl logs -n questdb-operator-system \ deployment/questdb-operator-controller-manager -c manager --tail=1000 ``` -| Situation | Action | -| --- | --- | -| The new manager never became ready and release notes confirm API compatibility | Consider `helm rollback` to the last known-good revision. | -| The manager is ready but a cluster is unhealthy | Diagnose the cluster first; controller rollback may not repair database or spec state. | -| The release changed a CRD schema or required object migration | Follow the release-specific recovery procedure or contact support. Do not blindly roll back. | -| Database pods or data changed | Stop and assess the database. A Helm rollback is not a data rollback. | +| Situation | Action | +| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | +| The new manager never became ready and release notes confirm API compatibility | Consider `helm rollback` to the last known-good revision. | +| The manager is ready but a cluster is unhealthy | Diagnose the cluster first; controller rollback may not repair database or spec state. | +| The release changed a CRD schema or required object migration | Follow the release-specific recovery procedure or contact support. Do not blindly roll back. | +| Database pods or data changed | Stop and assess the database. A Helm rollback is not a data rollback. | ```sh helm rollback questdb-operator \ @@ -181,27 +239,35 @@ kubectl rollout status deployment/questdb-operator-controller-manager \ -n questdb-operator-system --timeout=5m ``` -:::warning -Rolling back the Helm release or controller does **not** reverse CRD schemas already sent to the API server, mutations to custom resources, or database state. Never blindly cross a breaking schema change. -::: +:::warning Rolling back the Helm release or controller does **not** reverse CRD +schemas already sent to the API server, mutations to custom resources, or +database state. Never blindly cross a breaking schema change. ::: ## Uninstall or remove the operator -Choose one of these paths. Do not uninstall the controller first when permanent cleanup is intended: active `QuestDBPromotion` finalizers need a compatible running operator to finish or resolve their cutovers. +Choose one of these paths. Do not uninstall the controller first when permanent +cleanup is intended: active `QuestDBPromotion` finalizers need a compatible +running operator to finish or resolve their cutovers. ### A. Temporarily remove the operator and leave databases unmanaged -A Helm uninstall removes the controller but, with the default `crd.keep=true`, retains all three CRDs and their custom resources: +A Helm uninstall removes the controller but, with the default `crd.keep=true`, +retains all three CRDs and their custom resources: ```sh helm uninstall questdb-operator -n questdb-operator-system --wait --timeout=5m ``` -Existing database pods continue running, but they are **unmanaged**: no reconciliation, promotion/failover workflow, certificate or Secret convergence, or configuration convergence occurs. Reinstall a compatible operator promptly if the databases are to remain in service. +Existing database pods continue running, but they are **unmanaged**: no +reconciliation, promotion/failover workflow, certificate or Secret convergence, +or configuration convergence occurs. Reinstall a compatible operator promptly if +the databases are to remain in service. ### B. Permanently remove all managed resources -Keep a compatible operator running throughout the tenant cleanup. First inventory and export the resources, PVCs, and store locations to a protected path: +Keep a compatible operator running throughout the tenant cleanup. First +inventory and export the resources, PVCs, and store locations to a protected +path: ```sh kubectl get questdbclusters,questdbobjectstores,questdbpromotions -A @@ -210,9 +276,14 @@ kubectl get questdbclusters,questdbobjectstores,questdbpromotions -A -o yaml \ > /secure/path/questdb-custom-resources.yaml ``` -Record every object-store bucket/container and effective backup and replication prefix; the operator never deletes those objects. Stop all applications and other clients that can write to or read from the databases. +Record every object-store bucket/container and effective backup and replication +prefix; the operator never deletes those objects. Stop all applications and +other clients that can write to or read from the databases. -In each namespace, delete or resolve `QuestDBPromotion` objects first. A promotion already in `Draining` or `Promoting` keeps its finalizer while the compatible operator completes the shaped cutover; wait until every promotion is gone before proceeding: +In each namespace, delete or resolve `QuestDBPromotion` objects first. A +promotion already in `Draining` or `Promoting` keeps its finalizer while the +compatible operator completes the shaped cutover; wait until every promotion is +gone before proceeding: ```sh kubectl get questdbpromotions -n @@ -222,7 +293,11 @@ kubectl wait --for=delete questdbpromotion/ \ kubectl get questdbpromotions -n ``` -Then delete each `QuestDBCluster` and verify its pods are gone and its retained PVCs match the intended retention decision. See [Delete a database cluster](/docs/enterprise-kubernetes-operator/operations/database/#delete-a-database-cluster). Stop on any deletion or verification failure; do not inspect or act on PVCs afterward. +Then delete each `QuestDBCluster` and verify its pods are gone and its retained +PVCs match the intended retention decision. See +[Delete a database cluster](/docs/enterprise-kubernetes-operator/operations/database/#delete-a-database-cluster). +Stop on any deletion or verification failure; do not inspect or act on PVCs +afterward. ```sh CLUSTER_DELETE_ACCEPTED=false @@ -248,16 +323,18 @@ fi kubectl get pvc -n -l questdb.io/cluster= -o wide ``` -After every cluster in the namespace is gone, delete its `QuestDBObjectStore` configuration objects. Repeat for every namespace, then uninstall the operator: +After every cluster in the namespace is gone, delete its `QuestDBObjectStore` +configuration objects. Repeat for every namespace, then uninstall the operator: ```sh kubectl delete questdbobjectstore -n --timeout=5m helm uninstall questdb-operator -n questdb-operator-system --wait --timeout=5m ``` -:::danger -Deleting a CRD deletes **every** remaining custom resource of that kind in every namespace. Only after the permanent cleanup above, and only with explicit acceptance of the service interruption and potential data/control-plane loss, verify PVC/object-store retention and remove the retained CRDs: -::: +:::danger Deleting a CRD deletes **every** remaining custom resource of that +kind in every namespace. Only after the permanent cleanup above, and only with +explicit acceptance of the service interruption and potential data/control-plane +loss, verify PVC/object-store retention and remove the retained CRDs: ::: ```sh kubectl delete crd questdbclusters.questdb.io \ diff --git a/documentation/enterprise-kubernetes-operator/troubleshooting.md b/documentation/enterprise-kubernetes-operator/troubleshooting.md index 83b0e4684..b36bd5818 100644 --- a/documentation/enterprise-kubernetes-operator/troubleshooting.md +++ b/documentation/enterprise-kubernetes-operator/troubleshooting.md @@ -1,21 +1,27 @@ --- title: Troubleshoot the Kubernetes Operator -description: Diagnose Operator and QuestDB cluster conditions, events, logs, and object-store configuration. +description: + Diagnose Operator and QuestDB cluster conditions, events, logs, and + object-store configuration. --- # Troubleshooting -Diagnose from the API outward. Avoid deleting pods, PVCs, or promotion objects until status and events have been captured. +Diagnose from the API outward. Avoid deleting pods, PVCs, or promotion objects +until status and events have been captured. -Before running any command, replace every `` value; an unreplaced placeholder can be interpreted as shell redirection. +Before running any command, replace every `` value; an unreplaced +placeholder can be interpreted as shell redirection. ## First response 1. **Generation:** confirm `.metadata.generation == .status.observedGeneration`. 2. **Conditions:** read every status, reason, message, and condition generation. 3. **Events:** inspect the cluster and namespace event timeline. -4. **Kubernetes objects:** inspect pods, PVCs, Services, endpoints, nodes, and scheduling. -5. **Logs:** then read operator logs and the relevant QuestDB/init-container logs. +4. **Kubernetes objects:** inspect pods, PVCs, Services, endpoints, nodes, and + scheduling. +5. **Logs:** then read operator logs and the relevant QuestDB/init-container + logs. ```sh kubectl get questdbcluster -n \ @@ -31,40 +37,53 @@ kubectl get endpointslice -n \ -l kubernetes.io/service-name=-ro ``` -If status is stale, check manager readiness, logs, RBAC, webhook/API connectivity, and events in `questdb-operator-system` before diagnosing an old condition. +If status is stale, check manager readiness, logs, RBAC, webhook/API +connectivity, and events in `questdb-operator-system` before diagnosing an old +condition. ## Decision table -| Signal or symptom | Meaning | Next checks | -| --- | --- | --- | -| `ConfigRejected=True` | An engine setting is invalid, injection-prone, or operator-owned. | Read reason/message; remove the rejected key from `spec.config` or `spec.replication.config`. Compare with [Configuration](/docs/enterprise-kubernetes-operator/configuration/#extra-engine-options). | -| `OperatorIdentityReady=False` | The operator could not provision/converge its least-privilege `questdb_operator` **SQL service account** from the admin identity. It is not cloud or object-store identity. | Check QuestDB pgwire reachability, admin Secret name/keys/permissions, ACL settings, and the condition reason. `WaitingForInstance` and brief `QuestDBUnreachable` can be boot states; persistent `AdminInsufficient` needs credential/privilege correction. | -| `WriteHealthy=False/PrimaryNotReady` | The primary is not Ready, so WAL write health cannot be established. | Diagnose primary Pod readiness, scheduling, mounts, probes, resources, and logs. Do not infer healthy writes from a read-serving replica or `Available=True`. | -| `WriteHealthy=False/PrimarySuspended` | One or more named primary WAL tables are write-impaired; reads and other tables can remain available. | Read the condition message for every table; inspect `wal_tables()`, QuestDB logs, table/disk state, and application errors. `Available=True` does not override this failure. | -| `WriteHealthy=Unknown` | Operator identity is not ready, or the latest `wal_tables()` query/observation was unavailable or failed. | Read reason/message; check operator SQL identity, pgwire reachability, and logs. Do not infer healthy writes from `Available=True`. | -| `BackupHealthy=Unknown` | No observed backup yet, an engine run is still progressing within its threshold, or configuration cannot be observed. | Confirm schedule, `status.backup.configured`, `lastBackup.status`, `lastProgressAt`, manager observation delay, store config, and primary logs. | -| `BackupHealthy=False/Failed` | Latest observed engine backup run failed. | Read `.status.backup.lastBackup.error`; check pod identity/static Secret, prefix permissions, DNS/network, and provider service health. | -| `BackupHealthy=False/Stalled` | The engine run remains `in_progress`, but `progressPercent` has not changed since `lastProgressAt` for `stalledAfterSeconds`. | Inspect `lastBackup`, `lastProgressAt`, database logs, IAM/Secret, network, and object-store health. Default threshold is 3600 seconds; explicit `0` disables detection and does not cancel the run. | -| `ReplicationHealthy=False` or `Unknown` | Replicas are suspended, behind, unreachable, unseeded, or freshness was not determined. | Inspect `.status.replication.seed`, `replicas[]`, `caughtUpNow`, `lagTxns`, and `suspendedTables`; then database metrics/logs and object-store access. Absent freshness is not zero lag. | -| `InstanceUnreachable=True` | Operator cannot safely observe an instance, or its node is unreachable. | Inspect pod readiness, node Ready condition, pgwire/metrics network paths, probes, CPU/memory pressure, and logs. Do not force-create another primary. | -| `PromotionRequired=True` | Established primary PVC is missing/Terminating; the operator fences its Pod, refuses an empty replacement, preserves `currentPrimary`, and does not auto-promote. | Confirm PVC/PV/cloud-disk and zero ready RW endpoints. Restore the volume or select a replica and follow [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover), accepting emergency loss when a drain is impossible. | -| `Recovered=True` | Engine reported restore completion. | Still validate actual tables, timestamps, and row counts before traffic. | -| `RecoveryFailed=True` | Restore init/runtime failed or was incomplete. | Read reason/message and genesis init logs; verify source store/root/instance/target. `spec.bootstrap` is immutable, so correct it in a fresh destination. | -| `StorageResizeBlocked=True` | StorageClass/CSI expansion is unavailable. | Confirm `allowVolumeExpansion`, driver support, PVC events, and requested size. Never shrink; restore to a new class if needed. | -| Promotion `Pending`/`Validating` | Request awaits acceptance, catch-up, or drain prerequisites. | Read `.status.reason/message`, active promotion, target live status, and `caughtUpNow`. A request left Pending for 10 minutes becomes `StaleRequest`. | -| Promotion `Draining` | Old primary is stopping writes and uploading its tail, or target is replaying it. | Check old-primary and target logs. A live hung upload is not automatically timed out; decide whether to continue or explicitly escalate with data-loss acceptance. | -| Promotion `Promoting` | Target has been shaped as primary but has not served yet. | Diagnose target image, scheduling, PVC mount, Secret, network, and store access. This phase and its deletion finalizer are unbounded. | -| Promotion `Failed` | One-shot cutover ended. | Read `.status.reason` and `.status.message`; correct the cause and create a new promotion name. Do not patch the terminal object. | -| `ImagePullBackOff` / `ErrImagePull` | Kubelet cannot pull operator or QuestDB image. | Check the correct namespace's imagePullSecret metadata, repository/tag, node registry reachability, and entitlement. Operator and database pull credentials are separate. | -| PVC `Pending` | No matching volume can bind. | Describe PVC; check StorageClass, topology, quota, CSI controller, capacity, and cloud events. | -| Primary PVC `Terminating` | Deletion is in progress; a mounted same-name primary Pod may hold `pvc-protection`. | Do not remove the finalizer. Current versions fence an established primary Pod automatically. Watch for the Pod to terminate, the old PVC UID to disappear, `PromotionRequired=True`, and zero ready RW endpoints. | -| Pod `Pending` | Scheduler or volume attachment cannot place it. | Describe pod; check requests, taints/tolerations, affinity, topology, PDB context, PVC binding, and single-node disk attachment. | -| Store Secret/IAM/network error | QuestDB pod cannot read/write object storage. | Confirm the `QuestDBObjectStore` provider fields, referenced Secret **names and keys** (not values), pod cloud identity, prefix-level IAM, DNS, HTTPS egress, endpoint, region/account, and provider audit logs. | -| Follower has no primary, no `WriteHealthy`, and `-rw` has no endpoints | Healthy follower behavior when `.status.replication.following=true`. | Do not wait for `WriteHealthy`; use `-ro` for reads and complete the migration cutover when ready. | +| Signal or symptom | Meaning | Next checks | +| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ConfigRejected=True` | An engine setting is invalid, injection-prone, or operator-owned. | Read reason/message; remove the rejected key from `spec.config` or `spec.replication.config`. Keys must match `^[A-Za-z0-9._-]+$`; values must not contain line separators. Compare with [Configuration](/docs/enterprise-kubernetes-operator/configuration/#extra-engine-options). | +| `TLSReady=True/Ready` | PGWire TLS is applied and the operator can use verified SQL when TLS was selected at creation. | Keep clients on TLS and monitor certificate expiry/rotation. | +| `TLSReady=True/VerificationDisabled` | PGWire TLS encryption is active, but operator certificate/hostname authentication is disabled. | Treat this as development-only; remove `insecureSkipVerify` after installing a trusted certificate. | +| `TLSReady=False` | Requested PGWire TLS cannot be applied or the stored cluster attempts to add/remove the TLS block after creation. | Read reason/message. For `SecretNotFound` or `InvalidSecret`, verify Secret namespace/name, `tls.crt`, `tls.key`, optional `ca.crt`, certificate validity, server-auth usage, trust chain, and SAN/serverName. For `ImmutableModeChange` or `AppliedModeUnknown`, return the spec to the original transport mode or create a new cluster. Do not work around it by switching operator traffic to plaintext. | +| `OperatorIdentityReady=False` | The operator could not provision/converge its least-privilege `questdb_operator` **SQL service account** from the admin identity. It is not cloud or object-store identity. | Check QuestDB pgwire reachability, admin Secret name/keys/permissions, ACL settings, and the condition reason. `WaitingForInstance` and brief `QuestDBUnreachable` can be boot states; persistent `AdminInsufficient` needs credential/privilege correction. | +| `WriteHealthy=False/PrimaryNotReady` | The primary is not Ready, so WAL write health cannot be established. | Diagnose primary Pod readiness, scheduling, mounts, probes, resources, and logs. Do not infer healthy writes from a read-serving replica or `Available=True`. | +| `WriteHealthy=False/PrimarySuspended` | One or more named primary WAL tables are write-impaired; reads and other tables can remain available. | Read the condition message for every table; inspect `wal_tables()`, QuestDB logs, table/disk state, and application errors. `Available=True` does not override this failure. | +| `WriteHealthy=Unknown` | Operator identity is not ready, or the latest `wal_tables()` query/observation was unavailable or failed. | Read reason/message; check operator SQL identity, pgwire reachability, and logs. Do not infer healthy writes from `Available=True`. | +| `BackupHealthy=Unknown` | No observed backup yet, an engine run is still progressing within its threshold, or configuration cannot be observed. | Confirm schedule, `status.backup.configured`, `lastBackup.status`, `lastProgressAt`, manager observation delay, store config, and primary logs. | +| `BackupHealthy=False/Failed` | Latest observed engine backup run failed. | Read `.status.backup.lastBackup.error`; check pod identity/static Secret, prefix permissions, DNS/network, and provider service health. | +| `BackupHealthy=False/Stalled` | The engine run remains `in_progress`, but `progressPercent` has not changed since `lastProgressAt` for `stalledAfterSeconds`. | Inspect `lastBackup`, `lastProgressAt`, database logs, IAM/Secret, network, and object-store health. Default threshold is 3600 seconds; explicit `0` disables detection and does not cancel the run. | +| `ReplicationHealthy=False` or `Unknown` | Replicas are suspended, behind, unreachable, unseeded, or freshness was not determined. | Inspect `.status.replication.seed`, `replicas[]`, `caughtUpNow`, `lagTxns`, and `suspendedTables`; then database metrics/logs and object-store access. Absent freshness is not zero lag. | +| `InstanceUnreachable=True` | Operator cannot safely observe an instance, or its node is unreachable. | Inspect pod readiness, node Ready condition, pgwire/metrics network paths, probes, CPU/memory pressure, and logs. Do not force-create another primary. | +| `PromotionRequired=True` | Established primary PVC is missing/Terminating; the operator fences its Pod, refuses an empty replacement, preserves `currentPrimary`, and does not auto-promote. | Confirm PVC/PV/cloud-disk and zero ready RW endpoints. Restore the volume or select a replica and follow [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover), accepting emergency loss when a drain is impossible. | +| `Recovered=True` | Engine reported restore completion. | Still validate actual tables, timestamps, and row counts before traffic. | +| `RecoveryFailed=True` | Restore init/runtime failed or was incomplete. | Read reason/message and genesis init logs; verify source store/root/instance/target. `RestoreError` on an older-than-retained PITR target is fail-closed. `spec.bootstrap` is immutable, so correct it in a fresh destination. Source instance names must match `^[a-z0-9]+(-[a-z0-9]+)*$`. | +| `StorageResizeBlocked=True` | StorageClass/CSI expansion is unavailable. | Confirm `allowVolumeExpansion`, driver support, PVC events, and requested size. Never shrink; restore to a new class if needed. | +| `Progressing=True/RollingUpdate` | The controller is rolling or safely holding a mutable spec change. | Check Pods, PVCs, peer health, Node readiness, and read-route labels. The hold may be protecting topology, storage, peer, Node, or read-routing conditions rather than being stuck. | +| Promotion `Pending`/`Validating` | Request awaits acceptance, catch-up, or drain prerequisites. | Read `.status.reason/message`, active promotion, target live status, and `caughtUpNow`. A request left Pending for 10 minutes becomes `StaleRequest`. | +| Promotion `Draining` | Old primary is stopping writes and uploading its tail, or target is replaying it. | Check old-primary and target logs. A live hung upload is not automatically timed out; decide whether to continue or explicitly escalate with data-loss acceptance. | +| Promotion `Stalled=True/TargetNotPrimary` | The target has been shaped but has not become the serving primary. | Diagnose target Pod readiness, image, scheduling, PVC, Secret, network, and store access. | +| Promotion `Stalled=True/FenceNotEffective` | The target reports that the old owner still holds the stream. Fencing is not relaxed. | If the old Pod remains on a NotReady node, power off or otherwise isolate the machine, then use Kubernetes' out-of-service procedure. Do not remove the finalizer while the old process may run. | +| Promotion `Promoting` | Target has been shaped as primary but has not served yet. | Diagnose target image, scheduling, PVC mount, Secret, network, and store access. This phase and its deletion finalizer are unbounded. | +| Promotion `Failed` | One-shot cutover ended. | Read `.status.reason` and `.status.message`; correct the cause and create a new promotion name. Do not patch the terminal object. | +| `ImagePullBackOff` / `ErrImagePull` | Kubelet cannot pull operator or QuestDB image. | Check the correct namespace's imagePullSecret metadata, repository/tag, node registry reachability, and entitlement. Operator and database pull credentials are separate. | +| PVC `Pending` | No matching volume can bind. | Describe PVC; check StorageClass, topology, quota, CSI controller, capacity, and cloud events. | +| Primary PVC `Terminating` | Deletion is in progress; a mounted same-name primary Pod may hold `pvc-protection`. | Do not remove the finalizer. Current versions fence an established primary Pod automatically. Watch for the Pod to terminate, the old PVC UID to disappear, `PromotionRequired=True`, and zero ready RW endpoints. | +| Pod `Pending` | Scheduler or volume attachment cannot place it. | Describe pod; check requests, taints/tolerations, affinity, topology, PDB context, PVC binding, and single-node disk attachment. | +| Store Secret/IAM/network error | QuestDB pod cannot read/write object storage. | Confirm the `QuestDBObjectStore` provider fields, referenced Secret **names and keys** (not values), pod cloud identity, prefix-level IAM, DNS, HTTPS egress, endpoint, region/account, and provider audit logs. | +| Follower has no primary, no `WriteHealthy`, and `-rw` has no endpoints | Healthy follower behavior when `.status.replication.following=true`. | Do not wait for `WriteHealthy`; use `-ro` for reads and complete the migration cutover when ready. | ## Primary PVC deletion and fencing -For an established replicated primary, missing or Terminating storage is a fence condition. This includes the Pod-delete → replacement → PVC-delete race: the replacement may become Ready on the old PVC before deletion starts, but the current operator then deletes that Pod so Kubernetes can release the claim. Do not keep waiting indefinitely, create a same-name PVC manually, remove `pvc-protection`, or send clients directly to the Pod. +For an established replicated primary, missing or Terminating storage is a fence +condition. This includes the Pod-delete → replacement → PVC-delete race: the +replacement may become Ready on the old PVC before deletion starts, but the +current operator then deletes that Pod so Kubernetes can release the claim. Do +not keep waiting indefinitely, create a same-name PVC manually, remove +`pvc-protection`, or send clients directly to the Pod. Watch identities and routing with bounded commands: @@ -79,7 +98,14 @@ kubectl get endpointslice -n \ -l kubernetes.io/service-name=-rw -o yaml ``` -The fenced state keeps a non-empty `status.currentPrimary` as the primary of record but has no ready RW endpoint and no replacement primary PVC. Recover the original storage if possible; otherwise select a live replica and make the explicit Emergency decision. After promotion, verify `WriteHealthy=True/Healthy`, exactly one live primary, target PVC `questdb.io/role=primary`, and final replica PVCs `questdb.io/role=replica`. `questdb.io/bootstrap` remains ancestry/state and must not be used as current role. +The fenced state keeps a non-empty `status.currentPrimary` as the primary of +record but has no ready RW endpoint and no replacement primary PVC. Recover the +original storage if possible; otherwise select a live replica and make the +explicit Emergency decision. After promotion, verify +`WriteHealthy=True/Healthy`, exactly one live primary, target PVC +`questdb.io/role=primary`, and final replica PVCs `questdb.io/role=replica`. +`questdb.io/bootstrap` remains ancestry/state and must not be used as current +role. ## Inspect logs @@ -103,10 +129,18 @@ kubectl logs -n \ -c --tail=500 ``` -Do not paste credentials from logs or custom resources into a shared ticket. Redact according to your policy. +Do not paste credentials from logs or custom resources into a shared ticket. +Redact according to your policy. ## Object-store configuration has no readiness status -`QuestDBObjectStore` is a validated configuration holder, not a managed bucket/container and not a probe controller. It has no status condition to wait for. Diagnose it through the consuming cluster's `BackupHealthy`, `ReplicationHealthy`, or recovery conditions and through QuestDB pod logs, cloud identity/audit logs, and network tests. The operator itself has no object-store permissions and cannot list or clean the store. +`QuestDBObjectStore` is a validated configuration holder, not a managed +bucket/container and not a probe controller. It has no status condition to wait +for. Diagnose it through the consuming cluster's `BackupHealthy`, +`ReplicationHealthy`, or recovery conditions and through QuestDB pod logs, cloud +identity/audit logs, and network tests. The operator itself has no object-store +permissions and cannot list or clean the store. -If the cause remains unclear, collect the [support bundle](/docs/enterprise-kubernetes-operator/support/#collect-a-support-bundle) before changing the failing resources. +If the cause remains unclear, collect the +[support bundle](/docs/enterprise-kubernetes-operator/support/#collect-a-support-bundle) +before changing the failing resources. From c5b7618f0a18881ddea68caf15d94440bb5aa7ed Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Thu, 27 Aug 2026 19:21:57 -0400 Subject: [PATCH 02/12] docs: add Operator restore and migration guide --- .../getting-started/restore-and-migrate.md | 552 ++++++++++++++++++ .../enterprise-kubernetes-operator/index.md | 2 + documentation/sidebars.js | 5 + 3 files changed, 559 insertions(+) create mode 100644 documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md diff --git a/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md new file mode 100644 index 000000000..1e53c2bcd --- /dev/null +++ b/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md @@ -0,0 +1,552 @@ +--- +title: Restore or migrate QuestDB +description: + Restore QuestDB from an object-store backup or migrate an existing QuestDB + onto the Kubernetes Operator with a replica-first cutover. +--- + +# Restore or migrate QuestDB + +This guide covers two cloud-neutral ways to create a QuestDB Enterprise cluster +with existing data: + +1. **Restore:** create a new writable cluster from an object-store backup. +2. **Migrate:** create a replica-only follower of an external QuestDB, let it + catch up, and promote it after a controlled source drain. + +Both paths use the same Kubernetes resources on every supported cloud. The +provider-specific bucket or container, credentials, and pod identity are kept in +an existing `QuestDBObjectStore`. + +Before running a command, replace every `` value. An unreplaced +placeholder can be interpreted as shell redirection. + +## Before you start + +This guide assumes that: + +- the QuestDB Enterprise Kubernetes Operator is installed; +- the tenant namespace exists; +- the namespace has access to the QuestDB Enterprise image, either through an + `imagePullSecret` or ambient node credentials; +- a `ReadWriteOnce` StorageClass with `fsGroup` support is available; +- a same-namespace `QuestDBObjectStore` and any referenced credential Secret + already provide access to the source object store; and +- you know the source backup prefix and, for migration, the source replication + WAL prefix. + +See +[Configuration](/docs/enterprise-kubernetes-operator/configuration/#object-storage) +if object-store access is not ready. The operator does not test, list, read, or +write the store. QuestDB pods perform the object-store I/O, and the consuming +cluster's conditions are the readiness signal. + +Confirm the APIs, source store, and StorageClass before continuing: + +```sh +kubectl get crd questdbclusters.questdb.io \ + questdbobjectstores.questdb.io questdbpromotions.questdb.io +kubectl get questdbobjectstore -n +kubectl get storageclass +``` + +Copy the exact QuestDB Enterprise image and `imagePullSecrets` from a working +cluster when possible. Remove the `imagePullSecrets` block from the examples +only when every destination node has ambient pull access. + +## Scenario 1: Restore from an object-store backup + +A restore always creates a **new** `QuestDBCluster` and a new PVC. It never +restores over a running cluster or an existing volume. + +### 1. Collect the source details + +Record: + +- the source `QuestDBObjectStore` name; +- the backup prefix under that store; +- the source's backup instance name; and +- the desired namespace, cluster name, image, StorageClass, and volume size. + +For a running source, get its backup instance name directly from QuestDB: + +```sql +SELECT backup_instance_name(); +``` + +For an operator-managed source, it is also normally available in status: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='{.status.replication.seed.backupInstanceName}{"\n"}' +``` + +Copy the value exactly. Set `sourceInstanceName` whenever the backup prefix +contains more than one backup instance. If the value is omitted, the engine can +select the source only when the prefix contains exactly one instance. + +### 2. Choose destination prefixes + +The restored cluster must not write backups or replication WAL into another live +cluster's prefixes. In this example: + +- `` remains the read-only restore source; +- `backup///` is the restored cluster's new backup + prefix; and +- the omitted replication root defaults to the identity-scoped + `db///`. + +The example reuses the source `QuestDBObjectStore` for the restored cluster's +own backup and replication writes. To use a different object store, replace the +top-level `objectStoreRef` with another existing, writable store in the same +namespace. Keep `bootstrap.recovery.source.objectStoreRef` pointed at the source +store, and keep all destination prefixes distinct from live source prefixes. + +### 3. Create the restored cluster + +Save the following as `restore.yaml`: + +```yaml +apiVersion: questdb.io/v1alpha1 +kind: QuestDBCluster +metadata: + name: + namespace: +spec: + image: + imagePullSecrets: + - name: + storage: + storageClassName: + size: 100Gi + resources: + requests: + memory: 4Gi + limits: + memory: 4Gi + objectStoreRef: + name: + bootstrap: + recovery: + source: + objectStoreRef: + name: + root: + sourceInstanceName: + backup: + enabled: true + schedule: "0 * * * *" + timezone: UTC + retention: 5 + root: backup/// +``` + +`spec.bootstrap` is immutable. Review the store, prefix, and instance name +before applying the file: + +```sh +kubectl apply -f restore.yaml +``` + +The operator withholds the genesis pod until it can resolve the source store. +The QuestDB recovery init container then restores and validates the backup. The +operator never reads the backup itself. + +### 4. Watch the restore + +Use a bounded loop that stops on success or terminal recovery failure: + +```bash +RECOVERED="" +FAILED="" +for _ in $(seq 1 180); do + RECOVERED="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.conditions[?(@.type=="Recovered")].status}')" + FAILED="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.conditions[?(@.type=="RecoveryFailed")].status}')" + [ "$RECOVERED" = "True" ] && break + [ "$FAILED" = "True" ] && break + sleep 10 +done +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}' +[ "$RECOVERED" = "True" ] && [ "$FAILED" != "True" ] +``` + +If the final command fails, do not patch `spec.bootstrap` or reuse the PVC. +Follow +[restore failure cleanup](/docs/enterprise-kubernetes-operator/operations/backup-restore/#if-it-fails) +and create a fresh cluster with corrected immutable values. + +### 5. Verify the restored writer and data + +`Recovered=True` proves that the engine completed recovery. Also require the +current writer-health contract: + +```bash +GENERATION="$(kubectl get questdbcluster -n \ + -o jsonpath='{.metadata.generation}')" +for _ in $(seq 1 120); do + STATE="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.observedGeneration}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="WriteHealthy")]}{.status}{"/"}{.reason}{end}')" + IFS='|' read -r OBSERVED AVAILABLE PROGRESSING WRITE_HEALTHY <<< "$STATE" + if [ "$OBSERVED" = "$GENERATION" ] && \ + [ "$AVAILABLE" = "True/PrimaryReady" ] && \ + [ "$PROGRESSING" = "False/Settled" ] && \ + [ "$WRITE_HEALTHY" = "True/Healthy" ]; then + break + fi + sleep 10 +done +printf 'observed=%s available=%s progressing=%s writeHealthy=%s\n' \ + "$OBSERVED" "$AVAILABLE" "$PROGRESSING" "$WRITE_HEALTHY" +[ "$OBSERVED" = "$GENERATION" ] && \ +[ "$AVAILABLE" = "True/PrimaryReady" ] && \ +[ "$PROGRESSING" = "False/Settled" ] && \ +[ "$WRITE_HEALTHY" = "True/Healthy" ] +``` + +Confirm that the RW Service has an endpoint: + +```sh +kubectl get endpointslice -n \ + -l kubernetes.io/service-name=-rw +``` + +Before sending application traffic, connect through `-rw` and +validate critical tables, expected row counts, minimum and maximum timestamps, +application invariants, and free storage. See +[Connect to a database](/docs/enterprise-kubernetes-operator/operations/database/#connect) +for a temporary PGWire connection. + +For a point-in-time restore, add `bootstrap.recovery.recoveryTarget` when the +cluster is first created. See +[Point-in-time recovery](/docs/enterprise-kubernetes-operator/operations/backup-restore/#point-in-time-recovery-pitr) +for its retained-window and timestamp rules. + +## Scenario 2: Migrate an external QuestDB with a follower + +This path keeps the external source writable while an operator-managed replica +restores its backup and consumes its replication WAL. Cutover downtime is +limited to stopping and draining the source, consuming the final WAL, and +promoting the follower. + +The operator does not connect to, configure, stop, or fence the external source. +Those steps remain your responsibility. + +### 1. Prepare the source + +Before creating the follower, confirm that the external source: + +- runs a QuestDB Enterprise version compatible with the destination image; +- has a completed backup under a known backup prefix; +- uploads replication WAL under a known WAL prefix in the same object store; +- retains WAL back to the seed backup; +- returns an exact, non-empty value from `SELECT backup_instance_name();`; and +- can be stopped and restarted once with + `replication.role=primary-catchup-uploads` during cutover. + +The backup instance name must match `^[a-z0-9]+(-[a-z0-9]+)*$`. Confirm the +backup prefix, WAL prefix, and instance name against the source configuration; +they become immutable on the follower. + +### 2. Create a replica-only follower + +Save the following as `follower.yaml`: + +```yaml +apiVersion: questdb.io/v1alpha1 +kind: QuestDBCluster +metadata: + name: + namespace: +spec: + image: + imagePullSecrets: + - name: + instances: 1 + storage: + storageClassName: + size: 100Gi + resources: + requests: + memory: 4Gi + limits: + memory: 4Gi + objectStoreRef: + name: + backup: + enabled: true + schedule: "0 * * * *" + timezone: UTC + retention: 5 + root: + replication: + root: + bootstrap: + follow: + sourceInstanceName: +``` + +While the cluster is following, every instance is a replica and the backup +scheduler is paused. After promotion, this cluster adopts the source prefixes +and begins taking its own backups there. + +Review all immutable source selectors, then apply the file: + +```sh +kubectl apply -f follower.yaml +``` + +### 3. Wait for the follower to serve reads + +Wait until the replica has restored its baseline and reconciliation is settled: + +```bash +for _ in $(seq 1 180); do + STATE="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.replication.following}{"|"}{.status.readyInstances}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}')" + IFS='|' read -r FOLLOWING READY AVAILABLE PROGRESSING <<< "$STATE" + if [ "$FOLLOWING" = "true" ] && [ "$READY" = "1" ] && \ + [ "$AVAILABLE" = "True/Following" ] && \ + [ "$PROGRESSING" = "False/Settled" ]; then + break + fi + sleep 10 +done +printf 'following=%s ready=%s available=%s progressing=%s\n' \ + "$FOLLOWING" "$READY" "$AVAILABLE" "$PROGRESSING" +[ "$FOLLOWING" = "true" ] && [ "$READY" = "1" ] && \ +[ "$AVAILABLE" = "True/Following" ] && \ +[ "$PROGRESSING" = "False/Settled" ] +``` + +A healthy follower deliberately has no current primary and no RW endpoint. +Confirm both properties: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='following={.status.replication.following}{" primary="}{.status.currentPrimary}{"\n"}' +kubectl get endpointslice -n \ + -l kubernetes.io/service-name=-rw \ + -o jsonpath='{range .items[*].endpoints[*]}{.addresses}{"\n"}{end}' +``` + +The second command must print no endpoint addresses. + +### 4. Confirm replication catch-up + +Inspect the live follower position: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.replication.replicas[*]}{.instance}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{" suspended="}{.suspendedTables}{"\n"}{end}{range .status.conditions[?(@.type=="ReplicationHealthy")]}ReplicationHealthy={.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}{.status.replication.stream}{"\n"}' +``` + +Prefer to begin cutover with `caughtUpNow=true` and `lagTxns=0`. A busy source +may briefly move away from zero. A quiet source may report `StreamNotDetermined` +because the engine omits already-caught-up tables from its poll; that is not +proof of success. In that case, reconfirm the immutable source identity and +roots, then query `-ro` and verify a recent, known source +record. + +Do not proceed with `ReplicationHealthy=False`, suspended tables, a known +backlog that is not advancing, or unverified source selectors. The planned +promotion performs a final fail-closed check after the source is drained. + +### 5. Stop writes and drain the source + +Record the start of cutover in the Bash shell you will keep open: + +```bash +CUTOVER_TIME_CAPTURED=false +CUTOVER_STARTED_AT="" +if CUTOVER_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" && \ + [ -n "$CUTOVER_STARTED_AT" ]; then + CUTOVER_TIME_CAPTURED=true +fi +[ "$CUTOVER_TIME_CAPTURED" = true ] && \ + printf 'Cutover started at %s\n' "$CUTOVER_STARTED_AT" +``` + +Continue in this shell only if `CUTOVER_TIME_CAPTURED=true`; the final backup +check rejects a missing timestamp. + +Then perform these steps with the external source's service manager or container +runtime: + +1. Stop all application writes to the external source. +2. Stop the source QuestDB process. +3. Configure the source to start once with + `replication.role=primary-catchup-uploads`. +4. Start the source and watch its logs. +5. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS`. A normal shutdown without + this close reason does not prove that the final WAL reached object storage. +6. Confirm the source process has exited, and disable automatic restarts. + +The final upload has no safe fixed timeout. Supervise it at the source until it +succeeds. + +:::danger Do not promote while the external source may still be running as a +primary. The operator cannot fence an unmanaged process. Keep the old data +available for rollback investigation, but ensure the process and its supervisor +cannot restart it. ::: + +### 6. Promote the follower + +Create a one-shot planned promotion targeting the follower's instance serial +`1`: + +```sh +kubectl apply -f - < + namespace: +spec: + clusterRef: + name: + target: 1 + mode: Planned + catchUpTimeoutSeconds: 900 + primaryGracePeriodSeconds: 120 +EOF +``` + +The promotion waits for the source stream to remain quiet for at least 60 +seconds and for the target to consume the published WAL. It fails closed rather +than silently accepting a lossy cutover. + +Watch until the promotion completes or fails: + +```bash +PHASE="" +for _ in $(seq 1 180); do + PHASE="$(kubectl get questdbpromotion -n \ + -o jsonpath='{.status.phase}')" + printf '%s %s\n' "$(date -u +%FT%TZ)" "$PHASE" + case "$PHASE" in + Completed|Failed) break ;; + esac + sleep 10 +done +kubectl get questdbpromotion -n \ + -o jsonpath='{.status.phase}{" "}{.status.reason}{": "}{.status.message}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}' +[ "$PHASE" = "Completed" ] +``` + +If it fails, leave the source stopped and read the reported reason before taking +another action. A failed promotion is terminal; correct the cause and create a +new promotion object. Do not remove the promotion finalizer. See +[If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). + +### 7. Verify the new primary + +Wait for the promoted cluster's writer-health contract: + +```bash +GENERATION="$(kubectl get questdbcluster -n \ + -o jsonpath='{.metadata.generation}')" +for _ in $(seq 1 120); do + STATE="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.observedGeneration}{"|"}{.status.currentPrimary}{"|"}{.status.replication.following}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="WriteHealthy")]}{.status}{"/"}{.reason}{end}')" + IFS='|' read -r OBSERVED PRIMARY FOLLOWING AVAILABLE PROGRESSING WRITE_HEALTHY <<< "$STATE" + if [ "$OBSERVED" = "$GENERATION" ] && \ + [ "$PRIMARY" = "-1" ] && \ + [ "$FOLLOWING" != "true" ] && \ + [ "$AVAILABLE" = "True/PrimaryReady" ] && \ + [ "$PROGRESSING" = "False/Settled" ] && \ + [ "$WRITE_HEALTHY" = "True/Healthy" ]; then + break + fi + sleep 10 +done +printf 'observed=%s primary=%s following=%s available=%s progressing=%s writeHealthy=%s\n' \ + "$OBSERVED" "$PRIMARY" "$FOLLOWING" "$AVAILABLE" "$PROGRESSING" "$WRITE_HEALTHY" +[ "$OBSERVED" = "$GENERATION" ] && \ +[ "$PRIMARY" = "-1" ] && \ +[ "$FOLLOWING" != "true" ] && \ +[ "$AVAILABLE" = "True/PrimaryReady" ] && \ +[ "$PROGRESSING" = "False/Settled" ] && \ +[ "$WRITE_HEALTHY" = "True/Healthy" ] +``` + +Confirm that `-rw` now has an endpoint, then connect through +that Service and validate recent data and application writes: + +```sh +kubectl get endpointslice -n \ + -l kubernetes.io/service-name=-rw +``` + +Permanently decommission the old source so that it cannot restart and contend +for the adopted WAL stream. + +### 8. Verify the first post-cutover backup + +The WAL cleaner remains held until the promoted cluster completes its own first +backup. With the hourly schedule in this guide, allow one schedule interval plus +the operator's roughly two-minute observation delay: + +```bash +PRIMARY_UID_CAPTURED=false +PRIMARY_UID_BEFORE_BACKUP="" +if PRIMARY_UID_BEFORE_BACKUP="$(kubectl get pod -1 \ + -n -o jsonpath='{.metadata.uid}')" && \ + [ -n "$PRIMARY_UID_BEFORE_BACKUP" ]; then + PRIMARY_UID_CAPTURED=true +fi + +BACKUP_VERIFIED=false +STATUS="" +END_TIME="" +for _ in $(seq 1 450); do + STATUS="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.backup.lastBackup.status}')" + END_TIME="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.backup.lastBackup.endTime}')" + if [ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ + [ -n "$CUTOVER_STARTED_AT" ] && \ + [ "$STATUS" = "completed" ] && [ -n "$END_TIME" ] && \ + [[ "$END_TIME" > "$CUTOVER_STARTED_AT" ]]; then + BACKUP_VERIFIED=true + break + fi + [ "$STATUS" = "failed" ] && break + sleep 10 +done +printf 'status=%s endTime=%s cutoverStartedAt=%s\n' \ + "$STATUS" "$END_TIME" "$CUTOVER_STARTED_AT" +[ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ +[ -n "$CUTOVER_STARTED_AT" ] && [ "$BACKUP_VERIFIED" = true ] +``` + +Releasing the WAL cleaner rolls the primary once. Prove that the asynchronous +roll occurred by waiting for the pod UID to change: + +```bash +PRIMARY_ROLLED=false +PRIMARY_UID_AFTER_BACKUP="" +for _ in $(seq 1 120); do + if PRIMARY_UID_AFTER_BACKUP="$(kubectl get pod -1 \ + -n -o jsonpath='{.metadata.uid}' 2>/dev/null)" && \ + [ "$PRIMARY_UID_CAPTURED" = true ] && \ + [ -n "$PRIMARY_UID_AFTER_BACKUP" ] && \ + [ "$PRIMARY_UID_AFTER_BACKUP" != "$PRIMARY_UID_BEFORE_BACKUP" ]; then + PRIMARY_ROLLED=true + break + fi + sleep 10 +done +printf 'before=%s after=%s rolled=%s\n' \ + "$PRIMARY_UID_BEFORE_BACKUP" "$PRIMARY_UID_AFTER_BACKUP" "$PRIMARY_ROLLED" +[ "$PRIMARY_UID_CAPTURED" = true ] && [ "$PRIMARY_ROLLED" = true ] +``` + +After the new pod appears, repeat the writer-health check from the previous step +and require it to settle before declaring the migration complete. + +For emergency source loss, multiple followers, or detailed failure handling, use +the full +[high-availability migration runbook](/docs/enterprise-kubernetes-operator/high-availability/#migrate-an-existing-questdb-onto-the-operator). diff --git a/documentation/enterprise-kubernetes-operator/index.md b/documentation/enterprise-kubernetes-operator/index.md index 2951361d8..5f46f6d15 100644 --- a/documentation/enterprise-kubernetes-operator/index.md +++ b/documentation/enterprise-kubernetes-operator/index.md @@ -37,6 +37,8 @@ static Secret. QuestDB database pods perform all object-store I/O. [AKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/azure/). - **Shared install requirements:** see [Installation](/docs/enterprise-kubernetes-operator/installation/). +- **Restore or migrate existing data:** follow the cloud-neutral + [restore and migration guide](/docs/enterprise-kubernetes-operator/getting-started/restore-and-migrate/). - **PGWire TLS and network isolation:** plan them before creation with [Configuration](/docs/enterprise-kubernetes-operator/configuration/#pgwire-tls). - **Operate the operator:** use the diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 84ffb0548..e82bca8cf 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -831,6 +831,11 @@ module.exports = { id: "enterprise-kubernetes-operator/getting-started/azure", label: "Azure AKS", }, + { + type: "doc", + id: "enterprise-kubernetes-operator/getting-started/restore-and-migrate", + label: "Restore or migrate", + }, ], }, "enterprise-kubernetes-operator/configuration", From 163c80f0f83adf3f438f4123633920f41995aad0 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 13:54:07 -0400 Subject: [PATCH 03/12] docs: highlight placeholder replacement warning --- .../getting-started/restore-and-migrate.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md index 1e53c2bcd..5ae731211 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md @@ -18,8 +18,10 @@ Both paths use the same Kubernetes resources on every supported cloud. The provider-specific bucket or container, credentials, and pod identity are kept in an existing `QuestDBObjectStore`. +:::warning Before running a command, replace every `` value. An unreplaced placeholder can be interpreted as shell redirection. +::: ## Before you start From 4a0a1cd9aadadcd82507b7aaa6e761cc19f775c0 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:06:04 -0400 Subject: [PATCH 04/12] docs: focus migration guide on follower cutover --- .../{restore-and-migrate.md => migrate.md} | 207 ++---------------- .../enterprise-kubernetes-operator/index.md | 4 +- documentation/sidebars.js | 4 +- 3 files changed, 21 insertions(+), 194 deletions(-) rename documentation/enterprise-kubernetes-operator/getting-started/{restore-and-migrate.md => migrate.md} (65%) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md similarity index 65% rename from documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md rename to documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 5ae731211..6470d34a5 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/restore-and-migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -1,22 +1,18 @@ --- -title: Restore or migrate QuestDB +title: Migrate QuestDB onto the Kubernetes Operator description: - Restore QuestDB from an object-store backup or migrate an existing QuestDB - onto the Kubernetes Operator with a replica-first cutover. + Migrate an existing QuestDB Enterprise deployment onto the Kubernetes + Operator with a replica-first cutover. --- -# Restore or migrate QuestDB +# Migrate QuestDB onto the Kubernetes Operator -This guide covers two cloud-neutral ways to create a QuestDB Enterprise cluster -with existing data: +This guide creates an operator-managed, replica-only follower of an external +QuestDB Enterprise deployment, lets it restore the source backup and consume +replication WAL, then promotes it after a controlled source drain. -1. **Restore:** create a new writable cluster from an object-store backup. -2. **Migrate:** create a replica-only follower of an external QuestDB, let it - catch up, and promote it after a controlled source drain. - -Both paths use the same Kubernetes resources on every supported cloud. The -provider-specific bucket or container, credentials, and pod identity are kept in -an existing `QuestDBObjectStore`. +The workflow is cloud-neutral. The provider-specific bucket or container, +credentials, and pod identity are kept in an existing `QuestDBObjectStore`. :::warning Before running a command, replace every `` value. An unreplaced @@ -34,8 +30,7 @@ This guide assumes that: - a `ReadWriteOnce` StorageClass with `fsGroup` support is available; - a same-namespace `QuestDBObjectStore` and any referenced credential Secret already provide access to the source object store; and -- you know the source backup prefix and, for migration, the source replication - WAL prefix. +- you know the source backup and replication WAL prefixes. See [Configuration](/docs/enterprise-kubernetes-operator/configuration/#object-storage) @@ -56,177 +51,7 @@ Copy the exact QuestDB Enterprise image and `imagePullSecrets` from a working cluster when possible. Remove the `imagePullSecrets` block from the examples only when every destination node has ambient pull access. -## Scenario 1: Restore from an object-store backup - -A restore always creates a **new** `QuestDBCluster` and a new PVC. It never -restores over a running cluster or an existing volume. - -### 1. Collect the source details - -Record: - -- the source `QuestDBObjectStore` name; -- the backup prefix under that store; -- the source's backup instance name; and -- the desired namespace, cluster name, image, StorageClass, and volume size. - -For a running source, get its backup instance name directly from QuestDB: - -```sql -SELECT backup_instance_name(); -``` - -For an operator-managed source, it is also normally available in status: - -```sh -kubectl get questdbcluster -n \ - -o jsonpath='{.status.replication.seed.backupInstanceName}{"\n"}' -``` - -Copy the value exactly. Set `sourceInstanceName` whenever the backup prefix -contains more than one backup instance. If the value is omitted, the engine can -select the source only when the prefix contains exactly one instance. - -### 2. Choose destination prefixes - -The restored cluster must not write backups or replication WAL into another live -cluster's prefixes. In this example: - -- `` remains the read-only restore source; -- `backup///` is the restored cluster's new backup - prefix; and -- the omitted replication root defaults to the identity-scoped - `db///`. - -The example reuses the source `QuestDBObjectStore` for the restored cluster's -own backup and replication writes. To use a different object store, replace the -top-level `objectStoreRef` with another existing, writable store in the same -namespace. Keep `bootstrap.recovery.source.objectStoreRef` pointed at the source -store, and keep all destination prefixes distinct from live source prefixes. - -### 3. Create the restored cluster - -Save the following as `restore.yaml`: - -```yaml -apiVersion: questdb.io/v1alpha1 -kind: QuestDBCluster -metadata: - name: - namespace: -spec: - image: - imagePullSecrets: - - name: - storage: - storageClassName: - size: 100Gi - resources: - requests: - memory: 4Gi - limits: - memory: 4Gi - objectStoreRef: - name: - bootstrap: - recovery: - source: - objectStoreRef: - name: - root: - sourceInstanceName: - backup: - enabled: true - schedule: "0 * * * *" - timezone: UTC - retention: 5 - root: backup/// -``` - -`spec.bootstrap` is immutable. Review the store, prefix, and instance name -before applying the file: - -```sh -kubectl apply -f restore.yaml -``` - -The operator withholds the genesis pod until it can resolve the source store. -The QuestDB recovery init container then restores and validates the backup. The -operator never reads the backup itself. - -### 4. Watch the restore - -Use a bounded loop that stops on success or terminal recovery failure: - -```bash -RECOVERED="" -FAILED="" -for _ in $(seq 1 180); do - RECOVERED="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.conditions[?(@.type=="Recovered")].status}')" - FAILED="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.conditions[?(@.type=="RecoveryFailed")].status}')" - [ "$RECOVERED" = "True" ] && break - [ "$FAILED" = "True" ] && break - sleep 10 -done -kubectl get questdbcluster -n \ - -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"/"}{.reason}{" "}{.message}{"\n"}{end}' -[ "$RECOVERED" = "True" ] && [ "$FAILED" != "True" ] -``` - -If the final command fails, do not patch `spec.bootstrap` or reuse the PVC. -Follow -[restore failure cleanup](/docs/enterprise-kubernetes-operator/operations/backup-restore/#if-it-fails) -and create a fresh cluster with corrected immutable values. - -### 5. Verify the restored writer and data - -`Recovered=True` proves that the engine completed recovery. Also require the -current writer-health contract: - -```bash -GENERATION="$(kubectl get questdbcluster -n \ - -o jsonpath='{.metadata.generation}')" -for _ in $(seq 1 120); do - STATE="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.observedGeneration}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="WriteHealthy")]}{.status}{"/"}{.reason}{end}')" - IFS='|' read -r OBSERVED AVAILABLE PROGRESSING WRITE_HEALTHY <<< "$STATE" - if [ "$OBSERVED" = "$GENERATION" ] && \ - [ "$AVAILABLE" = "True/PrimaryReady" ] && \ - [ "$PROGRESSING" = "False/Settled" ] && \ - [ "$WRITE_HEALTHY" = "True/Healthy" ]; then - break - fi - sleep 10 -done -printf 'observed=%s available=%s progressing=%s writeHealthy=%s\n' \ - "$OBSERVED" "$AVAILABLE" "$PROGRESSING" "$WRITE_HEALTHY" -[ "$OBSERVED" = "$GENERATION" ] && \ -[ "$AVAILABLE" = "True/PrimaryReady" ] && \ -[ "$PROGRESSING" = "False/Settled" ] && \ -[ "$WRITE_HEALTHY" = "True/Healthy" ] -``` - -Confirm that the RW Service has an endpoint: - -```sh -kubectl get endpointslice -n \ - -l kubernetes.io/service-name=-rw -``` - -Before sending application traffic, connect through `-rw` and -validate critical tables, expected row counts, minimum and maximum timestamps, -application invariants, and free storage. See -[Connect to a database](/docs/enterprise-kubernetes-operator/operations/database/#connect) -for a temporary PGWire connection. - -For a point-in-time restore, add `bootstrap.recovery.recoveryTarget` when the -cluster is first created. See -[Point-in-time recovery](/docs/enterprise-kubernetes-operator/operations/backup-restore/#point-in-time-recovery-pitr) -for its retained-window and timestamp rules. - -## Scenario 2: Migrate an external QuestDB with a follower +## Migration workflow This path keeps the external source writable while an operator-managed replica restores its backup and consumes its replication WAL. Cutover downtime is @@ -389,10 +214,12 @@ runtime: The final upload has no safe fixed timeout. Supervise it at the source until it succeeds. -:::danger Do not promote while the external source may still be running as a -primary. The operator cannot fence an unmanaged process. Keep the old data -available for rollback investigation, but ensure the process and its supervisor -cannot restart it. ::: +:::danger +Do not promote while the external source may still be running as a primary. The +operator cannot fence an unmanaged process. Keep the old data available for +rollback investigation, but ensure the process and its supervisor cannot restart +it. +::: ### 6. Promote the follower diff --git a/documentation/enterprise-kubernetes-operator/index.md b/documentation/enterprise-kubernetes-operator/index.md index 5f46f6d15..9f53d9920 100644 --- a/documentation/enterprise-kubernetes-operator/index.md +++ b/documentation/enterprise-kubernetes-operator/index.md @@ -37,8 +37,8 @@ static Secret. QuestDB database pods perform all object-store I/O. [AKS onboarding guide](/docs/enterprise-kubernetes-operator/getting-started/azure/). - **Shared install requirements:** see [Installation](/docs/enterprise-kubernetes-operator/installation/). -- **Restore or migrate existing data:** follow the cloud-neutral - [restore and migration guide](/docs/enterprise-kubernetes-operator/getting-started/restore-and-migrate/). +- **Migrate an existing QuestDB:** follow the cloud-neutral + [migration guide](/docs/enterprise-kubernetes-operator/getting-started/migrate/). - **PGWire TLS and network isolation:** plan them before creation with [Configuration](/docs/enterprise-kubernetes-operator/configuration/#pgwire-tls). - **Operate the operator:** use the diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 51212a748..f0b4f587f 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -837,8 +837,8 @@ module.exports = { }, { type: "doc", - id: "enterprise-kubernetes-operator/getting-started/restore-and-migrate", - label: "Restore or migrate", + id: "enterprise-kubernetes-operator/getting-started/migrate", + label: "Migrate existing QuestDB", }, ], }, From 6c2a446137166fe15abd80e28ceb17d279ac28f8 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:27:47 -0400 Subject: [PATCH 05/12] docs: add external source preparation steps --- .../getting-started/migrate.md | 145 ++++++++++++++++-- 1 file changed, 132 insertions(+), 13 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 6470d34a5..542566557 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -63,19 +63,138 @@ Those steps remain your responsibility. ### 1. Prepare the source -Before creating the follower, confirm that the external source: - -- runs a QuestDB Enterprise version compatible with the destination image; -- has a completed backup under a known backup prefix; -- uploads replication WAL under a known WAL prefix in the same object store; -- retains WAL back to the seed backup; -- returns an exact, non-empty value from `SELECT backup_instance_name();`; and -- can be stopped and restarted once with - `replication.role=primary-catchup-uploads` during cutover. - -The backup instance name must match `^[a-z0-9]+(-[a-z0-9]+)*$`. Confirm the -backup prefix, WAL prefix, and instance name against the source configuration; -they become immutable on the follower. +The source must run a QuestDB Enterprise version compatible with the destination +image. Replication only carries changes to WAL-enabled tables. Inventory the +source before enabling replication: + +```questdb-sql title="Check source tables" +SELECT table_name, walEnabled +FROM tables() +ORDER BY table_name; +``` + +Data already present in a non-WAL table is included in the seed backup, but +later changes to that table are not replicated. Stop writes to non-WAL tables +before the seed backup and keep them stopped through cutover, or arrange to +synchronize them separately. + +Choose a backup prefix and a replication WAL prefix in the **same bucket**. The +follower has one `QuestDBObjectStore` for both uses, so separate source buckets +cannot be represented by the follower specification. The roots must be distinct, +stable prefixes with a trailing `/`. No other live primary may write to the WAL +prefix. + +The following S3 example uses ambient AWS credentials from an EC2 instance +profile. Add it to the source's `server.conf`, replacing every placeholder: + +```ini title="server.conf on the external source" +backup.enabled=true +backup.object.store=s3::bucket=;root=;region=; +backup.schedule.cron=0 * * * * +backup.schedule.tz=UTC +backup.cleanup.keep.latest.n=5 + +replication.role=primary +replication.object.store=s3::bucket=;root=;region=; +replication.primary.cleaner.enabled=false +``` + +The example temporarily disables the source WAL cleaner so it cannot delete WAL +needed by the follower. This increases object-storage usage until cutover, so +keep the migration bounded. Backup retention is entry-count based, and the WAL +cleaner can merge backup and checkpoint history; do not treat a retention count +as a guaranteed time window. Keep at least one completed backup available until +the follower has restored it. + +:::note Other object-store providers +The backup and replication mechanics are provider-independent. For another +provider supported by your Operator release, replace the two S3 connection +strings with that provider's +[object-store connection strings](/docs/high-availability/setup/#1-configure-object-storage). +Keep the backup and WAL roots distinct, and configure the destination +`QuestDBObjectStore` for the same underlying store. +::: + +If the source runs in Docker, use the equivalent environment variables in its +existing container definition. File-backed object-store values keep static +credentials, when required, out of the container specification: + +```text title="backup-object-store" +s3::bucket=;root=;region=; +``` + +```text title="replication-object-store" +s3::bucket=;root=;region=; +``` + +```yaml title="Docker Compose service excerpt" +services: + questdb: + environment: + QDB_BACKUP_ENABLED: "true" + QDB_BACKUP_OBJECT_STORE_FILE: /run/secrets/backup-object-store + QDB_BACKUP_SCHEDULE_CRON: "0 * * * *" + QDB_BACKUP_SCHEDULE_TZ: UTC + QDB_BACKUP_CLEANUP_KEEP_LATEST_N: "5" + QDB_REPLICATION_ROLE: primary + QDB_REPLICATION_OBJECT_STORE_FILE: /run/secrets/replication-object-store + QDB_REPLICATION_PRIMARY_CLEANER_ENABLED: "false" + volumes: + - /secure/backup-object-store:/run/secrets/backup-object-store:ro + - /secure/replication-object-store:/run/secrets/replication-object-store:ro +``` + +Ensure that only the QuestDB runtime account can read the mounted files. Do not +put long-lived access keys in the Compose file or commit them to source control. +When ambient credentials are unavailable, add the provider credentials to the +protected object-store files. + +The object-store and replication-role settings are not reloadable. Restart the +source with its normal service manager or container runtime, then trigger the +seed backup: + +```questdb-sql title="Take the seed backup" +BACKUP DATABASE; +``` + +`BACKUP DATABASE` starts the backup asynchronously. Poll its latest record until +it reports `backup complete`: + +```questdb-sql title="Check the seed backup" +SELECT status, progress_percent, start_ts, end_ts, backup_error +FROM backups() +ORDER BY start_ts DESC +LIMIT 1; +``` + +After the backup completes, record the source's exact backup instance name: + +```questdb-sql title="Get the source backup instance name" +SELECT backup_instance_name(); +``` + +The value must be non-empty and match `^[a-z0-9]+(-[a-z0-9]+)*$`. Finally, +confirm that both prefixes contain objects: + +```bash title="Check the S3 source prefixes" +aws s3api list-objects-v2 \ + --region '' \ + --bucket '' \ + --prefix '' \ + --max-keys 5 +aws s3api list-objects-v2 \ + --region '' \ + --bucket '' \ + --prefix '' \ + --max-keys 5 +``` + +Do not create the follower unless the seed backup is complete, both commands +return objects, the source WAL cleaner remains disabled, all source selectors +match the intended store, and the source can later be stopped and restarted once +with `replication.role=primary-catchup-uploads`. Copy the backup prefix, WAL +prefix, and backup instance name exactly into the follower specification; those +source selectors are immutable. ### 2. Create a replica-only follower From 34fa176574cf6746c1120b70cc3be2e6a9134915 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:46:05 -0400 Subject: [PATCH 06/12] docs: emphasize non-WAL migration warning --- .../enterprise-kubernetes-operator/getting-started/migrate.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 542566557..ed9a7c61b 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -73,10 +73,12 @@ FROM tables() ORDER BY table_name; ``` +:::warning[Non-WAL tables do not replicate] Data already present in a non-WAL table is included in the seed backup, but later changes to that table are not replicated. Stop writes to non-WAL tables before the seed backup and keep them stopped through cutover, or arrange to synchronize them separately. +::: Choose a backup prefix and a replication WAL prefix in the **same bucket**. The follower has one `QuestDBObjectStore` for both uses, so separate source buckets From 9c7eef4d525e6053c3b7c0b152c33b058baa613d Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:47:54 -0400 Subject: [PATCH 07/12] docs: clarify migration retention guidance --- .../getting-started/migrate.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index ed9a7c61b..9afbfee2b 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -101,12 +101,13 @@ replication.object.store=s3::bucket=;root=;regio replication.primary.cleaner.enabled=false ``` -The example temporarily disables the source WAL cleaner so it cannot delete WAL -needed by the follower. This increases object-storage usage until cutover, so -keep the migration bounded. Backup retention is entry-count based, and the WAL -cleaner can merge backup and checkpoint history; do not treat a retention count -as a guaranteed time window. Keep at least one completed backup available until -the follower has restored it. +The example disables WAL cleanup on the source for the rest of the migration. +WAL objects will accumulate in object storage, but QuestDB will not delete them +before the follower consumes them. + +Separately, `backup.cleanup.keep.latest.n=5` retains the five most recent +completed backups—not five hours of backups. Before creating the follower, +confirm that at least one completed backup is still available. :::note Other object-store providers The backup and replication mechanics are provider-independent. For another From 2ba9da85aaddee499994afbd85c75a5c233d7ea6 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:49:45 -0400 Subject: [PATCH 08/12] docs: simplify source configuration guidance --- .../getting-started/migrate.md | 38 +++---------------- 1 file changed, 5 insertions(+), 33 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 9afbfee2b..3f1fbaf52 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -118,39 +118,11 @@ Keep the backup and WAL roots distinct, and configure the destination `QuestDBObjectStore` for the same underlying store. ::: -If the source runs in Docker, use the equivalent environment variables in its -existing container definition. File-backed object-store values keep static -credentials, when required, out of the container specification: - -```text title="backup-object-store" -s3::bucket=;root=;region=; -``` - -```text title="replication-object-store" -s3::bucket=;root=;region=; -``` - -```yaml title="Docker Compose service excerpt" -services: - questdb: - environment: - QDB_BACKUP_ENABLED: "true" - QDB_BACKUP_OBJECT_STORE_FILE: /run/secrets/backup-object-store - QDB_BACKUP_SCHEDULE_CRON: "0 * * * *" - QDB_BACKUP_SCHEDULE_TZ: UTC - QDB_BACKUP_CLEANUP_KEEP_LATEST_N: "5" - QDB_REPLICATION_ROLE: primary - QDB_REPLICATION_OBJECT_STORE_FILE: /run/secrets/replication-object-store - QDB_REPLICATION_PRIMARY_CLEANER_ENABLED: "false" - volumes: - - /secure/backup-object-store:/run/secrets/backup-object-store:ro - - /secure/replication-object-store:/run/secrets/replication-object-store:ro -``` - -Ensure that only the QuestDB runtime account can read the mounted files. Do not -put long-lived access keys in the Compose file or commit them to source control. -When ambient credentials are unavailable, add the provider credentials to the -protected object-store files. +Every `server.conf` setting can instead be supplied as an environment variable. +See [Environment variables](/docs/configuration/overview/#environment-variables) +for the naming convention. If an object-store string contains static +credentials, load it from a protected file as described in +[Secrets from files](/docs/configuration/overview/#secrets-from-files). The object-store and replication-role settings are not reloadable. Restart the source with its normal service manager or container runtime, then trigger the From 507203871eeaa9a6145d953da2f2111258796cbb Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 14:51:09 -0400 Subject: [PATCH 09/12] docs: explain backup instance name format --- .../getting-started/migrate.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 3f1fbaf52..eeb0116b6 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -148,8 +148,9 @@ After the backup completes, record the source's exact backup instance name: SELECT backup_instance_name(); ``` -The value must be non-empty and match `^[a-z0-9]+(-[a-z0-9]+)*$`. Finally, -confirm that both prefixes contain objects: +QuestDB generates the name as three random lowercase words separated by hyphens, +for example `happy-green-turtle`. Copy the entire value exactly. Then confirm +that both prefixes contain objects: ```bash title="Check the S3 source prefixes" aws s3api list-objects-v2 \ From 24ff23b4c69f065aabc65a074a92be64097da6d9 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 15:04:49 -0400 Subject: [PATCH 10/12] docs: simplify migration verification commands --- .../getting-started/migrate.md | 140 +++++------------- 1 file changed, 35 insertions(+), 105 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index eeb0116b6..65096009c 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -224,23 +224,9 @@ kubectl apply -f follower.yaml Wait until the replica has restored its baseline and reconciliation is settled: -```bash -for _ in $(seq 1 180); do - STATE="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.replication.following}{"|"}{.status.readyInstances}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}')" - IFS='|' read -r FOLLOWING READY AVAILABLE PROGRESSING <<< "$STATE" - if [ "$FOLLOWING" = "true" ] && [ "$READY" = "1" ] && \ - [ "$AVAILABLE" = "True/Following" ] && \ - [ "$PROGRESSING" = "False/Settled" ]; then - break - fi - sleep 10 -done -printf 'following=%s ready=%s available=%s progressing=%s\n' \ - "$FOLLOWING" "$READY" "$AVAILABLE" "$PROGRESSING" -[ "$FOLLOWING" = "true" ] && [ "$READY" = "1" ] && \ -[ "$AVAILABLE" = "True/Following" ] && \ -[ "$PROGRESSING" = "False/Settled" ] +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.phase}'=Following --timeout=30m ``` A healthy follower deliberately has no current primary and no RW endpoint. @@ -278,21 +264,14 @@ promotion performs a final fail-closed check after the source is drained. ### 5. Stop writes and drain the source -Record the start of cutover in the Bash shell you will keep open: +Record the UTC cutover start time: -```bash -CUTOVER_TIME_CAPTURED=false -CUTOVER_STARTED_AT="" -if CUTOVER_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" && \ - [ -n "$CUTOVER_STARTED_AT" ]; then - CUTOVER_TIME_CAPTURED=true -fi -[ "$CUTOVER_TIME_CAPTURED" = true ] && \ - printf 'Cutover started at %s\n' "$CUTOVER_STARTED_AT" +```sh +date -u +%FT%TZ ``` -Continue in this shell only if `CUTOVER_TIME_CAPTURED=true`; the final backup -check rejects a missing timestamp. +Keep this timestamp so you can confirm that the first completed backup happened +after cutover began. Then perform these steps with the external source's service manager or container runtime: @@ -367,33 +346,12 @@ new promotion object. Do not remove the promotion finalizer. See ### 7. Verify the new primary -Wait for the promoted cluster's writer-health contract: +Wait for the promoted cluster to report `Running`: -```bash -GENERATION="$(kubectl get questdbcluster -n \ - -o jsonpath='{.metadata.generation}')" -for _ in $(seq 1 120); do - STATE="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.observedGeneration}{"|"}{.status.currentPrimary}{"|"}{.status.replication.following}{"|"}{range .status.conditions[?(@.type=="Available")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="Progressing")]}{.status}{"/"}{.reason}{end}{"|"}{range .status.conditions[?(@.type=="WriteHealthy")]}{.status}{"/"}{.reason}{end}')" - IFS='|' read -r OBSERVED PRIMARY FOLLOWING AVAILABLE PROGRESSING WRITE_HEALTHY <<< "$STATE" - if [ "$OBSERVED" = "$GENERATION" ] && \ - [ "$PRIMARY" = "-1" ] && \ - [ "$FOLLOWING" != "true" ] && \ - [ "$AVAILABLE" = "True/PrimaryReady" ] && \ - [ "$PROGRESSING" = "False/Settled" ] && \ - [ "$WRITE_HEALTHY" = "True/Healthy" ]; then - break - fi - sleep 10 -done -printf 'observed=%s primary=%s following=%s available=%s progressing=%s writeHealthy=%s\n' \ - "$OBSERVED" "$PRIMARY" "$FOLLOWING" "$AVAILABLE" "$PROGRESSING" "$WRITE_HEALTHY" -[ "$OBSERVED" = "$GENERATION" ] && \ -[ "$PRIMARY" = "-1" ] && \ -[ "$FOLLOWING" != "true" ] && \ -[ "$AVAILABLE" = "True/PrimaryReady" ] && \ -[ "$PROGRESSING" = "False/Settled" ] && \ -[ "$WRITE_HEALTHY" = "True/Healthy" ] +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.phase}'=Running --timeout=20m +kubectl get questdbcluster -n -o wide ``` Confirm that `-rw` now has an endpoint, then connect through @@ -413,59 +371,31 @@ The WAL cleaner remains held until the promoted cluster completes its own first backup. With the hourly schedule in this guide, allow one schedule interval plus the operator's roughly two-minute observation delay: -```bash -PRIMARY_UID_CAPTURED=false -PRIMARY_UID_BEFORE_BACKUP="" -if PRIMARY_UID_BEFORE_BACKUP="$(kubectl get pod -1 \ - -n -o jsonpath='{.metadata.uid}')" && \ - [ -n "$PRIMARY_UID_BEFORE_BACKUP" ]; then - PRIMARY_UID_CAPTURED=true -fi - -BACKUP_VERIFIED=false -STATUS="" -END_TIME="" -for _ in $(seq 1 450); do - STATUS="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.status}')" - END_TIME="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.endTime}')" - if [ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ - [ -n "$CUTOVER_STARTED_AT" ] && \ - [ "$STATUS" = "completed" ] && [ -n "$END_TIME" ] && \ - [[ "$END_TIME" > "$CUTOVER_STARTED_AT" ]]; then - BACKUP_VERIFIED=true - break - fi - [ "$STATUS" = "failed" ] && break - sleep 10 -done -printf 'status=%s endTime=%s cutoverStartedAt=%s\n' \ - "$STATUS" "$END_TIME" "$CUTOVER_STARTED_AT" -[ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ -[ -n "$CUTOVER_STARTED_AT" ] && [ "$BACKUP_VERIFIED" = true ] +First, record the primary pod's current UID: + +```sh +kubectl get pod -1 -n \ + -o custom-columns='NAME:.metadata.name,UID:.metadata.uid' ``` -Releasing the WAL cleaner rolls the primary once. Prove that the asynchronous -roll occurred by waiting for the pod UID to change: +Then wait for the first backup to complete and print its completion time: -```bash -PRIMARY_ROLLED=false -PRIMARY_UID_AFTER_BACKUP="" -for _ in $(seq 1 120); do - if PRIMARY_UID_AFTER_BACKUP="$(kubectl get pod -1 \ - -n -o jsonpath='{.metadata.uid}' 2>/dev/null)" && \ - [ "$PRIMARY_UID_CAPTURED" = true ] && \ - [ -n "$PRIMARY_UID_AFTER_BACKUP" ] && \ - [ "$PRIMARY_UID_AFTER_BACKUP" != "$PRIMARY_UID_BEFORE_BACKUP" ]; then - PRIMARY_ROLLED=true - break - fi - sleep 10 -done -printf 'before=%s after=%s rolled=%s\n' \ - "$PRIMARY_UID_BEFORE_BACKUP" "$PRIMARY_UID_AFTER_BACKUP" "$PRIMARY_ROLLED" -[ "$PRIMARY_UID_CAPTURED" = true ] && [ "$PRIMARY_ROLLED" = true ] +```sh +kubectl wait questdbcluster/ -n \ + --for=jsonpath='{.status.backup.lastBackup.status}'=completed --timeout=75m +kubectl get questdbcluster -n \ + -o jsonpath='{.status.backup.lastBackup.endTime}{" completed\n"}' +``` + +Confirm that the completion time is later than the cutover start time recorded +in step 5. + +Releasing the WAL cleaner rolls the primary once. Watch the pod until it has a +new UID and is ready, then press Control-C: + +```sh +kubectl get pod -1 -n --watch \ + -o custom-columns='NAME:.metadata.name,UID:.metadata.uid,READY:.status.containerStatuses[0].ready,PHASE:.status.phase' ``` After the new pod appears, repeat the writer-health check from the previous step From 5212535d52488166851c665d575f945e0418e99f Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 15:20:15 -0400 Subject: [PATCH 11/12] docs: verify final WAL upload by exit code --- .../getting-started/migrate.md | 6 +++--- .../enterprise-kubernetes-operator/high-availability.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 65096009c..0e4a0d4bd 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -281,9 +281,9 @@ runtime: 3. Configure the source to start once with `replication.role=primary-catchup-uploads`. 4. Start the source and watch its logs. -5. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS`. A normal shutdown without - this close reason does not prove that the final WAL reached object storage. -6. Confirm the source process has exited, and disable automatic restarts. +5. Wait for the source process to exit with code `0`. A non-zero exit means the + final WAL upload did not complete; do not promote. +6. Disable automatic restarts. The final upload has no safe fixed timeout. Supervise it at the source until it succeeds. diff --git a/documentation/enterprise-kubernetes-operator/high-availability.md b/documentation/enterprise-kubernetes-operator/high-availability.md index 48da9305a..35205629f 100644 --- a/documentation/enterprise-kubernetes-operator/high-availability.md +++ b/documentation/enterprise-kubernetes-operator/high-availability.md @@ -321,9 +321,9 @@ fi 1. Stop application writes to the source, then stop the source database. 2. Restart the source once with `replication.role=primary-catchup-uploads`. -3. Wait for `CLOSE_REASON_UPLOADS_COMPLETE_SUCCESS` in the **source** logs, then - stop it again. This final upload may itself be unbounded; supervise it at the - source. A normal shutdown does not prove the tail reached object storage. +3. Wait for the source process to exit with code `0`, then prevent it from + restarting. This final upload may itself be unbounded; supervise it at the + source. A non-zero exit means the tail was not safely uploaded; do not promote. 4. Create a `Planned` `QuestDBPromotion` for a healthy follower instance and use the bounded watcher above. From bb4c00ff8ba4e62f29d330989eec5e3a57544265 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 2 Sep 2026 15:44:20 -0400 Subject: [PATCH 12/12] docs: clarify post-cutover cleaner rollout --- .../getting-started/migrate.md | 29 ++-- .../high-availability.md | 149 +----------------- 2 files changed, 25 insertions(+), 153 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md index 0e4a0d4bd..80867cbef 100644 --- a/documentation/enterprise-kubernetes-operator/getting-started/migrate.md +++ b/documentation/enterprise-kubernetes-operator/getting-started/migrate.md @@ -367,11 +367,20 @@ for the adopted WAL stream. ### 8. Verify the first post-cutover backup -The WAL cleaner remains held until the promoted cluster completes its own first -backup. With the hourly schedule in this guide, allow one schedule interval plus -the operator's roughly two-minute observation delay: +The new primary inherits the source's backup and WAL history but initially has +no backup under its own backup-instance name. The operator therefore keeps the +WAL cleaner disabled until the new primary completes its first backup and +establishes its own retention point. -First, record the primary pod's current UID: +After that backup, the operator restores the configured WAL-cleaner setting. +Because `replication.primary.cleaner.enabled` is not reloadable, applying the +setting recreates the primary pod once. The replacement reuses the same PVC, but +expect a brief interruption to writes. With the hourly schedule in this guide, +allow one schedule interval plus the operator's roughly two-minute observation +delay. + +First, record the primary pod's current UID. The replacement keeps the same pod +name, so the UID is how you distinguish it from the original pod: ```sh kubectl get pod -1 -n \ @@ -390,8 +399,8 @@ kubectl get questdbcluster -n \ Confirm that the completion time is later than the cutover start time recorded in step 5. -Releasing the WAL cleaner rolls the primary once. Watch the pod until it has a -new UID and is ready, then press Control-C: +After the backup completes, watch the pod until it has a new UID and is ready, +then press Control-C: ```sh kubectl get pod -1 -n --watch \ @@ -401,6 +410,8 @@ kubectl get pod -1 -n --watch \ After the new pod appears, repeat the writer-health check from the previous step and require it to settle before declaring the migration complete. -For emergency source loss, multiple followers, or detailed failure handling, use -the full -[high-availability migration runbook](/docs/enterprise-kubernetes-operator/high-availability/#migrate-an-existing-questdb-onto-the-operator). +With multiple follower instances, promote the healthy instance you verified in +step 4. For source loss, see +[Emergency promotion](/docs/enterprise-kubernetes-operator/high-availability/#emergency-promotion). +For cutover problems, see +[If promotion stalls or fails](/docs/enterprise-kubernetes-operator/high-availability/#if-promotion-stalls-or-fails). diff --git a/documentation/enterprise-kubernetes-operator/high-availability.md b/documentation/enterprise-kubernetes-operator/high-availability.md index 35205629f..325e9fd07 100644 --- a/documentation/enterprise-kubernetes-operator/high-availability.md +++ b/documentation/enterprise-kubernetes-operator/high-availability.md @@ -233,148 +233,9 @@ cutover. Contact support before considering it. ## Migrate an existing QuestDB onto the operator -A follower reduces migration downtime by restoring the source's backup and -consuming its WAL while the external source continues serving. +Use the [migration guide](/docs/enterprise-kubernetes-operator/getting-started/migrate/) +for the canonical replica-first migration procedure. -### Source prerequisites - -The source must: - -- run a compatible QuestDB Enterprise version; -- create completed backups in a known object-store backup root; -- upload replication WAL to a known root in the same store; -- retain WAL back to the seed backup; -- expose its exact `SELECT backup_instance_name();` value, matching - `^[a-z0-9]+(-[a-z0-9]+)*$`; and -- support a controlled stop and one final `primary-catchup-uploads` run. - -The operator never connects to, configures, stops, fences, or lists storage for -the source. - -### Create and verify the follower - -All source selectors are immutable. Confirm `sourceInstanceName`, backup root, -and WAL root before creating the cluster. The source instance name must be -copied exactly and match the lowercase hyphen-separated engine identity format. -Copy the working tenant cluster's `spec.image` and `spec.imagePullSecrets`: set -`` to that private image and -`` to the pull Secret in this namespace. Remove the -entire `imagePullSecrets` block only when every node has ambient pull access, -such as an authorized EKS worker-node role. - -```yaml -apiVersion: questdb.io/v1alpha1 -kind: QuestDBCluster -metadata: - name: - namespace: -spec: - image: - imagePullSecrets: - - name: - instances: 2 - storage: - storageClassName: - size: 100Gi - objectStoreRef: - name: - backup: - enabled: true - schedule: "0 * * * *" - root: - replication: - root: - bootstrap: - follow: - sourceInstanceName: -``` - -A healthy follower has `.status.replication.following=true`, no current primary, -an empty `-rw`, and reads through `-ro`. It intentionally omits -`WriteHealthy` because it has no primary; do not wait for that condition. Verify -all instances and `ReplicationHealthy`; a quiet source can make progress -indeterminate, so also confirm the immutable source identity and roots directly -against the source configuration. - -```sh -kubectl get questdbcluster -n \ - -o jsonpath='following={.status.replication.following}{" primary="}{.status.currentPrimary}{"\n"}{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{.status.replication.stream}{"\n"}' -kubectl get endpointslice -n \ - -l kubernetes.io/service-name=-rw -``` - -### Cut over - -Immediately before stopping the source, record when cutover preparation began in -the same Bash shell you will use for the post-cutover check: - -```bash -CUTOVER_TIME_CAPTURED=false -CUTOVER_STARTED_AT="" -if CUTOVER_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" && \ - [ -n "$CUTOVER_STARTED_AT" ]; then - CUTOVER_TIME_CAPTURED=true -fi -[ "$CUTOVER_TIME_CAPTURED" = true ] && \ - printf 'Cutover preparation started at: %s\n' "$CUTOVER_STARTED_AT" -``` - -1. Stop application writes to the source, then stop the source database. -2. Restart the source once with `replication.role=primary-catchup-uploads`. -3. Wait for the source process to exit with code `0`, then prevent it from - restarting. This final upload may itself be unbounded; supervise it at the - source. A non-zero exit means the tail was not safely uploaded; do not promote. -4. Create a `Planned` `QuestDBPromotion` for a healthy follower instance and use - the bounded watcher above. - -The planned follower gate requires the source stream to remain quiet for at -least 60 seconds and the target to consume the published WAL before promotion. -It still cannot prove an idle source process is stopped. If the engine reports -`SourceStillOwnsStore`, stop the source fully and retry with a new promotion -object. - -:::danger Emergency follower promotion accepts losing source WAL that was not -uploaded or not consumed. Use it only when the source cannot be drained and that -loss is explicitly accepted. ::: - -5. After a completed cutover, verify `-rw`, data, writes, and - `currentPrimary`. -6. Permanently decommission the source so it cannot restart and contend for the - adopted WAL root. -7. Wait for this cluster's own first post-cutover backup. The hourly deadline - must exceed one schedule interval plus the roughly two-minute observation - delay, so this check allows about 75 minutes. Do not accept an old - `completed` status: require a non-empty `endTime` later than the captured - cutover-start time. Both values are RFC3339 UTC timestamps, so the Bash - string comparison proves the observed backup completed after cutover - preparation began. The WAL cleaner remains off until this backup completes; - its release rolls the primary once. Verify the roll and writer readiness - afterward. - -```bash -BACKUP_VERIFIED=false -STATUS="" -END_TIME="" -for _ in $(seq 1 450); do - STATUS="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.status}')" - END_TIME="$(kubectl get questdbcluster -n \ - -o jsonpath='{.status.backup.lastBackup.endTime}')" - if [ "${CUTOVER_TIME_CAPTURED:-false}" = true ] && \ - [ "$STATUS" = "completed" ] && [ -n "$END_TIME" ] && \ - [[ "$END_TIME" > "$CUTOVER_STARTED_AT" ]]; then - BACKUP_VERIFIED=true - break - fi - [ "$STATUS" = "failed" ] && break - sleep 10 -done -[ "$BACKUP_VERIFIED" = true ] && \ - kubectl get questdbcluster -n \ - -o jsonpath='{range .status.conditions[?(@.type=="Available")]}Available={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="Progressing")]}Progressing={.status}{"/"}{.reason}{"\n"}{end}{range .status.conditions[?(@.type=="WriteHealthy")]}WriteHealthy={.status}{"/"}{.reason}{"\n"}{end}' -``` - -After the roll, require `Available=True/PrimaryReady`, -`Progressing=False/Settled`, and `WriteHealthy=True/Healthy` at the current -generation before declaring the cluster writer-ready and the WAL cleaner -released. +If the source is lost before it can be drained, see +[Emergency promotion](#emergency-promotion). For diagnosis and recovery from a +cutover problem, see [If promotion stalls or fails](#if-promotion-stalls-or-fails).