diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 580a50450a..dd8832ec9e 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -21,6 +21,8 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [SWITCH ROLE](/docs/query/sql/switch-role/) - New reference page for switching a QuestDB Enterprise node between the primary and replica roles in place, with `SWITCH STATUS`, the timeout semantics, the statement errors, and the `SWITCH ROLE` permission +- [Failover and role switch](/docs/high-availability/failover/) - Operator guide for planned switchovers and promotions without a restart: prerequisites, the timeout budget, refusals and the `UNKNOWN` state, what each client protocol sees during a demote, the restart hazard, and the `GET /lifecycle` and `POST /lifecycle/switch` endpoints - [Cold storage](/docs/concepts/cold-storage/) - Historical partitions move to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store as Parquet and stay queryable with normal SQL, covering the partition lifecycle, the range-read path, the immutability boundary at `DROP LOCAL`, and the manager and refresher roles - [Operating cold storage](/docs/operations/cold-storage/) - Setup guide and runbook: preparing the object store prefix, configuring the cluster, the two-step manager handoff, remote garbage collection, cold-read metrics, backup interaction, and a troubleshooting table - [Cold storage configuration](/docs/configuration/cold-storage/) - Every `cold.storage.*` setting, covering the store connection, roles, upload, read path, garbage collection, and request timeouts @@ -45,6 +47,11 @@ This page tracks significant updates to the QuestDB documentation. ### Reference +- Added the [`node_role()`](/docs/query/functions/meta/#node_role) function, which reports the replication role of the instance to any authenticated session +- [RBAC](/docs/security/rbac/#failover-operator) - Added the `SWITCH ROLE` permission to the permissions table, a failover operator scenario, and the built-in admin's break-glass role in a replicated cluster +- [Minimal HTTP server](/docs/operations/logging-metrics/#lifecycle-endpoint) - Documented the `GET /lifecycle` component snapshot and the [`http.health.check.authentication.required`](/docs/configuration/http-min-server/#httphealthcheckauthenticationrequired) setting +- [Replication metrics](/docs/operations/logging-metrics/#replication-metrics) - Added `questdb_replication_pending_upload_txn` and `questdb_backup_active_at_last_demote` +- [Replication configuration](/docs/configuration/database-replication/#replicationrole) - Documented that `replication.role` is the boot role, the `primary-catchup-uploads` value, and the restart hazard after an in-place switch - [Storage Policy](/docs/concepts/storage-policy/) - The `TO REMOTE` and `DROP REMOTE` stages are now enforced. Documented what each does, the requirement for a WAL table with cold storage enabled, and the warning that `DROP LOCAL` makes a partition read-only for good - Added the [`table_cold_partitions()`](/docs/query/functions/meta/#table_cold_partitions) function, which returns the remote state of every partition in a table's manifest as it walks `pending`, `live`, `sealed`, and `deleting` - [SHOW PARTITIONS](/docs/query/sql/show/#show-partitions) - Two trailing columns, `seqTxn` and `isRemotelyServed`, are appended to the result set. Tools binding columns by position must be updated @@ -80,6 +87,8 @@ This page tracks significant updates to the QuestDB documentation. ### Updated +- [Replication setup guide](/docs/high-availability/setup/#migration-procedures) - Planned primary migration now points at the in-place switch; the restart-based flow is kept for older versions and the emergency migration is marked as the lossy path +- [Error codes](/docs/troubleshooting/error-codes/#er005) - ER005 now covers the refusal of an in-place promotion, and ER006 the restart of a demoted node with a stale `replication.role` - Client libraries rewritten for the QWP binary protocol, unifying ingestion and streaming SQL queries under one handle: [Java](/docs/connect/clients/java/), [Python](/docs/connect/clients/python/), [Go](/docs/connect/clients/go/), [C & C++](/docs/connect/clients/c-and-cpp/), [Rust](/docs/connect/clients/rust/), and [.NET](/docs/connect/clients/dotnet/) - [Web Console](/docs/getting-started/web-console/overview/) - Documented query sharing by link and tab import/export in the [code editor](/docs/getting-started/web-console/code-editor/), custom AI providers and per-provider permission levels in [QuestDB AI](/docs/getting-started/web-console/questdb-ai/), automatic column sizing in the [result grid](/docs/getting-started/web-console/result-grid/), and the storage policy section in [table details](/docs/getting-started/web-console/table-details/) - [AI coding agents](/docs/getting-started/ai-coding-agents/) - Repositioned around the agent skill and the Web Console MCP bridge together diff --git a/documentation/configuration/database-replication.md b/documentation/configuration/database-replication.md index 4922dad4d2..4a672033dd 100644 --- a/documentation/configuration/database-replication.md +++ b/documentation/configuration/database-replication.md @@ -55,7 +55,28 @@ accepts the `ca_cert_file` and `ca_builtin_roots` TLS parameters. See - **Reloadable**: no Defaults to `none` for stand-alone instances. To enable replication, set to -one of: `primary`, `replica`. +one of: `primary`, `replica`. Values are case-insensitive. + +This is the role the instance boots into. Since QuestDB Enterprise 3.3.3 the +role can also be switched at runtime with +[`SWITCH ROLE`](/docs/query/sql/switch-role/), which does not update this +setting. + +:::danger + +After a runtime switch, set `replication.role` to the new role on both nodes +before either of them restarts. A demoted node that restarts with +`replication.role=primary` can come back as a second primary on the same object +store, and a promoted node that restarts with `replication.role=replica` +silently demotes itself. See +[Restarts](/docs/high-availability/failover/#restarts). + +::: + +`primary-catchup-uploads` is a one-shot maintenance value: the instance starts +as a primary, uploads every pending transaction, and exits. It is used by the +restart-based +[planned primary migration](/docs/high-availability/setup/#planned-primary-migration). ### replication.summary.interval diff --git a/documentation/configuration/http-min-server.md b/documentation/configuration/http-min-server.md index 30772f28d8..24f6095895 100644 --- a/documentation/configuration/http-min-server.md +++ b/documentation/configuration/http-min-server.md @@ -3,10 +3,29 @@ title: Minimal HTTP server description: Configuration settings for the minimal HTTP server in QuestDB. --- -The minimal HTTP server provides the health check and Prometheus metrics -endpoints, running on a separate port (default 9003) from the main HTTP server. -This lightweight server remains responsive even when the main server is under -heavy load. +The minimal HTTP server provides the health check, Prometheus metrics, and +lifecycle endpoints, running on a separate port (default 9003) from the main +HTTP server. This lightweight server remains responsive even when the main +server is under heavy load. + +It serves `GET /status` (any unbound path answers the same health check), +`GET /metrics` when metrics are enabled, `GET /lifecycle`, and in QuestDB +Enterprise `POST /lifecycle/switch`. See the +[minimal HTTP server](/docs/operations/logging-metrics/#minimal-http-server) +section for the health check and lifecycle endpoints, and +[Failover and role switch](/docs/high-availability/failover/#lifecycle-api) for +the switch endpoint. TLS for this port is configured with the `http.min.tls.*` +settings on the [TLS](/docs/configuration/tls/) page. + +## http.health.check.authentication.required + +- **Default**: `true` +- **Reloadable**: no + +Whether `GET /status`, `GET /metrics`, and `GET /lifecycle` require +authentication when the HTTP server does. Set to `false` to let a load balancer +or a Kubernetes probe call them without credentials. `POST /lifecycle/switch` +always requires credentials, regardless of this setting. ## http.min.bind.to diff --git a/documentation/configuration/http-server.md b/documentation/configuration/http-server.md index 470fba3e0c..cb8971ee49 100644 --- a/documentation/configuration/http-server.md +++ b/documentation/configuration/http-server.md @@ -320,6 +320,11 @@ Limit the number of response rows over HTTP. Forces HTTP read-only mode when `true`, disabling commands which modify data or data structure (e.g. INSERT, UPDATE, CREATE TABLE). +Read-only mode is a data boundary, not an administrative lockdown. With access +control disabled, functions such as `reload_config()` and, in QuestDB +Enterprise, the [role switch](/docs/high-availability/failover/) statements and +endpoint stay available on a read-only interface. + ## Query cache ### http.query.cache.block.count diff --git a/documentation/enterprise-kubernetes-operator/high-availability.md b/documentation/enterprise-kubernetes-operator/high-availability.md index 307826e7d6..48da9305a4 100644 --- a/documentation/enterprise-kubernetes-operator/high-availability.md +++ b/documentation/enterprise-kubernetes-operator/high-availability.md @@ -45,6 +45,17 @@ until an explicit promotion; fencing is not automatic failover. ## Promotion and failover +:::note + +The operator performs a cutover by changing configuration and re-rolling pods. +It does not use the database's in-place +[role switch](/docs/high-availability/failover/) (`SWITCH ROLE`, +`POST /lifecycle/switch`). Do not run those against instances managed by the +operator: `status.currentPrimary` and the PVC role labels would no longer +describe the cluster. + +::: + | Mode | Use | Data effect | | ----------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Planned` | Healthy primary and live replica | Drains the old primary and verifies the target before promotion. Fails rather than becoming lossy. | diff --git a/documentation/high-availability/client-failover/concepts.md b/documentation/high-availability/client-failover/concepts.md index dd31c02d6f..be95b64a9b 100644 --- a/documentation/high-availability/client-failover/concepts.md +++ b/documentation/high-availability/client-failover/concepts.md @@ -16,7 +16,8 @@ import { EnterpriseNote } from "@site/src/components/EnterpriseNote" When a QuestDB cluster fails over from one primary to another — whether through -a planned promotion, a rolling upgrade, or an unplanned outage — clients with a +a planned [in-place promotion](/docs/high-availability/failover/), a rolling +upgrade, or an unplanned outage — clients with a single hard-coded address must be reconfigured and restarted. A failover-aware client instead carries the full list of peers and walks that list automatically when the current connection breaks. diff --git a/documentation/high-availability/failover.md b/documentation/high-availability/failover.md new file mode 100644 index 0000000000..93f58855d3 --- /dev/null +++ b/documentation/high-availability/failover.md @@ -0,0 +1,393 @@ +--- +title: Failover and role switch +sidebar_label: Failover +description: + Switch a QuestDB Enterprise node between the primary and replica roles in + place, promote a replica after a primary loss, and recover from a refused + switch. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + In-place role switching is a QuestDB Enterprise feature, available since + version 3.3.3. This page describes the behaviour of version 4.0.0. + + +A replicated cluster has one primary that accepts writes and one or more +replicas that follow it through the object store. `SWITCH ROLE` moves the +primary role between nodes while both keep running: no JVM restart, the health +endpoints keep answering `200`, and most client connections survive. Use it for +a planned switchover before maintenance, for a rolling upgrade, and to promote +a caught-up replica after the primary is lost. + +This page is the operator guide. The statement grammar is on the +[`SWITCH ROLE`](/docs/query/sql/switch-role/) reference page. + +## Roles at a glance + +| Role | Writes | Replication | +| --------- | -------- | ---------------------------------------------------- | +| `PRIMARY` | accepted | uploads WAL transactions to the object store | +| `REPLICA` | refused | downloads and applies WAL transactions | +| `UNKNOWN` | refused | none. A switch was aborted part-way, see [below](#refusals-and-the-torn-state) | + +Three surfaces report the role. Use the first two for monitoring; the third +needs the `SWITCH ROLE` permission. + +| Surface | Access | Reports | +| ----------------------------------------- | ------------------------------------------------- | ---------------------------------------------------- | +| [`node_role()`](/docs/query/functions/meta/#node_role) | any authenticated SQL session | the role | +| [`GET /lifecycle`](#lifecycle-api) | port 9003, health-check credentials | `currentRole`, `switchInFlight`, `ready`, components | +| [`SWITCH STATUS`](/docs/query/sql/switch-role/) | SQL session with `SWITCH ROLE` | `current_role`, `switch_in_flight`, `captured_at` | + +## Prerequisites + +- QuestDB Enterprise 4.0.0 or later on every node. Version 3.3.3 introduced the + switch; 4.0.0 added the `SWITCH ROLE` permission and the refusal of a demote + that would abandon un-uploaded transactions. +- Both nodes configured for [replication](/docs/high-availability/setup/) with + the same `replication.object.store`. +- The replica caught up. On the primary, the + `questdb_replication_pending_upload_txn` metric reads `0`; on the replica, no + table is suspended and the WAL lag is nil. +- An account with the `SWITCH ROLE` permission and with `PGWIRE` (for SQL) or + `HTTP` (for the REST endpoint). See + [Failover operator](/docs/security/rbac/#failover-operator). +- Clients configured with a multi-host address list, so writers follow the + primary role on their own. See + [Client failover](/docs/high-availability/client-failover/concepts/). + +The [cold storage](/docs/concepts/cold-storage/) manager role is independent of +the replication role and does not move with it. If the cluster is managed by +the [Kubernetes Operator](/docs/enterprise-kubernetes-operator/high-availability/), +use its promotion object instead; the operator does not use the in-place +switch and does not expect roles to move underneath it. + +## Planned switchover + +Demote first, then promote. The object store accepts one owner at a time, so a +promotion issued while the old primary still owns the store is refused; the +demote releases that ownership. + +1. Confirm the primary has nothing pending: `questdb_replication_pending_upload_txn` + is `0` on the primary's `/metrics`. +2. On the primary, demote it. A busy primary needs a budget larger than the + default 5 seconds: + + ```questdb-sql + SWITCH ROLE TO REPLICA TIMEOUT 60000; + ``` + + New writes are refused from this moment; connected clients stay connected + and receive `replica access is read-only` on their next write. + +3. Poll until the switch settles: + + ```questdb-sql + SWITCH STATUS; + ``` + + | current_role | switch_in_flight | captured_at | + | ------------ | ---------------- | --------------------------- | + | REPLICA | false | 2026-08-28T10:15:02.114233Z | + + If `current_role` is `UNKNOWN`, stop here and follow + [Refusals and the torn state](#refusals-and-the-torn-state). + +4. On the replica, promote it: + + ```questdb-sql + SWITCH ROLE TO PRIMARY; + ``` + +5. Poll `SWITCH STATUS` on the new primary until `current_role` is `PRIMARY`, + then check that `GET /lifecycle` reports `"ready":true` and run a test write. +6. Update `replication.role` in `server.conf` on both nodes, see + [Restarts](#restarts). + +Writers with a multi-host list reconnect to the new primary on their own. A +writer pointed at a single address must be repointed. + +## Promote a replica after a primary loss + +1. Make sure the failed primary is stopped and cannot restart as a primary. If + its disk survives, set `replication.role=replica` in its `server.conf` + before it is ever started again. +2. On the replica: + + ```questdb-sql + SWITCH ROLE TO PRIMARY; + ``` + +3. Poll `SWITCH STATUS` and `GET /lifecycle` as above. + +The promotion is refused if the replica has not yet applied everything that is +in the object store. The node then stays a replica and keeps downloading; +promote it again once it has caught up. A promotion never accepts data loss: +transactions the failed primary committed but never uploaded are not in the +store, and the in-place switch cannot recover them. When the loss is accepted +and the replica cannot catch up, use the restart-based +[emergency primary migration](/docs/high-availability/setup/#emergency-primary-migration). + +Once the old primary is repaired, start it with `replication.role=replica`. It +rejoins as a replica; any transactions it committed but never uploaded are +superseded by the new primary's stream. + +## Choosing the timeout + +`TIMEOUT` is the budget for each stage of the switch: the drain of in-flight +writers, the upload of pending transactions, the uploader shutdown, and the +materialized view quiesce. A demote spanning many tables shares one deadline +across them. The default is 5 seconds and cannot be changed in configuration; +the ceiling is 600 seconds, chosen to stay inside a typical Kubernetes +termination grace period. + +Two things happen when the budget runs out during a demote: + +- Client writers still busy when the drain expires: the demote is refused and + the node stays a writable primary. +- Committed transactions still not uploaded when the upload drain expires: the + demote is not completed and the node parks in the `UNKNOWN` state rather + than abandoning acknowledged writes. + +Size the budget from what the primary is doing. A quiet primary settles well +under a second; a primary under sustained ingestion with a backlog on the +uploader needs tens of seconds. Checking +`questdb_replication_pending_upload_txn` before the demote removes most of the +guesswork. + +A promotion rarely needs a long budget: its slow part is the object-store +ownership check, which is bounded by the request timeouts of the store. + +## Refusals and the torn state + +A switch is accepted immediately and can still fail while it runs. The +session that submitted it does not learn about the failure; `SWITCH STATUS`, +`GET /lifecycle` and the server log do. + +| Outcome | Role afterwards | How to see it | What to do | +| ---------------------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| Busy writers outlived the timeout | unchanged, still a writable primary | `GET /lifecycle` reports `"ready":false` with the `replication` component `DEGRADED`; the log says `switch refused: drain budget expired with busy writers` | Quiet the writers or retry with a larger `TIMEOUT` | +| Promotion refused: the replica is behind the store | unchanged, still replicating | `"ready":false`, `replication` component `DEGRADED`, [ER005](/docs/troubleshooting/error-codes/#er005) text in the log | Wait for the replica to catch up, `RESUME WAL` any suspended table, retry | +| Boot in progress, switch in flight, shutting down | unchanged | Statement error, or `503` / `409` from the REST endpoint | Retry later | +| Demote could not upload every committed transaction | `UNKNOWN`: read-only, not replicating | `node_role()` returns `UNKNOWN`, `"ready":false`, log names the pending transaction count | Heal, see below | +| Any other failure part-way through the cascade | `UNKNOWN` | same | Heal, see below | + +A node in the `UNKNOWN` state refuses writes, does not upload or download, and +advertises itself as a replica to clients, so no writer selects it and no +coordinator can mistake it for a healthy primary. + +To heal it: + +1. Switch it back toward primary. This rebuilds the uploader, which resumes + shipping the backlog: + + ```questdb-sql + SWITCH ROLE TO PRIMARY; + ``` + + Either direction is accepted from `UNKNOWN`, and the node lands on the + target of the retry if the cascade completes. Prefer `PRIMARY` when the + cause was pending uploads. + +2. Remove the cause: run `RESUME WAL` on a suspended table, restore + connectivity to the object store, and so on. +3. Wait for `questdb_replication_pending_upload_txn` to reach `0`. +4. Retry the demote with a larger `TIMEOUT`. + +If a table's backlog cannot be shipped and losing it is acceptable, add the +table to the reloadable +[`replication.disabled.tables`](/docs/configuration/database-replication/#replicationdisabledtables) +setting and retry the demote. That is the only way to make a demote abandon +committed transactions. + +:::warning + +Never restart a node in the `UNKNOWN` state with `replication.role=replica`. +A replica boot performs no upload audit and completes the abandoned demote, +discarding the backlog the refusal protected. Restart it as `primary`, which +replays the pending uploads at boot. + +::: + +## Restarts + +`SWITCH ROLE` changes the role in memory only. Nothing is written to +`server.conf` or to any other file, and on the next start the node boots into +the role given by +[`replication.role`](/docs/configuration/database-replication/#replicationrole): +`replica` boots as a replica, every other value boots as a primary. + +:::danger + +Set `replication.role` to the node's current role on both nodes as soon as a +switch has settled, before either node can restart. + +A demoted node that restarts with `replication.role=primary` can come back as a +second primary. While it ran as a replica it kept its object-store ownership +token current, so the ownership check at boot passes when it was caught up. The +conflict is only detected later, as +[ER006](/docs/troubleshooting/error-codes/#er006), and the node that detects it +halts, which may be the legitimate primary. + +A promoted node that restarts with `replication.role=replica` silently demotes +itself. Transactions it committed as primary but had not yet uploaded are +superseded by the store, and if the other node is also a replica the cluster +has no primary and reports no error. + +::: + +## What clients see + +The switch never closes a socket by itself. A demote refuses writes, drains the +in-flight writers, and swaps the authorization backend; the protocol servers are +not told anything. What a connected client observes depends on its protocol. + +| Protocol | Connection through a demote | Write in the demote window | +| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| PGWire | kept open, not re-authenticated. The same session writes again after a later promotion | Error `replica access is read-only` for INSERT, UPDATE, ALTER, TRUNCATE, RENAME, CREATE and DROP, view and storage policy changes. A `COMMIT` of a transaction opened before the demote is refused with the same error and the transaction is rolled back | +| HTTP `/exec`, Web Console | kept open | HTTP `403` with `"error":"replica access is read-only"`, same statement set | +| ILP over HTTP | kept open | HTTP `421` with `"code":"not accepting writes"`. A request already in flight when the demote lands gets `403` with `"code":"unauthorised"` and the read-only message | +| ILP over TCP | kept open | Rows are **silently dropped**: the protocol has no error channel and the server only logs `commit failed … replica access is read-only`. The socket is closed only when a refusal reaches the connection, for instance a write to a table first seen after the demote, and only when [`line.tcp.disconnect.on.error`](/docs/configuration/ingestion/#linetcpdisconnectonerror) is `true` | +| ILP over UDP | connectionless | Buffered rows are dropped, with no client-visible signal | +| QWP ingestion | closed with WebSocket code `1000` and reason `replica access is read-only` on the first refused frame; idle connections are not closed. For durable acknowledgements the close waits, bounded by 10 seconds, until the committed work is durably uploaded and the final ack is flushed | The refused frame is not acknowledged and is replayed by the client after reconnecting. On reconnect the node answers `421` with `X-QuestDB-Role: REPLICA` and the client walks to the next host | +| QWP queries, PGWire and HTTP `SELECT` | kept open; an in-flight result keeps streaming its snapshot | not affected. The `target=` role filter of a query client is evaluated only when it connects | + +ILP over TCP is the one protocol where a demote loses data without telling the +client. Move ingestion that must survive a switchover to ILP over HTTP or to +QWP with [store-and-forward](/docs/high-availability/store-and-forward/concepts/). + +A demoted node advertises itself as a replica to the QWP role gate, and so does +a node in the `UNKNOWN` state. Sessions keep their identity across a switch, +but an assumed service account does not survive it. + +## After the switch + +- Update `replication.role` on both nodes, see [Restarts](#restarts). +- Scheduled backups run on the primary only. The schedule pauses on the + demoted node and resumes on the promoted one. A backup that was still running + when the node was demoted is reported by the + `questdb_backup_active_at_last_demote` metric, see + [Backup and restore](/docs/operations/backup/#schedules-on-a-replica). +- Materialized views are refreshed by the new primary from where the old one + left off. +- [Storage policies](/docs/concepts/storage-policy/) are enforced on every node + regardless of role; nothing changes there. +- The cold storage manager role stays where it was. Move it separately with + [`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) if + the demoted node is being retired. + +## Lifecycle API + +The [minimal HTTP server](/docs/operations/logging-metrics/#minimal-http-server) +on port 9003 exposes the same switch to external coordinators. It accepts the +same credentials as the main HTTP server: HTTP basic authentication, a +[REST token](/docs/connect/compatibility/rest-api/#authentication-via-token-in-questdb-enterprise) +as `Authorization: Bearer`, or an OIDC access token. The principal needs the +`HTTP` endpoint permission and, for the switch, `SWITCH ROLE`. With access +control disabled no credentials are needed. TLS for this port is configured with +the `http.min.tls.*` settings on the [TLS](/docs/configuration/tls/) page. + +### GET /lifecycle + +Returns the role and the state of every server component: + +```shell +curl -H "Authorization: Bearer $QDB_REST_TOKEN" \ + https://primary.example.com:9003/lifecycle +``` + +```json +{ + "capturedAtMicros": 1756380000123456, + "currentRole": "PRIMARY", + "switchInFlight": false, + "ready": true, + "components": [ + { + "name": "engine", + "state": "READY", + "lastTransitionMicros": 1756379991204000, + "latestProgress": null, + "hardRequiredDependencies": ["factory-provider", "backup-restore"], + "softDependencies": [] + }, + { + "name": "replication", + "state": "READY", + "lastTransitionMicros": 1756379991650000, + "latestProgress": null, + "hardRequiredDependencies": ["engine"], + "softDependencies": [] + } + ] +} +``` + +- `currentRole` is `PRIMARY`, `REPLICA`, or `UNKNOWN`. During a switch it keeps + the previous role. +- `switchInFlight` is `true` while a switch is running. +- `ready` is `false` while any component is `DEGRADED` or `FAILED`: during + boot, and after a refused or torn switch until a later switch succeeds. The + health check on `/status` stays `200` throughout, so a readiness probe should + read this flag rather than the health check. +- `components` lists every server component, in registration order, with its + `state` (`INIT`, `STARTING`, `DEGRADED`, `READY`, `SWITCHING`, `STOPPING`, + `STOPPED`, `FAILED`), its last transition time, and its dependencies. During + a switch the role-aware components move through `SWITCHING` one at a time. + +This endpoint follows +[`http.health.check.authentication.required`](/docs/configuration/http-min-server/#httphealthcheckauthenticationrequired): +set it to `false` to let a probe read the role without credentials. The +response is chunked. QuestDB open source serves the same endpoint without the +three role fields. + +### POST /lifecycle/switch + +Submits a switch. `role` is `primary` or `replica` (case-insensitive); +`timeout_ms` is optional and follows the same rules as the SQL `TIMEOUT`: + +```shell +curl -X POST -H "Authorization: Bearer $QDB_REST_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"role":"replica","timeout_ms":60000}' \ + https://primary.example.com:9003/lifecycle/switch +``` + +| Status | Body | Meaning | +| ------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `202` | `{"accepted":true}` | Submitted. Poll `GET /lifecycle` | +| `400` | `{"error":"invalid_request","message":"…"}` | Missing or invalid `role`, `timeout_ms` out of range, unknown or duplicate field, nested object | +| `401` | `Unauthorized` | No or invalid credentials. This endpoint always requires them | +| `403` | `{"error":"permission_denied"}` | The principal lacks `SWITCH ROLE`. Audit-logged | +| `405` | `{"error":"method_not_allowed","message":"POST required"}` | Not a `POST` | +| `409` | `{"error":"switch_in_flight","current_role":"PRIMARY","target_role":"REPLICA"}` | A switch is already running | +| `413` | `{"error":"request_entity_too_large","message":"…"}` | Body larger than 8 KiB | +| `500` | `{"error":"internal_error"}` | The authorization check failed for a reason other than a denial. See the server log | +| `503` | `{"error":"service_unavailable","message":"…"}` | `boot in progress; switch not yet available`, `server is shutting down`, or `server is busy`. Retry | + +The `400` messages name the problem: `role required and must be one of +[primary, replica]`, `timeout_ms must be integer`, `timeout_ms must be positive; +0 strands committed-but-not-uploaded WAL`, `timeout_ms must not exceed 600000 +ms`, `unknown field: …`. + +A coordinator drives a switch as follows: + +1. `GET /lifecycle` on the target node. Do not act on a stale read, and treat + `UNKNOWN` on any node as a signal not to promote another one. +2. `POST /lifecycle/switch`. Treat `409` and `503` as "retry later". +3. Poll `GET /lifecycle` until `switchInFlight` is `false` and `currentRole` + is the target. Then check `ready`; `false` means the switch was refused and + the log has the reason. + +## Next steps + +- [`SWITCH ROLE`](/docs/query/sql/switch-role/) reference, with the statement + errors and the result columns. +- [Replication setup guide](/docs/high-availability/setup/) for the + restart-based migration procedures and point-in-time recovery. +- [Client failover](/docs/high-availability/client-failover/concepts/) for how + clients follow the primary role. +- [Failover operator](/docs/security/rbac/#failover-operator) for the account + that runs switches. diff --git a/documentation/high-availability/overview.md b/documentation/high-availability/overview.md index a70f43e885..46d6a9209a 100644 --- a/documentation/high-availability/overview.md +++ b/documentation/high-availability/overview.md @@ -64,11 +64,15 @@ This decoupled architecture means: - Add or remove replicas without touching the primary - Replicas can be in different regions or availability zones - Object store provides durability and point-in-time recovery +- The primary role moves between nodes at runtime, see + [Failover and role switch](/docs/high-availability/failover/) ## Availability strategies **Hot availability** - Run replicas continuously alongside the primary for -instant failover. Faster recovery, higher cost. +instant failover. Faster recovery, higher cost. A replica is promoted in place +with [`SWITCH ROLE`](/docs/query/sql/switch-role/), without restarting either +node. **Cold availability** - Reconstruct a new primary from the latest snapshot and WAL files when needed. Slower recovery, lower cost. @@ -142,6 +146,8 @@ by you. - [Setup Guide](/docs/high-availability/setup/) — configure object storage, the primary, and replica nodes. +- [Failover and role switch](/docs/high-availability/failover/) — switch roles + in place, promote a replica, and recover from a refused switch. - [Client failover](/docs/high-availability/client-failover/concepts/) — configure your applications to follow a primary promotion automatically. - [Store-and-forward](/docs/high-availability/store-and-forward/concepts/) — diff --git a/documentation/high-availability/setup.md b/documentation/high-availability/setup.md index a1820eb605..aa7ce66984 100644 --- a/documentation/high-availability/setup.md +++ b/documentation/high-availability/setup.md @@ -259,7 +259,7 @@ configuration options, tuning, and troubleshooting. | Node | Recoverable | Unrecoverable | | ------- | ----------- | ----------------------------------- | -| Primary | Restart | Promote replica, create new replica | +| Primary | Restart | [Promote a replica](/docs/high-availability/failover/#promote-a-replica-after-a-primary-loss), create new replica | | Replica | Restart | Destroy and recreate | ### Network partitions @@ -290,7 +290,14 @@ If the cluster uses [cold storage](/docs/concepts/cold-storage/), the manager ro ### Planned primary migration -Use when the current primary is healthy but you want to switch to a new one. +Since QuestDB Enterprise 3.3.3, the primary role moves without stopping either +node: demote the primary with `SWITCH ROLE TO REPLICA`, then promote the +replica with `SWITCH ROLE TO PRIMARY`. Clients stay connected and no data is +lost. The procedure, its timeout, and what to do when a switch is refused are in +[Failover and role switch](/docs/high-availability/failover/). + +On older versions, or when the object store changes at the same time, use the +restart-based flow: 1. Stop the primary 2. Restart with `replication.role=primary-catchup-uploads` @@ -299,7 +306,11 @@ Use when the current primary is healthy but you want to switch to a new one. ### Emergency primary migration -Use when the primary has failed. +Use when the primary has failed and a replica cannot be promoted in place: the +surviving replica is behind the object store and you accept losing the +transactions that never reached it. If a caught-up replica exists, +[promote it in place](/docs/high-availability/failover/#promote-a-replica-after-a-primary-loss) +instead; that path refuses rather than losing data. 1. Stop the failed primary (ensure it cannot restart) 2. Stop the replica diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 5198f1a97b..206e7888ed 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -232,6 +232,14 @@ SELECT reload_config(); You can also use this to enable and disable the schedule by adding or commenting out the `backup.schedule.cron` config setting. +#### Schedules on a replica + +Scheduled backups run on the primary only. When a node is demoted with +[`SWITCH ROLE`](/docs/query/sql/switch-role/) its schedule pauses, and it +resumes when the node is promoted again. A backup that was still running at the +moment of a demote is reported by the `questdb_backup_active_at_last_demote` +metric, see [Replication metrics](/docs/operations/logging-metrics/#replication-metrics). + ### Backup instance name diff --git a/documentation/operations/logging-metrics.md b/documentation/operations/logging-metrics.md index cb75332a78..240647f3c2 100644 --- a/documentation/operations/logging-metrics.md +++ b/documentation/operations/logging-metrics.md @@ -214,6 +214,47 @@ for the health check service. To do so, increase `http.min.worker.count` to `1`. ::: +#### Lifecycle endpoint + +`GET /lifecycle` on the same port returns the startup and shutdown state of +every server component as JSON, for probes and coordinators that need more +than the `200` of the health check: + +```shell +curl http://127.0.0.1:9003/lifecycle +``` + +```json +{ + "capturedAtMicros": 1756380000123456, + "components": [ + { + "name": "engine", + "state": "READY", + "lastTransitionMicros": 1756379991204000, + "latestProgress": null, + "hardRequiredDependencies": ["factory-provider"], + "softDependencies": [] + } + ] +} +``` + +- `capturedAtMicros` and `lastTransitionMicros` are epoch microseconds. +- `state` is one of `INIT`, `STARTING`, `DEGRADED`, `READY`, `SWITCHING`, + `STOPPING`, `STOPPED`, `FAILED`. +- `latestProgress` is `null`, or a progress object while a component is + restoring data. +- The dependency lists name the components a component waits for. + +The response is chunked and lists every component in registration order. Like +the health check, the endpoint follows +[`http.health.check.authentication.required`](/docs/configuration/http-min-server/#httphealthcheckauthenticationrequired). + +In QuestDB Enterprise the response also carries `currentRole`, `switchInFlight`, +and `ready`, and the port accepts `POST /lifecycle/switch`. See +[Failover and role switch](/docs/high-availability/failover/#lifecycle-api). + ### Environment variables Values in the log configuration file can be overridden with environment @@ -321,6 +362,18 @@ Watch rates and ratios rather than raw totals. Sustained investigating. See [Operating cold storage](/docs/operations/cold-storage/#cold-read-metrics). +### Replication metrics + +_Enterprise only._ + +Two gauges describe the state of an in-place +[role switch](/docs/high-availability/failover/): + +| Metric | Type | Description | +| ------ | ---- | ----------- | +| `questdb_replication_pending_upload_txn` | gauge | Committed transactions not yet uploaded to the object store, summed over the replicated tables. Poll it before demoting a primary: a demote that cannot bring it to zero within its timeout is not completed | +| `questdb_backup_active_at_last_demote` | gauge | `1` if a backup was still running when the node was last demoted, `0` otherwise. Cleared by the next promotion | + ### Prometheus Alertmanager QuestDB includes a log writer that sends any message logged at critical level diff --git a/documentation/operations/upgrade.md b/documentation/operations/upgrade.md index 2e7591db62..f36890607f 100644 --- a/documentation/operations/upgrade.md +++ b/documentation/operations/upgrade.md @@ -92,6 +92,13 @@ store. If a replica running an older version encounters metadata it does not recognize, it will error. Upgrading replicas first ensures they can handle any new metadata the primary produces after its upgrade. +When upgrading to 4.0.0 or later, grant the `SWITCH ROLE` permission only after +every node runs the new version. The permission is stored in a form older +versions cannot read, and a `GRANT ALL` issued before the upgrade does not +include it. Accounts that triggered role switches through `SYSTEM ADMIN` on +3.3.x need an explicit `GRANT SWITCH ROLE`. See +[Failover operator](/docs/security/rbac/#failover-operator). + ### QuestDB Enterprise BYOC For QuestDB Enterprise BYOC customers, QuestDB manages the upgrade process. The diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 9f8ef4aac4..d2156a88fb 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -340,6 +340,55 @@ memory_metrics(); | MMAP_O3 | 0 | | NATIVE_O3 | 96 | +## node_role + +:::note + +Replication and the `node_role()` function are available in **QuestDB +Enterprise** only, since version 4.0.0. + +::: + +`node_role()` returns the [replication](/docs/high-availability/overview/) role +the instance currently holds, so a session can tell whether it is connected to +the primary or to a replica. Any authenticated session can call it; unlike +`SWITCH STATUS`, it needs no permission. + +**Arguments:** + +- `node_role()` does not require arguments. + +**Return value:** + +Returns a string: `PRIMARY`, `REPLICA`, or `UNKNOWN` while an in-place role +switch has been aborted part-way, see +[Failover and role switch](/docs/high-availability/failover/#refusals-and-the-torn-state). +On an instance without replication it returns the role derived from +`replication.role`, which is `PRIMARY` for a standalone instance. + +The value is read once per statement execution, so a cached plan or a prepared +statement observes a role change on its next execution. + +**Examples:** + +```questdb-sql +SELECT node_role(); +``` + +| node_role | +| --------- | +| PRIMARY | + +:::warning + +`node_role()` cannot be used in a materialized view or a live view. Avoid it in +`UPDATE` on a WAL table as well: the statement is re-executed on every node of +a replicated cluster and each node evaluates its own role, so the primary and +its replicas would write different values. Tagging rows on `INSERT` is safe, +because inserted rows replicate as data. + +::: + ## query_activity **Arguments:** diff --git a/documentation/query/sql/switch-role.md b/documentation/query/sql/switch-role.md new file mode 100644 index 0000000000..3c9e3b0028 --- /dev/null +++ b/documentation/query/sql/switch-role.md @@ -0,0 +1,187 @@ +--- +title: SWITCH ROLE +sidebar_label: SWITCH ROLE +description: + SWITCH ROLE and SWITCH STATUS SQL keyword reference documentation. Moves a + running QuestDB Enterprise instance between the primary and replica roles. +--- + +Moves the instance that executes the statement between the +[replication](/docs/high-availability/overview/) primary and replica roles +without a restart, and reports the role the instance currently holds. A demoted +primary stops accepting writes and starts following the object store; a +promoted replica starts uploading and admits writes. This is how a planned +switchover, or a promotion after a primary loss, is performed while clients +stay connected. + +:::note + +Replication and `SWITCH ROLE` are available in **QuestDB Enterprise** only, +since version 3.3.3. The permission and the refusal behaviour described here are +those of version 4.0.0. + +::: + +## Syntax + +```questdb-sql title="Switch role" +SWITCH ROLE TO { PRIMARY | REPLICA } [TIMEOUT milliseconds]; +``` + +```questdb-sql title="Report role" +SWITCH STATUS; +``` + +## Description + +Both statements act on the instance that executes them, not on the cluster. +Issue them against the specific node whose role you want to change or inspect. + +| Clause | Effect | +| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `TO PRIMARY` | Stops the WAL downloader, verifies that this instance owns the object store, starts the uploader, and admits writes last | +| `TO REPLICA` | Refuses new writes first, drains in-flight writers, uploads the transactions still pending, closes the uploader, and starts the downloader | +| `TIMEOUT` | Bounds each stage of the switch, in milliseconds. `1` to `600000`, default `5000` | + +`SWITCH ROLE` returns as soon as the switch is accepted. The switch itself runs +in the background: poll `SWITCH STATUS`, or +[`GET /lifecycle`](/docs/high-availability/failover/#lifecycle-api), until +`switch_in_flight` is `false` and `current_role` matches the target. + +### Result columns + +`SWITCH ROLE` returns a single row: + +| Column | Type | Description | +| ------------- | --------- | ------------------------------------------------------------------ | +| `accepted` | _BOOLEAN_ | Always `true`. A switch that cannot be accepted raises an error instead | +| `target_role` | _STRING_ | `PRIMARY` or `REPLICA` | + +`SWITCH STATUS` returns a single row: + +| Column | Type | Description | +| ------------------ | ----------- | ---------------------------------------------------------------------------------------------- | +| `current_role` | _STRING_ | `PRIMARY`, `REPLICA`, or `UNKNOWN` | +| `switch_in_flight` | _BOOLEAN_ | `true` while a switch is running. `current_role` keeps the previous role until it completes | +| `captured_at` | _TIMESTAMP_ | When the status was read | + +`UNKNOWN` means a switch was aborted part-way and the instance holds neither +role: it is read-only and is not replicating. Retry the switch to heal it, see +[Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). + +### Timeout behaviour + +`TIMEOUT` bounds the stages of the switch, not the calling session. The writer +drain, the upload of pending transactions, the uploader shutdown, and the +materialized view quiesce each settle within the budget, while the statement +returns immediately. + +A demote whose writer drain outlives the budget is refused and the instance +stays primary. A demote whose pending uploads outlive the budget leaves the +instance in the `UNKNOWN` state rather than abandoning acknowledged writes. On a +busy primary, pass an explicit `TIMEOUT` larger than the default 5 seconds. + +The default of `5000` milliseconds is not configurable. The upper bound of +`600000` (10 minutes) keeps a switch inside a typical Kubernetes termination +grace period. The value applies to that switch only. + +This differs from +[`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/), where +`TIMEOUT` bounds the caller's wait and the transition continues regardless. + +### Instances without replication + +The statement is accepted on an instance that has no +`replication.object.store` configured. `SWITCH ROLE TO REPLICA` then leaves the +instance read-only with nothing to replicate from. It is not a way to make a +standalone instance read-only; use the read-only settings of the interfaces +instead. + +:::warning + +A role set this way does **not** survive a restart. The instance boots into the +role given by `replication.role` in `server.conf`. Update that setting on every +switched node before any restart, otherwise a demoted node can come back as a +second primary. See [Restarts](/docs/high-availability/failover/#restarts). + +::: + +### Permissions + +Both statements require the `SWITCH ROLE` permission, granted with +`GRANT SWITCH ROLE TO entity`. `SYSTEM ADMIN` does not imply it; `DATABASE ADMIN` +does. A denied session receives `Access denied for [SWITCH ROLE]`. +When access control is disabled, both statements are open to every session. See +[Failover operator](/docs/security/rbac/#failover-operator). + +Before version 4.0.0, both statements required `SYSTEM ADMIN`. + +### Errors + +Errors raised by the statement itself: + +| Error | Cause | +| ---------------------------------------------------------- | ----------------------------------------------------------------------------- | +| `timeout must be within [1, 600000] ms` | `TIMEOUT` out of range. `0` is rejected because it would abandon pending uploads | +| `switch already in flight [current=PRIMARY, target=REPLICA]` | A switch is running. Poll `SWITCH STATUS` | +| `boot in progress; switch not yet available` | The instance has not finished starting | +| `server is shutting down` | The instance is stopping | +| `server is busy` | The switch executor could not take the request. Retry | + +A switch that is accepted and later refused does not raise an error in the +session that submitted it. The outcome is visible through `SWITCH STATUS`, +`GET /lifecycle`, and the server log. See +[Refusals and the torn state](/docs/high-availability/failover/#refusals-and-the-torn-state). + +## Examples + +A planned switchover runs on two instances, in this order: + +```questdb-sql title="1. Demote the current primary" +SWITCH ROLE TO REPLICA TIMEOUT 60000; +``` + +| accepted | target_role | +| -------- | ----------- | +| true | REPLICA | + +```questdb-sql title="2. Confirm it settled, on the same instance" +SWITCH STATUS; +``` + +| current_role | switch_in_flight | captured_at | +| ------------ | ---------------- | --------------------------- | +| REPLICA | false | 2026-08-28T10:15:02.114233Z | + +```questdb-sql title="3. Promote the replica, on the other instance" +SWITCH ROLE TO PRIMARY; +``` + +```questdb-sql title="4. Confirm it is accepting writes" +SWITCH STATUS; +``` + +| current_role | switch_in_flight | captured_at | +| ------------ | ---------------- | --------------------------- | +| PRIMARY | false | 2026-08-28T10:15:09.771902Z | + +Read the role from any session, without the `SWITCH ROLE` permission: + +```questdb-sql +SELECT node_role(); +``` + +| node_role | +| --------- | +| PRIMARY | + +## See also + +- [Failover and role switch](/docs/high-availability/failover/) for the full + procedure, the REST endpoint, and recovery from a refused switch +- [`node_role()`](/docs/query/functions/meta/#node_role) for the open role read +- [RBAC](/docs/security/rbac/#failover-operator) for the `SWITCH ROLE` permission +- [`replication.role`](/docs/configuration/database-replication/#replicationrole) + for the boot role +- [`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) for + the cold storage manager role, which is independent of the replication role diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index f905e0b8be..5b10f3c266 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -211,6 +211,22 @@ GRANT DATABASE ADMIN TO dba; ::: +### Failover operator + +A service account that can move the primary role between nodes, for an external +coordinator or a runbook, without any other administrative right: + +```questdb-sql +CREATE SERVICE ACCOUNT failover_bot WITH PASSWORD 'pwd'; +GRANT HTTP TO failover_bot; -- POST /lifecycle/switch on port 9003 +GRANT SWITCH ROLE TO failover_bot; -- SWITCH ROLE, SWITCH STATUS, the endpoint +``` + +`SYSTEM ADMIN` is neither required nor sufficient for a role switch, and +`DATABASE ADMIN` includes `SWITCH ROLE`. Monitoring accounts do not need it: +`node_role()` and `GET /lifecycle` are open to any authenticated principal. See +[Failover and role switch](/docs/high-availability/failover/). + ## Core concepts ## SQL commands reference diff --git a/documentation/sidebars.js b/documentation/sidebars.js index d518a599c7..e0a4b89577 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -474,6 +474,7 @@ module.exports = { "query/sql/show", "query/sql/snapshot", "query/sql/switch-cold-storage-role", + "query/sql/switch-role", "query/sql/truncate", "query/sql/union-except-intersect", "query/sql/update", @@ -746,6 +747,7 @@ module.exports = { label: "Replication", items: [ "high-availability/setup", + "high-availability/failover", "high-availability/tuning", "high-availability/wal-cleanup", ], diff --git a/documentation/troubleshooting/error-codes.md b/documentation/troubleshooting/error-codes.md index fe0246e712..00a348d9ad 100644 --- a/documentation/troubleshooting/error-codes.md +++ b/documentation/troubleshooting/error-codes.md @@ -88,6 +88,14 @@ To do so, place an empty `_migrate_primary` file in your database installation d This will update the primary instance to the latest state from the object store and have it take over as the new primary instance. +An in-place promotion with +[`SWITCH ROLE TO PRIMARY`](/docs/high-availability/failover/#promote-a-replica-after-a-primary-loss) +runs the same check before it changes anything. A replica that has not yet +applied everything in the object store is refused: it stays a replica and +keeps replicating, `GET /lifecycle` reports the `replication` component as +`DEGRADED`, and the reason is in the server log. Promote it again once it has +caught up. + ### ER006 This error occurs when you start a primary instance and discover another instance is already acting as the primary. @@ -101,6 +109,12 @@ You have the following options: * Reconfigure it as `replication.role=replica` and restart it * Perform a planned primary migration and resume the primary role on this instance +This error is also the expected outcome when a node that was demoted in place +with [`SWITCH ROLE`](/docs/query/sql/switch-role/) is restarted while +`server.conf` still says `replication.role=primary`. Set `replication.role` to +the node's current role before any restart, see +[Restarts](/docs/high-availability/failover/#restarts). + ### ER007 This error indicates a Data ID mismatch between the local database and the backup or replication object store.