Multi-primary writes: ingest on any node, zero-downtime upgrades

[bluestreak01]

Summary

Multi-primary writes: every node in a QuestDB Enterprise cluster accepts writes, and the cluster coordinator (#131) assigns each transaction a global sequence number per table before it commits. All nodes and replicas apply a table's transactions in that order, so every copy converges on the same state. Losing a node no longer interrupts ingestion, and upgrades roll through the cluster with no write downtime.

This is the second iteration of the coordinator. The first elects a single primary; this one turns the same quorum into a transaction sequencer.

Current Limitations

• One primary accepts writes; every other node is read-only
• Losing the primary means a failover, and writes pause until a replica is promoted
• Upgrading the primary requires a planned switchover
• Writers must find the primary; the cluster cannot spread ingestion across nodes

Features

Feature	Description
Writes on any node	Clients send data to any node in the cluster; there is no primary to locate
Per-table transaction sequencing	The coordinator quorum hands out a strictly increasing sequence number per table; the assignment is replicated, so it survives a coordinator failure
Ordered apply everywhere	Nodes and replicas apply a table's transactions in sequence order, pulling missing segments from the object store, so all copies are identical
Deterministic conflict resolution	Deduplication and last-writer-wins follow the global sequence, so concurrent writes to the same rows resolve the same way on every node
Continuous availability	A failed node takes only its own capacity with it; store-and-forward clients resend unacknowledged batches to another node with no failover event
Zero-downtime upgrades	Nodes drain and upgrade one at a time while the rest keep accepting writes
Built for write rates	Sequence numbers are assigned per transaction, not per row, and requests are batched, so sequencing stays off the per-row hot path
Cluster view	Sequencing lag and each node's share of writes appear in the coordinator's cluster view in the Web Console

How it works

1. A client sends a batch to any node.
2. The node builds the WAL transaction and asks the coordinator quorum for the next sequence number for that table.
3. The quorum assigns it through its replicated log, and the node commits and uploads the WAL segment tagged with the number.
4. Every node applies that table's transactions strictly in sequence order, including transactions written by other nodes.
5. The client receives its acknowledgement once the transaction is durable. If the node fails before that, the store-and-forward client replays the batch on another node.

Example

Two clients write to the same table through different nodes. Both nodes, and every replica, end up with the same order:

client A → node-1: INSERT INTO trades ...    -- sequenced as trades#1042
client B → node-2: INSERT INTO trades ...    -- sequenced as trades#1043

-- every node applies trades in order: ... 1042, 1043 ...

Benefits

• Ingestion never stops — node loss, maintenance and upgrades no longer pause writes
• No failover to run — clients move to another node on their own, with no promotion step
• Scale ingestion across nodes — spread writers over the cluster instead of funnelling through one primary
• Same data everywhere — a single global order per table means no divergence and no reconciliation
• No load balancer — clients discover nodes from the coordinator and pick any of them

Scope

• Coordinator: per-table sequencer in the replicated state machine, with batched assignment
• Node: multi-primary write path, WAL transactions tagged with the global sequence, ordered apply from the object store
• Deduplication and last-writer-wins on the sequenced stream
• Gap recovery when a node fails after receiving a sequence number but before upload
• Clients: any-node routing from the coordinator registry, replay on another node
• Rolling upgrade procedure with no write downtime
• Sequencing lag and per-node write share in the cluster view

Next iteration of #131 (cluster coordinator). Relies on store-and-forward clients (#109). Supersedes #12.