feat(qwp): add browser and Node.js QWP client - #62
Open
glasstiger wants to merge 239 commits into
Open
Conversation
The ingress and egress sessions close the connection for its side effect when a server message cannot be decoded, and discarded the returned promise with `void`. A reconnecting ingress transport's close() awaits the replay store outside any catch, and QwpNodeFileReplayStore.close() rethrows a periodic-checkpoint, segment-handle or advisory-lock release failure. A read-only or full journal volume therefore turned a malformed server response into an unhandled rejection, which terminates the host process by default from Node 15 on. Attach the same `.catch(() => undefined)` every sibling call site already uses. The protocol error itself still reaches the caller through the rejected send and through fail(); only the secondary teardown failure is absorbed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
Both bytes are defined in QWP_STATUS but had no case in qwpSenderErrorCategory, so they resolved to UNKNOWN -- indistinguishable from an unassigned byte. The replay transport treats UNKNOWN as poison-strike exempt so that an unrecognised future status retries forever rather than being blamed on the frame, which meant a frame rejected for exceeding the server's cap replayed without bound and stalled the pipeline behind it, since ACKs are cumulative. Give both a category. LIMIT_EXCEEDED stays RETRIABLE rather than TERMINAL: it is deterministic under byte-identical replay, but routing it through the poison detector ends it via the quarantine path, which preserves the rows for retryQwpNodeOrphanSlot() instead of failing the producer outright. Add a test that asserts every defined status byte outside the three success responses is classified, so a future status cannot silently inherit the exempt path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
Three problems in the replay poison detector, all of which decided the fate of a running store-and-forward producer: - WRITE_ERROR and INTERNAL_ERROR are RETRIABLE by policy, yet consumed poison strikes. With the escalation window defaulting to 5 seconds and the reconnect backoff capped at maxBackoffMs, four strikes accumulated well inside that window, so a concurrent DDL, a checkpoint or a briefly full server volume permanently killed the producer. Raise the default window to 5 minutes, which is what actually separates "this frame is poison" from "the server cannot write right now". - A DICTIONARY_GAP is the server asking for symbol catch-up, not a verdict on the frame. Charging a strike condemned the frame before the recovery it asked for had even been attempted. Exempt the status, not just frames already flagged dictionaryCatchup. - An intervening connection-establishment failure wiped the whole episode. That made the canonical poison case unreachable: a frame that takes the server down guarantees the next connect fails, so the count reset before it could ever meet maxFrameRejections and the frame replayed without bound. Keep the strikes and bank the outage instead, so the escalation window measures connected dwell only -- which is the behaviour the original comment described wanting. Left alone deliberately: the 1000/1001/1012/1013 close-code exemption. It lets a peer's close code influence policy, but removing it would charge poison strikes to innocent frames during a rolling restart, and it errs toward retrying rather than abandoning data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
addColumn() validated the column name, type and scale but never the column count, so a row with more than QWP_MAX_COLUMNS_PER_TABLE distinct columns staged cleanly and atNow() accepted it. The cap was then enforced inside QwpTableBuffer during buildTable(), which runs in flushNow() before anything reaches the transport -- so the throw escaped before releaseStagedRows() and the rows stayed staged. Every subsequent flush() rebuilt the same table and threw again, and so did close(), which sent nothing. One over-wide row therefore made the sender permanently unusable and stranded every row staged before it, in that table and in every other, with reset() the only escape and it drops everything. Check the count in addColumn() where the new column is added. The existing try/catch routes it through failRow(), which discards just the row in progress and its table selection, so the sender stays usable and earlier rows still flush. Apply the same bound to compileWriterSchema(), which had no column-count check either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
Sizing a column and writing it independently derived the same three things. The Gorilla path was the worst: columnPayloadSize() rebuilt the bigint array and ran qwpGorillaSize(), writeColumn() rebuilt and ran it again, and encodeQwpGorilla() ran it a third time before encoding -- four passes and two N-element bigint arrays where one of each will do. inlineSymbolDictionary() rebuilt its Map and row-id array in both passes, and nullCount() rescanned the null vector in both. Compute a per-column plan once and share it between the passes through ColumnEncodeOptions. The map is scoped to a single encodeQwpIngressFrame() call, so a column mutated between calls can never be sized from a stale plan. Measured on the repository's own benchmark workloads, 10k rows, 100 iterations after 20 warmup, gorilla on: trades 1.34 ms -> 1.17 ms (1.15x) sparse 2.92 ms -> 1.98 ms (1.47x) wide 39.7 ms -> 38.0 ms (1.04x) Output is byte-identical: the SHA-256 of each encoded frame is unchanged across all three workloads with gorilla both on and off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
The sender promises that a rejected value discards the row in progress, including its table selection, so a half-built row can never reach QuestDB. Two setters escaped that guarantee. binaryColumn() built its defensive copy in addColumn()'s argument list, outside addColumn()'s own try. A Uint8Array whose ArrayBuffer has been transferred away -- structuredClone with `transfer`, or worker.postMessage -- makes that copy throw, so the throw escaped past failRow(): table() then reported "Table name has already been set" and the next atNow() published the row without its binary column. Copy inside a guard, the way uuidColumn() already does. It is the only one of the 22 addColumn() call sites that converted a value outside a try. decimalColumn() returned early for a zero-length Int8Array -- the documented byte-array spelling of NULL -- before reaching addColumn(), which is where a non-nullish call gets its availability, row-state and column-name checks. That spelling therefore accepted a call made before table(), a name with illegal characters, and a non-string name, all of which the null spelling and the ILP v3 equivalent reject. Route it through omitsNullish(name, null), as long256Column's all-absent path already does for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
…lished A QWP column carries one decimal scale per frame, so the fluent row API locks the scale on the first value staged for a column and rescales later values onto it. That lock lived in table.schema, which releaseStagedRows() never cleared, so it outlived every flush and held for the sender's lifetime. The consequence is a data-shape trap: a stream whose first decimal happens to be integral locks scale 0, and every more precise value afterwards fails with "cannot rescale decimal ... without precision loss" -- forever, and across flushes. The identical sequence on http::...;protocol_version=3 is ingested in full, so the same application code silently drops rows on QWP and not on ILP. The cited Java parity is QwpTableBuffer.ColumnBuffer, which is per-frame. Drop the decimal entries from a table's schema once its staged rows have been published, so the next frame's first value sets the scale afresh. Within a frame the lock is unchanged: rows still share one scale and a value that cannot be rescaled exactly still fails its row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
When close() ran while the first connect was still in flight, closeNow() aborted it and then fired the teardown without awaiting it. The advisory lock, journal handles and socket were therefore released on a detached promise chain, after close()'s own promise had already resolved. A sender created on the same sf_dir immediately after `await sender.close()` failed with QwpReplayStoreLockedError naming its own process -- the shape a SIGTERM handler that closes and reopens, or a test that reuses a directory, actually has. The existing regression test only required the lock to be gone within three seconds, so it did not catch the window. Await the teardown, bounded by the same close deadline the publication step uses, so a connection factory that ignores the abort signal cannot make close() wait out its connect timeout instead. Tighten the test to reopen the slot with no polling window at all; it fails against the old detached chain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
connectNow() closed the whole client when prewarm rejected, and connect() memoized the resulting promise. One transient outage during prewarm -- a rolling restart, a load-balancer warm-up, a 502 from a proxy -- was therefore fatal: close() latches closing/closed irreversibly, the retry returned the original rejection without reaching the server again, and every borrowSender()/borrowQuery() afterwards threw QwpClientClosedError. The only recovery was to discard the client and build a new one, which QWP.md never said. Closing was there to avoid leaving half-warmed entries behind, but prewarm() already releases every entry it did acquire back into the pool before it rethrows, so what it leaves are healthy warm connections. Let the rejection reach the caller instead, and clear the memoized attempt on rejection so the next connect() starts a new one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
Durable ACK is negotiated on the /write/v4 ingress route. The unified
config parser spreads the typed `webSocket` block into both sides, so a
`{ webSocket: { requestDurableAck: true } }` override reached `egress`
as well; connectQwpNodeEndpoint() then sent X-QWP-Request-Durable-Ack on
/read/v1 and rejected the session with QwpDurableAckUnavailableError
because no response carries x-qwp-durable-ack there. Ingest worked while
every pooled query session failed to connect, and the failure is not
retryable, so the query pool went terminal.
The connect-string key request_durable_ack was always ingress-only, so
the two spellings disagreed.
Drop it from the shared block -- the ingress block sets it explicitly
anyway -- and strip it in egressTransportOptions() alongside the other
ingress-only fields, so the typed object form is covered too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
Each was checked against the parser or the export surface: - QWP.md named `connectQwpNodeQuery()` as a connect-string entry point. No such export exists; it is `connectQwpNodeEgress()`. - The sf_dir defaults paragraph claimed a 60-second close drain. The parser produces 5000 ms, which the key table and the store-and-forward section both already stated correctly. - The same paragraph claimed "4 MiB frame/segment batches". `sf_dir` alone defaults the segment size but leaves the ingress frame cap unset, so say what it actually sizes and point at the two ways to bound a frame before the server advertises its cap. - `sf_max_segment_bytes` was documented only as journal segment size. It installs an ingress frame cap as well, with or without `sf_dir`, so a user who sets it alone gets QwpBatchTooLargeError on larger batches. - The Pool table said a standalone sender or query client ignores every key in it. `query_close_timeout_ms` is parsed into `egressSession` and bounds the CANCEL drain of any egress session built from it. - "Both keys apply to ingress and egress alike" is true on Node.js only: browser ingress plumbs neither `target` nor `zone`, and cannot, since the WebSocket API hides the upgrade response that carries role and zone. - QwpNodeIngressOptions.senderId's TSDoc claimed a `default` default. That is the connect-string parser's; the typed API has no default, and pooled slots derive `sender-<slot>`. config-docs.test.ts checked key names but never function names, which is how the first of these survived. Add a check that every backticked connect/create entry point QWP.md names is exported by one of the two package roots. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5WuAdKhXeFBSS9mswuZgG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a complete QWP client surface that works in both browsers and Node.js while leaving the existing ILP transports Node-only.
QWP support ships as a preview:
QWP.mddocuments the compatibility baseline for the first QWP release, and imports from internal source paths are never supported.Entry points
The repository now builds two published packages from a shared private core, each exposing its complete API from its package root.
@questdb/nodejs-clientSender(including QWP ingress selected withws::,wss::, orudp::), QWP codecs, egress, TLS, and persistent store-and-forward@questdb/browser-client@questdb/nodejs-clientkeeps the existing Node.js transports and dependencies, so nothing about the current client changes for existing consumers.@questdb/browser-clienthas no Node.js imports, Node.js typings, Node engine requirement,undici, orws, so supporting browsers does not require compromising the Node.js build.Ingress
Senderintegration with fluent rows, batching, byte/interval auto-flush, commits, transactions, and ACK watermarkssender.writer(table, schema).row({...})) for repeated rows on one schema, with the full QWP column-type setudp::, Node-only) behind the same fluent row APIEgress
Observability
onProgress,onError, and the Java-parityonSenderErrorrejection stream for event-driven telemetryAPI and platform integration
qdb_sessionbenchmarks/) covering encoder floors, the high-level sender, egress views, store-and-forward persistence policies, and a live end-to-end laneQwpBrowserSessionAuthTest,QwpIngressUpgradeProcessorOnHeadersReadyTest, andQwpEgressMaxBatchRowsTest, plus the Enterprise REST/OIDC login suitesCompatibility
http/https/tcp/tcpssenders are unchanged, with the two exceptions below.auth: {keyId, token}supplies only the private scalar, and the JWK was completed with a hardcoded public point unrelated to it. Node.js accepted that inconsistent pair without validating it up to v24 and rejects it from v26 withERR_CRYPTO_INVALID_JWK, so TCP auth failed outright on that runtime. The point is now derived from the private key. Signing only ever used the private scalar, so signatures, credentials, and auth outcomes are unchanged on every Node.js version; callers passing a completejwkobject were never affected.nullandundefinednow omit the column, which QuestDB records as NULL; most column methods previously threw a type error. On protocol v2 this also changes the wire bytes forarrayColumn(name, null), which used to emit an explicit NULL-array marker — QuestDB rejects that encoding withARRAY_INVALID_TYPE(verified against 9.4.3), so omitting it is itself a fix. A row whose every value is nullish now fails when the row is closed rather than at the column call. Code that relied on the throw as a data-quality guard should validate before calling the sender.flock/LockFileEx; the Node.js client uses a pure-JavaScript directory lock and cannot participate in those kernel locks, so neither sees the other. The persistence format stays cross-client for sequential handoff — a directory written by one runtime can be opened by the other once the first has closed it — and two Node.js processes still exclude each other. Depending on a native addon for kernel locks was the alternative, and it left store-and-forward broken on any platform or Node.js major without a prebuilt binary.Dependencies
ws(Node WebSocket transport). There is no native dependency; store-and-forward locking is pure JavaScriptfzstdis bundled into the build output for egress decompression;THIRD_PARTY_NOTICES.mdrecords its licenseResolved issues
Null or undefined column and symbol values are omitted across the existing ILP senders and the new QWP senders.
Fixes Client should skip columns if value is null #28
The new QWP sender introduces the sender.write().row() API.
Fixes State-machine builder #60
Validation
pnpm vitest run benchmarks): 3 files / 14 tests passedpnpm typecheckpnpm typecheck:qwp-browserpnpm typecheck:testpnpm typecheck:benchpnpm eslintpnpm lint:benchpnpm buildpnpm test:dist(loads both built packages through theirexportsmaps): 3 files / 30 tests passedpnpm typecheck:distpnpm check:packagesDependencies and provenance