From 38b14bb989e4abd4d21216361929004a8b361047 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 09:18:03 -0600 Subject: [PATCH 01/12] docs(analytics): document transaction-commit-time metric Add a Storage Metrics section documenting transaction-commit-time (write commit submit->durable duration distribution) and how its upper percentiles serve as the leading indicator for the storage overload (503) rejection. Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/analytics/overview.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6a..d9549cc47 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -169,6 +169,25 @@ Harper automatically tracks the following metrics for all services. Applications | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | | `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | +### Storage Metrics + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | + +`transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, +`p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard +uses: when the oldest outstanding commit exceeds `storage.maxTransactionQueueTime` (default 45s), +Harper rejects new write transactions with `Outstanding write transactions have too long of queue, +please try again later` (HTTP 503). + +Because it is measured on that same clock, it is the leading indicator for that rejection: a rising +`p99`/`p999` means commits are taking longer to drain — from write volume, large transactions, or a +saturated storage volume — and is the signal to shed or throttle write load before commits start +timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared +write path rather than per-thread submission concurrency, so alert on the upper percentiles trending +toward `maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. + ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | From 340817646b104543dda30f97f25a428e65f93eb0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 10:29:14 -0600 Subject: [PATCH 02/12] fix(analytics): satisfy prettier table formatting --- reference/analytics/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index d9549cc47..69e2bb292 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,8 +171,8 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------- | | `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, From daf8ab9d2fd206a66ce09e87ac6ac0ab0eb1a234 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:38:46 -0600 Subject: [PATCH 03/12] docs(analytics): address PR #572 review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Correct transaction-commit-time description: the recorder fires on both fulfilment and rejection (submit -> settle), not only on durable commit — a failed/retried commit still consumes queue time and should not be undercounted. - Restore the storage. prefix on maxTransactionQueueTime for consistency with its earlier mention. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 69e2bb292..fcb982e7d 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,9 +171,9 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------- | -| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------------------- | +| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → settle, success or failure) | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, `p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard @@ -186,7 +186,7 @@ Because it is measured on that same clock, it is the leading indicator for that saturated storage volume — and is the signal to shed or throttle write load before commits start timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared write path rather than per-thread submission concurrency, so alert on the upper percentiles trending -toward `maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. +toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics From 1771f0afbe4032c34fa87f4fb8ff0ff34aec9a75 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:49:53 -0600 Subject: [PATCH 04/12] docs(analytics): qualify transaction-commit-time as RocksDB-only, settle-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (codex) flagged the original framing as overstating the metric's reliability as a leading indicator: it is RocksDB-only, records one sample per commit attempt (not per logical transaction — conflict retries each get their own sample), and only records once an attempt settles, so a wedged commit contributes nothing while still tripping the 503 guard. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 35 ++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index fcb982e7d..41eba7066 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,22 +171,29 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------------------- | -| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → settle, success or failure) | + + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------ | +| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit → settle, per commit attempt | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, -`p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard -uses: when the oldest outstanding commit exceeds `storage.maxTransactionQueueTime` (default 45s), -Harper rejects new write transactions with `Outstanding write transactions have too long of queue, -please try again later` (HTTP 503). - -Because it is measured on that same clock, it is the leading indicator for that rejection: a rising -`p99`/`p999` means commits are taking longer to drain — from write volume, large transactions, or a -saturated storage volume — and is the signal to shed or throttle write load before commits start -timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared -write path rather than per-thread submission concurrency, so alert on the upper percentiles trending -toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. +`p999`, `count`) on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each +sample covers one commit attempt, not one logical write transaction — a transient-conflict retry +re-issues the commit and records its own sample, so `count` can exceed the number of logical writes — +and a sample is only recorded once an attempt settles, so a commit that is still outstanding +contributes nothing yet. It measures the same clock the storage engine's overload guard uses: when +the oldest outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default 45s), +Harper rejects new write transactions on that thread with `Outstanding write transactions have too +long of queue, please try again later` (HTTP 503). + +A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large +transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write +load. But because it only records on settle, a wedged commit — the case the overload guard exists to +catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this +distribution as a leading indicator on its own, pair it with queue-depth or stuck-commit observability +before alerting solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the +threshold against a baseline for your workload. ### Resource Usage Metrics From f2fd67a35e960b14620111b1b914807d5490c327 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:57:33 -0600 Subject: [PATCH 05/12] docs(analytics): fix raw/aggregate field split and overload-guard wording - median/p90/p95/p99/p999 are only present on the per-minute aggregate (hdb_analytics); raw per-second entries only carry mean/distribution/count. - The overload guard tracks a commit it happened to arm on, not reliably the oldest outstanding one (retries/chained commits can wedge unarmed). - ASCII "to" instead of the arrow glyph per the content style guide. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 41eba7066..01eb938cb 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -173,19 +173,21 @@ Harper automatically tracks the following metrics for all services. Applications -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------ | -| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit → settle, per commit attempt | - -`transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, -`p999`, `count`) on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each -sample covers one commit attempt, not one logical write transaction — a transient-conflict retry -re-issues the commit and records its own sample, so `count` can exceed the number of logical writes — -and a sample is only recorded once an attempt settles, so a commit that is still outstanding -contributes nothing yet. It measures the same clock the storage engine's overload guard uses: when -the oldest outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default 45s), -Harper rejects new write transactions on that thread with `Outstanding write transactions have too -long of queue, please try again later` (HTTP 503). +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- | +| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt | + +`transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for +LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a +transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the +number of logical writes — and a sample is only recorded once an attempt settles, so a commit that is +still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, +`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on +the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table +for percentile-based alerting. It measures the same clock the storage engine's overload guard uses: +when a tracked outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default +45s), Harper rejects new write transactions on that thread with `Outstanding write transactions have +too long of queue, please try again later` (HTTP 503). A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write From c2abfafa3ee2a93dc11df9f8495ccd4b3704f563 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:04:33 -0600 Subject: [PATCH 06/12] docs(analytics): add 5.2 release note, point to the real stuck-commit signal - Add a transaction-commit-time entry to the v5.2 release notes so it's discoverable from the changelog. - Replace the vague "queue-depth or stuck-commit observability" pointer (no such metric is documented yet) with the actual overload-guard log line operators can watch today. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 5 +++-- release-notes/v5-lincoln/5.2.md | 6 ++++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 01eb938cb..43a598b0b 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -193,8 +193,9 @@ A rising `p99`/`p999` signals commits are taking longer to drain — from write transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write load. But because it only records on settle, a wedged commit — the case the overload guard exists to catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this -distribution as a leading indicator on its own, pair it with queue-depth or stuck-commit observability -before alerting solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the +distribution as a leading indicator on its own. Watch the server log for the "Rejecting writes on this +thread" error the guard emits once it trips, and don't rely solely on percentiles trending toward +`storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf382..665e2a097 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -13,3 +13,9 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ### Replicated `set_configuration` The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration). + +## Analytics + +### `transaction-commit-time` metric + +A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics). From d90edac819c3e7f25b1869c73fbaad2319db6f52 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:45:17 -0600 Subject: [PATCH 07/12] docs(analytics): fix accuracy issues found by independent review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (codex + grok + harper-domain adjudication) found the initial framing overstated this metric's reliability as a leading indicator. Verified each claim against harper's DatabaseTransaction.ts and analytics/write.ts before applying: - RocksDB-only; never emitted for LMDB-backed databases. - One sample per commit attempt, not per logical transaction (conflict retries each record their own sample). - Only records once an attempt settles — a still-outstanding commit contributes nothing. - Raw entries (hdb_raw_analytics) carry mean/distribution/count only; percentiles are computed on the per-minute aggregate (hdb_analytics). Fixed the page's own raw-entry JSON example, which showed percentile fields that don't exist on raw rows. - The overload guard rejects record updates and publishes only — deletes and canonical-source writes (replication, caching sources) bypass it. - The guard tracks at most one outstanding commit per thread: a retry issued while the prior attempt still holds that slot is never armed (so a wedge there won't trip the 503); a later backoff-delayed retry re-arms fresh. - Added a v5.2 release-note entry and VersionBadge. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 43a598b0b..c9abeeb55 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -55,10 +55,8 @@ Example raw entry: "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", - "median": 202, "mean": 202, - "p95": 202, - "p90": 202, + "distribution": [202], "count": 1 }, { @@ -180,22 +178,27 @@ Harper automatically tracks the following metrics for all services. Applications `transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the -number of logical writes — and a sample is only recorded once an attempt settles, so a commit that is +number of logical writes. A sample is only recorded once an attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, `distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table -for percentile-based alerting. It measures the same clock the storage engine's overload guard uses: -when a tracked outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default -45s), Harper rejects new write transactions on that thread with `Outstanding write transactions have -too long of queue, please try again later` (HTTP 503). +for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard, +which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds +`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on +that thread with `Outstanding write transactions have too long of queue, please try again later` +(HTTP 503) — deletes and writes applied from a canonical source (e.g. replication or a caching +source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry +issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff +retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a +later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt. A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write -load. But because it only records on settle, a wedged commit — the case the overload guard exists to -catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this -distribution as a leading indicator on its own. Watch the server log for the "Rejecting writes on this -thread" error the guard emits once it trips, and don't rely solely on percentiles trending toward -`storage.maxTransactionQueueTime`. Tune the +load. But the metric shares only a timebase with the guard, not its population: a wedged commit +contributes no sample until it settles, and an early retry that wedges can go untracked by the guard +entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the +server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and +don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics From a4cca363c762a6f5013972cb0ac32462c7d21457 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:55:36 -0600 Subject: [PATCH 08/12] docs(analytics): note the metric is absent during transaction-log replay recordCommitLatency wraps the async transaction.commit() path only; replayLogs.ts's directCommitSync() calls commitSync() directly and isn't instrumented, so replay/startup commits emit no sample either. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c9abeeb55..de585db94 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -175,8 +175,9 @@ Harper automatically tracks the following metrics for all services. Applications | ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- | | `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt | -`transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for -LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a +`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not +emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during +transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, From 2bd3f8d801f4d2555207db60342d8e0fc1532c31 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 16 Aug 2026 19:30:14 -0600 Subject: [PATCH 09/12] Address transaction metric review feedback Co-Authored-By: GPT-5 Codex --- reference/analytics/overview.md | 48 ++++++++++++++++------------ reference/database/storage-tuning.md | 2 ++ 2 files changed, 30 insertions(+), 20 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index de585db94..78c605f5d 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -109,10 +109,16 @@ Example aggregate entry: "metric": "bytes-sent", "method": "connack", "type": "mqtt", - "median": 4, "mean": 4, - "p95": 4, + "p1": 4, + "p10": 4, + "p25": 4, + "median": 4, + "p75": 4, "p90": 4, + "p95": 4, + "p99": 4, + "p999": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 @@ -181,26 +187,28 @@ transaction-log replay. Each sample covers one commit attempt, not one logical w transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, -`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on -the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table -for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard, -which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds -`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on -that thread with `Outstanding write transactions have too long of queue, please try again later` -(HTTP 503) — deletes and writes applied from a canonical source (e.g. replication or a caching -source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry -issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff -retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a -later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt. +`distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`, +`p999`) are only available on the aggregate (`hdb_analytics`) once raw entries are rolled up. Query +the aggregate table for percentile-based alerting. With the default +[`analytics.aggregatePeriod`](#analyticsaggregateperiod) of 60 seconds, those alerts can observe new +aggregate values no more than once per minute. + +The metric shares a timebase with the RocksDB storage engine's overload guard. When the oldest +tracked outstanding commit on a thread exceeds +[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) +(default 45s), Harper rejects new record updates on that thread with +`Outstanding write transactions have too long of queue, please try again later` (HTTP 503). Deletes +and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. +Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, so a conflict +retry or chained commit that wedges is caught on its own timer rather than hiding behind an earlier +attempt. A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large -transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write -load. But the metric shares only a timebase with the guard, not its population: a wedged commit -contributes no sample until it settles, and an early retry that wedges can go untracked by the guard -entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the -server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and -don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the -threshold against a baseline for your workload. +transactions, or a saturated storage volume — and provides an early warning to shed or throttle +write load. However, a wedged commit contributes no sample until it settles. Use this distribution +with `write-transaction-queue-depth`, which remains elevated while commits are outstanding, and +watch the server log for the "Rejecting writes on this thread" error the guard emits when it trips. +Tune the threshold against a baseline for your workload. ### Resource Usage Metrics diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index d99153b84..e9e4cff5f 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -39,6 +39,8 @@ Default: `45s` The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. + Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. ```yaml From 3ac1b4bf1e08de98c2bbe6b6fe211d5aaa36d50b Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 16 Aug 2026 19:41:20 -0600 Subject: [PATCH 10/12] Resolve independent review findings Co-Authored-By: GPT-5 Codex --- reference/analytics/overview.md | 15 +++++++++------ reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 2 +- reference/database/system-tables.md | 14 +++++++++----- 4 files changed, 20 insertions(+), 13 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 78c605f5d..c069b5230 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -196,9 +196,13 @@ aggregate values no more than once per minute. The metric shares a timebase with the RocksDB storage engine's overload guard. When the oldest tracked outstanding commit on a thread exceeds [`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) -(default 45s), Harper rejects new record updates on that thread with -`Outstanding write transactions have too long of queue, please try again later` (HTTP 503). Deletes -and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. +(default 45s), Harper rejects new record updates and publishes from new application requests on that +thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP +503). Deletes and writes applied from a canonical source (e.g. replication or a caching source) +bypass this check. + + + Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, so a conflict retry or chained commit that wedges is caught on its own timer rather than hiding behind an earlier attempt. @@ -206,9 +210,8 @@ attempt. A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and provides an early warning to shed or throttle write load. However, a wedged commit contributes no sample until it settles. Use this distribution -with `write-transaction-queue-depth`, which remains elevated while commits are outstanding, and -watch the server log for the "Rejecting writes on this thread" error the guard emits when it trips. -Tune the threshold against a baseline for your workload. +with the server log: the "Rejecting writes on this thread" error is the authoritative signal when the +guard trips. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 482c7d9de..b8f5d926e 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -253,7 +253,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s` +- `maxTransactionQueueTime` — Max time a tracked write commit may remain unsettled before Harper rejects new application updates and publishes with HTTP 503; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index e9e4cff5f..385196730 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -37,7 +37,7 @@ Type: `string` (duration) Default: `45s` -The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The maximum time a tracked write commit may remain unsettled before Harper rejects new application-originated record updates and publishes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index 84da6dd00..865c9a8a4 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -50,10 +50,8 @@ A typical record: "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", - "median": 202, "mean": 202, - "p95": 202, - "p90": 202, + "distribution": [202], "count": 1 }, { @@ -105,10 +103,16 @@ A typical aggregate record: "metric": "bytes-sent", "method": "connack", "type": "mqtt", - "median": 4, "mean": 4, - "p95": 4, + "p1": 4, + "p10": 4, + "p25": 4, + "median": 4, + "p75": 4, "p90": 4, + "p95": 4, + "p99": 4, + "p999": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 From 175121c5c46bd403e073b2ac4bfa4eae4b934c46 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 16 Aug 2026 19:52:26 -0600 Subject: [PATCH 11/12] Clarify commit guard version semantics Co-Authored-By: GPT-5 Codex --- reference/analytics/overview.md | 13 +++++++------ reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 4 ++++ 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c069b5230..77cfa72a3 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -184,9 +184,10 @@ Harper automatically tracks the following metrics for all services. Applications `transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a -transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the -number of logical writes. A sample is only recorded once an attempt settles, so a commit that is -still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, +sample is recorded whether the attempt succeeds or fails, and a transient-conflict retry records its +own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an +attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries +(`hdb_raw_analytics`) carry `mean`, `distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`, `p999`) are only available on the aggregate (`hdb_analytics`) once raw entries are rolled up. Query the aggregate table for percentile-based alerting. With the default @@ -203,9 +204,9 @@ bypass this check. -Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, so a conflict -retry or chained commit that wedges is caught on its own timer rather than hiding behind an earlier -attempt. +Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, including +conflict retries and chained commits. It rejects once the oldest tracked attempt exceeds the limit, +so later attempts can no longer be omitted from overload detection. A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and provides an early warning to shed or throttle diff --git a/reference/configuration/options.md b/reference/configuration/options.md index b8f5d926e..ab1e77a65 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -253,7 +253,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Max time a tracked write commit may remain unsettled before Harper rejects new application updates and publishes with HTTP 503; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` +- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 385196730..beee903b6 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -33,12 +33,16 @@ storage: ### `storage.maxTransactionQueueTime` + + Type: `string` (duration) Default: `45s` The maximum time a tracked write commit may remain unsettled before Harper rejects new application-originated record updates and publishes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. +Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this check. In v5.2.0, a retry or chained commit submitted while another attempt was already tracked could be omitted from overload detection. + The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. From 94bf4e4c37b32b0a0a77df4a2b8433a0dbc1e3fd Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 16 Aug 2026 23:02:18 -0600 Subject: [PATCH 12/12] Clarify v5.2.0 transaction guard gap Co-Authored-By: GPT-5 Codex --- reference/database/storage-tuning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index beee903b6..0fefc66f6 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -41,7 +41,7 @@ Default: `45s` The maximum time a tracked write commit may remain unsettled before Harper rejects new application-originated record updates and publishes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. -Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this check. In v5.2.0, a retry or chained commit submitted while another attempt was already tracked could be omitted from overload detection. +Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this check. In v5.2.0, only one commit per thread was tracked at a time, so any commit - including a conflict retry or a chained commit - submitted while another was already outstanding could be omitted from overload detection. The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles.