docs(analytics): document transaction-commit-time metric

reference/analytics/overview.md

@@ -55,10 +55,8 @@ Example raw entry:
 			"metric": "bytes-sent",
 			"path": "search_by_conditions",
 			"type": "operation",
-			"median": 202,
 			"mean": 202,
-			"p95": 202,
-			"p90": 202,
+			"distribution": [202],
 			"count": 1
 		},
 		{
@@ -111,10 +109,16 @@ Example aggregate entry:
 	"metric": "bytes-sent",
 	"method": "connack",
 	"type": "mqtt",
-	"median": 4,
 	"mean": 4,
-	"p95": 4,
+	"p1": 4,
+	"p10": 4,
+	"p25": 4,
+	"median": 4,
+	"p75": 4,
 	"p90": 4,
+	"p95": 4,
+	"p99": 4,
+	"p999": 4,
 	"count": 1,
 	"id": 1688589569646,
 	"time": 1688589569646
@@ -169,6 +173,47 @@ Harper automatically tracks the following metrics for all services. Applications
 | `bytes-received`      | node.database       | `replication` | `blob`    | bytes | Bytes received for blob replication                        |
 | `replication-latency` | node.database.table |               | `ingest`  | ms    | Time difference from source commit timestamp to local time |
 
+### Storage Metrics
+
+<VersionBadge version="v5.2.0" />
+
+| `metric`                  | `path` | `method` | `type` | Unit | Description                                                         |
+| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- |
+| `transaction-commit-time` |        |          |        | ms   | RocksDB write-commit duration, submit to settle, per commit attempt |
+
+`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not
+emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during
+transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a
+sample is recorded whether the attempt succeeds or fails, and a transient-conflict retry records its
+own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an
+attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries
+(`hdb_raw_analytics`) carry `mean`,
+`distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`,
+`p999`) are only available on the aggregate (`hdb_analytics`) once raw entries are rolled up. Query
+the aggregate table for percentile-based alerting. With the default
+[`analytics.aggregatePeriod`](#analyticsaggregateperiod) of 60 seconds, those alerts can observe new
+aggregate values no more than once per minute.
+
+The metric shares a timebase with the RocksDB storage engine's overload guard. When the oldest
+tracked outstanding commit on a thread exceeds
+[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime)
+(default 45s), Harper rejects new record updates and publishes from new application requests on that
+thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP
+503). Deletes and writes applied from a canonical source (e.g. replication or a caching source)
+bypass this check.
+
+<VersionBadge type="changed" version="v5.2.1" />
+
+Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, including
+conflict retries and chained commits. It rejects once the oldest tracked attempt exceeds the limit,
+so later attempts can no longer be omitted from overload detection.
+
+A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large
+transactions, or a saturated storage volume — and provides an early warning to shed or throttle
+write load. However, a wedged commit contributes no sample until it settles. Use this distribution
+with the server log: the "Rejecting writes on this thread" error is the authoritative signal when the
+guard trips. Tune the threshold against a baseline for your workload.
+
 ### Resource Usage Metrics
 
 | `metric`                  | Key attributes                                                                                   | Other               | Unit    | Description                                                                       |

reference/configuration/options.md

@@ -253,7 +253,7 @@ storage:
 - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold`
 - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0)
 - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false`
-- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s`
+- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s`
 - `noReadAhead` — Advise OS against read-ahead; _Default_: `false`
 - `prefetchWrites` — Prefetch before write transactions; _Default_: `true`
 - `path` — Database files directory; _Default_: `<rootPath>/database`

reference/database/storage-tuning.md

@@ -33,11 +33,17 @@ storage:
 
 ### `storage.maxTransactionQueueTime`
 
+<VersionBadge type="changed" version="v5.2.1" />
+
 Type: `string` (duration)
 
 Default: `45s`
 
-The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes.
+The maximum time a tracked write commit may remain unsettled before Harper rejects new application-originated record updates and publishes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check.
+
+Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this check. In v5.2.0, only one commit per thread was tracked at a time, so any commit - including a conflict retry or a chained commit - submitted while another was already outstanding could be omitted from overload detection.
+
+The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles.
 
 Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency.
 

reference/database/system-tables.md

@@ -50,10 +50,8 @@ A typical record:
 			"metric": "bytes-sent",
 			"path": "search_by_conditions",
 			"type": "operation",
-			"median": 202,
 			"mean": 202,
-			"p95": 202,
-			"p90": 202,
+			"distribution": [202],
 			"count": 1
 		},
 		{
@@ -105,10 +103,16 @@ A typical aggregate record:
 	"metric": "bytes-sent",
 	"method": "connack",
 	"type": "mqtt",
-	"median": 4,
 	"mean": 4,
-	"p95": 4,
+	"p1": 4,
+	"p10": 4,
+	"p25": 4,
+	"median": 4,
+	"p75": 4,
 	"p90": 4,
+	"p95": 4,
+	"p99": 4,
+	"p999": 4,
 	"count": 1,
 	"id": 1688589569646,
 	"time": 1688589569646

release-notes/v5-lincoln/5.2.md

@@ -43,3 +43,9 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t
 ### Web Application Firewall
 
 Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview).
+
+## Analytics
+
+### `transaction-commit-time` metric
+
+A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics).