docs(analytics): document transaction queue depth metrics

reference/analytics/overview.md

@@ -171,14 +171,75 @@ Harper automatically tracks the following metrics for all services. Applications
 
 ### Resource Usage Metrics
 
-| `metric`                  | Key attributes                                                                                   | Other               | Unit    | Description                                                                       |
-| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- |
-| `database-size`           | `size`, `used`, `free`, `audit`                                                                  | `database`          | bytes   | Database file size breakdown                                                      |
-| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time`              | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown |
-| `resource-usage`          | (see below)                                                                                      |                     | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric))     |
-| `storage-volume`          | `available`, `free`, `size`                                                                      | `database`          | bytes   | Storage volume size breakdown                                                     |
-| `table-size`              | `size`                                                                                           | `database`, `table` | bytes   | Table file size                                                                   |
-| `utilization`             |                                                                                                  |                     | %       | Percentage of time the worker thread was processing requests                      |
+| `metric`                        | Key attributes                                                                                   | Other               | Unit    | Description                                                                                                           |
+| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------- |
+| `database-size`                 | `size`, `used`, `free`, `audit`                                                                  | `database`          | bytes   | Database file size breakdown                                                                                          |
+| `main-thread-utilization`       | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time`              | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown                                     |
+| `read-transaction-queue-depth`  | `depth`, `maxDepth`                                                                              |                     | count   | Open tracked transactions holding a read handle (see [transaction queue depth](#transaction-queue-depth-metrics))     |
+| `resource-usage`                | (see below)                                                                                      |                     | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric))                                         |
+| `storage-volume`                | `available`, `free`, `size`                                                                      | `database`          | bytes   | Storage volume size breakdown                                                                                         |
+| `table-size`                    | `size`                                                                                           | `database`, `table` | bytes   | Table file size                                                                                                       |
+| `transaction-commit-time`       | `mean`, `median`, `p90`, `p95`, `p99`, `p999`                                                    |                     | ms      | Duration from write commit submission to settlement (see [transaction queue depth](#transaction-queue-depth-metrics)) |
+| `utilization`                   |                                                                                                  |                     | %       | Percentage of time the worker thread was processing requests                                                          |
+| `write-transaction-queue-depth` | `depth`, `maxDepth`                                                                              |                     | count   | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics))                 |
+
+#### Transaction Queue Depth Metrics
+
+<VersionBadge version="v5.2.0" />
+
+`write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are
+in flight against the storage engine per worker thread — a concurrency and throughput signal, not a
+reliable predictor on their own of the `Outstanding write transactions have too long of queue, please
+try again later` (HTTP 503) rejection: `maxDepth` amplitude reflects concurrent commits, not whether
+any single one is approaching the
+[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime)
+duration limit (default 45s) that actually trips the 503.
+
+`transaction-commit-time` records each commit's submit-to-settle duration on that same clock, and a
+rising `p99`/`p999` (in the `hdb_analytics` aggregate table, where percentiles are computed — they
+aren't present on `hdb_raw_analytics`) is a leading indicator of _gradual_ slowdowns approaching that
+limit. It doesn't help with a single commit that hangs indefinitely, though: the metric only records
+once a commit settles, so a genuinely wedged commit contributes no sample at all, while
+`write-transaction-queue-depth`'s `depth` stays elevated on that thread for as long as the commit
+remains outstanding. Harper also logs `Rejecting writes on this thread: a commit has been outstanding
+for ...` once per stuck commit when the 503 check itself fires, which is the authoritative signal for
+that specific failure.
+
+| Field      | Unit  | Description                                             |
+| ---------- | ----- | ------------------------------------------------------- |
+| `depth`    | count | Instantaneous depth sampled at emit time                |
+| `maxDepth` | count | High-water mark since this thread's last emitted sample |
+
+- **`write-transaction-queue-depth`** counts write commits handed to the storage engine whose commit
+  promises have not yet settled — how many commits this thread is juggling concurrently. This is
+  in-flight, not durability: a settled commit promise means the storage engine accepted the write,
+  not that it has been synced to disk.
+- **`read-transaction-queue-depth`** counts concurrently open tracked transactions holding a read
+  handle, including write transactions and transactions opened with snapshot disabled. A high count
+  can mean either many short-lived transactions or a few long-lived ones — the count alone can't
+  distinguish them, so use it as a concurrency signal; a duration-based metric would be needed to
+  identify a single transaction held open long enough to hold back compaction.
+
+Both metrics are gauges tracked only on the RocksDB write/read path, sampled per worker thread, so
+activity against LMDB-backed databases is not counted at all. On a mixed install they report RocksDB
+traffic only and silently under-count. They also carry no `database` dimension, so a reported depth
+cannot be attributed to a specific database. On an install with no RocksDB databases
+(`storage.engine: lmdb`), both always read `0` — indistinguishable from a healthy, empty queue —
+regardless of actual read/write load. All per-thread analytics reporting, including these gauges,
+piggybacks on the thread having recorded some other analytics-eligible activity in the period — a
+thread with no recordable activity in a given second emits no row at all rather than an explicit
+`depth: 0`. Absence of a sample is not the same as a healthy reading, particularly for
+`read-transaction-queue-depth` on an otherwise-quiet thread holding a single long-lived read.
+
+The raw per-thread entries in `hdb_raw_analytics` retain each thread's true instantaneous `depth` and
+per-period `maxDepth`; treat those as the reliable source for spike detection. The aggregate
+`hdb_analytics` table is not a sum of per-thread peaks — each thread's `maxDepth` is first averaged
+across its raw samples for the period, then those per-thread averages are summed — so a brief
+single-thread spike is diluted rather than preserved. Always alert on the `maxDepth` field of each
+queue-depth entry in an `hdb_raw_analytics` record's `metrics` array (or lower the
+sampling/aggregation period) rather than relying on the aggregate table to catch short spikes. Tune
+the concrete alert threshold against a baseline for your workload, since absolute depth scales with
+worker-thread count and per-transaction size.
 
 #### `resource-usage` Metric
 

reference/configuration/options.md

@@ -253,7 +253,7 @@ storage:
 - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold`
 - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0)
 - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false`
-- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s`
+- `maxTransactionQueueTime` — Max time a single write commit may stay unsettled before Harper starts rejecting writes with 503; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s`
 - `noReadAhead` — Advise OS against read-ahead; _Default_: `false`
 - `prefetchWrites` — Prefetch before write transactions; _Default_: `true`
 - `path` — Database files directory; _Default_: `<rootPath>/database`

reference/database/storage-tuning.md

@@ -37,7 +37,7 @@ Type: `string` (duration)
 
 Default: `45s`
 
-The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes.
+The maximum time a single write commit may remain unsettled before Harper starts rejecting new application-originated writes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold — it acts as backpressure when downstream disk I/O cannot keep up with incoming writes. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check.
 
 Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency.
 

release-notes/v5-lincoln/5.2.md

@@ -26,6 +26,12 @@ Components can now declare recurring jobs in their configuration with a new buil
 
 The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration).
 
+## Analytics
+
+### Transaction queue depth metrics
+
+New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread write-commit and open-read-transaction concurrency against the storage engine, as a throughput/concurrency signal. A new `transaction-commit-time` metric records per-commit submit-to-settle duration, a leading indicator of gradual slowdowns toward the write-queue HTTP 503 rejection — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics).
+
 ## CLI
 
 ### Explicit Authentication for Operations API Commands