diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6a..77cfa72a3 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -55,10 +55,8 @@ Example raw entry: "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", - "median": 202, "mean": 202, - "p95": 202, - "p90": 202, + "distribution": [202], "count": 1 }, { @@ -111,10 +109,16 @@ Example aggregate entry: "metric": "bytes-sent", "method": "connack", "type": "mqtt", - "median": 4, "mean": 4, - "p95": 4, + "p1": 4, + "p10": 4, + "p25": 4, + "median": 4, + "p75": 4, "p90": 4, + "p95": 4, + "p99": 4, + "p999": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 @@ -169,6 +173,47 @@ Harper automatically tracks the following metrics for all services. Applications | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | | `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | +### Storage Metrics + + + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- | +| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt | + +`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not +emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during +transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a +sample is recorded whether the attempt succeeds or fails, and a transient-conflict retry records its +own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an +attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries +(`hdb_raw_analytics`) carry `mean`, +`distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`, +`p999`) are only available on the aggregate (`hdb_analytics`) once raw entries are rolled up. Query +the aggregate table for percentile-based alerting. With the default +[`analytics.aggregatePeriod`](#analyticsaggregateperiod) of 60 seconds, those alerts can observe new +aggregate values no more than once per minute. + +The metric shares a timebase with the RocksDB storage engine's overload guard. When the oldest +tracked outstanding commit on a thread exceeds +[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) +(default 45s), Harper rejects new record updates and publishes from new application requests on that +thread with `Outstanding write transactions have too long of queue, please try again later` (HTTP +503). Deletes and writes applied from a canonical source (e.g. replication or a caching source) +bypass this check. + + + +Beginning with v5.2.1, the guard tracks every outstanding commit attempt on the thread, including +conflict retries and chained commits. It rejects once the oldest tracked attempt exceeds the limit, +so later attempts can no longer be omitted from overload detection. + +A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large +transactions, or a saturated storage volume — and provides an early warning to shed or throttle +write load. However, a wedged commit contributes no sample until it settles. Use this distribution +with the server log: the "Rejecting writes on this thread" error is the authoritative signal when the +guard trips. Tune the threshold against a baseline for your workload. + ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 482c7d9de..ab1e77a65 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -253,7 +253,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s` +- `maxTransactionQueueTime` — Per-thread write-commit duration threshold for HTTP 503 backpressure; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index d99153b84..0fefc66f6 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -33,11 +33,17 @@ storage: ### `storage.maxTransactionQueueTime` + + Type: `string` (duration) Default: `45s` -The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The maximum time a tracked write commit may remain unsettled before Harper rejects new application-originated record updates and publishes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. + +Beginning with v5.2.1, Harper tracks every outstanding commit attempt for this check. In v5.2.0, only one commit per thread was tracked at a time, so any commit - including a conflict retry or a chained commit - submitted while another was already outstanding could be omitted from overload detection. + +The [`transaction-commit-time` metric](../analytics/overview.md#storage-metrics) provides the corresponding commit-latency distribution, but only records each attempt after it settles. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. diff --git a/reference/database/system-tables.md b/reference/database/system-tables.md index 84da6dd00..865c9a8a4 100644 --- a/reference/database/system-tables.md +++ b/reference/database/system-tables.md @@ -50,10 +50,8 @@ A typical record: "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", - "median": 202, "mean": 202, - "p95": 202, - "p90": 202, + "distribution": [202], "count": 1 }, { @@ -105,10 +103,16 @@ A typical aggregate record: "metric": "bytes-sent", "method": "connack", "type": "mqtt", - "median": 4, "mean": 4, - "p95": 4, + "p1": 4, + "p10": 4, + "p25": 4, + "median": 4, + "p75": 4, "p90": 4, + "p95": 4, + "p99": 4, + "p999": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index dde9cb594..4063b5ddf 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -43,3 +43,9 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t ### Web Application Firewall Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview). + +## Analytics + +### `transaction-commit-time` metric + +A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics).