feat(metrics): Add native histograms and Prometheus protobuf exposition by ethanolchik · Pull Request #856 · prometheus/client_js

ethanolchik · 2026-09-10T16:49:18Z

Applications currently cannot expose native histogram samples from this client. This adds opt-in native collection to Histogram and the Prometheus protobuf exposition needed to scrape it.

Fixes #576.

const registry = new client.Registry(
  client.Registry.PROMETHEUS_PROTOBUF_CONTENT_TYPE,
);
const histogram = new client.Histogram({
  name: 'request_duration_seconds',
  help: 'Request duration',
  nativeHistogramBucketFactor: 1.1,
  buckets: [],
  registers: [registry],
});

Existing histogram configurations retain classic behavior. Native histograms can retain explicit classic buckets for migration, or use buckets: [] for native-only protobuf output. Protobuf registry methods return a Buffer, represented as Uint8Array in the public TypeScript declarations.

The implementation includes:

Standard exponential schemas -4 through 8, positive and negative sparse buckets, a configurable zero bucket, and native exemplars. The default budget of 160 populated buckets reduces resolution as needed and remains a soft limit at schema -4.
Native JSON snapshots and sum, first, and omit aggregation through registries, clusters, and workers, including reconciliation of different schemas and zero thresholds.
Protobuf encoding for native histograms and existing metric types. protobufjs/light uses the checked-in lib/metrics.json descriptor; lib/metrics.proto and npm run generate-protobuf make its source and regeneration available.
Public types, regression tests, configuration and migration documentation, a runnable HTTP example, and changelog entries.

Validation performed locally:

761 tests in 34 suites passed on Node 22.23.2, 24.21.0, and 26.4.0. The configured Bun CI command also passed.
The Linux check workflow passed through act on Node 24.21.0: ESLint, Prettier, and TypeScript.
npm run benchmarks completed on macOS/Node 26.4.0. A focused registry comparison against upstream, with increased sampling, found no significant regression above a 5% threshold; default-label cases measured roughly 2–4% overhead.
Manual interoperability checks with Prometheus 3.12.0 covered native counts, sums, quantiles, signed and empty histograms, worker aggregation, classic coexistence, and other metric types.
Protobuf descriptor regeneration matched the checked-in file, and the package dry run included the runtime schema and implementation files.

The existing benchmark suite covers classic metrics. Dedicated native workload benchmarks and application-specific rollout validation remain follow-up work. Applications select the protobuf response format themselves; HTTP Accept negotiation is outside this change.

Developed with AI assistance (Codex), also disclosed in the commit trailer.

jdmarshall · 2026-09-11T20:22:08Z

 * @returns {Function} aggregator function
 */
-function AggregatorFactory(aggregatorFn) {
+function AggregatorFactory(aggregatorFn, nativeAggregatorFn) {


I don't see why you need the second parameter.

I added it so sum and first can pass their native histogram aggregators while keeping the existing classic histogram callbacks unchanged. The existing callbacks returns numbers whereas the native callbacks return native histogram snapshots. It's an optional field but I'm happy to structure it differently.

I'd be curious to see how the other clients solve this. Do we need to reuse the aggregators across two incompatible types? I'll see if I can look this up.

jdmarshall · 2026-09-11T20:25:08Z

+function AggregatorFactory(aggregatorFn, nativeAggregatorFn) {
 	return metrics => {
 		if (metrics.length === 0) return;
+		const hasNativeHistograms = metrics.some(


And this check runs on every single metric, on every single scrape. This is not when and where to do a sanity check. That should be farther up the chain.

Also why would this happen? You get one histogram with a particular name. Either all of the values will be natives or they won't, right?

Yeah true, this was probably more defensive than necessary. I've changed it so it uses the first snapshot only.

jdmarshall · 2026-09-11T20:33:36Z

 			if (defaultLabelNames !== undefined) {
 				for (const labelName of defaultLabelNames) {
 					seriesLabels[labelName] ??= this._defaultLabels[labelName];
+					if (


What is this fixing?

This fixes a case where a histogram label is null or undefined and the registry has a default for that label. The existing line applies the default to seriesLabels, but the formatter uses the value from sharedLabels instead, so the default is ignored.

I've moved this change into a separate commit in line with your other comment.

jdmarshall · 2026-09-11T20:34:42Z

 	}

-	async getMetricsAsString(metrics) {
+	async getMetricsAsString(metrics, contentType = this.contentType) {


since encodeMetricFamily doesn't return a string, this is not the way to wire this up.

You're also returning from the middle of a function now.

Ah yeah true, that's my bad. I've moved protobuf encoding into metrics() where the output format is selected.

jdmarshall · 2026-09-11T20:35:55Z

+		if (contentType === Registry.PROMETHEUS_PROTOBUF_CONTENT_TYPE) {
+			const { encodeMetricFamily } = require('./protobuf');
+			return encodeMetricFamily(metric, this._defaultLabels);
+		}


This should be hoisted up to the calling function, which fixes the early exit and the mismatched function name and method signature.

Ok sure, I've moved this into metrics().

jdmarshall · 2026-09-11T20:38:10Z

+			return encodeMetricFamily(metric, this._defaultLabels);
+		}
+
+		const isOpenMetrics = contentType === Registry.OPENMETRICS_CONTENT_TYPE;


You've got way too many changes in a single commit. I don't see how this one or the one I questioned below are part of native histograms. I know some devs prefer large commits that tie to the ticket, but I've yet to meet one who actually does forensics in git history so I'm pretty sure that's a Chesterton's Fence situation.

If you're going to fix other bugs in a driveby I'd prefer the be done as a separate commit in the same PR. It'll also help because there's already an open PR touching some of the label code and this is going to make a hash of things.

Ok yeah I totally understand. I've split it into three commits now:

preserve counter names during OpenMetrics output (c45baee)

apply defaults to nullish histogram labels (43a3419)

native histograms and protobuf (6924fdf)

jdmarshall · 2026-09-11T22:43:10Z

So am I correct in thinking that native histograms are incompatible with the default Prometheus data type? If so then I'm not sure how to keep this compatible with worker.shutdown()

ethanolchik · 2026-09-12T18:04:59Z

So am I correct in thinking that native histograms are incompatible with the default Prometheus data type? If so then I'm not sure how to keep this compatible with worker.shutdown()

Well, I though that because worker.shutdown() transfers JSON snapshots rather than formatted text, the native data is included regardless of the worker registry’s content type and preserved as objects by the coordinator. So Protobuf is
only needed on the registry serving the final scrape to expose the native buckets.

jdmarshall · 2026-09-20T18:38:30Z

Can you rebase on main? There were some benchmark changes and another PR that may or may not conflict, and I'm interested in seeing if this slows down metrics()

Signed-off-by: Ethan Olchik <eitan.olchik@gmail.com>

Extend Histogram with opt-in exponential native buckets, exemplars, and resolution reduction. Preserve native snapshots through registry, worker, and cluster aggregation. Add protobuf exposition for all existing metric types, binary registry return types, schema generation, documentation, and an HTTP example. Fixes prometheus#576 Assisted-by: Codex Signed-off-by: Ethan Olchik <eolchik@cloudflare.com>

ethanolchik · 2026-09-20T18:53:17Z

Can you rebase on main? There were some benchmark changes and another PR that may or may not conflict, and I'm interested in seeing if this slows down metrics()

rebased

jdmarshall · 2026-09-20T19:18:27Z

Thanks!

I finally took a peek at the python code. The aggregation is pretty much only for multiprocess work, so we are free to name aggregators anything we want to name them.

I suspect the simpler solution here is to introduce new aggregators for the native histograms, and establish a default aggregator for those types. So a 'sum-native' or 'sumNative' type instead of changing the factory function to take 2 parameters one of which is never used for 99% of all stats generated.

ethanolchik · 2026-09-20T19:40:24Z

Yeah that sounds simpler, thanks for that!

I'll give native histograms their own aggregators and default them to sumNative. I'll also make sure shutdown picks those up, since it currently only keeps metrics using sum.

Signed-off-by: Ethan Olchik <eitan.olchik@gmail.com>

krajorama

I've looked at this from native histograms point of view - I've worked with native histograms for the past 3 years in Prometheus.

LGTM from native histograms point of view, with a few caveats:

one can implement more sophisticated ways to keep the number of buckets down and try to avoid reducing the resolution (called coarsen here), which is done in client_golang. Probably fine for now, but should be explained more in the README that there is no protection if a histogram is saturated other than restarting the program or implementing some monitoring and reset over histograms.
for exemplars it would be nice if the number was configurable and maybe the client_golang algorithm was applied to get a distribution that's more likely to keep outliers - which we assumed are more valuable insight - nothing to do in this PR though
classic and native histograms can observer NaN and +- infinity, but it's an existing limitation that they are rejected - nothing to do in this PR

(Also I think this PR is a bit large, it could be split up into at least: 2 PRs for the two bugfixes in the first two commit, a PR for adding Protobuf and a PR for adding native histograms).

krajorama · 2026-09-22T16:47:10Z

+a zero bucket. The default zero threshold is `2 ** -128`, configurable with
+`nativeHistogramZeroThreshold`. The default budget of 160 populated buckets per
+label set can be configured with `nativeHistogramMaxBucketNumber` (0 disables
+the budget). When needed, resolution is reduced down to schema -4; at that


nit:

Suggested change

the budget). When needed, resolution is reduced down to schema -4; at that

the budget). When needed, resolution is progressively reduced down to schema -4; at that

krajorama · 2026-09-22T16:59:18Z

+		this.zeroCount = 0;
+		this.positiveBuckets = new Map();
+		this.negativeBuckets = new Map();
+		this.createdTimestamp = nowTimestamp();


I'm loving this, but it means counters and summaries and classic histograms are now different and don't have created timestamp. So maybe this should be commented out with a TODO() on top.

krajorama · 2026-09-22T17:15:42Z

 	}

 	/**
 	 * Initialize the metrics for the given combination of labels to zero.


I think this function needs a warning that prohibits usage in flight when there's also sum or sumNative aggregation being used. The reason is that if you reset a histogram in the sum*, then the sum is no longer guaranteed to be monotonic. This is best illustrated if you imagine having zero() function on counters. Assume you have two counters that are at count==1 , then you reset of them and leave it at 0. The sum will look like: 2 2 2 2 1 1 1 1 . Which means that PromQL will measure an increase of 1 over this time range as it will detect a reset at the 5th sample.

So what you'd actually have to do is to reset all counters/histograms that you aggregate.

ethanolchik changed the title ~~Add native histograms and Prometheus protobuf exposition~~ feat(metrics): Add native histograms and Prometheus protobuf exposition Sep 10, 2026

jdmarshall reviewed Sep 11, 2026

View reviewed changes

Comment thread lib/histogram.js Outdated

jdmarshall reviewed Sep 11, 2026

View reviewed changes

Comment thread lib/registry.js Outdated

jdmarshall reviewed Sep 11, 2026

View reviewed changes

Comment thread lib/registry.js Outdated

ethanolchik force-pushed the feat/native-histograms branch from faff229 to 6924fdf Compare September 12, 2026 18:03

ethanolchik and others added 3 commits September 20, 2026 19:50

fix(registry): preserve counter names during OpenMetrics exposition

814c5e4

Signed-off-by: Ethan Olchik <eitan.olchik@gmail.com>

fix(registry): apply default labels to nullish histogram labels

c4531ba

Signed-off-by: Ethan Olchik <eitan.olchik@gmail.com>

ethanolchik force-pushed the feat/native-histograms branch from 6924fdf to 527ad33 Compare September 20, 2026 18:52

refactor: use dedicated native histogram aggregators

2b91ffa

Signed-off-by: Ethan Olchik <eitan.olchik@gmail.com>

ethanolchik requested a review from jdmarshall September 20, 2026 20:20

krajorama reviewed Sep 22, 2026

View reviewed changes

	the budget). When needed, resolution is reduced down to schema -4; at that
	the budget). When needed, resolution is progressively reduced down to schema -4; at that

Conversation

ethanolchik commented Sep 10, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdmarshall Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdmarshall Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdmarshall Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethanolchik Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jdmarshall commented Sep 11, 2026

Uh oh!

ethanolchik commented Sep 12, 2026

Uh oh!

jdmarshall commented Sep 20, 2026

Uh oh!

ethanolchik commented Sep 20, 2026

Uh oh!

jdmarshall commented Sep 20, 2026

Uh oh!

ethanolchik commented Sep 20, 2026

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jdmarshall Sep 11, 2026 •

edited

Loading

jdmarshall Sep 11, 2026 •

edited

Loading

jdmarshall Sep 11, 2026 •

edited

Loading

ethanolchik Sep 12, 2026 •

edited

Loading