Bound instruments

[jack-berg]

Supersedes #8314.

Java implementation of open-telemetry/opentelemetry-specification#5050

Adds bound instrument support to
opentelemetry-api-incubator. Usage:

// Initialize instrument
ExtendedLongCounter rolls =
    (ExtendedLongCounter)
        meter
            .counterBuilder("dice.rolls")
            .setDescription("The number of times each side of the die was rolled")
            .setUnit("{roll}")
            .build();

// Bind known attribute sets
BoundLongCounter face1 = rolls.bind(Attributes.of(AttributeKey.longKey("roll.value"), 1L));
BoundLongCounter face2 = rolls.bind(Attributes.of(AttributeKey.longKey("roll.value"), 2L));
// ... omitted for brevity

// Record
face1.

add(1);
face1.

add(1);
face2.

add(1);

In the original prototype #8314, I went with an API where bind returns the existing instrument interface. I.e. you call LongCounter bind(Attributes) from LongCounter. The user is meant to call add(long), but the other methods bind / add(long, Attributes), add(long, Attributes, Context) are still available. The user could call bind again without issue, but calling the add overloads which accept Attributes remove the performance gain.

Instead, in this PR I've added dedicated Bound{Type}{Instrument} interfaces, e.g:

public interface BoundLongCounter {
  void add(long value);

  void add(long value, Context context);
}

These are impossible to misuse, and allows for supporting a value + Context overload. Additionally, they lend themselves to extending later with a close() method. Something that cannot be reasonably done if we reuse the existing instrument interfaces because whereas calling close on a bound instrument should only close that series, calling close on the instrument should either close all the series or have some sort of selector for the series that should be closed. The semantics are too different to reuse.

In the original prototype #8314 I sketched out what it would look like in its final form with APIs added directly to the stable opentelemetry-api. Here I've added it to the incubator such that its actually in a position to merge and evaluate.

Performance characteristics

Main vs. PR — unbound instruments

No material change to the existing (unbound) record path — deltas are within run-to-run variance. The larger single-thread swings, e.g. cumulative last-value gauge, are a known high-variance cases.

Details

Benchmark	Threads	Temporality	Cardinality	Instrument	Baseline (ops/s)	After (ops/s)	Δ ops/s	Δ %
threads1	1	CUMULATIVE	1	COUNTER_SUM	158,325,186	159,391,084	+1,065,898	+0.7%
threads1	1	CUMULATIVE	1	GAUGE_LAST_VALUE	183,139,208	180,468,971	-2,670,236	-1.5%
threads1	1	CUMULATIVE	1	HISTOGRAM_BASE2_EXPONENTIAL	39,291,064	41,919,603	+2,628,539	+6.7%
threads1	1	CUMULATIVE	1	HISTOGRAM_EXPLICIT	92,534,333	98,472,419	+5,938,086	+6.4%
threads1	1	CUMULATIVE	1	UP_DOWN_COUNTER_SUM	161,924,927	158,040,499	-3,884,427	-2.4%
threads1	1	CUMULATIVE	128	COUNTER_SUM	115,543,509	111,712,232	-3,831,277	-3.3%
threads1	1	CUMULATIVE	128	GAUGE_LAST_VALUE	148,070,080	144,925,490	-3,144,590	-2.1%
threads1	1	CUMULATIVE	128	HISTOGRAM_BASE2_EXPONENTIAL	41,873,746	42,525,530	+651,784	+1.6%
threads1	1	CUMULATIVE	128	HISTOGRAM_EXPLICIT	75,151,621	69,319,108	-5,832,513	-7.8%
threads1	1	CUMULATIVE	128	UP_DOWN_COUNTER_SUM	102,711,802	108,061,488	+5,349,686	+5.2%
threads1	1	DELTA	1	COUNTER_SUM	91,633,280	103,749,856	+12,116,576	+13.2%
threads1	1	DELTA	1	GAUGE_LAST_VALUE	116,227,824	115,745,440	-482,383	-0.4%
threads1	1	DELTA	1	HISTOGRAM_BASE2_EXPONENTIAL	39,629,643	40,941,130	+1,311,487	+3.3%
threads1	1	DELTA	1	HISTOGRAM_EXPLICIT	63,310,760	66,721,151	+3,410,391	+5.4%
threads1	1	DELTA	1	UP_DOWN_COUNTER_SUM	86,535,970	103,123,679	+16,587,709	+19.2%
threads1	1	DELTA	128	COUNTER_SUM	91,622,587	95,676,523	+4,053,936	+4.4%
threads1	1	DELTA	128	GAUGE_LAST_VALUE	104,291,119	104,236,851	-54,269	-0.1%
threads1	1	DELTA	128	HISTOGRAM_BASE2_EXPONENTIAL	40,229,354	40,819,765	+590,410	+1.5%
threads1	1	DELTA	128	HISTOGRAM_EXPLICIT	68,004,549	66,535,518	-1,469,030	-2.2%
threads1	1	DELTA	128	UP_DOWN_COUNTER_SUM	97,683,094	93,745,498	-3,937,595	-4.0%
threads4	4	CUMULATIVE	1	COUNTER_SUM	535,089,001	521,377,157	-13,711,844	-2.6%
threads4	4	CUMULATIVE	1	GAUGE_LAST_VALUE	81,124,966	80,452,929	-672,037	-0.8%
threads4	4	CUMULATIVE	1	HISTOGRAM_BASE2_EXPONENTIAL	21,356,951	20,911,711	-445,239	-2.1%
threads4	4	CUMULATIVE	1	HISTOGRAM_EXPLICIT	19,237,646	19,695,778	+458,133	+2.4%
threads4	4	CUMULATIVE	1	UP_DOWN_COUNTER_SUM	534,756,212	543,023,364	+8,267,152	+1.5%
threads4	4	CUMULATIVE	128	COUNTER_SUM	197,496,478	209,149,966	+11,653,488	+5.9%
threads4	4	CUMULATIVE	128	GAUGE_LAST_VALUE	194,699,412	196,287,679	+1,588,267	+0.8%
threads4	4	CUMULATIVE	128	HISTOGRAM_BASE2_EXPONENTIAL	76,683,671	76,662,421	-21,250	-0.0%
threads4	4	CUMULATIVE	128	HISTOGRAM_EXPLICIT	85,206,557	84,920,755	-285,802	-0.3%
threads4	4	CUMULATIVE	128	UP_DOWN_COUNTER_SUM	218,427,080	215,636,553	-2,790,527	-1.3%
threads4	4	DELTA	1	COUNTER_SUM	36,528,899	34,486,228	-2,042,671	-5.6%
threads4	4	DELTA	1	GAUGE_LAST_VALUE	17,687,569	17,356,643	-330,926	-1.9%
threads4	4	DELTA	1	HISTOGRAM_BASE2_EXPONENTIAL	13,396,210	11,855,077	-1,541,133	-11.5%
threads4	4	DELTA	1	HISTOGRAM_EXPLICIT	13,301,940	13,010,617	-291,323	-2.2%
threads4	4	DELTA	1	UP_DOWN_COUNTER_SUM	34,346,363	35,693,302	+1,346,939	+3.9%
threads4	4	DELTA	128	COUNTER_SUM	88,028,068	88,121,572	+93,504	+0.1%
threads4	4	DELTA	128	GAUGE_LAST_VALUE	96,058,157	95,760,817	-297,341	-0.3%
threads4	4	DELTA	128	HISTOGRAM_BASE2_EXPONENTIAL	56,761,544	57,160,448	+398,904	+0.7%
threads4	4	DELTA	128	HISTOGRAM_EXPLICIT	59,048,808	59,180,127	+131,318	+0.2%
threads4	4	DELTA	128	UP_DOWN_COUNTER_SUM	90,712,180	87,564,654	-3,147,526	-3.5%

Bound vs. unbound instruments

Bound recording has improvements across the board, with most improvement in uncontended cases. There are small reductions in a few cases that are within the benchmark error margin I've been seeing.

Details

Benchmark	Threads	Temporality	Cardinality	Instrument	Baseline (ops/s)	After (ops/s)	Δ ops/s	Δ %
threads1	1	CUMULATIVE	1	COUNTER_SUM	159,391,084	218,136,578	+58,745,494	+36.9%
threads1	1	CUMULATIVE	1	GAUGE_LAST_VALUE	180,468,971	284,574,460	+104,105,488	+57.7%
threads1	1	CUMULATIVE	1	HISTOGRAM_BASE2_EXPONENTIAL	41,919,603	49,735,775	+7,816,171	+18.6%
threads1	1	CUMULATIVE	1	HISTOGRAM_EXPLICIT	98,472,419	127,766,252	+29,293,833	+29.7%
threads1	1	CUMULATIVE	1	UP_DOWN_COUNTER_SUM	158,040,499	218,985,930	+60,945,431	+38.6%
threads1	1	CUMULATIVE	128	COUNTER_SUM	111,712,232	190,232,127	+78,519,895	+70.3%
threads1	1	CUMULATIVE	128	GAUGE_LAST_VALUE	144,925,490	267,103,404	+122,177,914	+84.3%
threads1	1	CUMULATIVE	128	HISTOGRAM_BASE2_EXPONENTIAL	42,525,530	48,274,598	+5,749,069	+13.5%
threads1	1	CUMULATIVE	128	HISTOGRAM_EXPLICIT	69,319,108	94,473,247	+25,154,140	+36.3%
threads1	1	CUMULATIVE	128	UP_DOWN_COUNTER_SUM	108,061,488	195,110,858	+87,049,370	+80.6%
threads1	1	DELTA	1	COUNTER_SUM	103,749,856	189,860,598	+86,110,742	+83.0%
threads1	1	DELTA	1	GAUGE_LAST_VALUE	115,745,440	213,234,413	+97,488,972	+84.2%
threads1	1	DELTA	1	HISTOGRAM_BASE2_EXPONENTIAL	40,941,130	44,663,107	+3,721,977	+9.1%
threads1	1	DELTA	1	HISTOGRAM_EXPLICIT	66,721,151	108,356,750	+41,635,599	+62.4%
threads1	1	DELTA	1	UP_DOWN_COUNTER_SUM	103,123,679	190,656,218	+87,532,539	+84.9%
threads1	1	DELTA	128	COUNTER_SUM	95,676,523	148,650,217	+52,973,694	+55.4%
threads1	1	DELTA	128	GAUGE_LAST_VALUE	104,236,851	202,659,440	+98,422,589	+94.4%
threads1	1	DELTA	128	HISTOGRAM_BASE2_EXPONENTIAL	40,819,765	44,501,247	+3,681,482	+9.0%
threads1	1	DELTA	128	HISTOGRAM_EXPLICIT	66,535,518	73,602,326	+7,066,807	+10.6%
threads1	1	DELTA	128	UP_DOWN_COUNTER_SUM	93,745,498	156,968,261	+63,222,763	+67.4%
threads4	4	CUMULATIVE	1	COUNTER_SUM	521,377,157	760,631,798	+239,254,641	+45.9%
threads4	4	CUMULATIVE	1	GAUGE_LAST_VALUE	80,452,929	111,041,358	+30,588,429	+38.0%
threads4	4	CUMULATIVE	1	HISTOGRAM_BASE2_EXPONENTIAL	20,911,711	19,203,751	-1,707,960	-8.2%
threads4	4	CUMULATIVE	1	HISTOGRAM_EXPLICIT	19,695,778	23,168,693	+3,472,915	+17.6%
threads4	4	CUMULATIVE	1	UP_DOWN_COUNTER_SUM	543,023,364	766,333,854	+223,310,490	+41.1%
threads4	4	CUMULATIVE	128	COUNTER_SUM	209,149,966	300,808,820	+91,658,853	+43.8%
threads4	4	CUMULATIVE	128	GAUGE_LAST_VALUE	196,287,679	162,969,982	-33,317,696	-17.0%
threads4	4	CUMULATIVE	128	HISTOGRAM_BASE2_EXPONENTIAL	76,662,421	78,519,722	+1,857,302	+2.4%
threads4	4	CUMULATIVE	128	HISTOGRAM_EXPLICIT	84,920,755	84,950,109	+29,354	+0.0%
threads4	4	CUMULATIVE	128	UP_DOWN_COUNTER_SUM	215,636,553	312,019,553	+96,383,000	+44.7%
threads4	4	DELTA	1	COUNTER_SUM	34,486,228	34,347,664	-138,565	-0.4%
threads4	4	DELTA	1	GAUGE_LAST_VALUE	17,356,643	20,620,994	+3,264,351	+18.8%
threads4	4	DELTA	1	HISTOGRAM_BASE2_EXPONENTIAL	11,855,077	12,730,745	+875,669	+7.4%
threads4	4	DELTA	1	HISTOGRAM_EXPLICIT	13,010,617	13,623,306	+612,689	+4.7%
threads4	4	DELTA	1	UP_DOWN_COUNTER_SUM	35,693,302	35,276,503	-416,798	-1.2%
threads4	4	DELTA	128	COUNTER_SUM	88,121,572	97,623,830	+9,502,259	+10.8%
threads4	4	DELTA	128	GAUGE_LAST_VALUE	95,760,817	91,240,091	-4,520,726	-4.7%
threads4	4	DELTA	128	HISTOGRAM_BASE2_EXPONENTIAL	57,160,448	58,155,979	+995,531	+1.7%
threads4	4	DELTA	128	HISTOGRAM_EXPLICIT	59,180,127	59,089,987	-90,140	-0.2%
threads4	4	DELTA	128	UP_DOWN_COUNTER_SUM	87,564,654	94,063,923	+6,499,269	+7.4%

[codecov]

Codecov Report

❌ Patch coverage is 74.39446% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.79%. Comparing base (2626179) to head (be04826).

Files with missing lines	Patch %	Lines
...ry/api/incubator/metrics/ExtendedDefaultMeter.java	40.00%	24 Missing ⚠️
.../internal/state/DeltaSynchronousMetricStorage.java	81.11%	8 Missing and 9 partials ⚠️
...in/java/io/opentelemetry/sdk/metrics/SdkMeter.java	0.00%	16 Missing ⚠️
...sdk/metrics/internal/state/EmptyMetricStorage.java	0.00%	5 Missing ⚠️
...rnal/state/CumulativeSynchronousMetricStorage.java	71.42%	2 Missing and 2 partials ⚠️
...elemetry/sdk/metrics/ExtendedSdkDoubleCounter.java	84.61%	1 Missing and 1 partial ⚠️
...emetry/sdk/metrics/ExtendedSdkDoubleHistogram.java	84.61%	1 Missing and 1 partial ⚠️
...ntelemetry/sdk/metrics/ExtendedSdkLongCounter.java	83.33%	1 Missing and 1 partial ⚠️
...elemetry/sdk/metrics/ExtendedSdkLongHistogram.java	84.61%	1 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (74.39%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8527      +/-   ##
============================================
- Coverage     90.96%   90.79%   -0.17%     
- Complexity    10207    10272      +65     
============================================
  Files          1013     1013              
  Lines         27160    27423     +263     
  Branches       3182     3205      +23     
============================================
+ Hits          24706    24899     +193     
- Misses         1730     1788      +58     
- Partials        724      736      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jack-berg]

When discussing in the last java SIG, @trask brought up the point that perhaps the apparent 37% reduction in performance for threads=4,cardinality=128,temporality=cumulative,instrument=COUNTER_SUM case while the same case with cardinality=1 case improves is not expected, and might be caused test harness itself.

Indeed this turned out to be the case. The multi-thread cases all record to the same set of series in the same order. #8550 fixes the benchmark, and this PR includes the changes from #8550 and uses benchmark figures from it as a baseline for comparison.

I've updated the PR description with the new figures. The only remaining regression is in threads=4,cardinality=1,instrument=COUNTER_SUM,UP_DOWN_COUNTER_SUM. In these cases 4 threads hammer a single LongAdder and erode its striping. The results now match intuition.

[jack-berg]

Previously I reported that highly contended COUNTER_SUM, UP_DOWN_COUNTER_SUM cases showed a regression, which I attributed to higher contention on on the underlying LongAdder eroding its striping.

However, after merging #8559, that regression has gone away. My read is that the real bottleneck the whole time was the volatile valuesRecorded, and that highly contended cases were actually increasing contention of writes to that field. So the principle of "more throughput exacerbates a hotspot, slowing it down" was right, but I misattributed the hotspot to be the LongAdder instead of valuesRecorded.

The case for bound instruments looks even stronger now as they offer a benefit in all cases.