Extend hook's teardown metadata with warmup/calibration flags

[sergey-miryanov]

I'm working on gc-monitor and have implemented a pyperf hook. The hook's teardown function writes collected run data to an intermediate file, but there's currently no way to distinguish between warmup, calibration, and measurement runs.

Would it make sense to extend the metadata dictionary passed to teardown with additional context, such as {"is_warmup": is_warmup, "calibrate_loops": calibrate_loops}?

[vstinner]

The hook's teardown function writes collected run data to an intermediate file, but there's currently no way to distinguish between warmup, calibration, and measurement runs.

pyperf JSON format stores enough information to detect if a run is a calibration run or a normal run computing values. See Run 1: calibrate in:

$ python -m pyperf timeit -p2 -o json '[1,2]*1000'
$ python -m pyperf dump json 
Run 1: calibrate the number of loops: 2^17
- calibrate 1: 2.90 us (loops: 1, raw: 2.90 us)
- calibrate 2: 2.26 us (loops: 2, raw: 4.53 us)
- calibrate 3: 1.84 us (loops: 4, raw: 7.36 us)
(...)
- calibrate 21: 1.36 us (loops: 2^17, raw: 178 ms)
Run 2: 1 warmup, 3 values, 2^17 loops
- warmup 1: 1.38 us
- value 1: 1.38 us
- value 2: 1.39 us
- value 3: 1.48 us (+8%)
Run 3: 1 warmup, 3 values, 2^17 loops
- warmup 1: 1.45 us (+5%)
- value 1: 1.36 us
- value 2: 1.33 us
- value 3: 1.32 us

A calibration run only has warmups without values.

I'm not sure that I understood your problem correctly.

[sergey-miryanov]

Yes, the output JSON file contains this information, but I need access to it at runtime.

Currently, when _compute_values calls hook.teardown(metadata), the metadata dict only includes unit, name, hook, and any custom data the hook has stored. If metadata were enriched with additional flags, I could merge them into gc-monitor's output and process the data accordingly - for example, calculating separate stats for warmup, calibration, and value runs.

[vstinner]

If Run.values is empty, the run is a calibration. See the private method:

class Run:
    def _is_calibration(self):
        # Run used to calibrate or recalibration the number of loops,
        # or to calibrate the number of warmups
        return (not self._values)

Example using JSON generated above:

import pyperf
filename = 'json'
suite = pyperf.BenchmarkSuite.load(filename)
benchmark = suite.get_benchmarks()[0]
for index, run in enumerate(benchmark.get_runs()):
    nwarmup = len(run.warmups)
    nvalue = len(run.values)
    is_calibration = (not nvalue)
    if is_calibration:
        what = 'Calibration'
    else:
        what = f'{nwarmup} warmup, {nvalue} value'
    print(f"Run {index}: {what}")

Output:

Run 0: Calibration
Run 1: 1 warmup, 3 value
Run 2: 1 warmup, 3 value

[sergey-miryanov]

The hooks are scoped to _compute_values: hooks are instantiated there and torn down before the function returns.

pyperf/pyperf/_worker.py

Lines 65 to 68 in 0ba6bbe

	hook_managers = instantiate_selected_hooks(args.hook, self._custom_hooks)
	if len(hook_managers):
	self.metadata["hooks"] = ", ".join(hook_managers.keys())

pyperf/pyperf/_worker.py

Lines 117 to 120 in 0ba6bbe

	for hook in hook_managers.values():
	hook.teardown(self.metadata)
	def collect_metadata(self):

They don't have access to the Run object during construction or teardown, which is why I'd like to add extra fields to metadata.

You're convincing me (no offense!) to use post-processing instead - I understand that approach. However, I see it as a last resort, since I'd prefer to handle this at runtime. Also, if the user doesn't specify the -o option, post-processing isn't even available as a fallback.

[vstinner]

I don't know well the hook API. It sems like teardown() only gets a metadata dictionary, no Benchmark or Run object. Maybe the hook API should be extended or redesigned to fit your use case. I'm not sure.

[maurycy]

@sergey-miryanov @vstinner While working on #230, I thought about hooks.

What do you think about registration-based system like hooks.around(phase, callback, metadata), where:

• phase (or event) could be a tuple like ("calibration", "warmup") or ("measurement", "warmup") or ("calibration", ) or maybe () (too clever?),
• and callback could be a @contextmanager,
• metadata could be just supplied and given back to the callbacks.

This avoids lots of problems with simpler event-based hooks (exception handling or overloading metadata), continues the direction of #236 of making the pyperf less aware about particular hooks, and makes it possible to create a simple adapter for existing hooks without breaking them.

@sergey-miryanov Happy to pair on this if you want (not this week, though; I'm focusing on python/cpython#145411 right now.)

Here: the metadata in the teardown could already go with phase instead of booleans.

[sergey-miryanov]

Now that a pair of booleans encodes three states, we can use the following named constants for them:

Warmup	Calibrate	Function	Phase
T	F	calibrate_warmups, compute_warmups_values	Warmup
T	T	calibrate_loops	Calibration
F	F	compute_warmups_values	Measurement
F	T	No	-

I'm fine using those names instead of bools in this PR.

If we're talking about redesigning the hook API, I wouldn't want to change it much since we don't have many hooks that would leverage it.

What I'd like to change is where hooks are instantiated and torn down. That way, hooks could become stateful and do more runtime work when torn down or entering/exiting a context.

[maurycy]

@sergey-miryanov

is_warmup	calibrate_loops	args.recalibrate_warmups	args.recalibrate_loops	Function (calls _compute_values)	phase in metadata
T	F	F	F	calibrate_warmups (initial)	("calibration", "warmup")
T	F	T	F	calibrate_warmups (recalibrate)	("recalibration", "warmup")
T	F	F	F	compute_warmups_values (warmup samples)	("measurement", "warmup")
T	T	F	F	calibrate_loops (initial)	("calibration", "loops")
T	T	F	T	calibrate_loops (recalibrate)	("recalibration", "loops")
F	F	F	F	compute_warmups_values (value samples)	("measurement", "value")

Another option is having recalibrate=True in the metadata, instead of having ("calibration", "loops") and ("recalibration", "loops") etc. Or skip it, and assume it's available implicitly via recalibrate_loops or recalibrate_warmups.

The patch would be:

2026-05-27T16:56:58.291507000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/pyperf (phasephasephase 7084a7c*?) % git diff
diff --git i/pyperf/_metadata.py w/pyperf/_metadata.py
index e2ff96c..fd8bc2b 100644
--- i/pyperf/_metadata.py
+++ w/pyperf/_metadata.py
@@ -48,6 +48,12 @@ def is_tags(value):
     return all(isinstance(x, str) and x not in ('all', '') for x in value)
 
 
+def is_phase(value):
+    if not isinstance(value, (tuple, list)):
+        return False
+    return len(value) == 2 and all(isinstance(x, str) and x for x in value)
+
+
 def parse_load_avg(value):
     if isinstance(value, NUMBER_TYPES):
         return value
@@ -69,6 +75,7 @@ LOOPS = _MetadataInfo(format_number, (int,), is_strictly_positive, 'integer')
 WARMUPS = _MetadataInfo(format_number, (int,), is_positive, 'integer')
 SECONDS = _MetadataInfo(format_seconds, NUMBER_TYPES, is_positive, 'second')
 TAGS = _MetadataInfo(format_generic, (list,), is_tags, 'tag')
+PHASE = _MetadataInfo(format_generic, (tuple, list), is_phase, None)
 
 # Registry of metadata keys
 METADATA = {
@@ -92,6 +99,7 @@ METADATA = {
     'calibrate_warmups': WARMUPS,
     'recalibrate_warmups': WARMUPS,
     'tags': TAGS,
+    'phase': PHASE,
 }
 
 DEFAULT_METADATA_INFO = _MetadataInfo(format_generic, METADATA_VALUE_TYPES, None, None)
diff --git i/pyperf/_worker.py w/pyperf/_worker.py
index 3547fae..01b8177 100644
--- i/pyperf/_worker.py
+++ w/pyperf/_worker.py
@@ -114,6 +114,21 @@ class WorkerTask:
 
             index += 1
 
+        if calibrate_loops:
+            self.metadata["phase"] = (
+                "recalibration" if args.recalibrate_loops else "calibration",
+                "loops",
+            )
+        elif is_warmup and (args.calibrate_warmups or args.recalibrate_warmups):
+            self.metadata["phase"] = (
+                "recalibration" if args.recalibrate_warmups else "calibration",
+                "warmup",
+            )
+        elif is_warmup:
+            self.metadata["phase"] = ("measurement", "warmup")
+        else:
+            self.metadata["phase"] = ("measurement", "value")
+
         for hook in hook_managers.values():
             hook.teardown(self.metadata)
 

Does it make sense? Does it solve the issue at hand? Do you think it's future-proof? (I like the idea of hooks.around(phase, callback, metadata) but I agree that the number of hooks is limited, so a redesign can wait.)

cc @vstinner

[sergey-miryanov]

I'm not sure we need to distinguish between calibration and recalibration runs, but I don't have a strong objection to it. It's more than I need but works for me! :)

One point to note: since we use string literals to encode state, we'll have an open-ended number of states that hooks may or may not need to handle. We can definitely see this as both a pro and a con at the same time! :)

[maurycy]

@sergey-miryanov I will open a PR later today and let’s see what happens:)

[maurycy]

@sergey-miryanov #241 Decided against recalibration because it's a one-way door.

[vstinner]

Metadata is displayed to users when running a command such as python -m pyperf metadata ./pyperf/tests/telco.json, so I dislike putting anything in metadata if it only makes sense to hooks.

If you need more state, pass it to the hook. For example, add a "phase" parameter to HookBase.teardown().