Allow for warmup in PEP659

[tonybaloney]

PyPerf doesn't seem to account for PEP659's dynamic interpreter optimising the byte codes in the target function.

There is a condition for PyPy and GraalPython to run a warmup cycle, but not for Python 3.11+. I know PEP659 isn't a JIT, but functions which are heavily optimised will be unstable.

Can some flag be set to run a warmup cycle?

[vstinner]

You can manually change the number of computed warmup values using --warmup command line option.

But yeah, it would be interesting to increase the number of warmups when running on CPython with specialized bytecode.

My latest attempt (which is now old) to automatically detect "changepoint" and decide when a benchmark "looks stable", to compute automatically the number of warmup values when running a benchmark with a JIT (PyPy): https://vstinner.readthedocs.io/pypy_warmups.html

[vstinner]

I suggest you to always have a look at raw values: write results in a JSON file and then use "python -m pyperf dump" to see raw values. You can try to plot them.

Maybe a command can be added to "pyperf convert" to convert first values to warmup values. For example, replace 1 warmup 5 values to 3 warmups 3 values (ignore the first 3 values, instead of one).

[ericsnowcurrently]

You can try to plot them.

It would be awesome if pyperf basically did that for you, with a command that produced a plot that is easy to interpret. I know we want to keep pyperf as simple and straight-forward as possible, but it can be hard to set up this sort of graphing and to get the output right.

(This is just a thought. It isn't important enough to me to motivate me to do something about it. 😄)

[vstinner]

For example, I wrote https://pyperf.readthedocs.io/en/latest/examples.html#hist-scipy-script example in the doc.

I don't want to add a dependency to matplotlib to pyperf.

[vstinner]

And there is also https://pyperf.readthedocs.io/en/latest/examples.html#plot example in the doc.

Screenshot of this example on pyperf/tests/telco.json file:

[vstinner]

plot.py usage:

$ python3 plot.py --help
usage: plot.py [-h] [-b BENCHMARK] [--split-runs] [--skip SKIP] [--warmups] [--run INDEX] filename

positional arguments:
  filename

options:
  -h, --help            show this help message and exit
  -b BENCHMARK, --benchmark BENCHMARK
  --split-runs
  --skip SKIP           skip first SKIP values
  --warmups
  --run INDEX           only render run number INDEX

You may want to play with the --skip option.

I wrote this example a long time ago, I'm not sure if it's still correct :-) Maybe it should be updated for the latest pyperf API.

[ericsnowcurrently]

Yeah, clear instructions (with example) in the docs is sufficient.