refactor(bench): 📦 release the benchmark harness as monoprop-bench-tools

[Panadestein]

🤖 AI text below 🤖

Turns the repository into a uv workspace and splits the benchmark harness in two: the reusable half becomes a package we can publish, the benchmarks stay here.

monoprop/
├── pyproject.toml            # the library (unchanged in scope)
├── uv.lock                   # one lock for the whole workspace
├── src/monoprop/
├── benches/                  # monoprop's own benchmark suite — stays put
└── packages/
    ├── monoprop-bench-tools/ # → PyPI
    └── bench-third-party/    # → GitHub only, standalone uv project

What moved

Was	Is
benches/_memory_cpu.py	monoprop_bench_tools.memory.cpu
benches/_memory_gpu.py	monoprop_bench_tools.memory.gpu
benches/_builders.py	monoprop_bench_tools.models
benches/report.py	monoprop_bench_tools.report → monoprop-bench-report
benches/bmf.py	monoprop_bench_tools.bmf → monoprop-bench-bmf
tests/test_bench_*.py	packages/monoprop-bench-tools/tests/
benches/third_party/	packages/bench-third-party/

All moves are git mv, so git log --follow still works.

Design decisions worth reviewing

benches/ does not move. Bencher keys its metric history on the benchmark name, so moving or renaming a bench_* test orphans its tracked series and resets the thresholds' sample windows. The suite therefore cannot version with a library release, which is precisely the line between "harness" and "benchmarks". monoprop-bench-tools contains no benchmarks.

The bencher.dev tracking stays in the repo. Only the renderer (bmf.py) is tooling and moved. bench_main.yml carries BENCHER_API_KEY, vars.BENCHER_PROJECT, the --branch/--testbed identity and the four threshold blocks — that is this repository's CI identity, and a published package cannot own a secret. The workflow itself is unchanged; just bench-bmf now shells out to the console script.

Three dependency groups instead of two.

• test — monoprop's suite only, deliberately free of workspace members. cibuildwheel installs it against a built wheel via test-groups = ["test"], where a monoprop-bench-tools requirement would resolve from PyPI (nonexistent) and drag monoprop back in from source.
• workspace-test — test + the member. What uv run pytest at the root needs.
• bench — workspace-test + pytest-benchmark.

bench-third-party is excluded from the workspace. It pins a narrower requires-python, a git dependency and linux-x86_64-only CUDA wheels; folding it in would drag all of that into the root lock. It stays a standalone uv project with its own lockfile (cd packages/bench-third-party && uv sync). It now depends on monoprop-bench-tools properly instead of poking benches/ onto sys.path.

No gpu extra on the tools package. Adding one put cupy-cuda12x + cuda-pathfinder into the root lock. CuPy is installed separately; memory.gpu already degrades to Method.UNAVAILABLE without it.

Publishing

deploy.yml gains build-bench-tools (uv build --package monoprop-bench-tools — pure Python, no cibuildwheel matrix) and a separate upload_pypi_bench_tools job. Separate because it is a separate PyPI project, and because a failure there must not hold back the library's own upload. hatch-vcs reads the same git tags via raw-options = { root = "../.." }, so the two distributions share a version line; verified that the wheel still builds from the sdist, where there is no git.

Important

Before the next release someone has to register a trusted publisher for monoprop-bench-tools on PyPI (this workflow file, the pypi environment), or upload_pypi_bench_tools will fail on its first run.

Validation

• uv lock --check, uv sync --all-groups --all-extras
• pytest -m "not mpi" → 617 passed (both suites)
• just bench-smoke and just bench-bmf smoke end to end through the console scripts
• Both distributions build; packages/bench-third-party/uv.lock re-resolved
• prek run --all-files clean

Deliberate non-changes

• packages/** is excluded from Sonar scope, preserving current behaviour. Bringing it in is a separate call.
• benches/results/*.json is untouched, so Bencher continuity holds.

One seam this creates

benches/conftest.py writes the results artifact schema and monoprop_bench_tools.report/.bmf read it — now across a package boundary. Called out in both docstrings and in AGENTS.md: a change to the recorded sections has to land on both sides.

Docs updated in the same change: AGENTS.md (new "Workspace layout" section), README.md, docs/content/docs/benchmarks.mdx, docs/content/docs/testing.mdx, benches/README.md.

[github-actions]

Docs preview: https://pr-227.monoprop-docs.pages.dev