Skip to content

bench: use mimalloc as the global allocator in every benchmark binary - #9953

Open
connortsui20 wants to merge 1 commit into
ct/codspeed-local-only-baselinesfrom
ct/codspeed-bench-mimalloc
Open

connortsui20 wants to merge 1 commit into
ct/codspeed-local-only-baselinesfrom
ct/codspeed-bench-mimalloc

Conversation

@connortsui20

@connortsui20 connortsui20 commented Sep 20, 2026

Copy link
Copy Markdown
Member

Summary

Third of three stacked PRs on CodSpeed flakiness. The first resizes benchmarks, the second removes baselines from CodSpeed.

Twenty-two benchmark binaries already use mimalloc as their global allocator, on the advice of the benchmarking guide; the other 81 use the system allocator. Several simulation benchmarks that allocate inside the timed region flipped between two values on pull requests that could not have affected them:

Benchmark Flips on unrelated PRs (all PRs) Max swing
split_collection.rs: cold_misaligned[(16, 64)] 11 (31) 13%
fsst_compress.rs: compress_fsst[(500, 64, 8)] 11 (17) 12%
take_chunked.rs: take[duplicates/repeated/...] 11 (15) 21%
take_primitive.rs: dict_canonicalize_gt_u8[1000000] (avx2 leg) 2 (6) 28%

The system allocator's cost depends on its state, which differs between runner images and between runs; mimalloc does the same work every time. One allocator for every benchmark binary also means no benchmark measures a different allocator from its neighbours.

Changes

  • Every benchmark binary that lacked it gets mimalloc as its global allocator (81 files). The 16 crates that did not depend on it get the workspace mimalloc dev-dependency, so Cargo.lock gains mimalloc in their dependency lists and nothing else.
  • The benchmarking guide's "prefer mimalloc for throughput benchmarks" becomes "every benchmark binary uses mimalloc".
  • The CUDA and DuckDB benchmarks are left alone: they run on different legs and link C++ code.

The CodSpeed report on this PR will show a change for most simulation benchmarks, as expected from swapping the allocator under them. Whether the flips above stop can only be confirmed over the coming pull requests. Two of the flaky benchmarks that already had mimalloc (filter_fixed_width.rs and run_end_compress.rs) needed resizing in the first PR instead, so this removes one variable rather than guaranteeing stability.

Checks run:

  • cargo clippy --locked --benches with -D warnings on every touched crate (vortex-alp, vortex-datetime-parts, vortex-decimal-byte-parts, vortex-fastlanes, vortex-fsst, vortex-onpair, vortex-pco, vortex-runend, vortex-sparse, vortex-array, vortex-arrow, vortex-btrblocks, vortex-buffer, vortex-compressor, vortex-compute, vortex-file, vortex-layout, vortex-mask, vortex)
  • cargo +nightly-2026-09-10 fmt --all and taplo fmt --check on the touched manifests

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn


Generated by Claude Code

Twenty-two benchmark binaries already used mimalloc, on the advice of the
benchmarking guide; the other 81 used the system allocator, whose cost
depends on state that differs between runner images and between runs.
Several simulation benchmarks that allocate inside the timed region
flipped between two values on pull requests that could not have affected
them, among them split_collection's cold_misaligned, fsst_compress's
compress_fsst, and take_chunked's repeated-index case.

Give every benchmark binary the same allocator, add the workspace
mimalloc dev-dependency to the 16 crates that lacked it, and make the
guide's recommendation a rule. The CUDA and DuckDB benchmarks are left
alone: they run on different legs and link C++ code.

Signed-off-by: Claude <noreply@anthropic.com>
@codspeed

codspeed Bot commented Sep 20, 2026

Copy link
Copy Markdown

Merging this PR will regress 97 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 111 improved benchmarks
❌ 97 regressed benchmarks
✅ 2000 untouched benchmarks
⏩ 293 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation collect_bool_u32_gt[1024] 10.5 µs 24.3 µs -56.89%
Simulation preverify_advancing_ptr_unchecked[1000] 11.7 µs 23.9 µs -51%
Simulation take_fsl_f16_random[16, 100] 100.6 µs 176.5 µs -43.02%
Simulation execute_mask_then_value[256] 22.1 µs 37.5 µs -41.1%
Simulation fixed_16_advancing_ptr_safe[100] 19.9 µs 31.8 µs -37.53%
Simulation sum_v2_i32_nulls_scattered 463.8 µs 741 µs -37.41%
Simulation max_i32_nulls_scattered 499.6 µs 778.1 µs -35.79%
Simulation allocate_drop_vortex_minimal_alignment[1024] 63.7 µs 95.3 µs -33.22%
Simulation from_vec_drop_bytes[65536] 10.3 µs 14.6 µs -29.49%
Simulation preverify_advancing_ptr_unchecked[10000] 27 µs 38 µs -28.91%
Simulation take_fsl_f16_force_manual_range_copy[512, 10] 47.3 µs 64.1 µs -26.29%
Simulation bitwise_and_vortex_buffer[1024] 4.5 µs 6 µs -25.04%
Simulation from_vec_drop_arrow[65536] 13 µs 17.1 µs -24.07%
Simulation take_fsl_u8_random[256, 100] 138.6 µs 182.4 µs -24.04%
Simulation take_fsl_nullable_random[16, 100] 143.2 µs 187.9 µs -23.79%
Simulation from_vec_drop_vortex[65536] 13.6 µs 17.8 µs -23.75%
Simulation take_fsl_f16_force_manual_range_copy[1024, 10] 66 µs 84.6 µs -21.96%
Simulation fixed_16_advancing_ptr_safe[1000] 92.5 µs 118.3 µs -21.79%
Simulation canonicalize[16, 32] 52.4 µs 66.8 µs -21.49%
Simulation chunked_dict_primitive_canonical_into[u32, (1000, 10, 10)] 160.6 µs 200.4 µs -19.86%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/codspeed-bench-mimalloc (1801273) with ct/codspeed-local-only-baselines (5c6275f)

Open in CodSpeed

Footnotes

  1. 293 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@connortsui20
connortsui20 added this pull request to stack #9954 September 20, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants