bench: use mimalloc as the global allocator in every benchmark binary - #9953
connortsui20 wants to merge 1 commit into
Conversation
Twenty-two benchmark binaries already used mimalloc, on the advice of the benchmarking guide; the other 81 used the system allocator, whose cost depends on state that differs between runner images and between runs. Several simulation benchmarks that allocate inside the timed region flipped between two values on pull requests that could not have affected them, among them split_collection's cold_misaligned, fsst_compress's compress_fsst, and take_chunked's repeated-index case. Give every benchmark binary the same allocator, add the workspace mimalloc dev-dependency to the 16 crates that lacked it, and make the guide's recommendation a rule. The CUDA and DuckDB benchmarks are left alone: they run on different legs and link C++ code. Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will regress 97 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | collect_bool_u32_gt[1024] |
10.5 µs | 24.3 µs | -56.89% |
| ❌ | Simulation | preverify_advancing_ptr_unchecked[1000] |
11.7 µs | 23.9 µs | -51% |
| ❌ | Simulation | take_fsl_f16_random[16, 100] |
100.6 µs | 176.5 µs | -43.02% |
| ❌ | Simulation | execute_mask_then_value[256] |
22.1 µs | 37.5 µs | -41.1% |
| ❌ | Simulation | fixed_16_advancing_ptr_safe[100] |
19.9 µs | 31.8 µs | -37.53% |
| ❌ | Simulation | sum_v2_i32_nulls_scattered |
463.8 µs | 741 µs | -37.41% |
| ❌ | Simulation | max_i32_nulls_scattered |
499.6 µs | 778.1 µs | -35.79% |
| ❌ | Simulation | allocate_drop_vortex_minimal_alignment[1024] |
63.7 µs | 95.3 µs | -33.22% |
| ❌ | Simulation | from_vec_drop_bytes[65536] |
10.3 µs | 14.6 µs | -29.49% |
| ❌ | Simulation | preverify_advancing_ptr_unchecked[10000] |
27 µs | 38 µs | -28.91% |
| ❌ | Simulation | take_fsl_f16_force_manual_range_copy[512, 10] |
47.3 µs | 64.1 µs | -26.29% |
| ❌ | Simulation | bitwise_and_vortex_buffer[1024] |
4.5 µs | 6 µs | -25.04% |
| ❌ | Simulation | from_vec_drop_arrow[65536] |
13 µs | 17.1 µs | -24.07% |
| ❌ | Simulation | take_fsl_u8_random[256, 100] |
138.6 µs | 182.4 µs | -24.04% |
| ❌ | Simulation | take_fsl_nullable_random[16, 100] |
143.2 µs | 187.9 µs | -23.79% |
| ❌ | Simulation | from_vec_drop_vortex[65536] |
13.6 µs | 17.8 µs | -23.75% |
| ❌ | Simulation | take_fsl_f16_force_manual_range_copy[1024, 10] |
66 µs | 84.6 µs | -21.96% |
| ❌ | Simulation | fixed_16_advancing_ptr_safe[1000] |
92.5 µs | 118.3 µs | -21.79% |
| ❌ | Simulation | canonicalize[16, 32] |
52.4 µs | 66.8 µs | -21.49% |
| ❌ | Simulation | chunked_dict_primitive_canonical_into[u32, (1000, 10, 10)] |
160.6 µs | 200.4 µs | -19.86% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/codspeed-bench-mimalloc (1801273) with ct/codspeed-local-only-baselines (5c6275f)
Footnotes
-
293 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Summary
Third of three stacked PRs on CodSpeed flakiness. The first resizes benchmarks, the second removes baselines from CodSpeed.
Twenty-two benchmark binaries already use
mimallocas their global allocator, on the advice of the benchmarking guide; the other 81 use the system allocator. Several simulation benchmarks that allocate inside the timed region flipped between two values on pull requests that could not have affected them:split_collection.rs:cold_misaligned[(16, 64)]fsst_compress.rs:compress_fsst[(500, 64, 8)]take_chunked.rs:take[duplicates/repeated/...]take_primitive.rs:dict_canonicalize_gt_u8[1000000](avx2 leg)The system allocator's cost depends on its state, which differs between runner images and between runs;
mimallocdoes the same work every time. One allocator for every benchmark binary also means no benchmark measures a different allocator from its neighbours.Changes
mimallocas its global allocator (81 files). The 16 crates that did not depend on it get the workspacemimallocdev-dependency, soCargo.lockgainsmimallocin their dependency lists and nothing else.The CodSpeed report on this PR will show a change for most simulation benchmarks, as expected from swapping the allocator under them. Whether the flips above stop can only be confirmed over the coming pull requests. Two of the flaky benchmarks that already had
mimalloc(filter_fixed_width.rsandrun_end_compress.rs) needed resizing in the first PR instead, so this removes one variable rather than guaranteeing stability.Checks run:
cargo clippy --locked --bencheswith-D warningson every touched crate (vortex-alp,vortex-datetime-parts,vortex-decimal-byte-parts,vortex-fastlanes,vortex-fsst,vortex-onpair,vortex-pco,vortex-runend,vortex-sparse,vortex-array,vortex-arrow,vortex-btrblocks,vortex-buffer,vortex-compressor,vortex-compute,vortex-file,vortex-layout,vortex-mask,vortex)cargo +nightly-2026-09-10 fmt --allandtaplo fmt --checkon the touched manifests🤖 Generated with Claude Code
https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn
Generated by Claude Code