Skip edition-enabled encodings with no registered plugin when writing - #9917
Conversation
…iting Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Merging this PR will degrade performance by 12.11%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
12.9 µs | 20.3 µs | -36.77% |
| ❌ | WallTime | mul_u32_nonnull_avx512 |
5.6 µs | 6.3 µs | -10.92% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
635.5 ns | 527.2 ns | +20.55% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/writer-registered-ids (6fce0df) with develop (c11856c)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Summary
#9914 stopped mapping the writer's permitted serialized IDs through the array registry before filtering compression schemes. That mapping had a side effect on the scheme filter only: an edition-enabled ID with no registered plugin produced
Nonein thefilter_mapand was excluded from the set handed to the compressor. The array context never had that filter. It seeded and permitted every enabled ID whether or not a plugin was registered.This PR filters at the source, so both the scheme set and the array context exclude enabled IDs that nothing on the session could serialize.
Changes
With editions enforced,
new_array_contextnow keeps only the enabled array IDs that have a registered plugin. That set seeds the file's encoding table and, throughto_ids, restricts the compressor. Without edition enforcement the set comes from the registry itself and needs no filter.Behavior
Before #9914, a session that enabled an edition but did not register one of its encodings never selected that encoding's compression scheme. After #9914 the scheme could be selected, and the write then failed in
array_serializewith "not registered for serialization". This PR returns to skipping the scheme.Excluding those IDs from the array context is new behavior. An unregistered encoding can never be serialized, since
array_serializefails before any interning, so its entry in the file's encoding table was never used. The footer changes only for sessions in that partial-registry state. Every shipped writer registers all encodings, so none are affected.Testing
A new test enables an edition listing
vortex.primitiveandvortex.alpon a bare array session, where ALP is not registered, and checks that only the primitive ID is seeded and ALP cannot be interned.