Add ArrayPlugin to enable serde for new DecimalBytePartsArray format - #9810
Conversation
Merging this PR will degrade performance by 19.87%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | column_x_constant[2] |
131.3 µs | 454.3 µs | -71.09% |
| ❌ | Simulation | column_x_constant[32] |
132 µs | 390.1 µs | -66.16% |
| ❌ | Simulation | column_x_column[2] |
101.8 µs | 203.1 µs | -49.85% |
| ❌ | Simulation | column_x_constant[256] |
311.9 µs | 590.3 µs | -47.17% |
| ❌ | Simulation | nullable[2] |
324.1 µs | 591 µs | -45.17% |
| ❌ | Simulation | non_nullable[2] |
327.9 µs | 593.2 µs | -44.73% |
| ❌ | Simulation | column_x_extension_constant[2] |
147.1 µs | 246.9 µs | -40.41% |
| ❌ | Simulation | nullable[2] |
416.8 µs | 629.4 µs | -33.78% |
| ❌ | Simulation | non_nullable[2] |
418.9 µs | 630.4 µs | -33.55% |
| ❌ | Simulation | column_x_extension_constant[256] |
277.8 µs | 415.2 µs | -33.09% |
| ❌ | Simulation | random_i8[0.8] |
72.9 µs | 98.1 µs | -25.76% |
| ❌ | Simulation | column_x_extension_constant[32] |
135.2 µs | 167.7 µs | -19.39% |
| ❌ | Simulation | column_x_column[32] |
76.8 µs | 94.1 µs | -18.38% |
| ❌ | Simulation | decompress[datetime_for_bp] |
159.3 µs | 193.6 µs | -17.68% |
| ❌ | WallTime | arrow_checked_add_u32_avx2[16384] |
17.6 µs | 21.3 µs | -17.21% |
| ❌ | Simulation | column_x_column[256] |
75.2 µs | 88.5 µs | -15.04% |
| ❌ | Simulation | optimize_lookup_predicate[ids=16, shape=balanced_or] |
70 µs | 80.9 µs | -13.45% |
| ❌ | WallTime | dbp_split_kernel_mixed_null_neon[(I256, 1024)] |
1.9 µs | 2.2 µs | -12.44% |
| ⚡ | Simulation | random_i16[0.95] |
98.5 µs | 76 µs | +29.6% |
| ⚡ | Simulation | random_i8[0.01] |
45.3 µs | 39.7 µs | +14.22% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/dbp-serde (8ba61cc) with mk/dbp-v2-feature (8fc58e0)
Footnotes
-
229 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
b822fb6 to
e13f457
Compare
a68b2ae to
b970a8f
Compare
ArrayPlugin to enable serde for new DecimalBytePartsArray format
03a13f8 to
72f2396
Compare
72f2396 to
d846ec7
Compare
Use one ArrayPlugin for the frozen single-part format and the new wide format. Preserve frozen files with wider physical storage and add wire contract tests plus an opt-in compatibility fixture. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths. Include the v2 compatibility fixture in the default suite without enabling unstable encodings. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
d846ec7 to
46de82a
Compare
| pub fn decimal_byte_parts_v2_id() -> ArrayId { | ||
| static ID: CachedId = CachedId::new("vortex.decimal_byte_parts_v2"); | ||
| *ID |
There was a problem hiding this comment.
I thought the new array in mem would have v2? So we would have v1 == "" here
| pub fn decimal_byte_parts_v2_id() -> ArrayId { | |
| static ID: CachedId = CachedId::new("vortex.decimal_byte_parts_v2"); | |
| *ID | |
| pub fn decimal_byte_parts_v2_id() -> ArrayId { | |
| static ID: CachedId = CachedId::new("vortex.decimal_byte_parts.v2"); | |
| *ID |
Keep one plugin dispatcher with separate v1 and v2 metadata and serde functions operating directly on the current DBP array. Preserve the current draft, including the latest metadata names and validation edits, for comparison against an ArrayRepresentation prototype. Snapshot validation: 272 tests passed and 3 failed before fail-fast cancelled the remaining 22 tests. The failures cover accepting zero lower parts in v2 and two expectations of the previous error wording. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
1389183 to
8ba61cc
Compare
| "v2 carries at most {MAX_LOWER_PARTS} lower parts, got {lower_part_count}" | ||
| ); | ||
| vortex_ensure!( | ||
| parts.children.len() == 1 + lower_part_count, |
There was a problem hiding this comment.
we really need to remove children from the vtable and only use slots
…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
) ## Summary `DecimalBytePartsArray` stored the whole unscaled decimal value in one signed integer child. That capped it at values that fit in 64 bits. This PR adds support for `i128` and `i256` decimals. Each value is now split into a signed most significant part (MSP) plus up to three unsigned 64-bit lower parts. Every part is an independent child array, so each one compresses on its own. The frozen `vortex.decimal_byte_parts` file format is untouched. Arrays with lower parts serialize under a new `vortex.decimal_byte_parts.v2` format owned by a plugin. This is the integration branch for three reviewed sub-PRs: #9808 (splitting and assembly), #9809 (array and kernels), and #9810 (serde plugin). ## Representation | Decimal storage | Children | | --- | --- | | `i8`, `i16`, `i32`, `i64` | Signed MSP only. Shares the original value buffer. | | `i128` | `i64` MSP holding the high 64 bits, plus one `u64` lower part. | | `i256` | `i64` MSP holding the high 64 bits, plus three `u64` lower parts. | Parts are ordered most significant first. The MSP carries the sign and the null mask. Lower parts are non-nullable unsigned integers. A lower part may use a narrower dtype such as `u8`, `u16`, or `u32` when its values fit. Its position still counts as a full 64-bit window. Splitting writes zeroes at null positions so stray bytes in null slots do not hurt compression of the lower parts. ## Splitting and assembly `DecimalByteParts::encode` splits a `DecimalArray` into parts. `split_decimal` exposes the raw parts for callers that want to build the array themselves. Assembly picks a path from the number of lower parts: - **None.** Reuse the MSP buffer as decimal storage without copying. - **One.** Combine the MSP and the lower part into an `i128`. - **Two.** The lower parts form the low 128 bits of an `i256`. The MSP is sign-extended into the high 128 bits. - **Three.** The MSP and the first lower part form the high 128 bits. The remaining two form the low 128 bits. Assembly casts narrowed lower parts back to `u64` first. The `i256` assembly loop vectorizes on local ARM64 builds. Marking `i256`'s shifts `#[inline]` removed three out-of-line calls per row. | Rows | With `#[inline]` | Without | | --- | --- | --- | | 1,024 | 0.917 µs | 6.207 µs | | 8,192 | 6.332 µs | 48.540 µs | Medians of five alternating release runs. `From<i64>` and `From<u64>` for `i256` are added in `vortex-array`. ## Compute `execute::<DecimalArray>` reassembles the canonical array from all parts. The compare, filter, is-constant, and take kernels understand lower parts. Slice and mask apply per child. Two limits are documented in code. Take with nullable indices on an array with lower parts falls back to canonical execution, because taking each part would make the lower parts nullable. The CUDA kernel rejects arrays with lower parts, because GPU reassembly is not implemented yet. ## Serialization `DecimalBytePartsPlugin` owns both wire formats and picks one from the array layout. | Array layout | Serialized ID | | --- | --- | | MSP only | `vortex.decimal_byte_parts` | | MSP plus one to three lower parts | `vortex.decimal_byte_parts.v2` | The v1 format is frozen. Its metadata and decoder live in `plugin/v1.rs` and are byte-identical to what shipped. The v1 decoder rejects any payload that claims lower parts. The v2 format records the MSP's integer type and one integer type per lower part. The lower part count is the length of that list. The decoder validates every type and restores each child with its recorded dtype. The v2 format itself accepts zero lower parts. The plugin only chooses it when lower parts are present, so files stay readable by older readers whenever possible. The in-memory encoding ID is now `vortex.decimal_byte_parts.v2`. The registry maps both wire IDs to the plugin, so existing v1 files read through it with no migration. No edition declares the v2 format yet. Writing an array with lower parts under an edition that does not permit v2 fails with an explicit error rather than silently falling back. ## Compression The BtrBlocks decimal scheme still narrows decimals that fit in `i64` and wraps them in a single-part array. Wide decimals stay canonical. Nothing in this PR writes the v2 format through the compressor. The scheme now declares `vortex.decimal_byte_parts` as its produced encoding rather than the in-memory ID. Since #9914 that list holds the serialized IDs a scheme writes, and this scheme only ever writes the frozen format. Without that change every writer that filters schemes by edition would drop the decimal scheme, because no edition permits the in-memory v2 name. ## API Changes **Breaking.** Registering `DecimalByteParts` directly no longer supports serde for either format. Replace `session.arrays().register(DecimalByteParts)` with `session.arrays().register(DecimalBytePartsPlugin)`. `vortex_decimal_byte_parts::initialize` already does this. **Breaking.** `dbp_encode` is replaced by `DecimalByteParts::encode`. **Breaking.** `DecimalBytesPartsMetadata` is no longer public. `DecimalBytePartsV2Metadata` is exposed instead. The in-memory encoding ID string changed from `vortex.decimal_byte_parts` to `vortex.decimal_byte_parts.v2`. This affects display and trace output, not files. New public items: `DecimalByteParts::try_new_with_lower_parts`, `DecimalByteParts::encode`, `split_decimal`, `DecimalParts`, `DecimalBytePartsPlugin`, `decimal_byte_parts_v1_id`, and `decimal_byte_parts_v2_id`. --------- Signed-off-by: "Matt Katz" <mhkatz97@gmail.com> Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Summary
Add serialization for
DecimalBytePartsArraywith lower parts while preserving the frozen v1 file format.DecimalBytePartsPluginowns both formats and selects the serialized ID from the array's layout:vortex.decimal_byte_partsvortex.decimal_byte_parts_v2Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to
u8,u16, oru32when its values fit.Changes
serializeanddeserializemethods return errors directing callers toDecimalBytePartsPlugin. Keep DBP metadata and serde helpers inplugin.rsso the dedicated plugin owns both formats.i128values and nullablei256values. The frozen fixture is unchanged.API Changes
Breaking: directly registering
DecimalByteParts, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace:with:
vortex_decimal_byte_parts::initialize(&session)already registers the plugin. Existing v1 files remain readable through it; no file migration is required.