Skip to content

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format - #9810

Merged
mhk197 merged 17 commits into
mk/dbp-v2-featurefrom
mk/dbp-serde
Sep 16, 2026
Merged

mhk197 merged 17 commits into
mk/dbp-v2-featurefrom
mk/dbp-serde

Conversation

@mhk197

@mhk197 mhk197 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Add serialization for DecimalBytePartsArray with lower parts while preserving the frozen v1 file format. DecimalBytePartsPlugin owns both formats and selects the serialized ID from the array's layout:

Array layout Serialized ID
Signed most significant part (MSP) only vortex.decimal_byte_parts
Signed MSP plus one to three unsigned lower parts vortex.decimal_byte_parts_v2

Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to u8, u16, or u32 when its values fit.

Changes

  • Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype.
  • Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical.
  • Make DBP's VTable serialize and deserialize methods return errors directing callers to DecimalBytePartsPlugin. Keep DBP metadata and serde helpers in plugin.rs so the dedicated plugin owns both formats.
  • Add a separate v2 compatibility fixture covering positive and negative wide i128 values and nullable i256 values. The frozen fixture is unchanged.

API Changes

Breaking: directly registering DecimalByteParts, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace:

session.arrays().register(DecimalByteParts);

with:

session.arrays().register(DecimalBytePartsPlugin);

vortex_decimal_byte_parts::initialize(&session) already registers the plugin. Existing v1 files remain readable through it; no file migration is required.

@codspeed

codspeed Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 19.87%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 12 improved benchmarks
❌ 18 regressed benchmarks
✅ 2205 untouched benchmarks
⏩ 229 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation column_x_constant[2] 131.3 µs 454.3 µs -71.09%
Simulation column_x_constant[32] 132 µs 390.1 µs -66.16%
Simulation column_x_column[2] 101.8 µs 203.1 µs -49.85%
Simulation column_x_constant[256] 311.9 µs 590.3 µs -47.17%
Simulation nullable[2] 324.1 µs 591 µs -45.17%
Simulation non_nullable[2] 327.9 µs 593.2 µs -44.73%
Simulation column_x_extension_constant[2] 147.1 µs 246.9 µs -40.41%
Simulation nullable[2] 416.8 µs 629.4 µs -33.78%
Simulation non_nullable[2] 418.9 µs 630.4 µs -33.55%
Simulation column_x_extension_constant[256] 277.8 µs 415.2 µs -33.09%
Simulation random_i8[0.8] 72.9 µs 98.1 µs -25.76%
Simulation column_x_extension_constant[32] 135.2 µs 167.7 µs -19.39%
Simulation column_x_column[32] 76.8 µs 94.1 µs -18.38%
Simulation decompress[datetime_for_bp] 159.3 µs 193.6 µs -17.68%
WallTime arrow_checked_add_u32_avx2[16384] 17.6 µs 21.3 µs -17.21%
Simulation column_x_column[256] 75.2 µs 88.5 µs -15.04%
Simulation optimize_lookup_predicate[ids=16, shape=balanced_or] 70 µs 80.9 µs -13.45%
WallTime dbp_split_kernel_mixed_null_neon[(I256, 1024)] 1.9 µs 2.2 µs -12.44%
Simulation random_i16[0.95] 98.5 µs 76 µs +29.6%
Simulation random_i8[0.01] 45.3 µs 39.7 µs +14.22%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/dbp-serde (8ba61cc) with mk/dbp-v2-feature (8fc58e0)

Open in CodSpeed

Footnotes

  1. 229 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197
mhk197 force-pushed the mk/dbp-serde branch 2 times, most recently from b822fb6 to e13f457 Compare September 9, 2026 02:43
@mhk197
mhk197 force-pushed the mk/dbp-serde branch 2 times, most recently from a68b2ae to b970a8f Compare September 9, 2026 04:15
@mhk197
mhk197 marked this pull request as ready for review September 9, 2026 04:49
@mhk197 mhk197 changed the title Register versioned decimal byte-part serialization Add ArrayPlugin to enable serde for new DecimalBytePartsArray format Sep 9, 2026
@mhk197
mhk197 removed this pull request from stack #9811 September 9, 2026 15:09
@mhk197
mhk197 added this pull request to stack #9813 September 9, 2026 15:09
@mhk197 mhk197 added the changelog/skip Do not list PR in the changelog label Sep 9, 2026
@mhk197 mhk197 added changelog/break A breaking API change and removed changelog/skip Do not list PR in the changelog labels Sep 15, 2026
Base automatically changed from mk/dbp-array to mk/dbp-v2-feature September 15, 2026 13:05
Use one ArrayPlugin for the frozen single-part format and the new wide
format. Preserve frozen files with wider physical storage and add wire
contract tests plus an opt-in compatibility fixture.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths.

Include the v2 compatibility fixture in the default suite without enabling unstable encodings.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Comment on lines +39 to +41
pub fn decimal_byte_parts_v2_id() -> ArrayId {
static ID: CachedId = CachedId::new("vortex.decimal_byte_parts_v2");
*ID

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the new array in mem would have v2? So we would have v1 == "" here

Suggested change
pub fn decimal_byte_parts_v2_id() -> ArrayId {
static ID: CachedId = CachedId::new("vortex.decimal_byte_parts_v2");
*ID
pub fn decimal_byte_parts_v2_id() -> ArrayId {
static ID: CachedId = CachedId::new("vortex.decimal_byte_parts.v2");
*ID

Keep one plugin dispatcher with separate v1 and v2 metadata and serde
functions operating directly on the current DBP array. Preserve the
current draft, including the latest metadata names and validation edits,
for comparison against an ArrayRepresentation prototype.

Snapshot validation: 272 tests passed and 3 failed before fail-fast
cancelled the remaining 22 tests. The failures cover accepting zero lower
parts in v2 and two expectations of the previous error wording.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
"v2 carries at most {MAX_LOWER_PARTS} lower parts, got {lower_part_count}"
);
vortex_ensure!(
parts.children.len() == 1 + lower_part_count,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we really need to remove children from the vtable and only use slots

@mhk197
mhk197 merged commit bbd1a50 into mk/dbp-v2-feature Sep 16, 2026
94 of 100 checks passed
@mhk197
mhk197 deleted the mk/dbp-serde branch September 16, 2026 14:04
mhk197 added a commit that referenced this pull request Sep 16, 2026
…mat (#9810)

Add serialization for `DecimalBytePartsArray` with lower parts while
preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both
formats and selects the serialized ID from the array's layout:

| Array layout | Serialized ID |
| --- | --- |
| Signed most significant part (MSP) only | `vortex.decimal_byte_parts`
|
| Signed MSP plus one to three unsigned lower parts |
`vortex.decimal_byte_parts_v2` |

Both formats deserialize into the same in-memory array representation.
Each lower part represents a 64-bit window, but its storage dtype may be
narrowed to `u8`, `u16`, or `u32` when its values fit.

- Record the MSP's physical integer type, lower-part count, and each
lower part's physical integer type in metadata. Deserialization
validates these types and restores each child with its recorded dtype.
- Keep the frozen format restricted to zero lower parts and require at
least one for v2. Frozen metadata remains byte-identical.
- Make DBP's VTable `serialize` and `deserialize` methods return errors
directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and
serde helpers in `plugin.rs` so the dedicated plugin owns both formats.
- Add a separate v2 compatibility fixture covering positive and negative
wide `i128` values and nullable `i256` values. The frozen fixture is
unchanged.

**Breaking:** directly registering `DecimalByteParts`, or calling its
VTable serde methods, now errors for both serialization and
deserialization, including the frozen v1 format. Replace:

```rust
session.arrays().register(DecimalByteParts);
```

with:

```rust
session.arrays().register(DecimalBytePartsPlugin);
```

`vortex_decimal_byte_parts::initialize(&session)` already registers the
plugin. Existing v1 files remain readable through it; no file migration
is required.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
mhk197 added a commit that referenced this pull request Sep 16, 2026
…mat (#9810)

Add serialization for `DecimalBytePartsArray` with lower parts while
preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both
formats and selects the serialized ID from the array's layout:

| Array layout | Serialized ID |
| --- | --- |
| Signed most significant part (MSP) only | `vortex.decimal_byte_parts`
|
| Signed MSP plus one to three unsigned lower parts |
`vortex.decimal_byte_parts_v2` |

Both formats deserialize into the same in-memory array representation.
Each lower part represents a 64-bit window, but its storage dtype may be
narrowed to `u8`, `u16`, or `u32` when its values fit.

- Record the MSP's physical integer type, lower-part count, and each
lower part's physical integer type in metadata. Deserialization
validates these types and restores each child with its recorded dtype.
- Keep the frozen format restricted to zero lower parts and require at
least one for v2. Frozen metadata remains byte-identical.
- Make DBP's VTable `serialize` and `deserialize` methods return errors
directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and
serde helpers in `plugin.rs` so the dedicated plugin owns both formats.
- Add a separate v2 compatibility fixture covering positive and negative
wide `i128` values and nullable `i256` values. The frozen fixture is
unchanged.

**Breaking:** directly registering `DecimalByteParts`, or calling its
VTable serde methods, now errors for both serialization and
deserialization, including the frozen v1 format. Replace:

```rust
session.arrays().register(DecimalByteParts);
```

with:

```rust
session.arrays().register(DecimalBytePartsPlugin);
```

`vortex_decimal_byte_parts::initialize(&session)` already registers the
plugin. Existing v1 files remain readable through it; no file migration
is required.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
mhk197 added a commit that referenced this pull request Sep 16, 2026
…mat (#9810)

Add serialization for `DecimalBytePartsArray` with lower parts while
preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both
formats and selects the serialized ID from the array's layout:

| Array layout | Serialized ID |
| --- | --- |
| Signed most significant part (MSP) only | `vortex.decimal_byte_parts`
|
| Signed MSP plus one to three unsigned lower parts |
`vortex.decimal_byte_parts_v2` |

Both formats deserialize into the same in-memory array representation.
Each lower part represents a 64-bit window, but its storage dtype may be
narrowed to `u8`, `u16`, or `u32` when its values fit.

- Record the MSP's physical integer type, lower-part count, and each
lower part's physical integer type in metadata. Deserialization
validates these types and restores each child with its recorded dtype.
- Keep the frozen format restricted to zero lower parts and require at
least one for v2. Frozen metadata remains byte-identical.
- Make DBP's VTable `serialize` and `deserialize` methods return errors
directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and
serde helpers in `plugin.rs` so the dedicated plugin owns both formats.
- Add a separate v2 compatibility fixture covering positive and negative
wide `i128` values and nullable `i256` values. The frozen fixture is
unchanged.

**Breaking:** directly registering `DecimalByteParts`, or calling its
VTable serde methods, now errors for both serialization and
deserialization, including the frozen v1 format. Replace:

```rust
session.arrays().register(DecimalByteParts);
```

with:

```rust
session.arrays().register(DecimalBytePartsPlugin);
```

`vortex_decimal_byte_parts::initialize(&session)` already registers the
plugin. Existing v1 files remain readable through it; no file migration
is required.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
mhk197 added a commit that referenced this pull request Sep 17, 2026
…mat (#9810)

Add serialization for `DecimalBytePartsArray` with lower parts while
preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both
formats and selects the serialized ID from the array's layout:

| Array layout | Serialized ID |
| --- | --- |
| Signed most significant part (MSP) only | `vortex.decimal_byte_parts`
|
| Signed MSP plus one to three unsigned lower parts |
`vortex.decimal_byte_parts_v2` |

Both formats deserialize into the same in-memory array representation.
Each lower part represents a 64-bit window, but its storage dtype may be
narrowed to `u8`, `u16`, or `u32` when its values fit.

- Record the MSP's physical integer type, lower-part count, and each
lower part's physical integer type in metadata. Deserialization
validates these types and restores each child with its recorded dtype.
- Keep the frozen format restricted to zero lower parts and require at
least one for v2. Frozen metadata remains byte-identical.
- Make DBP's VTable `serialize` and `deserialize` methods return errors
directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and
serde helpers in `plugin.rs` so the dedicated plugin owns both formats.
- Add a separate v2 compatibility fixture covering positive and negative
wide `i128` values and nullable `i256` values. The frozen fixture is
unchanged.

**Breaking:** directly registering `DecimalByteParts`, or calling its
VTable serde methods, now errors for both serialization and
deserialization, including the frozen v1 format. Replace:

```rust
session.arrays().register(DecimalByteParts);
```

with:

```rust
session.arrays().register(DecimalBytePartsPlugin);
```

`vortex_decimal_byte_parts::initialize(&session)` already registers the
plugin. Existing v1 files remain readable through it; no file migration
is required.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
mhk197 added a commit that referenced this pull request Sep 18, 2026
)

## Summary

`DecimalBytePartsArray` stored the whole unscaled decimal value in one
signed integer child. That capped it at values that fit in 64 bits. This
PR adds support for `i128` and `i256` decimals.

Each value is now split into a signed most significant part (MSP) plus
up to three unsigned 64-bit lower parts. Every part is an independent
child array, so each one compresses on its own. The frozen
`vortex.decimal_byte_parts` file format is untouched. Arrays with lower
parts serialize under a new `vortex.decimal_byte_parts.v2` format owned
by a plugin.

This is the integration branch for three reviewed sub-PRs: #9808
(splitting and assembly), #9809 (array and kernels), and #9810 (serde
plugin).

## Representation

| Decimal storage | Children |
| --- | --- |
| `i8`, `i16`, `i32`, `i64` | Signed MSP only. Shares the original value
buffer. |
| `i128` | `i64` MSP holding the high 64 bits, plus one `u64` lower
part. |
| `i256` | `i64` MSP holding the high 64 bits, plus three `u64` lower
parts. |

Parts are ordered most significant first. The MSP carries the sign and
the null mask. Lower parts are non-nullable unsigned integers. A lower
part may use a narrower dtype such as `u8`, `u16`, or `u32` when its
values fit. Its position still counts as a full 64-bit window. Splitting
writes zeroes at null positions so stray bytes in null slots do not hurt
compression of the lower parts.

## Splitting and assembly

`DecimalByteParts::encode` splits a `DecimalArray` into parts.
`split_decimal` exposes the raw parts for callers that want to build the
array themselves.

Assembly picks a path from the number of lower parts:

- **None.** Reuse the MSP buffer as decimal storage without copying.
- **One.** Combine the MSP and the lower part into an `i128`.
- **Two.** The lower parts form the low 128 bits of an `i256`. The MSP
is sign-extended into the high 128 bits.
- **Three.** The MSP and the first lower part form the high 128 bits.
The remaining two form the low 128 bits.

Assembly casts narrowed lower parts back to `u64` first. The `i256`
assembly loop vectorizes on local ARM64 builds. Marking `i256`'s shifts
`#[inline]` removed three out-of-line calls per row.

| Rows | With `#[inline]` | Without |
| --- | --- | --- |
| 1,024 | 0.917 µs | 6.207 µs |
| 8,192 | 6.332 µs | 48.540 µs |

Medians of five alternating release runs. `From<i64>` and `From<u64>`
for `i256` are added in `vortex-array`.

## Compute

`execute::<DecimalArray>` reassembles the canonical array from all
parts. The compare, filter, is-constant, and take kernels understand
lower parts. Slice and mask apply per child.

Two limits are documented in code. Take with nullable indices on an
array with lower parts falls back to canonical execution, because taking
each part would make the lower parts nullable. The CUDA kernel rejects
arrays with lower parts, because GPU reassembly is not implemented yet.

## Serialization

`DecimalBytePartsPlugin` owns both wire formats and picks one from the
array layout.

| Array layout | Serialized ID |
| --- | --- |
| MSP only | `vortex.decimal_byte_parts` |
| MSP plus one to three lower parts | `vortex.decimal_byte_parts.v2` |

The v1 format is frozen. Its metadata and decoder live in `plugin/v1.rs`
and are byte-identical to what shipped. The v1 decoder rejects any
payload that claims lower parts.

The v2 format records the MSP's integer type and one integer type per
lower part. The lower part count is the length of that list. The decoder
validates every type and restores each child with its recorded dtype.
The v2 format itself accepts zero lower parts. The plugin only chooses
it when lower parts are present, so files stay readable by older readers
whenever possible.

The in-memory encoding ID is now `vortex.decimal_byte_parts.v2`. The
registry maps both wire IDs to the plugin, so existing v1 files read
through it with no migration.

No edition declares the v2 format yet. Writing an array with lower parts
under an edition that does not permit v2 fails with an explicit error
rather than silently falling back.

## Compression

The BtrBlocks decimal scheme still narrows decimals that fit in `i64`
and wraps them in a single-part array. Wide decimals stay canonical.
Nothing in this PR writes the v2 format through the compressor.

The scheme now declares `vortex.decimal_byte_parts` as its produced
encoding rather than the in-memory ID. Since #9914 that list holds the
serialized IDs a scheme writes, and this scheme only ever writes the
frozen format. Without that change every writer that filters schemes by
edition would drop the decimal scheme, because no edition permits the
in-memory v2 name.

## API Changes

**Breaking.** Registering `DecimalByteParts` directly no longer supports
serde for either format. Replace
`session.arrays().register(DecimalByteParts)` with
`session.arrays().register(DecimalBytePartsPlugin)`.
`vortex_decimal_byte_parts::initialize` already does this.

**Breaking.** `dbp_encode` is replaced by `DecimalByteParts::encode`.

**Breaking.** `DecimalBytesPartsMetadata` is no longer public.
`DecimalBytePartsV2Metadata` is exposed instead.

The in-memory encoding ID string changed from
`vortex.decimal_byte_parts` to `vortex.decimal_byte_parts.v2`. This
affects display and trace output, not files.

New public items: `DecimalByteParts::try_new_with_lower_parts`,
`DecimalByteParts::encode`, `split_decimal`, `DecimalParts`,
`DecimalBytePartsPlugin`, `decimal_byte_parts_v1_id`, and
`decimal_byte_parts_v2_id`.

---------

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants