Support wide decimals in `DecimalBytePartsArray` and kernels by mhk197 · Pull Request #9809 · vortex-data/vortex

mhk197 · 2026-09-08T20:36:13Z

DecimalBytePartsArray previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.

The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:

Decimal storage	Children
`i8` / `i16` / `i32` / `i64`	Signed MSP only; shares the original value buffer
`i128`	`i64` MSP + one `u64` lower part
`i256`	`i64` MSP + three `u64` lower parts

Only the MSP carries validity. Every lower part must be a non-nullable u64 array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain ArrayRefs, so their individual encodings are independent of the decimal representation.

execute::<DecimalArray> reassembles a DecimalArray from the MSP and lower parts children.

take with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.

This PR also refines the splitting and assembly modules.

Assembly takes ArrayRefs instead of PrimitiveArrays so that in the future, we can add special fast paths for constant arrays
Assembly casts lower parts to u64s, allowing for assembly of narrowed lower parts.
Assembly loop is optimized such that it vectorizes for i256 assembly on local runs.

codspeed · 2026-09-08T20:44:01Z

Merging this PR will degrade performance by 3.08%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 3 regressed benchmarks
✅ 2223 untouched benchmarks
🆕 48 new benchmarks
⏩ 204 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`arrow_checked_add_u32_avx2[16384]`	17.7 µs	21.3 µs	-17.11%
❌	Simulation	`decompress[datetime_for_bp]`	160.3 µs	193.3 µs	-17.06%
❌	WallTime	`filtered_owned_i64_avx512[OneNullInEight]`	22.4 µs	26.2 µs	-14.74%
⚡	Simulation	`allocate_drop_arrow[0]`	456.9 ns	402.7 ns	+13.45%
⚡	Simulation	`chunked_bool_canonical_into[(1000, 10)]`	30.7 µs	27.2 µs	+12.9%
⚡	Simulation	`allocate_drop_bytes[0]`	575.7 ns	521.6 ns	+10.39%
🆕	WallTime	`dbp_assemble_kernel_narrow_msp_neon[(I128, 1024)]`	N/A	529 ns	N/A
🆕	WallTime	`dbp_assemble_kernel_narrow_msp_neon[(I128, 8192)]`	N/A	3.8 µs	N/A
🆕	WallTime	`dbp_assemble_kernel_narrow_msp_neon[(I256, 1024)]`	N/A	917 ns	N/A
🆕	WallTime	`dbp_assemble_kernel_narrow_msp_neon[(I256, 8192)]`	N/A	6.7 µs	N/A
🆕	WallTime	`dbp_assemble_kernel_neon[(I128, 1024)]`	N/A	422 ns	N/A
🆕	WallTime	`dbp_assemble_kernel_neon[(I128, 8192)]`	N/A	3.5 µs	N/A
🆕	WallTime	`dbp_assemble_kernel_neon[(I256, 1024)]`	N/A	906 ns	N/A
🆕	WallTime	`dbp_assemble_kernel_neon[(I256, 8192)]`	N/A	6.9 µs	N/A
🆕	WallTime	`dbp_split_kernel_all_valid_neon[(I128, 1024)]`	N/A	501 ns	N/A
🆕	WallTime	`dbp_split_kernel_all_valid_neon[(I128, 8192)]`	N/A	3.2 µs	N/A
🆕	WallTime	`dbp_split_kernel_all_valid_neon[(I256, 1024)]`	N/A	1.3 µs	N/A
🆕	WallTime	`dbp_split_kernel_all_valid_neon[(I256, 8192)]`	N/A	36.9 µs	N/A
🆕	WallTime	`dbp_split_kernel_mixed_null_neon[(I128, 1024)]`	N/A	1.2 µs	N/A
🆕	WallTime	`dbp_split_kernel_mixed_null_neon[(I128, 8192)]`	N/A	8 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing mk/dbp-array (d1a95eb) with mk/dbp-v2-feature (4edb7aa)}

204 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

Represent wide decimals with a signed high part and up to three unsigned low parts. Add validation, execution, kernel support, property tests, and assembly benchmarks while keeping serialization on the frozen format. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Move array helpers onto a crate-private extension trait, preserve decimal precision and scale when replacing the MSP, and group slicing with the other compute operations. Inline canonical execution and select scalar storage directly from the lower-part count. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 · 2026-09-10T18:02:57Z

            .cast(array.msp().dtype().with_nullability(*target_nullability))?;

-        Ok(Some(
-            DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(),


I think this was redundant to begin with because we check above that target dtype is same as current modulo nullability

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs · 2026-09-11T09:15:57Z

    divan::main();
 }

+#[vortex_bench_support::cpu_features]


can you do this over the kernel working with [T] not vortex arrays. This is more noisy so it would be nice to only run if over small loops with allocs

joseph-isaacs · 2026-09-11T09:16:09Z

joseph-isaacs · 2026-09-11T09:16:57Z


    #[test]
    fn test_cast_decimal_byte_parts_nullability() {
        let mut ctx = array_session().create_execution_ctx();
        let decimal_dtype = DecimalDType::new(10, 2);
        let array =
            DecimalByteParts::try_new(buffer![100i32, 200, 300, 400].into_array(), decimal_dtype)
                .unwrap();

        // Cast to nullable decimal
        let casted = array
            .into_array()
            .cast(DType::Decimal(decimal_dtype, Nullability::Nullable))
            .unwrap();
        assert_eq!(
            casted.dtype(),
            &DType::Decimal(decimal_dtype, Nullability::Nullable)
        );

        // Verify the values are preserved
        let decoded = casted.execute::<DecimalArray>(&mut ctx).unwrap();
        assert_eq!(decoded.len(), 4);
    }

    #[test]
    fn test_cast_decimal_byte_parts_nullable_to_non_nullable() {
        let mut ctx = array_session().create_execution_ctx();
        let decimal_dtype = DecimalDType::new(10, 2);
        let array = DecimalByteParts::try_new(
            PrimitiveArray::from_option_iter([Some(100i32), None, Some(300)]).into_array(),
            decimal_dtype,
        )
        .unwrap();

        // Cast to non-nullable should fail due to nulls - force evaluation via execute::<Canonical>
        let result = array
            .into_array()
            .cast(DType::Decimal(decimal_dtype, Nullability::NonNullable))
            .and_then(|a| a.execute::<Canonical>(&mut ctx).map(|c| c.into_array()));
        assert!(result.is_err());
    }

    #[rstest]
    #[case::i32(DecimalByteParts::try_new(
        buffer![100i32, 200, 300, 400, 500].into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
    #[case::i64(DecimalByteParts::try_new(
        buffer![1000i64, 2000, 3000, 4000].into_array(),
        DecimalDType::new(19, 4),
    ).unwrap())]
    #[case::nullable(DecimalByteParts::try_new(
        PrimitiveArray::from_option_iter([Some(100i32), None, Some(300), Some(400), None])
            .into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
    #[case::single(DecimalByteParts::try_new(
        buffer![42i32].into_array(),
        DecimalDType::new(5, 1),
    ).unwrap())]
    #[case::negative(DecimalByteParts::try_new(
        buffer![-100i32, -200, 300, -400, 500].into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
+    #[case::one_lower_part(i128_parts(
+        vec![1i128 << 70, -(1i128 << 70), 5, (1i128 << 64) - 1, 0],
+        Validity::NonNullable,
+    ))]
+    #[case::three_lower_parts(i256_parts(
+        vec![i256_of(1, 0), i256_of(-1, 5), i256_of(0, u128::MAX)],
+        Validity::NonNullable,
+    ))]
    fn test_cast_decimal_byte_parts_conformance(#[case] array: DecimalBytePartsArray) {


we don't need this many new tests

joseph-isaacs · 2026-09-11T09:17:22Z

+        // The MSP alone only determines the ordering when it holds the whole value. With
+        // lower parts present, fall back to comparing the canonical decimal.
+        if !lhs.lower_parts().is_empty() {
+            return Ok(None);
+        }


add a todo saying we could be smarter here

joseph-isaacs · 2026-09-11T09:19:43Z

+impl TakeReduce for DecimalByteParts {
+    fn take(array: ArrayView<'_, Self>, indices: &ArrayRef) -> VortexResult<Option<ArrayRef>> {
+        // Taking with nullable indices makes every taken part nullable, but lower parts must
+        // stay non-nullable `u64` — validity belongs to the MSP alone. Fall back to the
+        // canonical path rather than rebuilding parts we would have to strip nullability from.
+        if indices.dtype().is_nullable() && !array.lower_parts().is_empty() {
+            return Ok(None);
+        }
+
+        array
+            .map_parts(|part| part.take(indices.clone()))
+            .map(|a| Some(a.into_array()))
+    }
+}


add a todo saying we could impl this using fill null or smthing else

joseph-isaacs · 2026-09-11T09:22:08Z

+    let mut value = T::from(msp).vortex_expect("MSP fits in the output type");
+    for part in lower {
+        value = (value << LOWER_PART_BITS)
+            | T::from(part).vortex_expect("lower word fits in the output type");


Can we remove this failure it will likely break simd. Can this ever fail. It should be checked outside the loop once and not checked here?

Can add infallible upcast from u64/i64 to i256 instead

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

`DecimalBytePartsArray` previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability. The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces: | Decimal storage | Children | | --- | --- | | `i8` / `i16` / `i32` / `i64` | Signed MSP only; shares the original value buffer | | `i128` | `i64` MSP + one `u64` lower part | | `i256` | `i64` MSP + three `u64` lower parts | Only the MSP carries validity. Every lower part must be a non-nullable `u64` array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain `ArrayRef`s, so their individual encodings are independent of the decimal representation. `execute::<DecimalArray>` reassembles a `DecimalArray` from the MSP and lower parts children. `take` with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts. This PR also refines the splitting and assembly modules. * Assembly takes `ArrayRef`s instead of `PrimitiveArray`s so that in the future, we can add special fast paths for constant arrays * Assembly casts lower parts to `u64`s, allowing for assembly of narrowed lower parts. * Assembly loop is optimized such that it vectorizes for `i256` assembly on local runs. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>

) ## Summary `DecimalBytePartsArray` stored the whole unscaled decimal value in one signed integer child. That capped it at values that fit in 64 bits. This PR adds support for `i128` and `i256` decimals. Each value is now split into a signed most significant part (MSP) plus up to three unsigned 64-bit lower parts. Every part is an independent child array, so each one compresses on its own. The frozen `vortex.decimal_byte_parts` file format is untouched. Arrays with lower parts serialize under a new `vortex.decimal_byte_parts.v2` format owned by a plugin. This is the integration branch for three reviewed sub-PRs: #9808 (splitting and assembly), #9809 (array and kernels), and #9810 (serde plugin). ## Representation | Decimal storage | Children | | --- | --- | | `i8`, `i16`, `i32`, `i64` | Signed MSP only. Shares the original value buffer. | | `i128` | `i64` MSP holding the high 64 bits, plus one `u64` lower part. | | `i256` | `i64` MSP holding the high 64 bits, plus three `u64` lower parts. | Parts are ordered most significant first. The MSP carries the sign and the null mask. Lower parts are non-nullable unsigned integers. A lower part may use a narrower dtype such as `u8`, `u16`, or `u32` when its values fit. Its position still counts as a full 64-bit window. Splitting writes zeroes at null positions so stray bytes in null slots do not hurt compression of the lower parts. ## Splitting and assembly `DecimalByteParts::encode` splits a `DecimalArray` into parts. `split_decimal` exposes the raw parts for callers that want to build the array themselves. Assembly picks a path from the number of lower parts: - **None.** Reuse the MSP buffer as decimal storage without copying. - **One.** Combine the MSP and the lower part into an `i128`. - **Two.** The lower parts form the low 128 bits of an `i256`. The MSP is sign-extended into the high 128 bits. - **Three.** The MSP and the first lower part form the high 128 bits. The remaining two form the low 128 bits. Assembly casts narrowed lower parts back to `u64` first. The `i256` assembly loop vectorizes on local ARM64 builds. Marking `i256`'s shifts `#[inline]` removed three out-of-line calls per row. | Rows | With `#[inline]` | Without | | --- | --- | --- | | 1,024 | 0.917 µs | 6.207 µs | | 8,192 | 6.332 µs | 48.540 µs | Medians of five alternating release runs. `From<i64>` and `From<u64>` for `i256` are added in `vortex-array`. ## Compute `execute::<DecimalArray>` reassembles the canonical array from all parts. The compare, filter, is-constant, and take kernels understand lower parts. Slice and mask apply per child. Two limits are documented in code. Take with nullable indices on an array with lower parts falls back to canonical execution, because taking each part would make the lower parts nullable. The CUDA kernel rejects arrays with lower parts, because GPU reassembly is not implemented yet. ## Serialization `DecimalBytePartsPlugin` owns both wire formats and picks one from the array layout. | Array layout | Serialized ID | | --- | --- | | MSP only | `vortex.decimal_byte_parts` | | MSP plus one to three lower parts | `vortex.decimal_byte_parts.v2` | The v1 format is frozen. Its metadata and decoder live in `plugin/v1.rs` and are byte-identical to what shipped. The v1 decoder rejects any payload that claims lower parts. The v2 format records the MSP's integer type and one integer type per lower part. The lower part count is the length of that list. The decoder validates every type and restores each child with its recorded dtype. The v2 format itself accepts zero lower parts. The plugin only chooses it when lower parts are present, so files stay readable by older readers whenever possible. The in-memory encoding ID is now `vortex.decimal_byte_parts.v2`. The registry maps both wire IDs to the plugin, so existing v1 files read through it with no migration. No edition declares the v2 format yet. Writing an array with lower parts under an edition that does not permit v2 fails with an explicit error rather than silently falling back. ## Compression The BtrBlocks decimal scheme still narrows decimals that fit in `i64` and wraps them in a single-part array. Wide decimals stay canonical. Nothing in this PR writes the v2 format through the compressor. The scheme now declares `vortex.decimal_byte_parts` as its produced encoding rather than the in-memory ID. Since #9914 that list holds the serialized IDs a scheme writes, and this scheme only ever writes the frozen format. Without that change every writer that filters schemes by edition would drop the decimal scheme, because no edition permits the in-memory v2 name. ## API Changes **Breaking.** Registering `DecimalByteParts` directly no longer supports serde for either format. Replace `session.arrays().register(DecimalByteParts)` with `session.arrays().register(DecimalBytePartsPlugin)`. `vortex_decimal_byte_parts::initialize` already does this. **Breaking.** `dbp_encode` is replaced by `DecimalByteParts::encode`. **Breaking.** `DecimalBytesPartsMetadata` is no longer public. `DecimalBytePartsV2Metadata` is exposed instead. The in-memory encoding ID string changed from `vortex.decimal_byte_parts` to `vortex.decimal_byte_parts.v2`. This affects display and trace output, not files. New public items: `DecimalByteParts::try_new_with_lower_parts`, `DecimalByteParts::encode`, `split_decimal`, `DecimalParts`, `DecimalBytePartsPlugin`, `decimal_byte_parts_v1_id`, and `decimal_byte_parts_v2_id`. --------- Signed-off-by: "Matt Katz" <mhkatz97@gmail.com> Signed-off-by: Matt Katz <mhkatz97@gmail.com>

This was referenced Sep 8, 2026

Compress wide decimals into lower parts when the writer permits the v2 format #9759

Draft

Add wide decimal byte-part splitting and assembly logic #9808

Merged

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format #9810

Merged

mhk197 force-pushed the mk/dbp-array branch 3 times, most recently from f7dfebe to a566a28 Compare September 9, 2026 02:43

mhk197 changed the title ~~Support multi-part decimal arrays and kernels~~ Support wide decimals in DecimalBytePartsArray and kernels Sep 9, 2026

mhk197 marked this pull request as ready for review September 9, 2026 04:17

mhk197 removed this pull request from stack #9811 September 9, 2026 15:09

mhk197 force-pushed the mk/dbp-array branch from f12c08c to 32a80b1 Compare September 9, 2026 15:09

mhk197 added this pull request to stack #9813 September 9, 2026 15:09

mhk197 force-pushed the mk/dbp-array branch from 32a80b1 to 9a179ed Compare September 9, 2026 18:31

mhk197 added changelog/skip Do not list PR in the changelog and removed changelog/skip Do not list PR in the changelog labels Sep 9, 2026

mhk197 force-pushed the mk/dbp-array branch from 9a179ed to 76a0db4 Compare September 9, 2026 20:06

mhk197 requested review from joseph-isaacs and robert3005 September 9, 2026 20:28

Base automatically changed from mk/dbp-parts to mk/dbp-v2-feature September 10, 2026 15:15

mhk197 added 5 commits September 10, 2026 16:15

Share decimal value assembly and validate lower-part dtypes

e30bd99

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Validate required decimal byte-parts slots before typed access

6a7b033

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix comment

a20094e

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs force-pushed the mk/dbp-array branch from 7283d4f to a20094e Compare September 10, 2026 15:15

mhk197 commented Sep 10, 2026

View reviewed changes

mhk197 added 2 commits September 10, 2026 16:26

QOL and perf changes for splitting

38b5bc4

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

comment fix

df7cbfc

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

reorg

b2e49fb

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs reviewed Sep 11, 2026

View reviewed changes

Comment thread encodings/decimal-byte-parts/benches/dbp_split.rs

joseph-isaacs Sep 11, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

mhk197 reacted with thumbs up emoji

joseph-isaacs reviewed Sep 11, 2026

View reviewed changes

mhk197 added 2 commits September 11, 2026 14:31

comments

18b8b99

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

improvements to dbp assembly and cadditional benchmarks

d1a95eb

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 requested a review from joseph-isaacs September 15, 2026 04:21

joseph-isaacs approved these changes Sep 15, 2026

View reviewed changes

mhk197 merged commit 8fc58e0 into mk/dbp-v2-feature Sep 15, 2026
113 of 123 checks passed

mhk197 deleted the mk/dbp-array branch September 15, 2026 13:05

mhk197 mentioned this pull request Sep 16, 2026

Support i128 and i256 decimals in DecimalByteParts encoding #9834

Merged

Navigation Menu

Support wide decimals in `DecimalBytePartsArray` and kernels - #9809

Support wide decimals in `DecimalBytePartsArray` and kernels#9809
mhk197 merged 10 commits into
mk/dbp-v2-featurefrom
mk/dbp-array

mhk197 commented Sep 8, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

mhk197 Sep 10, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

mhk197 Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mhk197 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 3.08%

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mhk197 commented Sep 8, 2026 •

edited

Loading

codspeed Bot commented Sep 8, 2026 •

edited

Loading