Support i128 and i256 decimals in DecimalByteParts encoding - #9834
Conversation
Merging this PR will regress 2 benchmarks
|
e0d2bf4 to
ac1f5a0
Compare
ac1f5a0 to
c787611
Compare
) ## Summary Compression schemes are filtered against the IDs a writer may emit. Today that filter mixes two ID spaces. Schemes declare their in-memory encoding ID, while most callers pass the serialized IDs from their enabled editions. The two only line up because every encoding currently uses the same string for both. `DecimalByteParts` stops doing that in #9834. Its in-memory ID becomes `vortex.decimal_byte_parts.v2` while the frozen wire ID stays `vortex.decimal_byte_parts`. With the current filter, every caller that passes edition IDs would silently drop the decimal scheme and write decimals canonical. This PR makes the contract explicit. Schemes declare the serialized IDs they write, and every caller filters with serialized IDs. ## Changes - The file writer no longer maps permitted serialized IDs through the registry to plugin in-memory IDs before filtering schemes. `new_array_context` returns only the array context, which is seeded with exactly the permitted IDs, and the writer reads them back with `to_ids` into a named set for the compressor builder. This matches the Python, TUI, CUDA, bench, and golden-test callers, which already pass serialized IDs. ## Behavior No scheme changes behavior. In-memory and wire IDs coincide for every scheme on develop, so the same schemes are retained from the same inputs and no snapshot changes. ## Follow-ups - #9834 declares `vortex.decimal_byte_parts` from the decimal scheme on top of this. - #9770 builds on this with a permitted-ID snapshot in `CompressorContext`, so a scheme can choose a wire format when it compresses. If the method names should say serialized IDs, renaming `produced_encodings` and `retain_allowed_encodings` together belongs there. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
c787611 to
c5ea5c5
Compare
Signed-off-by: "Matt Katz" <mhkatz97@gmail.com> Signed-off-by: Matt Katz <mhkatz97@gmail.com>
`DecimalBytePartsArray` previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability. The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces: | Decimal storage | Children | | --- | --- | | `i8` / `i16` / `i32` / `i64` | Signed MSP only; shares the original value buffer | | `i128` | `i64` MSP + one `u64` lower part | | `i256` | `i64` MSP + three `u64` lower parts | Only the MSP carries validity. Every lower part must be a non-nullable `u64` array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain `ArrayRef`s, so their individual encodings are independent of the decimal representation. `execute::<DecimalArray>` reassembles a `DecimalArray` from the MSP and lower parts children. `take` with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts. This PR also refines the splitting and assembly modules. * Assembly takes `ArrayRef`s instead of `PrimitiveArray`s so that in the future, we can add special fast paths for constant arrays * Assembly casts lower parts to `u64`s, allowing for assembly of narrowed lower parts. * Assembly loop is optimized such that it vectorizes for `i256` assembly on local runs. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
35adc87 to
026bb26
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.058x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.088x ➖, 2↑ 5↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 0↓)
File Size Changes (100 files changed, +0.0% overall, 48↑ 52↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.836x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.155x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (1.066x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.068x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 2↑ 3↓)
duckdb / parquet / ns (1.004x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 2↑ 4↓)
duckdb / parquet / ns (1.007x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.028x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.967x ➖, 2↑ 2↓)
duckdb / parquet / ns (1.030x ➖, 0↑ 0↓)
|
|
The only thing here is we should ensure that we actually have the number of children that matches the dtype when deserialising |
Summary
DecimalBytePartsArraystored the whole unscaled decimal value in one signed integer child. That capped it at values that fit in 64 bits. This PR adds support fori128andi256decimals.Each value is now split into a signed most significant part (MSP) plus up to three unsigned 64-bit lower parts. Every part is an independent child array, so each one compresses on its own. The frozen
vortex.decimal_byte_partsfile format is untouched. Arrays with lower parts serialize under a newvortex.decimal_byte_parts.v2format owned by a plugin.This is the integration branch for three reviewed sub-PRs: #9808 (splitting and assembly), #9809 (array and kernels), and #9810 (serde plugin).
Representation
i8,i16,i32,i64i128i64MSP holding the high 64 bits, plus oneu64lower part.i256i64MSP holding the high 64 bits, plus threeu64lower parts.Parts are ordered most significant first. The MSP carries the sign and the null mask. Lower parts are non-nullable unsigned integers. A lower part may use a narrower dtype such as
u8,u16, oru32when its values fit. Its position still counts as a full 64-bit window. Splitting writes zeroes at null positions so stray bytes in null slots do not hurt compression of the lower parts.Splitting and assembly
DecimalByteParts::encodesplits aDecimalArrayinto parts.split_decimalexposes the raw parts for callers that want to build the array themselves.Assembly picks a path from the number of lower parts:
i128.i256. The MSP is sign-extended into the high 128 bits.Assembly casts narrowed lower parts back to
u64first. Thei256assembly loop vectorizes on local ARM64 builds. Markingi256's shifts#[inline]removed three out-of-line calls per row.#[inline]Medians of five alternating release runs.
From<i64>andFrom<u64>fori256are added invortex-array.Compute
execute::<DecimalArray>reassembles the canonical array from all parts. The compare, filter, is-constant, and take kernels understand lower parts. Slice and mask apply per child.Two limits are documented in code. Take with nullable indices on an array with lower parts falls back to canonical execution, because taking each part would make the lower parts nullable. The CUDA kernel rejects arrays with lower parts, because GPU reassembly is not implemented yet.
Serialization
DecimalBytePartsPluginowns both wire formats and picks one from the array layout.vortex.decimal_byte_partsvortex.decimal_byte_parts.v2The v1 format is frozen. Its metadata and decoder live in
plugin/v1.rsand are byte-identical to what shipped. The v1 decoder rejects any payload that claims lower parts.The v2 format records the MSP's integer type and one integer type per lower part. The lower part count is the length of that list. The decoder validates every type and restores each child with its recorded dtype. The v2 format itself accepts zero lower parts. The plugin only chooses it when lower parts are present, so files stay readable by older readers whenever possible.
The in-memory encoding ID is now
vortex.decimal_byte_parts.v2. The registry maps both wire IDs to the plugin, so existing v1 files read through it with no migration.No edition declares the v2 format yet. Writing an array with lower parts under an edition that does not permit v2 fails with an explicit error rather than silently falling back.
Compression
The BtrBlocks decimal scheme still narrows decimals that fit in
i64and wraps them in a single-part array. Wide decimals stay canonical. Nothing in this PR writes the v2 format through the compressor.The scheme now declares
vortex.decimal_byte_partsas its produced encoding rather than the in-memory ID. Since #9914 that list holds the serialized IDs a scheme writes, and this scheme only ever writes the frozen format. Without that change every writer that filters schemes by edition would drop the decimal scheme, because no edition permits the in-memory v2 name.API Changes
Breaking. Registering
DecimalBytePartsdirectly no longer supports serde for either format. Replacesession.arrays().register(DecimalByteParts)withsession.arrays().register(DecimalBytePartsPlugin).vortex_decimal_byte_parts::initializealready does this.Breaking.
dbp_encodeis replaced byDecimalByteParts::encode.Breaking.
DecimalBytesPartsMetadatais no longer public.DecimalBytePartsV2Metadatais exposed instead.The in-memory encoding ID string changed from
vortex.decimal_byte_partstovortex.decimal_byte_parts.v2. This affects display and trace output, not files.New public items:
DecimalByteParts::try_new_with_lower_parts,DecimalByteParts::encode,split_decimal,DecimalParts,DecimalBytePartsPlugin,decimal_byte_parts_v1_id, anddecimal_byte_parts_v2_id.