Tell the compressor which serialized IDs the writer may emit by mhk197 · Pull Request #9770 · vortex-data/vortex

mhk197 · 2026-09-04T15:16:51Z

Filtering compression schemes by in-memory encoding cannot distinguish multiple wire formats of the same encoding. In order to support encodings like DecimalByteParts with multiple wire formats, the compressor must know which wire formats are enabled and must be able to select the appropriate schemes.

The writer now passes its permitted serialized IDs to compressor configuration, which selects the newest compatible version of each scheme before matching, generating statistics, estimating, or compressing.

Scheme::produced_encodings declares the serialized IDs that a scheme directly introduces. Every declared ID must be allowed.

Alternative versions are represented by Scheme::predecessor: register only the newest version, fall back through its predecessor chain when its IDs are unavailable, and remove the chain when no version is eligible. Historical versions do not compete with their replacements during compression.

BtrBlocksCompressorBuilder::allow_serialized_ids replaces retain_allowed_encodings and applies restrictions at build time, including to schemes added after the restriction. Both it and CascadingCompressor::with_allowed_serialized_ids intersect repeated restrictions. Registration order is preserved, and scheme IDs in a predecessor chain resolve to the selected version for exclusions and has_scheme checks. Cycles and overlapping chains are rejected.

The default file writer uses the serialized IDs of its enabled editions, or every registered serialized ID when edition enforcement is disabled. Custom write strategies retain their own configuration. Serialization still chooses the oldest wire form the resulting array fits and validates it against the writer's allowed IDs.

Existing production schemes have no predecessors, so this prepares version selection for consumers such as bitpacking (#9754) and decimal byte parts (#9759). The related edition specification is discussed in #9779.

Tests cover version selection before estimation, required output IDs, repeated restrictions, schemes added after restrictions, registration order, exclusions across versions, invalid predecessor chains, and writer round trips with unavailable encodings.

codspeed · 2026-09-04T15:23:44Z

Merging this PR will regress 2 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 2287 untouched benchmarks
⏩ 176 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`decompress[u64, (4000, 1024)]`	71.2 µs	86.6 µs	-17.82%
❌	WallTime	`mul_u32_nonnull_avx512`	5.6 µs	6.3 µs	-11%
⚡	WallTime	`dbp_assemble_kernel_avx512[(I256, 8192)]`	20.3 µs	13 µs	+56.23%
⚡	WallTime	`dbp_assemble_kernel_narrow_msp_avx512[(I256, 8192)]`	17 µs	14.4 µs	+18.44%
⚡	WallTime	`mul_i32_nonnull_avx512`	7.9 µs	7.1 µs	+11.17%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing mk/compressor-serialized-ids (29fe244) with mk/dbp-v2-feature (35adc87)}

176 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

robert3005 · 2026-09-08T14:21:25Z

                .with_btrblocks_builder(
                    BtrBlocksCompressorBuilder::default()
-                        .retain_allowed_encodings(&allowed_array_encodings),
+                        .retain_allowed_encodings(&allowed_array_encodings)


I think this has to be serialised ids

I think we should somehow unify these 2 functions

I guess we can enable a scheme if at least one of associated serialized ids is enabled?

Ah actually I think this needs to be separate since there's a diff bw what ids MUST be allowed for a scheme to proc (all) vs what is optional (at least one).

FSST for example requires VarBin and FSST to be enabled. DBP requires at least one of v1 or v2.

joseph-isaacs · 2026-09-08T14:51:49Z

Do you have an example of this being used?

mhk197

@joseph-isaacs https://github.com/vortex-data/vortex/pull/9759/changes#diff-fbbd1d380637b6d5557c6657611e9cf2b2f491f67c5d3f30d73e09a13f83aa8aR85

joseph-isaacs · 2026-09-09T09:31:06Z

    fn default() -> Self {
        Self {
            schemes: ALL_SCHEMES.to_vec(),
+            allowed_serialized_ids: None,


What does None mean?

All allowed (e.g. editions not enforced)

joseph-isaacs · 2026-09-09T09:31:39Z

 #[derive(Debug, Clone)]
 pub struct BtrBlocksCompressorBuilder {
    schemes: Vec<&'static dyn Scheme>,
+    allowed_serialized_ids: Option<HashSet<ArrayId>>,


Make this none optional

) ## Summary Compression schemes are filtered against the IDs a writer may emit. Today that filter mixes two ID spaces. Schemes declare their in-memory encoding ID, while most callers pass the serialized IDs from their enabled editions. The two only line up because every encoding currently uses the same string for both. `DecimalByteParts` stops doing that in #9834. Its in-memory ID becomes `vortex.decimal_byte_parts.v2` while the frozen wire ID stays `vortex.decimal_byte_parts`. With the current filter, every caller that passes edition IDs would silently drop the decimal scheme and write decimals canonical. This PR makes the contract explicit. Schemes declare the serialized IDs they write, and every caller filters with serialized IDs. ## Changes - The file writer no longer maps permitted serialized IDs through the registry to plugin in-memory IDs before filtering schemes. `new_array_context` returns only the array context, which is seeded with exactly the permitted IDs, and the writer reads them back with `to_ids` into a named set for the compressor builder. This matches the Python, TUI, CUDA, bench, and golden-test callers, which already pass serialized IDs. ## Behavior No scheme changes behavior. In-memory and wire IDs coincide for every scheme on develop, so the same schemes are retained from the same inputs and no snapshot changes. ## Follow-ups - #9834 declares `vortex.decimal_byte_parts` from the decimal scheme on top of this. - #9770 builds on this with a permitted-ID snapshot in `CompressorContext`, so a scheme can choose a wire format when it compresses. If the method names should say serialized IDs, renaming `produced_encodings` and `retain_allowed_encodings` together belongs there. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com> Signed-off-by: Matt Katz <mhkatz97@gmail.com>

`DecimalBytePartsArray` previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability. The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces: | Decimal storage | Children | | --- | --- | | `i8` / `i16` / `i32` / `i64` | Signed MSP only; shares the original value buffer | | `i128` | `i64` MSP + one `u64` lower part | | `i256` | `i64` MSP + three `u64` lower parts | Only the MSP carries validity. Every lower part must be a non-nullable `u64` array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain `ArrayRef`s, so their individual encodings are independent of the decimal representation. `execute::<DecimalArray>` reassembles a `DecimalArray` from the MSP and lower parts children. `take` with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts. This PR also refines the splitting and assembly modules. * Assembly takes `ArrayRef`s instead of `PrimitiveArray`s so that in the future, we can add special fast paths for constant arrays * Assembly casts lower parts to `u64`s, allowing for assembly of narrowed lower parts. * Assembly loop is optimized such that it vectorizes for `i256` assembly on local runs. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>

…mat (#9810) Add serialization for `DecimalBytePartsArray` with lower parts while preserving the frozen v1 file format. `DecimalBytePartsPlugin` owns both formats and selects the serialized ID from the array's layout: | Array layout | Serialized ID | | --- | --- | | Signed most significant part (MSP) only | `vortex.decimal_byte_parts` | | Signed MSP plus one to three unsigned lower parts | `vortex.decimal_byte_parts_v2` | Both formats deserialize into the same in-memory array representation. Each lower part represents a 64-bit window, but its storage dtype may be narrowed to `u8`, `u16`, or `u32` when its values fit. - Record the MSP's physical integer type, lower-part count, and each lower part's physical integer type in metadata. Deserialization validates these types and restores each child with its recorded dtype. - Keep the frozen format restricted to zero lower parts and require at least one for v2. Frozen metadata remains byte-identical. - Make DBP's VTable `serialize` and `deserialize` methods return errors directing callers to `DecimalBytePartsPlugin`. Keep DBP metadata and serde helpers in `plugin.rs` so the dedicated plugin owns both formats. - Add a separate v2 compatibility fixture covering positive and negative wide `i128` values and nullable `i256` values. The frozen fixture is unchanged. **Breaking:** directly registering `DecimalByteParts`, or calling its VTable serde methods, now errors for both serialization and deserialization, including the frozen v1 format. Replace: ```rust session.arrays().register(DecimalByteParts); ``` with: ```rust session.arrays().register(DecimalBytePartsPlugin); ``` `vortex_decimal_byte_parts::initialize(&session)` already registers the plugin. Existing v1 files remain readable through it; no file migration is required. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

CascadingCompressor carries the snapshot of serialized IDs the writer may emit, filled by the file writer from the enabled editions through BtrBlocksCompressorBuilder::allow_serialized_ids. A scheme whose encoding has more than one wire format picks its compression mode from it with allows_serialized_id, the newest permitted one; without a restriction every ID is allowed. No scheme consults the set yet. This is the mechanism docs/specs/editions.md describes under compression with replacement encodings (#9779). Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Scheme::produced_encodings now names the serialized IDs a scheme may write its output under, oldest first. BtrBlocksCompressorBuilder::allow_serialized_ids replaces retain_allowed_encodings: it keeps a scheme when at least one of those IDs is permitted and hands the set to the compressor, so the writer makes one call from the serialized IDs its editions permit instead of mapping them back to in-memory encodings, which could not tell two wire formats of one encoding apart. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

The compressor seeds each root CompressorContext with its permitted serialized IDs and every descent inherits them, so a scheme asks compress_ctx.allows_serialized_id both while estimating and while compressing and picks the same mode in both. The per-compressor accessor goes; allowed_serialized_ids remains for inspection. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

…ed ids Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Restore produced_encodings in feature-gated schemes and fix stale trait links. Document scheme ID resolution for Clippy. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

… unstable_encodings gate Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 added the changelog/chore A trivial change label Sep 4, 2026

mhk197 force-pushed the mk/compressor-serialized-ids branch from 85990c2 to a49c384 Compare September 8, 2026 13:59

mhk197 mentioned this pull request Sep 8, 2026

Compress wide decimals into lower parts when the writer permits the v2 format #9759

Draft

robert3005 reviewed Sep 8, 2026

View reviewed changes

mhk197 force-pushed the mk/compressor-serialized-ids branch from f5b732f to c27a6d6 Compare September 8, 2026 15:10

mhk197 commented Sep 8, 2026

View reviewed changes

This was referenced Sep 8, 2026

Add wide decimal byte-part splitting and assembly logic #9808

Merged

Support wide decimals in DecimalBytePartsArray and kernels #9809

Merged

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format #9810

Merged

joseph-isaacs reviewed Sep 9, 2026

View reviewed changes

mhk197 removed this pull request from stack #9811 September 9, 2026 15:09

mhk197 force-pushed the mk/compressor-serialized-ids branch from 94ab744 to 8129237 Compare September 9, 2026 15:09

mhk197 changed the base branch from develop to mk/dbp-serde September 9, 2026 15:09

mhk197 added this pull request to stack #9813 September 9, 2026 15:09

mhk197 force-pushed the mk/dbp-serde branch from f7b9f7e to 42e4302 Compare September 9, 2026 18:31

mhk197 force-pushed the mk/compressor-serialized-ids branch from 8129237 to b28f0b6 Compare September 9, 2026 18:31

mhk197 had a problem deploying to duckdb-build September 9, 2026 18:32 — with GitHub Actions Error

mhk197 force-pushed the mk/dbp-serde branch from 42e4302 to 1956d09 Compare September 9, 2026 20:36

mhk197 force-pushed the mk/compressor-serialized-ids branch from b28f0b6 to 7accd56 Compare September 9, 2026 21:29

joseph-isaacs force-pushed the mk/dbp-serde branch from 03a13f8 to 72f2396 Compare September 10, 2026 15:15

joseph-isaacs force-pushed the mk/compressor-serialized-ids branch from ece1fe2 to bf0a90f Compare September 10, 2026 15:15

mhk197 force-pushed the mk/dbp-serde branch from 72f2396 to d846ec7 Compare September 15, 2026 04:55

mhk197 force-pushed the mk/compressor-serialized-ids branch from bf0a90f to d228008 Compare September 15, 2026 05:08

mhk197 force-pushed the mk/dbp-serde branch from d846ec7 to 46de82a Compare September 15, 2026 13:05

mhk197 force-pushed the mk/compressor-serialized-ids branch from d228008 to 61b5492 Compare September 15, 2026 13:05

Base automatically changed from mk/dbp-serde to mk/dbp-v2-feature September 16, 2026 14:04

mhk197 force-pushed the mk/compressor-serialized-ids branch from 61b5492 to f5a94a0 Compare September 16, 2026 14:04

mhk197 force-pushed the mk/dbp-v2-feature branch from bbd1a50 to ac1f5a0 Compare September 16, 2026 14:27

This was referenced Sep 16, 2026

Support i128 and i256 decimals in DecimalByteParts encoding #9834

Merged

Filter compression schemes by serialized IDs instead of array IDs #9914

Merged

mhk197 force-pushed the mk/dbp-v2-feature branch from ac1f5a0 to c787611 Compare September 16, 2026 16:16

mhk197 and others added 4 commits September 16, 2026 13:39

Add wide decimal byte-part splitting and assembly logic (#9808)

ea5f878

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com> Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Declare the frozen v1 wire ID from the decimal compression scheme

c5ea5c5

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 force-pushed the mk/dbp-v2-feature branch from c787611 to c5ea5c5 Compare September 16, 2026 17:42

mhk197 added 8 commits September 16, 2026 13:51

Do not link the private lower-part limit from public docs

35adc87

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

add predecessor schemes and make produced_encodings refer to serializ…

b652b40

…ed ids Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Fix optional scheme trait methods and documentation links

b43711a

Restore produced_encodings in feature-gated schemes and fix stale trait links. Document scheme ID resolution for Clippy. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix

3a0d848

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Adapt to develop: rename remaining scheme-filter callers and drop the…

29fe244

… unstable_encodings gate Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 force-pushed the mk/compressor-serialized-ids branch from f5a94a0 to 29fe244 Compare September 16, 2026 19:32

This was referenced Sep 16, 2026

Handle array evolution in compressor schemes with replacement semantics #9919

Closed

Carry the writer's permitted serialized IDs into compression schemes #9928

Draft

mhk197 force-pushed the mk/dbp-v2-feature branch from 35adc87 to 026bb26 Compare September 17, 2026 15:39

mhk197 removed this pull request from stack #9813 September 17, 2026 15:48

Base automatically changed from mk/dbp-v2-feature to develop September 18, 2026 13:57

mhk197 closed this Sep 18, 2026

Navigation Menu

Tell the compressor which serialized IDs the writer may emit - #9770

Tell the compressor which serialized IDs the writer may emit#9770
mhk197 wants to merge 12 commits into
developfrom
mk/compressor-serialized-ids

mhk197 commented Sep 4, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

robert3005 Sep 8, 2026

Uh oh!

robert3005 Sep 8, 2026

Uh oh!

mhk197 Sep 8, 2026

Uh oh!

robert3005 Sep 8, 2026

Uh oh!

mhk197 Sep 8, 2026

Uh oh!

joseph-isaacs commented Sep 8, 2026

Uh oh!

mhk197 left a comment

Uh oh!

joseph-isaacs Sep 9, 2026

Uh oh!

mhk197 Sep 9, 2026

Uh oh!

joseph-isaacs Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mhk197 commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 2 benchmarks

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joseph-isaacs commented Sep 8, 2026

Uh oh!

mhk197 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhk197 commented Sep 4, 2026 •

edited

Loading

codspeed Bot commented Sep 4, 2026 •

edited

Loading