diff --git a/docs/_static/versioning-checks.svg b/docs/_static/versioning-checks.svg new file mode 100644 index 00000000000..48c28c67e5c --- /dev/null +++ b/docs/_static/versioning-checks.svg @@ -0,0 +1,70 @@ + + + + Writers check permissions, while readers validate contracts. + + On writing, in-memory array IDs select serializer plugins. The plugins return wire IDs, + metadata, buffers, and children, which are recursively serialized. Edition checks reject any + forbidden output ID, including children's IDs, without retrying a different wire format. + On reading, stored wire IDs select registered deserializer plugins. An unknown ID causes an + error. The plugins validate metadata, buffers, data types, and children against each stored + contract. Invalid data causes an error. Successful decoding constructs the reader's in-memory + encodings, adapting structure while preserving values, data types, and nulls. These are logical + groups of checks, not an exact execution timeline. The enclosing file format must be supported, + serializers must be available, edition enforcement is enabled, and unknown IDs are rejected. + + + + + + + + + Writers check permissions, while readers validate contracts. + Edition checks are enabled, unknown IDs are rejected, and serializers and file support are available. + WRITE + Memory + to storage + READ + Storage + to memory + + + + + + + Select serializer plugins + Use each array's in-memory ID. + Plugins choose wire formats. + Serialize each array and child + The plugin returns an ID, + metadata, buffers, and children. + Is every wire ID permitted? + Check the selected editions. + Yes: the output is permitted. + Find deserializer plugins + Use each stored wire ID. + Is the plugin registered? + Validate each contract + Do metadata, buffers, types, + and children satisfy that ID? + Construct encoded arrays + Adapt the structure as needed. + Keep values, types, and nulls. + No: the write fails without retrying. + No: the wire ID is unknown. + No: the data is invalid. + + + + + + + + Yes + Yes + + diff --git a/docs/_static/versioning-compatibility.svg b/docs/_static/versioning-compatibility.svg new file mode 100644 index 00000000000..980ec5080f9 --- /dev/null +++ b/docs/_static/versioning-compatibility.svg @@ -0,0 +1,127 @@ + + + + How array plugins convert between in-memory encodings and wire formats + + Writing starts with an in-memory array. Its ID selects a serializer plugin. + If no plugin can serialize this representation, the write fails. Otherwise, the plugin chooses + the oldest supported writable format that preserves the representation without recompression. + The plugin returns a wire ID, metadata, buffers, and child arrays. It can adapt these components + to the selected wire contract. Each returned child is serialized recursively. The writer checks + all returned IDs against the editions, along with layouts, extension types, and stored + aggregates. If any ID is forbidden, the write fails without retrying a different permitted ID. + After a successful write, the + reader must support the file format and have plugins for the stored wire IDs. Each plugin + validates the stored ID's contract and constructs an in-memory array, adapting the + structure when needed and reusing buffers where possible. Missing support or invalid serialized + data causes an error. Values, data types, and nulls are preserved. The diagram assumes edition + checks are enabled and allow_unknown is disabled. It groups related checks, which occur at + several points during writing and reading. + + + + + + + + + WRITE + Start with an in-memory array. + + + + + + + + + + + + + + + Yes + Then + No + No + Yes + Yes + Yes + No + No + + + + Find the serializer plugin + Look up the in-memory ID. + Can the plugin serialize this + representation without + recompression? + + + Select the wire format + Choose the oldest writable format + that preserves the representation. + Adapt metadata, buffers, or children. + Return the wire ID and these parts. + Serialize each returned child recursively. + + + Check edition permissions + Is every returned wire ID permitted, + including all serialized children? + Also check layouts, extension types, + and stored aggregates. + + + Write fails + No serializer or writable format exists. + + If another encoding is needed, + recompress the array before + serialization. + + + Write fails: forbidden output + The writer does not retry another ID. + + + File: wire IDs and data + After a successful write + + READ + Select implementations by the stored IDs. + + + Find the reader plugins + Look up the stored wire IDs. + Does the reader support the file + format and all required wire IDs? + + + Validate the wire contract + Are its metadata, buffers, + types, and children valid? + Each wire ID has a fixed contract. + + + Construct the in-memory array + Adapt the stored structure if needed. + Reuse buffers where possible. + Use the reader's array implementation. + + + Missing reader support + The file format or an ID is unsupported. + + + Reject invalid serialized data + The stored ID determines the contract. + + Preserve values, types, and nulls. + The in-memory array tree can differ. + + diff --git a/docs/_static/versioning-flow.svg b/docs/_static/versioning-flow.svg new file mode 100644 index 00000000000..d3b26be8ffa --- /dev/null +++ b/docs/_static/versioning-flow.svg @@ -0,0 +1,72 @@ + + + + One decimal encoding, two wire contracts + + DecimalBytePartsArray holds either one signed integer child or a signed most-significant child + followed by up to three unsigned lower-part children. The plugin serializes the single-child + shape using vortex.decimal_byte_parts, called v1 here, and the multi-child shape using + vortex.decimal_byte_parts.v2. Both wire contracts deserialize into DecimalBytePartsArray. + v1 metadata records the signed child's type and a lower-part count of zero. v2 metadata records + the signed child's type and the ordered types of zero to three lower parts. Thus v2 also accepts + one child, although the serializer selects v1 for that shape. Both contracts require a decimal + data type and have no buffers of their own. Each child has the same length as the parent, and the + signed child carries the parent's nullability, while lower parts are non-nullable. The arrows show serialization + choices and the corresponding reads, not every valid v2 input. Values, data types, and nulls + are preserved. Frozen wire contracts are immutable. Edition permissions are checked separately, + and no declared edition currently permits v2. + + + + + + + + + One decimal encoding, two wire contracts + The plugin converts between an array in memory and its serialized metadata and children. + IN MEMORY + ON WIRE + + DecimalBytePartsArray + + + + + + + + Decimals + Signed child + Decimals + Signed child + Unsigned child + One child + Multiple children (two shown) + + + + + serialize + deserialize + serialize + deserialize + + v1: exactly one signed child + vortex.decimal_byte_parts + Metadata: child type, lower-part count = 0 + Children: [signed] + + v2: signed child + 0 to 3 unsigned children + vortex.decimal_byte_parts.v2 + Metadata: signed type, ordered lower-part types + Children: [signed, lower parts, most significant first] + Although v2 accepts one child, the serializer chooses v1 when there are no lower parts. + Both contracts require a decimal dtype and have no own buffers. Each child has the same length as the parent. + The signed child carries the parent's nullability, whereas lower parts are non-nullable. + Conversions preserve values, types, and nulls. Frozen wire contracts remain fixed. + Edition permissions are checked separately, and no declared edition currently permits v2. + + diff --git a/docs/_theme/vortex/vortex.css b/docs/_theme/vortex/vortex.css index a70c4dbc268..0b0d9c92d22 100644 --- a/docs/_theme/vortex/vortex.css +++ b/docs/_theme/vortex/vortex.css @@ -252,6 +252,10 @@ table.docutils th, table.docutils td { } table.docutils th { background: var(--bg-alt); } +table.versioning-terms p { margin: 0; } +table.versioning-terms td:first-child { white-space: nowrap; } +figure.versioning-diagram { margin-inline: 0; } + .page-index .content > section > h1 { position: absolute; width: 1px; height: 1px; diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 5a2741655cf..fc6961f3bae 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -23,8 +23,9 @@ scanning without dictating physical layout, allowing the same logical data to use different encodings. **[Arrays](arrays.md)** are the in-memory representation. Unlike Arrow, Vortex arrays can be -*compressed*—an integer array might be bit-packed rather than stored as a flat buffer. Arrays -share the same representation on disk and over the wire, enabling zero-copy I/O. +*compressed*, so an integer array can be bit-packed rather than stored as a flat buffer. +Serialized formats can reuse array buffers for zero-copy I/O, while plugins adapt historical +formats to current array implementations. **[Compute](expressions.md)** functions operate directly on compressed arrays where possible, dispatching to encoding-specific kernels or falling back to canonical implementations. @@ -39,6 +40,9 @@ segment retrieval, FlatBuffer metadata for O(1) schema access, and support for m **[IPC Format](../specs/ipc-format.md)** provides streaming transfer of compressed arrays. +**[Versioning](../specs/versioning/design.md)** explains how library implementations, serialized +formats, and editions evolve while preserving read compatibility. + ## Integrations **Language bindings:** [Rust](https://docs.rs/vortex), [Python](../api/python/index.rst), diff --git a/docs/developer-guide/internals/serialization.md b/docs/developer-guide/internals/serialization.md index 22c73e66d9a..26d88c2edf7 100644 --- a/docs/developer-guide/internals/serialization.md +++ b/docs/developer-guide/internals/serialization.md @@ -1,9 +1,10 @@ # Serialization -Vortex uses the same binary representation for arrays in memory, on disk, and over the wire. -Metadata is stored in FlatBuffers for O(1) field access without parsing, and data buffers are -stored separately with alignment guarantees that enable zero-copy reads. Appropriate padding is -written into Vortex files to ensure that segments can be memory-mapped with correct alignment. +Vortex stores array-tree metadata in FlatBuffers and data buffers separately, with alignment that +enables zero-copy reads. A serializer can reuse an array's buffers while adapting its metadata or +children to a supported wire format. The in-memory array and its serialized representation can +evolve independently, as described in the [versioning design](../../specs/versioning/design.md). +Padding in Vortex files allows segments to be memory-mapped with the required alignment. ## Array Serialization @@ -27,9 +28,10 @@ On the wire, a serialized array is: [padding] [buffer 0] [padding] [buffer 1] ... [flatbuffer] [u32 flatbuffer length] ``` -Deserialization constructs an `ArrayParts` value that holds the FlatBuffer and buffer handles -without copying. The array is then decoded by resolving the array ID through the session's -registry and calling `build()` on the corresponding vtable. +`SerializedArray` holds the serialized tree and buffer handles. Decoding resolves the stored wire ID +through the session's plugin registry and calls the plugin's `deserialize` method with the metadata, +buffers, and children. The plugin validates that wire format and constructs an array supported by +the current implementation. ## IPC Format @@ -107,11 +109,10 @@ bindings, which `build.rs` compiles into `OUT_DIR`. The read/write traits they a ## Zero-Copy Design -The alignment and padding system is designed so that serialized buffers can be used directly -as in-memory arrays without copying. When a segment is read from disk or received over the -network, the I/O subsystem allocates an aligned buffer matching the segment's alignment -requirement. The resulting buffer handle can be used directly by the array without -reallocating or copying the data. +The alignment and padding system allows serialized buffers to be used directly in in-memory arrays +without copying. When a segment is read from disk or received over the network, the I/O subsystem +allocates an aligned buffer matching the segment's alignment requirement. The resulting buffer +handle can be used directly by the array without reallocating or copying the data. -This property holds across all three contexts: in-memory arrays, on-disk file segments, and -over-the-wire IPC messages all use the same layout and alignment conventions. +Reusing buffers does not require the reader's array tree to have the same structure as the serialized +tree. A plugin can adapt a historical format while retaining its data buffers. diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index fb7ce0afbe9..27294448866 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -27,3 +27,7 @@ Rust Quickstart C++ Quickstart Java Quickstart ``` + +For upgrades and files shared between applications, see +[Versioning and compatibility](../specs/versioning.md). It explains the read guarantee and how to +target formats supported by older Vortex versions. diff --git a/docs/index.md b/docs/index.md index 9de33b387b0..31b571ca1a8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -72,7 +72,7 @@ internals. Build and benchmark locally. [ALP](https://github.com/spiraldb/alp) — no decompression needed for many operations. - **Extensible file format**: Zero-allocation reads, FlatBuffer metadata for O(1) column access, - and optional WASM decompression kernels for forward compatibility. + and [editions](specs/versioning.md) for writing formats supported by older readers. - **Query engine integration**: Filter and projection pushdown through the Scan API, with native integrations for DataFusion, DuckDB, Spark, Trino, and Ray. diff --git a/docs/specs/editions.md b/docs/specs/editions.md index 1aa3cfe4e11..ffa20d539ff 100644 --- a/docs/specs/editions.md +++ b/docs/specs/editions.md @@ -1,386 +1,10 @@ -# Editions - -Vortex files contain several kinds of serialized **component**: array encodings, layout encodings, extension dtypes, and -aggregate functions. An **edition** is a named set of their concrete wire IDs. It controls what a writer may put in a -file and, once frozen, identifies its origin library or project and minimum version: the earliest release of that origin -that recognizes every ID in the set. - -Editions belong to independently versioned families and are cumulative within a family. Each edition includes all -components from the preceding edition in that family, plus any additions. A writer selects at most one edition from -each family and may use the union of their component IDs. For example, selecting `core2026.08.3` and -`tensor2026.04.0` allows the core components together with tensor arrays and dtypes. Every family names the origin -library or project whose release versions its editions use. - -The first frozen edition, `core2025.05.0`, contains the components that Vortex `0.36.0` could write. This marks the -start of the Vortex file format's stability guarantee. Every Vortex release from `0.36.0` onward can read -`core2025.05.0`, and later frozen `core` editions extend that guarantee to newer components. - -When a writer selects only frozen editions from one origin, the highest of their minimum library versions is the -earliest release guaranteed to read the resulting file. With multiple origins, the file requires the recorded minimum -version of each. Editions without minimum library versions are drafts and carry no guarantee about their future -compatibility. - -## What an edition contains - -An edition records every component by kind and wire ID. IDs are unique within a kind, but not across kinds: a layout -named `vortex.flat` and an array encoding with the same ID are distinct components. The writer therefore builds and -enforces a separate allowlist for each kind: - -| Kind | What it identifies | Used at | -|-------------|--------------------------------------------|------------------------------| -| `array` | a serialized array representation | array serialization | -| `layout` | the footer's layout tree | layout serialization context | -| `dtype` | extension dtypes nested in the file schema | file writer | -| `aggregate` | zone maps in zoned layouts | layout writer context | - -Writing a component that is absent from the selected editions fails the write. This rule applies to every kind, -including aggregates. Although a zone map is only an optimization and could be dropped, doing so would silently change -the writer's configured pruning behavior. - -Only aggregates that would actually be written are checked. If a column's dtype cannot support an aggregate, the writer -omits it and there is no edition violation. - -An empty allowlist permits no components. Collectively, the selected editions must declare every serialized array ID, -layout encoding, extension dtype, and aggregate function that the writer writes. An array serializer may expose several -wire IDs for one in-memory encoding. The serializer chooses the representation, and the serialization context rejects -the write if the chosen wire ID is not declared by the selected editions. - -For example, `core2026.08.0` declares the aggregate functions that the default writer may store in zone maps: `min`, -`max`, `bounded_min`, `bounded_max`, `nan_count`, and `null_count`. It does not declare `sum`, because the writer does -not store sums in zone maps. File-level statistics use a fixed legacy field for sums rather than a serialized aggregate -function ID, so this allowlist does not apply to them. - -Optional Vortex modules enable their own edition families alongside `core`. Tensor support enables -`tensor2026.04.0`, for example, while Zstd buffer wrapping enables `zstd2026.02.0`. - -## Resolving an unknown-component error - -An unknown-ID error means that the reader does not recognize a serialized component in the file. Find the component's -kind and ID in the [registry](#edition-registry): - -1. **It belongs to a frozen edition.** Upgrade the edition's named origin to at least its minimum library version. -2. **It belongs to a draft edition.** No released reader is guaranteed to support it. Use a build that registers the - component or ask the file's producer which build to use. -3. **It is not in the registry.** The file contains a custom, third-party, or experimental component outside the - editions system. Ask the producer for its implementation and register it with the reader's session. - -Tools that only inspect or copy data can opt in to `allow_unknown`. Unknown array encodings, layout encodings, and -extension dtypes are then preserved as inert representations. An unknown aggregate function disables the affected -zone-map pruning rather than causing the file to be rejected. - -## Writing with an edition - -By default, the Vortex facade targets `core2026.08.3`. A new encoding or serialization feature that is -still evolving gets a new draft edition; later additions create later editions rather than changing an already -published feature set. Each feature advances through its own independently versioned family until it is ready to join -the shared `preview` family. Preview components remain opt-in until they are ready to join `core`. Components supplied -by an optional plugin belong to that plugin's family, such as `tensor`, `zstd`, `spatial`, or `json`. - -Edition configuration belongs to the writer's Vortex session. Registering an edition makes its declaration available to -the session; enabling it allows the writer to use its components. Enabling another edition in the same family replaces -the previous selection. - -You can change the default configuration to: - -- **Target an older `core` edition** when the file must remain readable by an older Vortex deployment. -- **Enable another family** to use components outside `core`. Vortex currently defines `preview`, `tensor`, `zstd`, - `spatial`, and `json` in addition to `core`. - -Sessions created without the Vortex facade must register and enable their editions before writing files. - -For experimental or custom components that do not belong to an edition, the Rust writer exposes -`disable_editions()`. This disables every edition check for that write: every array representation registered in the -session is eligible for compression and serialization, while layouts, extension dtypes, and aggregate functions are -unrestricted. It does not register missing readers, so files written this way have no edition compatibility guarantee. - -Compression and edition compatibility are separate. Compressors produce current in-memory arrays and do not select a -wire ID. The writer maps each allowed serialized ID to its current in-memory encoding and restricts the default -BtrBlocks compressor to schemes producing those encodings. Custom compressors remain unrestricted, with serialization -providing the final compatibility boundary when edition enforcement is enabled. At that boundary, the array plugin -produces an ID, metadata, buffers, and children. The serialization context interns the returned ID and fails the write -if the selected editions do not permit it. A serializer may emit a historical ID when the value satisfies that ID's -frozen contract, but it does not inspect the edition allowlist. Without disabling edition enforcement, a custom layout -or compressor therefore cannot bypass the final wire-ID check. - -## How editions change - -A frozen edition never changes: neither its membership list nor the meaning of its component IDs may be altered. -Introducing a new serialized object or a reader-visible revision requires a new edition; it is never added -retroactively to an existing edition. A component supplied by an optional plugin creates that edition in the plugin's -independently versioned family. - -Core-maintained objects do not enter `core` directly. When an object is ready for users to try and its wire format is -believed complete, it enters a draft edition in an independently versioned family. Publishing that edition is a -format-stability commitment, not the start of format design: the serialized contract should change only when absolutely -necessary to resolve a problem found during testing. After successful testing, the same object ID and wire contract move -into a new `preview` edition for broad opt-in use, and later into a new `core` edition for use by default. - -A new stable `core` or plugin edition may freeze in the release of its origin project in which it first ships. Until -that release is cut, its version is not known and the declaration keeps `min_library_version: None`. After the release -is cut, the declaration is updated with that newly released version, usually during development of the next release. -This backfills the documented minimum library version; it does not delay the freeze or its read-forever compatibility -guarantee. - -A component may later be deprecated, meaning that writers stop using it. Readers must continue to support it, so -deprecation does not invalidate existing files. - -Writer behavior evolves independently from the in-memory representation. A change that an old reader must distinguish -uses a new serialized ID, even when the new deserializer produces the same in-memory array. A serializer may continue -emitting the older ID for values that satisfy its frozen contract; the selected editions validate the ID it emits. - -## How serialized components evolve - -Editions govern serialized components, not in-memory representations. An in-memory representation may gain capabilities -or be replaced without changing an edition. Each in-memory array plugin owns the mapping between that representation and -its wire history: - -- the serialized IDs its deserializer recognizes; -- one serializer that returns the appropriate lossless variant as an ID, metadata, buffers, and children; and -- a deserializer that receives the exact ID found in the file and constructs the current in-memory representation. - -An in-memory representation often has one serialized ID equal to its in-memory encoding ID, but this is only the simple -case. Editions constrain the ID stored in the file, because that is what an old reader can recognize. - -### Reader-visible evolution requires a new ID - -Any new form that an old reader does not already understand uses a new serialized ID. This includes additive metadata or -children when an old reader would accept the ID but reject or misinterpret the new combination. The ID is the capability -tag: readers do not consult the edition or negotiate a separate version while decoding an array. - -Keeping an ID is safe only when the emitted representation remains within that ID's existing frozen contract. A writer -may choose a different but already-valid encoding of the same contract, and a reader may fix a bug or normalize the old -form into a newer in-memory structure. Neither action expands what the wire ID means. - -A new wire ID does not normally require a second in-memory array. The current plugin registers every historical ID, -serializes the current value under the oldest allowed lossless one, and deserializes all of them into the current type. -The old ID remains registered forever. If the compressor and serializer cannot preserve one common in-memory -representation and losslessly downgrade it, the change instead needs a new in-memory array, compressor, and -deserializer. - -Name successive incompatible revisions by appending a version to the same base name: `vortex.foo`, `vortex.foo_v2`, -`vortex.foo_v3`. Do not give successor versions descriptive names. A linear naming scheme keeps the component's -serialized history unambiguous. - -#### Example: multi-part decimals - -`vortex.decimal_byte_parts` entered `core2025.05.0` with each decimal value represented by one signed integer child. Its -metadata includes `lower_part_count`, but readers of this component require that field to be zero. Suppose the in-memory -representation gains support for wide decimals, represented by a signed most-significant part and one or more unsigned -64-bit lower parts: - -- The serializer first tries to construct the old single-signed-child form. If every value can be - represented that way, it emits `vortex.decimal_byte_parts` with `lower_part_count = 0`, even if - the current in-memory array has lower-part children. -- An array that cannot be collapsed into that old form losslessly uses the new - `vortex.decimal_byte_parts.v2` component, initially staged in a draft edition. -- A new reader deserializes both IDs into the same in-memory representation. An older reader reports - `vortex.decimal_byte_parts.v2` as unknown instead of trying to decode a wire format it does not support. -- When targeting an edition that permits only the old ID, serializing a value that can be collapsed succeeds; an - irreducibly multi-part value fails because no lossless downgrade exists. - -#### Example: Pco 8-bit integers - -The historical `vortex.pco` contract does not include `i8` or `u8`; readers implementing that contract must not be -sent an 8-bit Pco payload under the familiar ID. Adding 8-bit support keeps one current in-memory `Pco` array but adds -`vortex.pco.v2` as a serialized component: - -- The single Pco serializer emits `vortex.pco` for the primitive types covered by the old contract, even when both IDs - are permitted. -- For `i8` or `u8`, the earliest lossless form is `vortex.pco.v2`. A target edition without that ID rejects the write. -- The current deserializer registers both IDs. When given `vortex.pco`, it still rejects an 8-bit dtype; understanding - the v2 payload does not silently broaden the frozen v1 contract. -- The Pco compression scheme can sample and construct 8-bit Pco arrays without consulting editions. Wire selection - remains the serializer's responsibility. - -If writing an older edition must succeed for every input, its compression policy must choose an in-memory encoding -whose serializer has a permitted lossless form. It must not disguise the newer Pco form with the old ID. - -### Reading: deserialize into the current representation - -Every component in a frozen edition remains readable. Its deserializer may convert old data directly into the current -in-memory representation rather than preserving a parallel legacy representation. For example, a `vortex.alp` array with -interior patches is read as a `Patched` array around a patch-free ALP array. Similarly, old zone maps, including -`vortex.stats` layouts, are read by the machinery used for modern `vortex.zoned` layouts. - -Readers do not negotiate versions. They resolve the component ID, pass that exact ID to its deserializer, and either -construct the current in-memory array or report an -[unknown-component error](#resolving-an-unknown-component-error). - -A current deserializer must preserve each historical ID's contract. Recognizing a newer ID does not authorize it to -accept the newer metadata, child shape, dtype coverage, or buffer interpretation when the file carries an older ID. +--- +orphan: true +--- -A file contains its array ID, dtype, metadata, children, and buffers. A newer plugin may be registered under both -`vortex.foo` and `vortex.foo_v2`, but an older build is registered only under `vortex.foo`. This is what guarantees that -the older build rejects a v2 file before interpreting its contents. - -### Writing: validate the selected component and writer behavior - -For each in-memory array, the writer calls its plugin's single serializer. The serializer owns the versioning logic and -returns the appropriate lossless variant. It may change metadata, buffers, and children without constructing a legacy -in-memory array. Returning `None` means the array cannot be serialized. The serialization context then interns the -returned ID, failing the write if that ID is not permitted by the selected editions. - -This selection happens recursively after compression. Compressor output therefore remains an in-memory concern: a -compressor does not label its array with an edition or choose a wire version. Layouts, extension dtypes, and aggregates -perform their analogous compatibility checks at their own serialization boundaries. - -### What this means for each kind - -- **Arrays.** The array serialization context permits only wire IDs from the selected editions. The in-memory array's - serializer chooses its lossless representation, and the context rejects it if its ID is not permitted. -- **Layouts.** The layout strategy builds the layout tree at write time. When targeting an older edition, it must use - structures available in that edition, such as plain chunked data in place of newer auxiliary layouts. -- **Extension dtypes.** Before writing any bytes, the file writer recursively validates every extension dtype in the - schema. Readers resolve serialized dtype IDs against the session's dtype registry. -- **Aggregate functions.** Zone maps serialize aggregate function IDs and their options. A zone map containing a - function outside the selected editions fails the write. With `allow_unknown`, readers disable a zone map whose - aggregate function they do not recognize; ignoring a zone map only reduces pruning and does not affect correctness. - -## The `preview` family - -The additive `preview` family is the shared opt-in set for core-maintained components whose serialized contracts have -survived independent testing but are not yet available to the default core writer. Preview currently contains no -components. Adding the first component will create a later preview edition; unrelated work remains in independent -families until it meets the preview compatibility bar. - -## Independently versioned component families - -Components that are ready for focused testing but are not yet ready for the shared preview set advance through their own -families. Optional modules use families such as `tensor`, `zstd`, `spatial`, and `json`. Each family can evolve without -coupling its chronology or selection to unrelated components. - -The wire format is expected to be complete when its first draft edition is published and should change only when -necessary to resolve an issue discovered during testing. If a correction changes what readers must understand, give the -corrected representation a new ID and add a later edition to the same family. Once testing establishes that an object is -ready for broad opt-in use, promote that same ID and serialized contract into a new `preview` edition. Later adoption by -the default writer promotes it into a new `core` edition. - -The default writer does not emit a component merely because its reader understands it. Users opt in by enabling the -edition containing that component. - -## Declaring, freezing, and the edition records - -The default declarations live in `vortex-edition/src/declarations/`, while optional-module declarations live in their -owning crates. Each declared edition is exported as a TOML record under `vortex/editions/`, grouped by family. A record -names the origin library or project whose releases `min_library_version` refers to. Draft records omit that field and -carry no read-forever guarantee. Regenerate the records by running: - -```sh -cargo run -p xtask -- generate-editions -``` - -Changing the declarations follows the edition's lifecycle: - -1. **Create a new family and edition for every new object.** Never add an unrelated serialized - object or reader-visible revision to an existing family. A revision advances the family that - owns its earlier ID. -2. **Publish test-ready work as a draft.** When an object is ready to be tried and its format is - believed complete, give it a wire ID and add it to a draft edition in its family. Change that - format only when necessary to resolve an issue found during testing; a reader-visible - correction gets another ID and a later edition in the same family. -3. **Promote the tested contract to preview.** Once it is ready for broad opt-in use, add the same - object ID and wire contract to a new additive `preview` edition. Promotion must not redesign - the format. -4. **Promote the adopted contract to core.** Once it is ready for use by the default writer, add - the same object ID and wire contract to a new `core` edition with `min_library_version: None` - and regenerate its draft record, then ship it in a release. The edition freezes as part of that - release. Its minimum library version cannot be populated yet because the release version is not - known until the release is cut. -5. **Backfill the released version.** After cutting the release, set `min_library_version` to that - newly released Vortex version — the version that first shipped readers for every member — and - regenerate the records, converting the draft record into a frozen record. This update usually - lands during development of the next release, but it documents the freeze that already - happened; it does not freeze the edition later. -6. **Never touch it again.** A frozen record is immutable: CI (`cargo run -p xtask -- check-editions`) rejects any - change that edits, renames, unfreezes, - or deletes a frozen record, and rejects new editions that do not extend their family's - chronology. To change what writers may emit, declare the next edition instead. - -## Edition registry - -Registry entries list the edition in which each component first appeared. Later editions in the same family inherit all -earlier components. - -### Frozen `core` editions - -#### `core2025.05.0` - -Minimum library version: `0.36.0`. - -- `array`: `fastlanes.bitpacked`, `fastlanes.for`, `vortex.alp`, `vortex.alprd`, `vortex.bool`, - `vortex.bytebool`, `vortex.chunked`, `vortex.constant`, `vortex.datetimeparts`, `vortex.decimal`, - `vortex.decimal_byte_parts`, `vortex.dict`, `vortex.ext`, `vortex.fsst`, `vortex.list`, - `vortex.null`, `vortex.primitive`, `vortex.runend`, `vortex.sparse`, `vortex.struct`, - `vortex.varbin`, `vortex.varbinview`, `vortex.zigzag` -- `layout`: `vortex.chunked`, `vortex.dict`, `vortex.flat`, `vortex.stats`, `vortex.struct` -- `dtype`: `vortex.date`, `vortex.time`, `vortex.timestamp` - -#### `core2025.06.0` - -Minimum library version: `0.40.0`. - -- `array`: `vortex.pco`, `vortex.sequence`, `vortex.zstd` - -#### `core2025.10.0` - -Minimum library version: `0.54.0`. - -- `array`: `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked` - -#### `core2026.08.0` - -Minimum library version: `0.84.0`. - -- `layout`: `vortex.zoned` -- `aggregate`: `vortex.bounded_max`, `vortex.bounded_min`, `vortex.max`, `vortex.min`, - `vortex.nan_count`, `vortex.null_count` - -#### `core2026.08.1` - -Minimum library version: `0.84.0`. - -- `array`: `vortex.onpair` - -#### `core2026.08.2` - -Minimum library version: `0.85.0`. - -- `array`: `vortex.map` - -#### `core2026.08.3` - -Minimum library version: `0.85.0`. - -- `array`: `vortex.parquet.variant`, `vortex.variant` -- `dtype`: `vortex.uuid` - -### Editions without a frozen guarantee - -These editions have no minimum library version. Evolving features advance through new draft editions in their own -families. Their formats are expected to remain compatible unless a defect is serious enough to block promotion into -core. Optional plugin families state their own policy. - -#### `preview2026.08.0` - -This edition currently adds no components. - -#### `tensor2026.04.0` - -- `array`: `vortex.tensor.cosine_similarity`, `vortex.tensor.inner_product`, `vortex.tensor.l2_norm`, - `vortex.tensor.l2_normalize` -- `dtype`: `vortex.tensor.fixed_shape_tensor`, `vortex.tensor.vector` - -#### `zstd2026.02.0` - -- `array`: `vortex.zstd_buffers` - -#### `spatial2026.08.0` - -- `dtype`: `vortex.st.box`, `vortex.st.linestring`, `vortex.st.multilinestring`, - `vortex.st.multipoint`, `vortex.st.multipolygon`, `vortex.st.point`, `vortex.st.polygon`, - `vortex.st.wkb` -- `aggregate`: `vortex.st.aabb` +# Editions -#### `json2026.08.0` +The editions specification is now part of [Versioning](versioning.md). -- `dtype`: `vortex.json` +For edition membership and minimum Vortex crate versions, see the +[edition registry](versioning/editions.md#edition-registry). diff --git a/docs/specs/file-format.md b/docs/specs/file-format.md index 04e065ac5f1..9e8fcdec1ec 100644 --- a/docs/specs/file-format.md +++ b/docs/specs/file-format.md @@ -1,9 +1,11 @@ # File Format :::{important} -The Vortex File Format has been considered stable since the release of version 0.36.0. That means that you can expect all -future versions of the Vortex library to be able to read files written by version 0.36.0 or later (up to and including -the version doing the reading). +The Vortex file format's stability guarantee starts with version `0.36.0` of the Vortex Rust crates +and edition `core2025.05.0`. Later versions of those crates retain read support for the components +in frozen editions. +[Versioning](/specs/versioning) explains the guarantee and the requirements for draft and custom +components. ::: :::{seealso} @@ -17,9 +19,8 @@ definition that allows efficiently querying the layout. Other considerations for the Vortex file format include: -* Backwards compatibility, and (coming soon) forwards compatibility. The set of encodings a - writer may put in a file — and the resulting read-compatibility promise — is governed by - [Editions](/specs/editions). +* File compatibility. Editions constrain which serialized components a writer can use. A writer + built with newer Vortex crates can target an older edition that its intended readers support. * Fine-grained encryption. * Efficient access for both local disk and cloud storage. * Minimal overhead reading few columns or rows from wide or long arrays. @@ -106,23 +107,18 @@ as of June 2025, it might look as follows. ## Backward Compatibility -Backward compatibility guarantees that any **older** Vortex file can be read by **newer** versions of the Vortex library, -and is expected from all releases of Vortex from version 0.36.0 onwards. +Later Vortex library versions retain read support for frozen formats, beginning with the formats in +`core2025.05.0`, supported from version `0.36.0`. The reader must retain the required component +implementations, including optional plugins. See [Versioning](versioning.md) for the guarantee and +its boundaries. ## Forward Compatibility -:::{warning} -Forward compatibility is not yet implemented, but is planned to ship prior to the 1.0 release. -::: - -Forward compatibility extends the preceding stability guarantee such that **newer** Vortex files can be read by -**older** versions of the Vortex library. - -The intent of this work is to allow us to continue to evolve the Vortex File Format, avoiding calcification -and remaining up-to-date with new compression codecs and layout optimizations -- without breaking existing -readers or requiring lockstep upgrades. +Newer writers can already produce files for older readers by +[selecting editions](versioning/using-editions.md) whose formats those readers support. That allows +applications to upgrade independently while continuing to exchange files in supported formats. -The plan is that at write-time, a minimum supported reader version is declared. Any encodings or layouts added after that minimum -reader version can then be embedded into the file with WebAssembly decompression logic. Old readers are able to decompress new -data (slower than native code, but still with SIMD acceleration) and read the file. New readers are able to make the best use of -these encodings with native decompression logic and additional push-down compute functions (which also provides an incentive to upgrade). +Reading a newly introduced format with an older reader is a separate capability. A proposed approach +embeds WebAssembly decoding logic for new encodings and layouts in the file. An older reader with +the required execution support could then interpret them without a native implementation. This +approach is not implemented and is not part of the edition compatibility guarantee. diff --git a/docs/specs/index.md b/docs/specs/index.md index 12fab8da430..918082e200b 100644 --- a/docs/specs/index.md +++ b/docs/specs/index.md @@ -8,7 +8,7 @@ maxdepth: 2 --- file-format -editions +versioning ipc-format dtype-format scalar-format diff --git a/docs/specs/versioning.md b/docs/specs/versioning.md new file mode 100644 index 00000000000..9e8e9a1f5ab --- /dev/null +++ b/docs/specs/versioning.md @@ -0,0 +1,100 @@ +# Versioning and compatibility + +Vortex's file compatibility guarantees let applications upgrade their readers and writers +independently. The companion pages explain the configuration and the reasoning behind the design: + +- [Using editions](versioning/using-editions.md) explains how to configure writers for older readers + and diagnose compatibility errors. +- [Versioning design](versioning/design.md) explains how encodings and wire formats evolve, with a + worked example and the invariants that preserve compatibility. +- [Compatibility matrix](versioning/compatibility.md) summarizes the combinations of writer version, + edition, wire format, and reader version. +- [Edition registry](versioning/editions.md) lists the permitted wire IDs and minimum versions for + each edition. + +An array's _encoding_ describes its in-memory representation, whereas its _wire format_ defines how +to interpret its serialized metadata, buffers, and children. A _wire ID_ identifies that contract. +An array plugin maps between the two representations, so a reader can reconstruct the data using a +different in-memory encoding from the writer's. + +Vortex guarantees **backward compatibility** for its frozen wire formats: newer library versions +retain the code needed to decode them. As a result, applications can upgrade their readers without +rewriting existing files. Writers targeting older readers, however, must use wire formats those +readers understand. + +An _edition_ names the wire formats that a writer is allowed to use for arrays and +[other serialized components](versioning/editions.md#component-checks). Once an edition is +[_frozen_](versioning/editions.md#freezing-an-edition), that set and its reader requirements stay +fixed. The first frozen edition is `core2025.05.0`, supported from version `0.36.0` of the Vortex +Rust library. + +This guarantee concerns file compatibility. The library's programming interfaces, however, follow +[Rust's semantic versioning rules](https://doc.rust-lang.org/cargo/reference/semver.html), so an API +change can require application changes even when existing files remain readable. + +## Writing for older readers + +Consider two applications. A service writes files using Vortex `0.85.0`, while a query engine reads +them using Vortex `0.84.0`. These numbers identify the Rust library versions used by each +application. + +The service +[selects `core2026.08.0` for writing](versioning/using-editions.md#writer-configuration). That +edition's recorded minimum reader version is `0.84.0`, so the query engine meets the version +requirement. With the required implementations registered, it can read valid files successfully +written within that edition's restrictions. + +Edition selection constrains the serialized output, but the service still uses Vortex `0.85.0`'s +compression code and in-memory array encodings. That code can choose different compression schemes +or use different array data structures, provided each serialized component uses a permitted wire ID +and obeys its contract. The query engine can then decode the output into the encodings supported by +Vortex `0.84.0`. The [decimal example](versioning/design.md#example-decimal-children) shows how one +in-memory encoding can serialize to two wire formats. + +If the service instead selects `core2026.08.3`, the edition permits additional wire IDs and records +a minimum of `0.85.0`, so the query engine running `0.84.0` is no longer guaranteed to read every +file the service can produce. However, it can still read a particular file if it has implementations +for all the wire IDs that file uses. The recorded minimum covers every wire ID in the edition, +including those that a particular file does not use. + +## Compatibility requirements + +**A successful write with edition checks enabled uses only permitted wire IDs.** A reader with +implementations for all those IDs can read the output. The reader must also understand the enclosing +[file format](file-format.md). + +To rely on a frozen edition's guarantee, the application must meet its +[minimum library version](versioning/using-editions.md#reader-versions) and register the required +implementations, including any optional plugins. An independent plugin can have its own versions and +compatibility policy, so upgrading Vortex alone does not install it. + +However, edition selection does not guarantee that every input or custom writing strategy can +produce a permitted file. An array's encoding can require a wire representation that the edition +forbids, or a custom strategy can choose an unsupported layout. The +[write fails](versioning/using-editions.md#write-errors) when its serialized output violates the +selection. + +[Draft editions](versioning/editions.md#draft-editions) have no frozen compatibility guarantee. +Custom wire formats written with edition checks disabled also fall outside the edition guarantee. + +## Upgrades + +The default Vortex session selects the newest frozen `core` edition, so a library upgrade can change +the default output permissions. To keep serving older readers, explicitly select an edition whose +requirements those readers meet. Change that selection when the readers can support the additional +wire formats. + +Selecting an older edition restricts what the application writes, but it does not restrict which +wire IDs the reader's registered plugins can decode. + +```{toctree} +--- +maxdepth: 1 +hidden: true +--- + +versioning/using-editions +versioning/design +versioning/compatibility +versioning/editions +``` diff --git a/docs/specs/versioning/compatibility.md b/docs/specs/versioning/compatibility.md new file mode 100644 index 00000000000..e3a6f67a796 --- /dev/null +++ b/docs/specs/versioning/compatibility.md @@ -0,0 +1,83 @@ +# Compatibility matrix + +The matrix shows which wire formats each library can write, which the target edition permits, and +which each reader supports. The names are illustrative rather than actual Vortex versions or +editions: + +- **Format A** and **Format B** have distinct wire IDs and contracts, with Format A introduced + first. +- **Library 1** reads and writes only Format A. +- **Library 2** reads and writes both formats through one array implementation. +- **Edition 1** permits only Format A, whereas **Edition 2** permits both formats. + +Both readers are assumed to support all other components in the file. **Unsupported** means that the +writer has no implementation for that format, whereas **Forbidden** means that the target edition +excludes it. **Allowed** means both requirements are met, but the writer must still construct an +array that the format can represent. Reader results apply only after a successful write. + +| Writer | Target edition | Format | Write result[^representation] | Library 1 reader | Library 2 reader | +| -------------- | ---------------------------- | ------------- | ----------------------------- | ------------------------------------------- | --------------------- | +| Library 1 | Edition 1 | Format A | Allowed | Reads | Reads[^current-array] | +| Library 1 | Edition 1 | Format B | Unsupported and forbidden | N/A | N/A | +| Library 1 | Edition 2[^declaration] | Format A | Allowed | Reads | Reads[^current-array] | +| Library 1 | Edition 2[^declaration] | Format B | Unsupported | N/A | N/A | +| Library 2 | Edition 1 | Format A | Allowed[^compression] | Reads | Reads[^current-array] | +| Library 2 | Edition 1 | Format B | Forbidden | N/A | N/A | +| Library 2 | Edition 2 | Format A | Allowed[^compression] | Reads | Reads[^current-array] | +| Library 2 | Edition 2 | Format B | Allowed[^compression] | [Unknown ID](using-editions.md#unknown-ids) | Reads | + +The serializer [selects a format](design.md#format-selection) from the array's structure, after +which the writer checks whether the target edition permits it. The format column shows that +selection, not a separate writer setting. Edition 2 permits both formats, so a writer targeting it +can still produce Format A for Library 1 to read. + +## Compatibility checks + +The diagram follows an array from memory to storage and back through the serializer and reader +plugins. It assumes correct implementations, edition checks enabled, and full decoding with +`allow_unknown` disabled. + +```{figure} ../../_static/versioning-compatibility.svg +:alt: Plugins select and validate wire formats while adapting arrays between memory and storage. +:target: ../../_static/versioning-compatibility.svg + +The writer selects a plugin by the array's in-memory ID, whereas the reader selects a plugin by the +stored wire ID. In either direction, the plugin can adapt the array structure while preserving +values, data types, and nulls. The diagram groups related checks, although in the implementation +component checks occur at several points during writing and reading. +``` + +If the serializer returns a forbidden ID, the write fails without retrying a different permitted ID. +When the target requires a different encoding, the array must be recompressed before serialization. +See [Format selection](design.md#format-selection). + +The matrix assumes valid serialized data, whereas the diagram also shows the reader rejecting data +that violates the stored ID's contract. Format compatibility does not prevent I/O errors. + +The [component checks](editions.md#component-checks) cover arrays and their children, layouts, +extension types, and stored aggregates. The reader needs implementations for the components the file +uses, not every component its edition permits. See [Unknown IDs](using-editions.md#unknown-ids) for +missing implementations and the exceptions available with `allow_unknown`. + +For the compatibility guarantee and minimum reader versions, see [Versioning](../versioning.md). The +[design](design.md#compatibility-invariants) explains the invariants behind these outcomes. + +[^representation]: + An input array can require a format that the target edition forbids. A serializer can adapt + metadata, buffers, or children without recompression, but compatibility can require a different + encoding. In that case, the array must be recompressed before serialization or the write fails. + +[^current-array]: + Library 2 reads Format A into its own array implementation, adapting the structure only if + necessary. A library upgrade alone does not require an array conversion. + +[^declaration]: + Library 1 needs Edition 2's declaration to select it. Registering the declaration does not add + support for Format B, so Library 1 still writes only Format A. + +[^compression]: + The default compressor filters schemes by their declared output wire IDs, but general per-writer + scheme configuration is not implemented. The planned configuration would select compatible + behavior before estimation, sampling, and full compression, avoiding recompression solely to + meet the edition. These cells still require the writer to construct a permitted representation. + See [Compression](design.md#compression) for the current behavior and planned work. diff --git a/docs/specs/versioning/design.md b/docs/specs/versioning/design.md new file mode 100644 index 00000000000..d2502458124 --- /dev/null +++ b/docs/specs/versioning/design.md @@ -0,0 +1,276 @@ +# Versioning design + +Applications that share files can use different library versions and upgrade at different times. As +new library releases improve compression and in-memory data structures, they must continue to read +existing files and write files for applications that have not upgraded. + +Vortex separates in-memory array encodings and compression code from the wire formats stored in +files. As a result, these implementations can change while retaining wire formats that existing +readers understand. The [versioning overview](../versioning.md) describes the compatibility +guarantee. + +## Versions and formats + +An array's in-memory and serialized representations have separate contracts. The terms below +distinguish those representations, the code that converts between them, and the permissions that +control what a writer can produce: + +```{list-table} +:header-rows: 1 +:class: versioning-terms + +* - Term + - Meaning +* - **Encoding** + - An array's in-memory representation, including its buffers and child arrays. +* - **Wire format** + - The contract for interpreting serialized data: valid data types, metadata, buffers, children, + and their meaning. +* - **Wire ID** + - The identifier for a wire contract. Once frozen, that contract cannot change under the same ID. +* - **Array plugin** + - Code that serializes an in-memory encoding and deserializes one or more wire formats into + arrays. +* - **Edition** + - A named set of permitted component wire IDs, against which writers check their output. +* - **Library version** + - A release of the implementations: encodings, plugins, compression algorithms, readers, and + writers. +``` + +For example, a newer library can improve how it compresses a dictionary's values while keeping the +same dictionary wire format. Similarly, it can change its internal array fields while retaining code +to read old files. Only a change to what a reader must understand requires a new serialized +contract. + +The file container has a separate [version tag](../file-format.md#file-specification) that describes +the enclosing format. Component wire IDs, on the other hand, describe the arrays and other +structures within that container, so those components can evolve independently. + +## Serialization + +In Vortex, compression produces an encoded array in memory. The default compressor chooses among +_compression schemes_, each of which changes the representation while preserving values, data types, +and nulls. An array can contain buffers and other arrays, called _children_. A dictionary array, for +example, has a child for its values and another for the codes that refer to those values. Each child +can use its own encoding. + +### Array plugin conversions + +An array plugin provides the code to serialize an in-memory array and read serialized data back into +an array. A plugin can support several wire formats. Support grows by adding wire IDs, while the +rules for frozen IDs stay fixed and readers retain the code needed to read them. + +When writing an array, Vortex finds its serializer using the array's in-memory encoding ID. The +serializer chooses a wire format and returns that format's wire ID, metadata, buffers, and child +arrays. It can construct different metadata, buffers, or children from those in the input array to +meet the chosen format's requirements. The library can therefore change its in-memory array data +structures while continuing to serialize data according to the same wire format. Each returned child +is then serialized through its own plugin. + +When reading, however, Vortex finds the deserializer using the wire ID stored in the file. As it +constructs an in-memory array, the deserializer must check that the metadata, buffers, data types, +and children satisfy the rules for that ID. It can arrange the buffers and children differently in +memory, or return an array whose encoding ID differs from the plugin's own in-memory ID. **A plugin +can therefore read several wire formats into the same in-memory array type.** + +These operations do not upgrade or downgrade the file. A reader can use an array implementation +added after the file was written, but the file's contents and wire IDs remain unchanged. Similarly, +a writer can use its current array implementation to serialize data in a wire format introduced by +an earlier library release. Neither operation requires a separate in-memory array type for each wire +format. Both must preserve values, data types, and nulls, although the array's structure in memory +can change when it is serialized and read back. + +An application can opt out of a particular conversion during reading by +[registering another plugin for the same wire ID](../../developer-guide/internals/session.md#registering-plugins). +That plugin must read the same serialized data correctly, but it can construct a different encoding +in memory. This gives the application control over how the data is represented without changing the +file or the rules for interpreting it. + +For example, the ALP wire format stores exceptional values, called patches, inside the ALP array. +With experimental `Patched` support enabled, the registered plugin constructs a `Patched` parent +that holds those patches and an ALP child that has none. In contrast, the standard ALP plugin keeps +the patches inside the ALP array in memory. Registering the standard plugin opts out of that change +to the array structure, while still reading the same stored data under the same wire ID. + +## Example: decimal children + +The decimal-byte-parts plugin can serialize the same in-memory array type using two wire formats. +Its encoding stores decimal values in integer child arrays, either in one child or split across +several children. The v1 wire contract permits only one child, while v2 adds support for multiple +children under a distinct wire ID.[^decimal-availability] + +```{figure} ../../_static/versioning-flow.svg +:alt: One decimal encoding holds either one signed child or a signed child with unsigned lower parts. The serializer chooses v1 for one child and v2 for multiple children. Both wire formats deserialize into the same array type. +:target: ../../_static/versioning-flow.svg +:figclass: versioning-diagram + +The arrows show the serializer's choices and the corresponding reads. Although v2 also accepts a +single child, the serializer chooses v1 for that shape. Each child has its own wire ID because it is +itself a serialized array. +``` + +For a single-child array, the serializer reuses the child and writes v1 metadata without +recompression. In contrast, an array with several children uses v2, even if its values are small +enough to fit in one child, because combining the parts requires re-encoding that the serializer +does not perform. + +The array type's in-memory ID is `vortex.decimal_byte_parts.v2` for both shapes. Since the +serializer can return a different wire ID, the writer must check the returned ID against the target +editions. + +Both wire formats deserialize into the same in-memory array type. However, the reader must still +validate the contract identified by the stored ID: `vortex.decimal_byte_parts` requires exactly one +signed integer child. Support for multiple children under v2 does not make them valid under v1. + +## Format selection + +A serializer must choose the oldest supported writable format that preserves the array's +representation without recompression. A plugin can adapt metadata, buffers, or children to fit an +older format. Formats retained only for reading, however, are not candidates for writing. + +The serializer makes that choice before the writer checks edition permissions. If the chosen ID is +forbidden, the write fails without retrying another format that happens to be permitted. In +particular, a custom edition that permits only the v2 decimal ID cannot write the single-child array +through this serializer, which selects v1. + +This policy preserves compatibility with readers of the earlier wire format when the existing +representation allows it. If compatibility requires a different encoding of the same values, the +write path must arrange recompression before serialization or fail. Selecting an edition does not +perform that conversion automatically. + +## Write and read checks + +Choosing a wire format and permitting it are separate steps. On writing, the selected editions +restrict which IDs the serializers may return, including those of every child. On reading, however, +edition selection does not restrict the input. Instead, the reader needs registered implementations +for the stored IDs, and those implementations must validate the corresponding contracts. + +```{figure} ../../_static/versioning-checks.svg +:alt: Writers select plugins by in-memory IDs, serialize arrays and children, and check all returned IDs against edition permissions. Readers select plugins by stored wire IDs, validate each contract, and construct in-memory arrays. Forbidden IDs, unknown IDs, and invalid data cause errors. +:target: ../../_static/versioning-checks.svg +:figclass: versioning-diagram + +The diagram groups related checks, which can occur at several points during writing and reading. +It assumes that edition enforcement is enabled, unknown IDs are rejected, and the enclosing file +format is supported. The [compatibility decision tree](compatibility.md#compatibility-checks) +includes serializer availability and the other conditions needed to read or write a file. +``` + +A missing reader implementation causes an [unknown-ID error](using-editions.md#unknown-ids), whereas +data that violates a known contract causes a validation error. Neither case becomes valid merely +because the reader supports another wire format for the same encoding. + +## Children and other components + +Suppose the decimal serializer selects the v1 wire ID, but its integer child's serializer returns an +ID that the target edition forbids. Checking only the decimal ID accepts output that the intended +reader cannot decode, so the writer must check every child recursively. + +The same requirement extends beyond arrays. A file also describes its layout, logical types, and +stored summaries used for pruning. Editions cover each of these component kinds: + +| Kind | What its wire ID identifies | +| ----------- | ------------------------------------------------------- | +| `array` | An array's serialized representation | +| `layout` | A node in the file's layout tree | +| `dtype` | An extension dtype, which defines a custom logical type | +| `aggregate` | An aggregate function stored in a zone map | + +A _zone map_ stores summaries for a group of rows, such as its minimum and maximum. Readers use +those summaries to skip groups that cannot match a filter. Their aggregate definitions need stable +meaning just as array formats do. The [component checks](editions.md#component-checks) describe the +writing rules for each kind. + +The kind and ID together identify a contract. For example, the array and layout named +`vortex.chunked` are separate components, so supporting one does not imply support for the other. + +## Editions + +Applications need a way to select compatible output without maintaining their own inventory of every +component. A frozen edition gives that inventory a stable name and records a library version that +supports all its members. New formats require a later edition, leaving the earlier target available +to writers targeting older versions. + +An _edition family_ groups editions for related components. Membership is cumulative within a +family: each later edition includes all earlier members. + +The `core` family covers the default writer's formats, while optional features have independent +families. A writer can select one `core` edition and one `tensor` edition, for example, and use the +union of their permitted components. This avoids tying a change in an optional feature to a change +in the application's core target. However, the reader must satisfy both selections' requirements. + +Each family names an _origin_, the project that supplies its implementations. A frozen edition's +minimum version refers to that origin. For `core`, it is the Vortex Rust library. Independent +plugins can use their own release numbers, so there is no single version comparison that covers +every possible combination. Versions must meet the minimum for each origin, and the implementations +must be registered in the reader. + +An edition declaration supplies permissions, so registering it does not add the implementations +needed to read or write its wire formats. See +[Writer configuration](using-editions.md#writer-configuration) for how to register and select +editions. + +The [compatibility matrix](compatibility.md) shows the combinations of writer version, edition, wire +format, and reader version. + +## Compatibility invariants + +1. **A frozen wire contract is immutable.** Its valid data types, metadata, buffers, children, + options, and meanings stay fixed, so a reader-visible extension requires a new ID. +2. **Compression, serialization, and reading preserve meaning.** Each serializer produces a valid + instance of its chosen contract, and each reader enforces that exact contract. All three + operations preserve values, data types, nulls, and the meaning of other components. +3. **Readers preserve backward compatibility.** Later implementations retain read support for frozen + formats, including those writers no longer choose. Edition selection does not restrict what a + reader can read. +4. **Frozen edition records are immutable.** Membership, origin, and recorded minimum stay fixed. + Later editions include all earlier members within their family, while selecting editions from + multiple families permits the union of their members. +5. **Edition enforcement covers the whole output.** Every serialized component must be permitted, + including children and nested dependencies, so compressor declarations alone are insufficient. +6. **Recorded reader requirements are sound.** Each recorded origin version supplies readers for + every member of the edition, but applications must register those implementations to use them. + +Together, these rules establish the relationship: + +```text +IDs used by the file ⊆ IDs permitted by the editions ⊆ IDs supported by the reader +``` + +The IDs here include their component kinds. With valid serialized data, correct implementations, and +support for the enclosing file format, the reader can interpret the output. + +This read guarantee is separate from a writer's ability to produce suitable output. A writer must +retain the behavior needed for the target editions it supports, but it does not need to retain every +historical writing implementation. + +## New formats + +A new format starts in a draft edition so it can be tested before its origin commits to reading it +indefinitely. Drafts have no recorded minimum version or frozen guarantee. A format intended for +`core` can progress from its own family to `preview` for broader testing, then to `core` for default +use. Promotion preserves its wire ID and interpretation. The +[registry instructions](editions.md#format-testing-and-promotion) cover promotion, freezing, and +recording the minimum version. + +## Compression + +The default BtrBlocks compressor excludes a scheme if any of its declared output wire IDs is +forbidden. This filtering also applies to schemes used for child compression. + +The current decimal scheme produces only single-child arrays and declares the v1 wire ID, so values +too wide for it remain in the standard uncompressed decimal representation. The multi-child +serializer exists, but the default scheme does not construct those arrays and no declared edition +permits their v2 ID. + +### Planned scheme configuration + +The planned improvement is to configure a scheme's behavior for the selected editions, allowing it +to retain an older mode when its newer mode requires a forbidden format. That configuration needs to +apply consistently to estimation, sampling, full compression, children, and fallbacks. General +per-writer scheme configuration is not implemented, and its API is unsettled. + +[^decimal-availability]: + No declared edition currently permits the multi-child format. The + [compression section](#compression) describes what the default compressor produces today. diff --git a/docs/specs/versioning/editions.md b/docs/specs/versioning/editions.md new file mode 100644 index 00000000000..5666639e54d --- /dev/null +++ b/docs/specs/versioning/editions.md @@ -0,0 +1,161 @@ +# Edition registry + +Each entry lists the components added by that edition, which also permits every component from +earlier editions in the same family. See [Using editions](using-editions.md) for configuration and +[Versioning design](design.md) for the compatibility rules. + +## Frozen `core` editions + +The origin of every edition below is `vortex`. Each minimum refers to the shared version of the +Vortex Rust crates, including the `vortex` crate. + +### `core2025.05.0` + +Minimum Vortex Rust crate version: `0.36.0`. + +- `array`: `fastlanes.bitpacked`, `fastlanes.for`, `vortex.alp`, `vortex.alprd`, `vortex.bool`, + `vortex.bytebool`, `vortex.chunked`, `vortex.constant`, `vortex.datetimeparts`, `vortex.decimal`, + `vortex.decimal_byte_parts`, `vortex.dict`, `vortex.ext`, `vortex.fsst`, `vortex.list`, + `vortex.null`, `vortex.primitive`, `vortex.runend`, `vortex.sparse`, `vortex.struct`, + `vortex.varbin`, `vortex.varbinview`, `vortex.zigzag` +- `layout`: `vortex.chunked`, `vortex.dict`, `vortex.flat`, `vortex.stats`, `vortex.struct` +- `dtype`: `vortex.date`, `vortex.time`, `vortex.timestamp` + +### `core2025.06.0` + +Minimum Vortex Rust crate version: `0.40.0`. + +- `array`: `vortex.pco`, `vortex.sequence`, `vortex.zstd` + +### `core2025.10.0` + +Minimum Vortex Rust crate version: `0.54.0`. + +- `array`: `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked` + +### `core2026.08.0` + +Minimum Vortex Rust crate version: `0.84.0`. + +- `layout`: `vortex.zoned` +- `aggregate`: `vortex.bounded_max`, `vortex.bounded_min`, `vortex.max`, `vortex.min`, + `vortex.nan_count`, `vortex.null_count` + +### `core2026.08.1` + +Minimum Vortex Rust crate version: `0.84.0`. + +- `array`: `vortex.onpair` + +### `core2026.08.2` + +Minimum Vortex Rust crate version: `0.85.0`. + +- `array`: `vortex.map` + +### `core2026.08.3` + +Minimum Vortex Rust crate version: `0.85.0`. + +- `array`: `vortex.parquet.variant`, `vortex.variant` +- `dtype`: `vortex.uuid` + +## Draft editions + +Draft editions have no recorded minimum version of their origin project's code, and each new format +or revision requires a new draft edition. Vortex-maintained draft formats are expected to remain +compatible unless a defect blocks promotion into `core`, while independent plugin projects state +their own policy. + +### `preview2026.08.0` + +This edition currently adds no components. + +### `tensor2026.04.0` + +- `array`: `vortex.tensor.cosine_similarity`, `vortex.tensor.inner_product`, + `vortex.tensor.l2_norm`, `vortex.tensor.l2_normalize` +- `dtype`: `vortex.tensor.fixed_shape_tensor`, `vortex.tensor.vector` + +### `zstd2026.02.0` + +- `array`: `vortex.zstd_buffers` + +### `spatial2026.08.0` + +- `dtype`: `vortex.st.box`, `vortex.st.linestring`, `vortex.st.multilinestring`, + `vortex.st.multipoint`, `vortex.st.multipolygon`, `vortex.st.point`, `vortex.st.polygon`, + `vortex.st.wkb` +- `aggregate`: `vortex.st.aabb` + +### `json2026.08.0` + +- `dtype`: `vortex.json` + +## Component checks + +The writer checks the formats it actually serializes against the selected editions. The checks cover +four kinds of component: + +| Kind | Writing rule | +| ------------------- | -------------------------------------------------------------------------------------- | +| Arrays | Check the serializer's returned wire ID and every serialized child recursively. | +| Layouts | Check every serialized layout ID. The writing strategy must use permitted layouts. | +| Extension dtypes | Check all extension dtypes in the schema, including nested ones, before writing bytes. | +| Aggregate functions | Check every function stored in a zone map against the edition and its format contract. | + +A forbidden zone-map aggregate causes the write to fail. Silently omitting it would change which +filters can use the configured zone map to skip rows. By contrast, the writer omits an aggregate +that does not apply to a column's data type, so there is no serialized component to check. + +For example, `core2026.08.0` declares `min`, `max`, `bounded_min`, `bounded_max`, `nan_count`, and +`null_count`. Zone maps do not store sums, so the edition does not declare `sum`. File-level +statistics, however, do store sums, in a fixed legacy field governed by the enclosing format's +contract. + +## Format testing and promotion + +A format intended for `core` starts in a dedicated edition family. Its first edition is a draft, +with no recorded `min_library_version` and no frozen compatibility guarantee. Even at this draft +stage, the format is expected to be complete. If testing reveals a defect whose correction changes +what readers must understand, the correction needs a new wire ID and a later edition. + +After initial testing, the format can enter a new `preview` edition for broader opt-in use, then a +later `core` edition for default use. Promotion changes which editions permit the format while +preserving its contract and wire ID. The current `preview` edition is empty, so its first component +must go into a new edition. + +## Freezing an edition + +A stable edition can freeze when its origin publishes the code that first supports all its members. +For `core`, this is a Vortex Rust crate release. Independent plugins use their own versions. + +Until the release version is known, the declaration uses `min_library_version: None`. Once it is +known, the field records the first release that supports all members, usually while the next release +is in development. Recording the version documents the freeze, but the guarantee applies from that +release even if the declaration is updated later. + +A frozen edition's membership, origin, and minimum version stay fixed. Deprecating a format can stop +writers from choosing it, but readers must retain support because existing files can contain it. + +## Maintaining edition records + +Default declarations live in `vortex-edition/src/declarations/`. Optional modules keep declarations +with their implementation code. The exported TOML records are under `vortex/editions/`, grouped by +family. + +1. For a new component, declare its own family and draft edition. For a revision, add a later + edition to the family that owns the earlier ID. +2. To promote a tested format, add it to new `preview` and `core` editions without changing its ID + or contract. +3. When an edition freezes, record the first release of its origin that supports every permitted + component, including inherited members. +4. Regenerate the records: + + ```sh + cargo run -p xtask -- generate-editions + ``` + +CI's `check-editions` command rejects changes to frozen records, including renames, unfreezing, and +deletion, so changes to permitted wire formats require a later edition. The command also rejects a +new edition that does not follow its family's chronology. diff --git a/docs/specs/versioning/using-editions.md b/docs/specs/versioning/using-editions.md new file mode 100644 index 00000000000..db08812895e --- /dev/null +++ b/docs/specs/versioning/using-editions.md @@ -0,0 +1,114 @@ +# Using editions + +To write files for an older Vortex version, select editions whose wire formats that version +supports. See [Versioning](../versioning.md) for the compatibility guarantee. + +## Writer configuration + +A _session_ holds the registered implementations, edition declarations, and enabled editions. The +following function creates write options targeting `core2026.08.0`, whose recorded minimum reader +version is `0.84.0`: + +```rust +use vortex::VortexSessionDefault; +use vortex::editions::CORE_2026_08_0; +use vortex::editions::EditionSessionExt; +use vortex::error::VortexResult; +use vortex::file::VortexWriteOptions; +use vortex::file::WriteOptionsSessionExt; +use vortex::session::VortexSession; + +fn writer_for_older_readers() -> VortexResult { + let session = VortexSession::default(); + session.enable_edition(CORE_2026_08_0)?; + + Ok(session.write_options()) +} +``` + +Use the returned options' `write` method to write an array stream to an output. The +[Rust quickstart](../../getting-started/rust.rst) covers the input and I/O setup. The example uses +file support from the `vortex` crate. + +The default session registers the standard implementations and edition declarations, and it +currently enables `core2026.08.3`. Calling `enable_edition` replaces the enabled edition from the +same family. Set the selection before starting the write, because that is when the writer captures +the permitted wire IDs. + +An edition declaration describes permitted formats, but registering it does not install the code to +read or write those formats. When constructing a session without the defaults, register the required +implementations and declarations, then enable the target editions. See +[Registering plugins](../../developer-guide/internals/session.md#registering-plugins) for the +registration API. Enabling an unregistered edition returns an error, while a selection that permits +no components prevents the writer from serializing any component governed by editions. + +## Edition families + +An _edition family_ groups editions for related formats. The `core` family covers the default +writer's formats. Optional features have their own families, such as `tensor` and `zstd`, so they +can add formats without changing an application's `core` selection. + +A writer selects at most one edition per family. Selecting `core2026.08.0` and `tensor2026.04.0` +permits every component in either edition. Within one family, a later edition includes all earlier +members. Across families, however, the selections are independent. + +Check the [registry](editions.md#edition-registry) before enabling an optional family. For example, +`tensor2026.04.0` is a draft and has no frozen minimum reader version. Adding it does not extend +`core`'s frozen guarantee to the tensor formats. Both applications still need the appropriate tensor +implementations. + +An edition name such as `core2026.08.3` contains its family, year, month, and a number +distinguishing editions in that family and month. These are Vortex editions, separate from Rust +language editions. + +## Reader versions + +For each selected frozen edition, find its recorded minimum version and its _origin_ in the +[registry](editions.md). The origin is the project that supplies the component implementations. The +`core` family's origin is `vortex`, so its `min_library_version` refers to the shared Vortex Rust +crate version. An independent plugin can name a different origin with its own release numbers. + +For editions with the same origin, use at least the highest recorded minimum, whereas editions from +different origins require a separate version check for each project. In both cases, the reader must +register the required implementations, since meeting the version requirement alone does not make a +plugin available. + +## Write errors + +The writer rejects forbidden formats in arrays, children, layouts, nested extension dtypes, and +stored aggregate functions. + +Selecting an edition restricts the permitted output, but does not automatically reconfigure a custom +strategy or compressor. Those implementations must therefore construct permitted representations +themselves. The default compressor's filtering is described in [Compression](design.md#compression). + +When a write fails because a format is forbidden, choose a permitted representation or strategy. +Alternatively, select a later edition after confirming that the readers meet its requirements. +[The decimal example](design.md#example-decimal-children) shows why an array's structure can require +a newer format even when its values appear suitable for an older one. + +For custom or experimental output, `VortexWriteOptions::disable_editions()` disables the array, +layout, extension-dtype, and aggregate checks. However, it does not register missing +implementations. Files written this way have no edition compatibility guarantee, so producers and +consumers must agree on the required implementations themselves. + +## Unknown IDs + +An unknown-ID error means that the reader has no registered implementation for that component. Look +up its kind and ID in the [registry](editions.md#edition-registry). The kind matters because an +array and a layout can share the same ID string while describing different formats. + +- For a frozen edition, use at least the recorded minimum version of its origin and register the + required optional module. +- For a draft edition, obtain a build that implements the component from the producer. A draft does + not promise support in a published release. +- For a component absent from the registry, obtain its implementation from the producer and register + it with the session. + +Inspection and copying tools can use `allow_unknown` to retain the serialized data of unknown +arrays, layouts, and extension dtypes without interpreting it. However, retaining those objects does +not make them available for ordinary computation. + +With `allow_unknown`, an unknown aggregate disables pruning for the affected zone-map layout, but +the layout's data remains readable if the reader supports the other required wire formats. Without +`allow_unknown`, the unknown aggregate causes an error. diff --git a/vortex-edition/src/lib.rs b/vortex-edition/src/lib.rs index ba2382ec02b..ede2646bb17 100644 --- a/vortex-edition/src/lib.rs +++ b/vortex-edition/src/lib.rs @@ -29,7 +29,7 @@ //! //! The first-party edition declarations live in this crate. The public `vortex` crate //! re-exports them and registers and enables them on the default session. See the published spec at -//! . +//! . pub mod declarations; mod session;