Skip to content

Add STRUCT and LIST support for Arrow/JSON, fix DuckDB Arrow coercion - #3210

Merged
texodus merged 1 commit into
masterfrom
virtual-server-arrow-jamboree
Aug 9, 2026
Merged

Add STRUCT and LIST support for Arrow/JSON, fix DuckDB Arrow coercion#3210
texodus merged 1 commit into
masterfrom
virtual-server-arrow-jamboree

Conversation

@texodus

@texodus texodus commented Aug 9, 2026

Copy link
Copy Markdown
Member

Nested struct/list ingest for Arrow and JSON

perspective-server now accepts Arrow LIST and STRUCT columns, as well as JSON/NDJSON Array and Object columns, via column expansion. This change makes it much easier to wire existing Arrow and JSON feeds from e.g. Kafka or csp directly into a Perspective Table without marshalling and transforming the input (and substantially reducing column copies when the transfer encoding is Arrow, since Perspective can route this directly).

  • struct / object → one dotted column per leaf, recursively. A null parent
    nulls every descendant (e.g. struct itself is not nullable).
  • list / array → controlled by a new list_flatten table option:
    • zip (default) — one row per element, non-list siblings repeated; an
      empty list yields one null row; multiple lists in a record must share a
      length.
    • cartesian — cartesian product.
    • stringifyJSON.stringify.

Virtual server window aggregates and type coercion

Features::window_aggregates carried a bare list of names per column type, which forced the SQL translation to know each function's shape. It now encodes these as generic WindowAggSpec, mirroring the existing declared AggSpec, which allows the model to specify:

  • frames — which frame kinds the function accepts (rows, range,
    cumulative, none).
  • offset — takes a row offset.
  • alpha — takes a smoothing factor, as ema does.
  • result_type — the output type.

duckdb.py and clickhouse.py virtual servers have updated their window aggregate catalogs against it, including ranking functions that take no source column and derived ones like rate. New backends should be able to add arbitrary window aggregates via the descriptor language.

Arrow type coercion

Completes the DuckDB/Arrow type coercion work started in #3151 with the additional missing types (sans arrow-compute). Add support for DECIMAL, TINYINT, ENUM, the unsigned types and timezone-carrying timestamps at any unit, fixes #3149

EDIT: This includes timestamp(μs, "UTC") which also has tests

@texodus texodus added the enhancement Feature requests or improvements label Aug 9, 2026
@texodus texodus changed the title Add STRUCT and LIST support for Arrow/JSON, fix V Add STRUCT and LIST support for Arrow/JSON, fix DuckDB Arrow coercion Aug 9, 2026
@texodus
texodus force-pushed the virtual-server-arrow-jamboree branch from c7ae845 to 79cdb81 Compare August 9, 2026 15:40
Signed-off-by: Andrew Stein <steinlink@gmail.com>
@texodus
texodus force-pushed the virtual-server-arrow-jamboree branch from 79cdb81 to 80f4303 Compare August 9, 2026 18:46
@texodus
texodus merged commit bd038e7 into master Aug 9, 2026
17 checks passed
@texodus
texodus deleted the virtual-server-arrow-jamboree branch August 9, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Feature requests or improvements

Development

Successfully merging this pull request may close these issues.

VirtualServer: coerce_column missing support for common Arrow types from external databases

1 participant