feat(inspect): add system metadata tables - #876
Draft
manuzhang wants to merge 9 commits into
Draft
Conversation
- Add Scan() virtual method and Scan() convenience overload to MetadataTable - Add SnapshotSelection struct for time-travel snapshot resolution - Add supports_time_travel() concrete method driven by kind() - Implement SnapshotsTable::Scan() to materialize snapshot rows via ArrowRowBuilder
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
manuzhang
force-pushed
the
agent/add-system-metadata-tables
branch
from
August 17, 2026 10:52
f4e4c83 to
b0f596c
Compare
There was a problem hiding this comment.
Pull request overview
Adds new “system” metadata tables under iceberg/inspect/ and shared utilities to scan Iceberg metadata into bounded Arrow batches, aligning C++ inspect APIs with Iceberg’s reference behavior. This builds on the streaming metadata-table API work referenced in #801.
Changes:
- Adds new metadata table implementations: branches, tags, files, partitions, manifests (plus supporting shared stream/util code).
- Refactors metadata-table base APIs to support typed factories, bounded batch sizing, and time-travel scans via
SnapshotSelection. - Adds/updates unit tests and wires them into CMake/Meson builds.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/iceberg/type_fwd.h | Forward declares the new inspect table classes. |
| src/iceberg/inspect/metadata_table.h | Expands/modernizes the metadata-table base API (kinds, typed factory, batch size, time travel interface). |
| src/iceberg/inspect/metadata_table.cc | Implements the refactored base behavior and TimeTravelMetadataTable scan dispatch. |
| src/iceberg/inspect/metadata_table_stream_internal.h | Adds a shared Arrow stream implementation for row-vector-backed metadata tables. |
| src/iceberg/inspect/metadata_table_util_internal.h | Declares shared internal helpers for snapshot resolution, schema building, partition projection, and live-file loading. |
| src/iceberg/inspect/metadata_table_util_internal.cc | Implements shared internal utilities used by files/partitions/manifests scans. |
| src/iceberg/inspect/branches_table.h | Declares the branches metadata table. |
| src/iceberg/inspect/branches_table.cc | Implements branch reference scanning into Arrow batches. |
| src/iceberg/inspect/tags_table.h | Declares the tags metadata table. |
| src/iceberg/inspect/tags_table.cc | Implements tag reference scanning into Arrow batches. |
| src/iceberg/inspect/files_table.h | Declares the snapshot-scoped files metadata table (time travel capable). |
| src/iceberg/inspect/files_table.cc | Implements live-file scanning for a selected snapshot into Arrow batches. |
| src/iceberg/inspect/partitions_table.h | Declares the snapshot-scoped partitions aggregates table (time travel capable). |
| src/iceberg/inspect/partitions_table.cc | Implements partition aggregation over live files for a selected snapshot. |
| src/iceberg/inspect/manifests_table.h | Declares the snapshot-scoped manifests table (time travel capable). |
| src/iceberg/inspect/manifests_table.cc | Implements manifest-list scanning for a selected snapshot, including partition summaries. |
| src/iceberg/inspect/snapshots_table.h | Extends snapshots table API with schema + streaming Scan(). |
| src/iceberg/inspect/snapshots_table.cc | Implements snapshots streaming scan with bounded batch size and resource cleanup. |
| src/iceberg/inspect/history_table.h | Updates history table interface to match new base API expectations. |
| src/iceberg/inspect/history_table.cc | Updates history table implementation; currently returns NotSupported for Scan(). |
| src/iceberg/inspect/meson.build | Installs the new public inspect headers. |
| src/iceberg/arrow_row_builder_internal.h | Adds num_rows() to support batch-size enforcement in streaming producers. |
| src/iceberg/arrow_row_builder.cc | Implements ArrowRowBuilder::num_rows(). |
| src/iceberg/test/arrow_row_builder_test.cc | Adds assertions covering the new num_rows() behavior. |
| src/iceberg/test/metadata_table_test_base.h | Introduces a shared fixture/helpers for metadata-table tests. |
| src/iceberg/test/metadata_table_test.cc | Updates tests to the typed metadata-table factory and validates time-travel support flags. |
| src/iceberg/test/history_table_test.cc | Adds schema-focused unit test coverage for HistoryTable. |
| src/iceberg/test/snapshots_table_test.cc | Adds snapshots streaming scan tests (schema/values/batching/null handling). |
| src/iceberg/test/system_metadata_tables_test.cc | Adds integration-style tests for the new system metadata tables (refs/files/partitions/manifests and snapshot selection). |
| src/iceberg/test/CMakeLists.txt | Wires the new/updated metadata-table tests into the CMake test target. |
| src/iceberg/test/meson.build | Wires (some) of the new tests into Meson’s table_test target. |
| src/iceberg/CMakeLists.txt | Adds new inspect sources to the main CMake build. |
| src/iceberg/meson.build | Adds new inspect sources to the main Meson build. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
56
to
60
| 'snapshot_test.cc', | ||
| 'snapshot_util_test.cc', | ||
| 'system_metadata_tables_test.cc', | ||
| 'table_metadata_builder_test.cc', | ||
| 'table_requirement_test.cc', |
Comment on lines
+53
to
+55
| Result<ArrowArrayStream> HistoryTable::Scan() { | ||
| return NotSupported("Scan is not supported for the history table"); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add streaming system metadata tables for branches, tags, files, partitions, and manifests.
Changes
Why
This makes Iceberg system metadata available through the C++ metadata-table API, using bounded Arrow batches and behavior aligned with Apache Iceberg's reference implementation.
Impact
Consumers can inspect table references, live files, partition aggregates, and manifest metadata without implementing manifest traversal or Arrow materialization themselves.
Dependency
This branch is based on #801, which provides the metadata-table streaming APIs used here. Because this PR targets
main, its diff includes that prerequisite history until #801 is merged.