Skip to content

AddFiles: create the table from the file schemas when it does not exist - #40124

Open
claudevdm wants to merge 1 commit into
apache:masterfrom
claudevdm:create-table-schema
Open

claudevdm wants to merge 1 commit into
apache:masterfrom
claudevdm:create-table-schema

Conversation

@claudevdm

Copy link
Copy Markdown
Collaborator

Before this change the pre-pass required an existing table. AddFiles without evolution creates a missing table from whichever file happens to reach ConvertToDataFile first; with the pre-pass every schema of the window is known before anything is created.

Create path (CommitSchemaUnion.create), taken when loadTable throws NoSuchTableException:

  • The window's schemas are folded into one union on a scratch create transaction that is never committed (the same staging loop as the evolve path, so a schema conflicting with another is reported or skipped per the handling, and the create is abandoned under FAIL_PIPELINE before anything exists).
  • The real table is built directly from createdSchema(union, config): every column optional at every level - one lucky file's declared or proven required-ness must not become a table constraint that the next file violates - except pinned paths and their ancestors, which are created required. Creation is the schema-authoring moment; evolution never tightens columns afterwards. A pin that no file schema carries fails the create under FAIL_PIPELINE (later windows could only add the column optional, so the pin would stay inert forever) and warns under ROUTE_TO_ERRORS. The created table is born with a single schema version.
  • Creation is not gated on any particular evolution option: the options guard an existing table's schema, and there is none to guard yet.
  • Partition spec, sort order and table properties come from TableCreation (the AddFiles constructor arguments), resolved against the union, so a partition or sort column carried by any schema of the window works; the name mapping is written in the same transaction.
  • Create race: two workers (or a concurrent non-evolution AddFiles) may create the table at once. AlreadyExistsException joins CommitFailedException in the retry, and the next attempt takes the evolve path against the table the other party created.
  • Empty window against a missing table returns NO_TABLE (-1) and creates nothing.

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Before this change the pre-pass required an existing table. AddFiles
without evolution creates a missing table from whichever file happens to
reach ConvertToDataFile first; with the pre-pass we can do better, since
every schema of the window is known before anything is created.

Create path (CommitSchemaUnion.create), taken when loadTable throws
NoSuchTableException:

- The window's schemas are folded into one union on a scratch create
  transaction that is never committed (the same staging loop as the
  evolve path, so a schema conflicting with another is reported or
  skipped per the handling, and the create is abandoned under
  FAIL_PIPELINE before anything exists).
- The real table is built directly from createdSchema(union, config):
  every column optional at every level - one lucky file's declared or
  proven required-ness must not become a table constraint that the next
  file violates - except pinned paths and their ancestors, which are
  created required. Creation is the schema-authoring moment; evolution
  never tightens columns afterwards. A pin that no file schema carries
  fails the create under FAIL_PIPELINE (later windows could only add
  the column optional, so the pin would stay inert forever) and warns
  under ROUTE_TO_ERRORS. The created table is born with a single schema
  version.
- Creation is not gated on any particular evolution option: the options
  guard an existing table's schema, and there is none to guard yet.
- Partition spec, sort order and table properties come from
  TableCreation (the AddFiles constructor arguments), resolved against
  the union, so a partition or sort column carried by any schema of the
  window works; the name mapping is written in the same transaction.
- Create race: two workers (or a concurrent non-evolution AddFiles) may
  create the table at once. AlreadyExistsException joins
  CommitFailedException in the retry, and the next attempt takes the
  evolve path against the table the other party created.
- Empty window against a missing table returns NO_TABLE (-1) and
  creates nothing.
@github-actions github-actions Bot added java io iceberg Iceberg IO connector labels Sep 15, 2026
@claudevdm

Copy link
Copy Markdown
Collaborator Author

R: @ahmedabu98

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iceberg Iceberg IO connector io java

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant