Conversation
Before this change the pre-pass required an existing table. AddFiles without evolution creates a missing table from whichever file happens to reach ConvertToDataFile first; with the pre-pass we can do better, since every schema of the window is known before anything is created. Create path (CommitSchemaUnion.create), taken when loadTable throws NoSuchTableException: - The window's schemas are folded into one union on a scratch create transaction that is never committed (the same staging loop as the evolve path, so a schema conflicting with another is reported or skipped per the handling, and the create is abandoned under FAIL_PIPELINE before anything exists). - The real table is built directly from createdSchema(union, config): every column optional at every level - one lucky file's declared or proven required-ness must not become a table constraint that the next file violates - except pinned paths and their ancestors, which are created required. Creation is the schema-authoring moment; evolution never tightens columns afterwards. A pin that no file schema carries fails the create under FAIL_PIPELINE (later windows could only add the column optional, so the pin would stay inert forever) and warns under ROUTE_TO_ERRORS. The created table is born with a single schema version. - Creation is not gated on any particular evolution option: the options guard an existing table's schema, and there is none to guard yet. - Partition spec, sort order and table properties come from TableCreation (the AddFiles constructor arguments), resolved against the union, so a partition or sort column carried by any schema of the window works; the name mapping is written in the same transaction. - Create race: two workers (or a concurrent non-evolution AddFiles) may create the table at once. AlreadyExistsException joins CommitFailedException in the retry, and the next attempt takes the evolve path against the table the other party created. - Empty window against a missing table returns NO_TABLE (-1) and creates nothing.
Collaborator
Author
|
R: @ahmedabu98 |
Contributor
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change the pre-pass required an existing table. AddFiles without evolution creates a missing table from whichever file happens to reach ConvertToDataFile first; with the pre-pass every schema of the window is known before anything is created.
Create path (CommitSchemaUnion.create), taken when loadTable throws NoSuchTableException:
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.