[Java] Fix BigQuery failed-row timestamp conversion - #40119
Open
goutamadwant wants to merge 1 commit into
Open
goutamadwant wants to merge 1 commit into
goutamadwant wants to merge 1 commit into
Conversation
Format Storage Write API TIMESTAMP output as canonical UTC with microsecond precision. Accept this form in the logical timestamp consumer while preserving numeric and ISO inputs. Add conversion boundaries and failed-row pipeline regressions across batch, streaming and at-least-once writes. Update timestamp fixture expectations and the changelog for apache#40110.
Contributor
|
Assigning reviewers: R: @ahmedabu98 for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40110.
Storage Write API failed rows currently contain TIMESTAMP strings with both a space and a
T, but no time zone. Converting those rows to Beam Rows throws instead of delivering the rejected records.Print standard TIMESTAMP values in BigQuery's canonical UTC format, with at most six fractional digits. Accept that format in the
SqlTypes.TIMESTAMPconsumer while retaining its ISO and integer-microsecond inputs. Protobuf Timestamp values are truncated to microsecond precision before printing. DATETIME's numeric-seconds behavior and TimestampPicos output are unchanged.Replace the malformed writer-test expectations and the fixture normalizer's use of the production formatter with literal expected values.
Testing
The new
StorageApiTimestampWriteTestruns public BigQueryIO pipelines with a fake append service in batch, fixed-shard streaming, and at-least-once modes. It rejects two timestamp-bearing records on successive appends, then accepts the remaining record. Assertions cover failed rows, error messages, successful output, persisted rows, and shrinking append batches. Failed rows are decoded through bothSqlTypes.TIMESTAMPand DATETIME consumers.All three modes fail with
DateTimeParseExceptionon unchanged production code and pass with this fix. The two conversion test classes and the pipeline regression pass together: 93 tests. They also cover integer/protobuf timestamps, sub-microsecond input, pre-epoch and BigQuery range boundaries, nested/repeated/null fields, input compatibility, and TimestampPicos preservation.The four-class rerun below passed all 33 applicable executions of the modified writer methods, including Avro, protobuf, and failed-row cases. Overall: 391 passed, 231 skipped by existing assumptions, and one failure in
testAutoPatchTableSchemaTest[3]. That test also fails in isolation on the pristine base, but with a different symptom: missing nested output fields rather than a missing-required-field retry error. The identical isolated run on patched code passes both applicable cases. These outcomes vary by run/context; the cause remains unresolved, so broader validation is not fully green. Spotless and changelog validation passed.No live BigQuery or Dataflow validation was run.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit BUILD_STATUS.md.
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.