Skip to content

[Java] Fix BigQuery failed-row timestamp conversion - #40119

Open
goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:fix-bigquery-failed-row-timestamps
Open

goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:fix-bigquery-failed-row-timestamps

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Fixes #40110.

Storage Write API failed rows currently contain TIMESTAMP strings with both a space and a T, but no time zone. Converting those rows to Beam Rows throws instead of delivering the rejected records.

Print standard TIMESTAMP values in BigQuery's canonical UTC format, with at most six fractional digits. Accept that format in the SqlTypes.TIMESTAMP consumer while retaining its ISO and integer-microsecond inputs. Protobuf Timestamp values are truncated to microsecond precision before printing. DATETIME's numeric-seconds behavior and TimestampPicos output are unchanged.

Replace the malformed writer-test expectations and the fixture normalizer's use of the production formatter with literal expected values.

Testing

The new StorageApiTimestampWriteTest runs public BigQueryIO pipelines with a fake append service in batch, fixed-shard streaming, and at-least-once modes. It rejects two timestamp-bearing records on successive appends, then accepts the remaining record. Assertions cover failed rows, error messages, successful output, persisted rows, and shrinking append batches. Failed rows are decoded through both SqlTypes.TIMESTAMP and DATETIME consumers.

All three modes fail with DateTimeParseException on unchanged production code and pass with this fix. The two conversion test classes and the pipeline regression pass together: 93 tests. They also cover integer/protobuf timestamps, sub-microsecond input, pre-epoch and BigQuery range boundaries, nested/repeated/null fields, input compatibility, and TimestampPicos preservation.

The four-class rerun below passed all 33 applicable executions of the modified writer methods, including Avro, protobuf, and failed-row cases. Overall: 391 passed, 231 skipped by existing assumptions, and one failure in testAutoPatchTableSchemaTest[3]. That test also fails in isolation on the pristine base, but with a different symptom: missing nested output fields rather than a missing-required-field retry error. The identical isolated run on patched code passes both applicable cases. These outcomes vary by run/context; the cause remains unresolved, so broader validation is not fully green. Spotless and changelog validation passed.

./gradlew :sdks:java:io:google-cloud-platform:test \
  --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest \
  --tests org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProtoTest \
  --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilsTest \
  --tests org.apache.beam.sdk.io.gcp.bigquery.StorageApiTimestampWriteTest \
  :sdks:java:io:google-cloud-platform:spotlessJavaCheck validateChanges \
  --continue --no-build-cache --no-parallel --max-workers=1

No live BigQuery or Dataflow validation was run.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit BUILD_STATUS.md.

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Format Storage Write API TIMESTAMP output as canonical UTC with microsecond precision. Accept this form in the logical timestamp consumer while preserving numeric and ISO inputs.

Add conversion boundaries and failed-row pipeline regressions across batch, streaming and at-least-once writes. Update timestamp fixture expectations and the changelog for apache#40110.
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: BigQueryIO Storage Write API failed-rows output crashes: TIMESTAMP printed with both a space and a T separator, then parsed by Instant.parse

1 participant