Skip to content

feat(bigquery): add ArrowQueryPageFetcher for Arrow query result pagination - #14378

Open
jinseopkim0 wants to merge 3 commits into
feat-bigquery-arrow-itfrom
feat-bigquery-arrow-page-fetcher
Open

jinseopkim0 wants to merge 3 commits into
feat-bigquery-arrow-itfrom
feat-bigquery-arrow-page-fetcher

Conversation

@jinseopkim0

Copy link
Copy Markdown
Contributor

This PR introduces ArrowQueryPageFetcher to handle pagination for queries executing with QueryResultsFormat.ARROW.

Summary of Changes

  • Implements ArrowQueryPageFetcher implementing NextPageFetcher<FieldValueList>.
  • Connects to the default storage read stream to fetch subsequent row pages.
  • Leverages ArrowDeserializer.loadArrowRows to deserialize Arrow record batches into FieldValueList collections with offset and maxResults bounding.
  • Adds ArrowQueryPageFetcherTest covering single-page, multi-page, max-results, and serialization behaviors.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces ArrowQueryPageFetcher, a NextPageFetcher implementation designed to paginate query results returned in Arrow format by reading from a gRPC storage read stream. It also adds comprehensive unit tests in ArrowQueryPageFetcherTest.java. The review feedback highlights serialization issues in ArrowQueryPageFetcher: the buffer queue should be marked as transient to prevent duplicate data upon deserialization, and it must be lazily initialized in getNextPage() to avoid a NullPointerException after deserialization.

@jinseopkim0

Copy link
Copy Markdown
Contributor Author

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the ArrowQueryPageFetcher class in BigQueryImpl.java to support paginating query results returned in Arrow format via the gRPC storage read stream. It also adds a comprehensive suite of unit tests in ArrowQueryPageFetcherTest.java to verify its behavior, including pagination, limit handling, error cases, and serialization. There are no review comments, so no feedback is provided.

@jinseopkim0

Copy link
Copy Markdown
Contributor Author

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the ArrowQueryPageFetcher class in BigQueryImpl.java to handle pagination for query results returned in Arrow format via the gRPC storage read stream. It also updates the read settings configuration to use NoHeaderProvider as a default, and adds a comprehensive suite of unit tests in ArrowQueryPageFetcherTest.java to verify single and multi-page fetching, max results handling, error conditions, and serialization. There are no review comments, and I have no feedback to provide.

@jinseopkim0
jinseopkim0 marked this pull request as ready for review September 14, 2026 22:42
@jinseopkim0
jinseopkim0 requested review from a team as code owners September 14, 2026 22:42
@jinseopkim0
jinseopkim0 requested a review from lqiu96 September 14, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant