Skip to content

feat(bigquery): add zero-copy queryArrow API for Arrow VectorSchemaRoot streaming - #14375

Open
jinseopkim0 wants to merge 3 commits into
feat-bigquery-arrow-veneerfrom
feat-bigquery-arrow-client
Open

jinseopkim0 wants to merge 3 commits into
feat-bigquery-arrow-veneerfrom
feat-bigquery-arrow-client

Conversation

@jinseopkim0

Copy link
Copy Markdown
Contributor

This PR introduces the queryArrow API on the BigQuery client veneer.

It enables applications to stream query results as Apache Arrow VectorSchemaRoot batches with zero memory copies, supporting fast-path query execution and automatic fallback to BigQuery Storage Read API sessions for larger queries.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for executing queries in Apache Arrow format via the new queryArrow API, utilizing the BigQuery Storage Read API for streaming results. Key changes include adding necessary gRPC dependencies, lazily initializing a shared BigQueryReadClient, and updating QueryRequestInfo to support Arrow serialization options. The reviewer's feedback suggests optimizing the configuration builder to avoid redundant copies, storing the result of getMergedHeaderProvider in a local variable to prevent duplicate calls, and simplifying the fallback path by removing a redundant try-catch block for InterruptedException.

@jinseopkim0

Copy link
Copy Markdown
Contributor Author

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces beta support for executing queries and retrieving results in Apache Arrow format via the new queryArrow API, enabling zero-copy vector access. It integrates the BigQuery Storage Read API by lazily initializing a BigQueryReadClient and configuring transport settings, including fallback mechanisms and fast-path query RPCs. Feedback on the changes suggests extending the plaintext transport channel check to support IPv6 local loopback addresses (::1) to ensure compatibility with local emulators running in IPv6 environments.

@jinseopkim0

Copy link
Copy Markdown
Contributor Author

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for querying BigQuery data in Apache Arrow format via the new queryArrow API. It adds the necessary gRPC and GAX dependencies, defines the queryArrow interface methods in BigQuery, and implements them in BigQueryImpl with support for both a fast path and a fallback path using the BigQuery Storage Read API. Additionally, QueryRequestInfo has been updated to serialize Arrow-related options, and corresponding unit tests have been added. I have no feedback to provide as there are no review comments.

@jinseopkim0
jinseopkim0 force-pushed the feat-bigquery-arrow-client branch from a4cb4bb to ebc1c6c Compare September 14, 2026 21:00
@jinseopkim0
jinseopkim0 marked this pull request as ready for review September 14, 2026 21:10
@jinseopkim0
jinseopkim0 requested review from a team as code owners September 14, 2026 21:10
@jinseopkim0
jinseopkim0 requested a review from lqiu96 September 14, 2026 21:10
@jinseopkim0
jinseopkim0 added this pull request to stack #14379 September 14, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant