feat(bigquery): add zero-copy queryArrow API for Arrow VectorSchemaRoot streaming - #14375
jinseopkim0 wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for executing queries in Apache Arrow format via the new queryArrow API, utilizing the BigQuery Storage Read API for streaming results. Key changes include adding necessary gRPC dependencies, lazily initializing a shared BigQueryReadClient, and updating QueryRequestInfo to support Arrow serialization options. The reviewer's feedback suggests optimizing the configuration builder to avoid redundant copies, storing the result of getMergedHeaderProvider in a local variable to prevent duplicate calls, and simplifying the fallback path by removing a redundant try-catch block for InterruptedException.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces beta support for executing queries and retrieving results in Apache Arrow format via the new queryArrow API, enabling zero-copy vector access. It integrates the BigQuery Storage Read API by lazily initializing a BigQueryReadClient and configuring transport settings, including fallback mechanisms and fast-path query RPCs. Feedback on the changes suggests extending the plaintext transport channel check to support IPv6 local loopback addresses (::1) to ensure compatibility with local emulators running in IPv6 environments.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for querying BigQuery data in Apache Arrow format via the new queryArrow API. It adds the necessary gRPC and GAX dependencies, defines the queryArrow interface methods in BigQuery, and implements them in BigQueryImpl with support for both a fast path and a fallback path using the BigQuery Storage Read API. Additionally, QueryRequestInfo has been updated to serialize Arrow-related options, and corresponding unit tests have been added. I have no feedback to provide as there are no review comments.
a4cb4bb to
ebc1c6c
Compare
This PR introduces the
queryArrowAPI on theBigQueryclient veneer.It enables applications to stream query results as Apache Arrow
VectorSchemaRootbatches with zero memory copies, supporting fast-path query execution and automatic fallback to BigQuery Storage Read API sessions for larger queries.