feat(bigquery): add ArrowQueryResult and ArrowQueryResultImpl for Arrow result streaming by jinseopkim0 · Pull Request #13944 · googleapis/google-cloud-java

jinseopkim0 · 2026-07-29T14:34:06Z

This PR introduces the ArrowQueryResult interface and ArrowQueryResultImpl class for streaming and paginating query results backed by Apache Arrow record batches.

It provides zero-copy VectorSchemaRoot streaming and implements the standard TableResult iteration contract over Arrow batches.

gemini-code-assist

Code Review

This pull request introduces support for Arrow-formatted query results in the BigQuery client, adding ArrowQueryPageFetcher to fetch Arrow pages and updating response parsing to handle Arrow schemas and record batches. The review feedback highlights critical issues that need to be addressed: a potential NotSerializableException in ArrowQueryPageFetcher due to a non-serializable schema field, memory leaks from unclosed Arrow vectors and record batches, and overly restrictive type checks (instanceof List instead of instanceof Collection) when determining page row counts.

jinseopkim0 · 2026-08-10T16:07:43Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for the Apache Arrow query results format in the BigQuery client. It refactors Arrow schema conversion and vector creation logic into a new helper class ArrowPojoUtils, adds an ArrowQueryPageFetcher to stream Arrow-formatted query results, and updates BigQueryImpl and QueryRequestInfo to handle Arrow serialization options and deserialize Arrow record batches. The review feedback highlights several improvement opportunities: preventing a potential resource leak in ArrowPojoUtils.createVectors by closing already allocated vectors if an exception is thrown, avoiding a potential NullPointerException in BigQueryImpl by using serviceOptions.getCredentialsProvider(), and extracting duplicate logic for computing firstPageRows to improve maintainability.

jinseopkim0 · 2026-08-10T16:24:05Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for the Apache Arrow query results format in the BigQuery client. It extracts Arrow conversion utilities into a new ArrowPojoUtils helper class, implements ArrowQueryPageFetcher to stream results via the BigQuery Storage Read API, and updates query execution paths to handle Arrow schemas and record batches. The review feedback focuses on critical resource management and performance optimizations, specifically ensuring exception-safe LIFO cleanup of Arrow vectors, allocators, and schema roots to prevent memory leaks, reusing allocator and root instances across page fetches, enforcing the maxResults limit, and properly handling credentials and universe domains.

jinseopkim0 · 2026-08-12T14:25:06Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for retrieving query results in Apache Arrow format using the fast query path and the BigQuery Storage API. It adds a new ArrowQueryPageFetcher to handle paginated Arrow record batches, introduces ArrowPojoUtils for Arrow schema and vector utilities, and updates BigQueryImpl to deserialize Arrow schemas and record batches. The review feedback highlights a critical resource leak in ArrowQueryPageFetcher due to its stateful nature across pages, suggesting a stateless implementation using try-with-resources to ensure gRPC streams, clients, and off-heap memory allocators are safely closed.

jinseopkim0 · 2026-08-12T14:50:33Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for the Arrow query results format in BigQuery, extracting Arrow conversion utilities into ArrowPojoUtils and implementing ArrowQueryPageFetcher in BigQueryImpl to read Arrow rows via the Storage Read API. The reviewer identifies a critical issue where direct imports and references to Apache Arrow classes in BigQueryImpl.java will cause NoClassDefFoundError at runtime for users who do not have Arrow on their classpath. To resolve this, the reviewer recommends encapsulating all Arrow-specific logic and deserialization within ArrowDeserializer and refactoring BigQueryImpl to avoid direct bytecode dependencies on optional Arrow classes.

jinseopkim0 · 2026-08-12T15:15:37Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for the Apache Arrow query results format in the BigQuery client, enabling faster query path execution. Key changes include the addition of ArrowPojoUtils for schema conversions, updates to ArrowDeserializer to handle row loading and deserialization, and the implementation of ArrowQueryPageFetcher in BigQueryImpl to fetch pages of Arrow-formatted results. The review feedback highlights three improvement opportunities: caching the BigQueryReadClient and stream iterator in transient fields within ArrowQueryPageFetcher to prevent significant gRPC overhead on pagination, removing an unused BufferAllocator in ArrowDeserializer.deserializeSchema, and restructuring vector resource management in loadArrowRows to avoid a potential double-closing issue.

jinseopkim0 · 2026-08-12T15:25:29Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request introduces support for retrieving query results in Apache Arrow format from BigQuery. It adds the ArrowDeserializer and ArrowPojoUtils helper classes to convert Arrow schemas and deserialize record batches, implements ArrowQueryPageFetcher to fetch pages of Arrow rows, and integrates these changes into BigQueryImpl and QueryRequestInfo. Feedback on these changes highlights several critical issues: potential resource leaks if the stream iterator is not fully exhausted, a risk of generating invalid stream names if the job location is null, high memory consumption due to a hardcoded page size of 100,000 rows, performance overhead from repeatedly instantiating RootAllocator, and reduced type safety from using Object instead of concrete Arrow types in method signatures.

…seable from BigQuery

… for PR 1

…QueryResultImpl

…pom.xml

…yResultImpl

…eadSession

…erator

lqiu96 · 2026-09-16T02:21:15Z

+ * underlying gRPC streaming channels are deterministically released.
+ */
+@BetaApi
+public interface ArrowQueryResult extends AutoCloseable, Iterable<VectorSchemaRoot> {


nit: perhaps we should add the javadoc that this represents the canonical BigQuery TableResult

Thanks for the comment, updated.

lqiu96 · 2026-09-16T02:24:31Z

+      if (!yieldedInitialBatch
+          && initialRecordBatchBytes != null
+          && initialRecordBatchBytes.length > 0) {


nit: Can we move this to a helper function explain the logic in here. I also see this being used elsewhere as well

Thanks for the comment, done.

lqiu96 · 2026-09-16T02:26:57Z

+      } catch (NoSuchElementException | BigQueryException e) {
+        throw e;


qq, can you explain why these two exceptions specifically?

Thanks for the question. These two exceptions are rethrown directly to avoid improper wrapping:

NoSuchElementException: Required by the standard Java Iterator.next() contract when no more elements remain. Rethrowing directly prevents it from falling through to catch (Exception e) and getting wrapped into a BigQueryException.

BigQueryException: Avoids redundant double-wrapping (BigQueryException wrapping another BigQueryException), preserving the original error code, message, and root cause.

Any other unexpected exception caught in the subsequent catch (Exception e) block is wrapped in a BigQueryException to ensure uniform error handling in the veneer.

I added a comment to clarify.

lqiu96 · 2026-09-16T02:28:12Z

+        if (isClosed()) {
+          throw new NoSuchElementException("Query stream was closed.");
+        }


qq, why we do need to throw an exception here?

Thanks for the question. When close() is called, it cancels the active gRPC stream. Catching this and checking if (isClosed()) ensures we throw NoSuchElementException (fulfilling the Java Iterator contract) rather than misreporting an intentional cancellation as an unexpected network failure (BigQueryException).

lqiu96 · 2026-09-16T02:30:41Z

@@ -0,0 +1 @@
+mock-maker-inline


qq, what is this needed for?

This enables Mockito's inline mock maker, which allows mocking final classes (such as the auto-generated BigQueryReadClient and GAX ServerStream) in unit tests (ArrowQueryResultTest) without having to spin up an in-memory gRPC server.

lqiu96 · 2026-09-16T02:31:35Z

+      loadBatch(new ByteArrayReadableSeekableByteChannel(bytes));
+    }
+
+    /**
+     * Deserializes an Arrow record batch from a protobuf {@link ByteString} and loads it into the
+     * root vector.
+     *
+     * @param byteString serialized Arrow record batch bytes as a {@link ByteString}
+     * @throws IOException if deserialization fails
+     */
+    private void loadBatch(ByteString byteString) throws IOException {
+      loadBatch(Channels.newChannel(byteString.newInput()));


do the channels created in here need to be closed?

Thanks for the feedback. I've updated the code accordingly.

lqiu96 · 2026-09-16T02:36:57Z

+  static ArrowQueryResultImpl fromReadSession(
+      ReadSession readSession, JobId jobId, BigQueryReadClient readClient) {
+    Schema pojoSchema = null;
+    if (readSession.hasArrowSchema()) {


qq, what happens if there is no arrow schema?

If readSession has no Arrow schema:

If streams are present: The constructor throws an IllegalArgumentException ("Arrow schema cannot be null when query results or streams are present.") since records cannot be deserialized without a schema.

If no streams are present (e.g. 0 rows / empty result): arrowSchema defaults to an empty Schema (new Schema(Collections.emptyList())), maintaining non-null internal state and yielding an empty iterator (0 batches).

lqiu96

added some small comments.

…cleanup

gemini-code-assist Bot reviewed Jul 29, 2026

View reviewed changes

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch 3 times, most recently from 2f4435e to 76600f0 Compare August 7, 2026 19:40

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch from 76600f0 to ec26676 Compare August 7, 2026 19:51

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch from ec26676 to 68798eb Compare August 7, 2026 20:13

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch 2 times, most recently from bb63672 to 88530f7 Compare August 7, 2026 20:32

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch from 88530f7 to 80ae0af Compare August 10, 2026 15:25

gemini-code-assist Bot reviewed Aug 10, 2026

View reviewed changes

gemini-code-assist Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java Outdated

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch from 8f1d641 to 101d2d0 Compare August 12, 2026 14:47

gemini-code-assist Bot reviewed Aug 12, 2026

View reviewed changes

jinseopkim0 added 6 commits September 14, 2026 16:53

fix(bigquery): preserve existing Service contract by removing AutoClo…

6d600ef

…seable from BigQuery

refactor(bigquery): isolate Arrow result model and streaming iterator…

c857596

… for PR 1

docs(bigquery): add Javadoc comments to non-override methods in Arrow…

dc02899

…QueryResultImpl

refactor(bigquery): deduplicate loadBatch in ArrowQueryResultImpl

3065daf

perf(bigquery): ensure zero-copy batch loading via ReadableByteChannel

3942577

fix(bigquery): remove unused gax-grpc and grpc-api dependencies from …

1b283db

…pom.xml

jinseopkim0 force-pushed the feat-bigquery-arrow-veneer branch from 10b3e9c to 1b283db Compare September 14, 2026 20:53

jinseopkim0 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 14, 2026

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 14, 2026

lqiu96 reviewed Sep 15, 2026

View reviewed changes

Comment thread ...bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResult.java

lqiu96 reviewed Sep 15, 2026

View reviewed changes

Comment thread ...uery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResultImpl.java Outdated

lqiu96 reviewed Sep 15, 2026

View reviewed changes

Comment thread ...uery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResultImpl.java Outdated

lqiu96 reviewed Sep 15, 2026

View reviewed changes

Comment thread ...uery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResultImpl.java

jinseopkim0 added 3 commits September 15, 2026 16:19

fix(bigquery): use empty Schema when arrowSchema is null in ArrowQuer…

6183bb7

…yResultImpl

fix: validate single-stream ReadSession in ArrowQueryResultImpl.fromR…

2226915

…eadSession

refactor: remove unnecessary locks from single-threaded VectorBatchIt…

e6eb762

…erator

jinseopkim0 requested a review from lqiu96 September 15, 2026 20:50

lqiu96 reviewed Sep 16, 2026

View reviewed changes

Comment thread ...uery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResultImpl.java

lqiu96 reviewed Sep 16, 2026

View reviewed changes

lqiu96 approved these changes Sep 16, 2026

View reviewed changes

chore: address review feedback on ArrowQueryResult documentation and …

11b22e2

…cleanup

jinseopkim0 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 16, 2026

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 16, 2026

jinseopkim0 merged commit a62fdf8 into main Sep 16, 2026
209 of 210 checks passed

		} catch (NoSuchElementException \| BigQueryException e) {
		throw e;

		@@ -0,0 +1 @@
		mock-maker-inline

Uh oh!

Conversation

jinseopkim0 commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jinseopkim0 commented Aug 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jinseopkim0 commented Aug 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jinseopkim0 commented Aug 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

jinseopkim0 commented Aug 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jinseopkim0 commented Aug 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jinseopkim0 commented Aug 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jinseopkim0 commented Jul 29, 2026 •

edited

Loading