impl(bigquery): multiplexing option on writer builder - #6811
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces multiplexing support for the BigQuery Arrow writer by updating WriterBuilder to accept a shared StreamPool and adding a with_multiplexing option. The Write client now manages and passes this shared pool to the builder. Feedback suggests introducing a helper function in the test module to eliminate repetitive boilerplate code when creating WriterBuilder instances across various test cases.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6811 +/- ##
=======================================
Coverage 97.02% 97.02%
=======================================
Files 326 326
Lines 107548 107585 +37
=======================================
+ Hits 104350 104388 +38
+ Misses 3198 3197 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dbolduc
marked this pull request as ready for review
September 15, 2026 05:47
alvarowolfx
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a fluent setter on the writer builder to enable multiplexing and plumbs the options from the client builder into the client into the writer builder into the writer.
All of this is
pub(crate)for now. The next PR will make thempub.Note: we will need separate connection pools for arrow vs. proto transports, so I am only implementing arrow right now.
Note: we discussed changing the signature of
.arrow().default()in the API review. I am still going to look into that, but after I get this feature done. (There is one PR after this).Towards #6765