GCP: Route GCS batch deletes per credential prefix - #16499
Conversation
fec4f8b to
93daf9f
Compare
|
PR isn't the cause of Kafka Connect CI failure. I've fixed Kafka Connect integration test flakiness in separate PR #16438 |
93daf9f to
99bf949
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
not stale |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code (claude-opus-5)
99bf949 to
bc0f0b5
Compare
Summary
Closes #16480
GCSFileIO.internalDeleteFilespartitionedBlobIds into fixed-size batches and then selected the GCSStorageclient once per batch from only the first object's path. When a singleGCSFileIOis configured with multiple per-prefixStorageCredentials (vended-credentials flow), a batch that crossed prefix boundaries was issued in full through whichever client matched the first object — sending the rest of the batch through the wrong credentials. The fix groupsBlobIds by theirPrefixedStorageclient (via the existing longest-prefix-matchclientForStoragePathhelper) before partitioning into batches, and uses each client's owndeleteBatchSize. No public API or exception contract changes.This mirrors how
S3FileIO.deleteFilesalready groups by bucket before batching.Tests
Two new unit tests in
TestGCSFileIO:deleteFilesRoutesToCorrectClientPerPrefix— interleaves objects across two credential-prefixed buckets and asserts that each per-prefixStorageclient receives only its ownBlobIds.deleteFilesBatchesPerClient— sets a smallgcs.delete.batch-sizeand asserts that batches stay per-client and never mixBlobIds from two prefixes.Confirmed locally with
./gradlew :iceberg-gcp:test --tests "org.apache.iceberg.gcp.gcs.TestGCSFileIO"(26 tests, 0 failures). The new tests also fail when run against the previous code, confirming they catch the bug.AI Disclosure