Skip to content

GCP: Route GCS batch deletes per credential prefix - #16499

Open
wombatu-kun wants to merge 1 commit into
apache:mainfrom
wombatu-kun:issue/16480-gcs-mixed-prefix-batch-deletes
Open

GCP: Route GCS batch deletes per credential prefix#16499
wombatu-kun wants to merge 1 commit into
apache:mainfrom
wombatu-kun:issue/16480-gcs-mixed-prefix-batch-deletes

Conversation

@wombatu-kun

@wombatu-kun wombatu-kun commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #16480

GCSFileIO.internalDeleteFiles partitioned BlobIds into fixed-size batches and then selected the GCS Storage client once per batch from only the first object's path. When a single GCSFileIO is configured with multiple per-prefix StorageCredentials (vended-credentials flow), a batch that crossed prefix boundaries was issued in full through whichever client matched the first object — sending the rest of the batch through the wrong credentials. The fix groups BlobIds by their PrefixedStorage client (via the existing longest-prefix-match clientForStoragePath helper) before partitioning into batches, and uses each client's own deleteBatchSize. No public API or exception contract changes.

This mirrors how S3FileIO.deleteFiles already groups by bucket before batching.

Tests

Two new unit tests in TestGCSFileIO:

  • deleteFilesRoutesToCorrectClientPerPrefix — interleaves objects across two credential-prefixed buckets and asserts that each per-prefix Storage client receives only its own BlobIds.
  • deleteFilesBatchesPerClient — sets a small gcs.delete.batch-size and asserts that batches stay per-client and never mix BlobIds from two prefixes.

Confirmed locally with ./gradlew :iceberg-gcp:test --tests "org.apache.iceberg.gcp.gcs.TestGCSFileIO" (26 tests, 0 failures). The new tests also fail when run against the previous code, confirming they catch the bug.


AI Disclosure

  • Model: Claude Opus 4.7
  • Platform/Tool: Claude Code
  • Human Oversight: fully reviewed
  • Prompt Summary: Select the GCS storage client per credential prefix when batching deletes.

@github-actions github-actions Bot added the GCP label May 21, 2026
@wombatu-kun
wombatu-kun force-pushed the issue/16480-gcs-mixed-prefix-batch-deletes branch from fec4f8b to 93daf9f Compare May 21, 2026 03:58
@wombatu-kun

Copy link
Copy Markdown
Contributor Author

PR isn't the cause of Kafka Connect CI failure. I've fixed Kafka Connect integration test flakiness in separate PR #16438

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jun 22, 2026
@wombatu-kun

Copy link
Copy Markdown
Contributor Author

not stale

@github-actions github-actions Bot removed the stale label Jun 23, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 10, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generated-by: Claude Code (claude-opus-5)
@wombatu-kun
wombatu-kun force-pushed the issue/16480-gcs-mixed-prefix-batch-deletes branch from 99bf949 to bc0f0b5 Compare August 10, 2026 14:59
@github-actions github-actions Bot removed the stale label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mixed-prefix GCS batch deletes reuse the first object's client for the whole batch

1 participant