HDDS-16389. Add command to display snapshot counts per bucket. - #11217
Draft
sadanand48 wants to merge 2 commits into
Draft
sadanand48 wants to merge 2 commits into
sadanand48 wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved OM aggregation, ACL, filtering, and empty-bucket issues block approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds ozone sh snapshot count to report active, deleted, and total snapshots per bucket.
Changes:
- Adds protobuf, OM, client, and RPC support.
- Implements filtering and ACL-aware aggregation.
- Registers the shell command and adds CLI tests.
File summaries
| File | Summary |
|---|---|
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java |
Updates the protocol test stub. |
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java |
Handles and serializes count requests. |
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java |
Aggregates counts, filtering, and ACL checks. Findings remain: cache-unaware reads (critical, 1 vote); ACL result-code mismatch (moderate, 2 votes); missing empty buckets (moderate, 3 votes); unbounded filtered scans (moderate, 1 vote); invalid extra separators accepted (moderate, 1 vote); insufficient server-side coverage (nit, 1 vote). |
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto |
Defines the new protocol messages and command. |
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotCountResponse.java |
Adds the aggregate response model. |
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotBucketCount.java |
Adds the per-bucket count model. |
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java |
Classifies the operation as read-only. |
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java |
Translates the new RPC. |
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java |
Exposes the protocol method. |
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java |
Forwards the client call. |
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java |
Extends the client contract. |
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java |
Exposes the object-store method. |
hadoop-ozone/cli-shell/src/test/java/org/apache/hadoop/ozone/shell/snapshot/TestCountSnapshotHandler.java |
Tests shell output and filtering. |
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotCommands.java |
Registers the command. |
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/snapshot/CountSnapshotHandler.java |
Implements the shell handler. |
Review details
Suppressed comments (3)
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3285
- For an exact
volume/bucketfilter this still opens an unbounded iterator and examines every snapshot in the cluster before applyingparsedFilter. SnapshotInfo keys are prefixed by/volume/bucket/, so using a prefix iterator/seek for exact filters would avoid making one-bucket queries O(total snapshots) and occupying an OM read thread unnecessarily.
try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
metadataManager.getSnapshotInfoTable().iterator()) {
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3363
indexOf('/')only checks the first separator, so a value such asvol/bucket/extrais accepted and treated as a bucket name containing/; it silently returns zero counts even though the option accepts only<bucket>or<volume>/<bucket>. Reject a second separator before constructing the filter.
int slashIndex = normalized.indexOf('/');
if (slashIndex < 0) {
return new BucketFilter(null, normalized);
}
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3301
- The added tests mock
ObjectStore.snapshotCountand never populatesnapshotInfoTableor invokeOzoneManager.snapshotCount. As a result, the new aggregation, filter parsing, and ACL paths have no server-side coverage, allowing regressions such as the denial-status mismatch and missing empty-bucket result to pass; add an OM/metadata-level test covering active/deleted rows and filtered access.
try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
metadataManager.getSnapshotInfoTable().iterator()) {
while (keyIter.hasNext()) {
SnapshotInfo snapshotInfo = keyIter.next().getValue();
String volumeName = snapshotInfo.getVolumeName();
String bucketName = snapshotInfo.getBucketName();
if (!parsedFilter.matches(volumeName, bucketName)) {
continue;
}
if (getAclsEnabled() && !hasListAccess(volumeName, bucketName, bucketAclCache)) {
continue;
}
String bucketKey = volumeName + "/" + bucketName;
BucketSnapshotCount bucketCount =
bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName));
bucketCount.increment(snapshotInfo.getSnapshotStatus());
- Files reviewed: 15/15 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+3284
to
+3287
| try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter = | ||
| metadataManager.getSnapshotInfoTable().iterator()) { | ||
| while (keyIter.hasNext()) { | ||
| SnapshotInfo snapshotInfo = keyIter.next().getValue(); |
Comment on lines
+3298
to
+3301
| String bucketKey = volumeName + "/" + bucketName; | ||
| BucketSnapshotCount bucketCount = | ||
| bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName)); | ||
| bucketCount.increment(snapshotInfo.getSnapshotStatus()); |
| bucketAclCache.put(bucketKey, true); | ||
| return true; | ||
| } catch (OMException ex) { | ||
| if (ex.getResult() == OMException.ResultCodes.ACCESS_DENIED) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR is to add a command to understand snapshot counts per bucket as currently there is no direct way to fetch this info
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16389
How was this patch tested?
Unit tests