Skip to content

HDDS-16389. Add command to display snapshot counts per bucket. - #11217

Draft
sadanand48 wants to merge 2 commits into
apache:masterfrom
sadanand48:HDDS-16389
Draft

sadanand48 wants to merge 2 commits into
apache:masterfrom
sadanand48:HDDS-16389

Conversation

@sadanand48

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR is to add a command to understand snapshot counts per bucket as currently there is no direct way to fetch this info

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16389

How was this patch tested?

Unit tests

@sadanand48 sadanand48 added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Sep 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved OM aggregation, ACL, filtering, and empty-bucket issues block approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds ozone sh snapshot count to report active, deleted, and total snapshots per bucket.

Changes:

  • Adds protobuf, OM, client, and RPC support.
  • Implements filtering and ACL-aware aggregation.
  • Registers the shell command and adds CLI tests.
File summaries
File Summary
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java Updates the protocol test stub.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java Handles and serializes count requests.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java Aggregates counts, filtering, and ACL checks. Findings remain: cache-unaware reads (critical, 1 vote); ACL result-code mismatch (moderate, 2 votes); missing empty buckets (moderate, 3 votes); unbounded filtered scans (moderate, 1 vote); invalid extra separators accepted (moderate, 1 vote); insufficient server-side coverage (nit, 1 vote).
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto Defines the new protocol messages and command.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotCountResponse.java Adds the aggregate response model.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotBucketCount.java Adds the per-bucket count model.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java Classifies the operation as read-only.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java Translates the new RPC.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java Exposes the protocol method.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java Forwards the client call.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java Extends the client contract.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java Exposes the object-store method.
hadoop-ozone/cli-shell/src/test/java/org/apache/hadoop/ozone/shell/snapshot/TestCountSnapshotHandler.java Tests shell output and filtering.
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotCommands.java Registers the command.
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/snapshot/CountSnapshotHandler.java Implements the shell handler.
Review details

Suppressed comments (3)

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3285

  • For an exact volume/bucket filter this still opens an unbounded iterator and examines every snapshot in the cluster before applying parsedFilter. SnapshotInfo keys are prefixed by /volume/bucket/, so using a prefix iterator/seek for exact filters would avoid making one-bucket queries O(total snapshots) and occupying an OM read thread unnecessarily.
      try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
               metadataManager.getSnapshotInfoTable().iterator()) {

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3363

  • indexOf('/') only checks the first separator, so a value such as vol/bucket/extra is accepted and treated as a bucket name containing /; it silently returns zero counts even though the option accepts only <bucket> or <volume>/<bucket>. Reject a second separator before constructing the filter.
      int slashIndex = normalized.indexOf('/');
      if (slashIndex < 0) {
        return new BucketFilter(null, normalized);
      }

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:3301

  • The added tests mock ObjectStore.snapshotCount and never populate snapshotInfoTable or invoke OzoneManager.snapshotCount. As a result, the new aggregation, filter parsing, and ACL paths have no server-side coverage, allowing regressions such as the denial-status mismatch and missing empty-bucket result to pass; add an OM/metadata-level test covering active/deleted rows and filtered access.
      try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
               metadataManager.getSnapshotInfoTable().iterator()) {
        while (keyIter.hasNext()) {
          SnapshotInfo snapshotInfo = keyIter.next().getValue();
          String volumeName = snapshotInfo.getVolumeName();
          String bucketName = snapshotInfo.getBucketName();
          if (!parsedFilter.matches(volumeName, bucketName)) {
            continue;
          }

          if (getAclsEnabled() && !hasListAccess(volumeName, bucketName, bucketAclCache)) {
            continue;
          }

          String bucketKey = volumeName + "/" + bucketName;
          BucketSnapshotCount bucketCount =
              bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName));
          bucketCount.increment(snapshotInfo.getSnapshotStatus());
  • Files reviewed: 15/15 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +3284 to +3287
try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
metadataManager.getSnapshotInfoTable().iterator()) {
while (keyIter.hasNext()) {
SnapshotInfo snapshotInfo = keyIter.next().getValue();
Comment on lines +3298 to +3301
String bucketKey = volumeName + "/" + bucketName;
BucketSnapshotCount bucketCount =
bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName));
bucketCount.increment(snapshotInfo.getSnapshotStatus());
bucketAclCache.put(bucketKey, true);
return true;
} catch (OMException ex) {
if (ex.getResult() == OMException.ResultCodes.ACCESS_DENIED) {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants