Skip to content

HDDS-16419. Avoid nested datanode failover in GetBlock and ReadChunk - #11239

Open
symious wants to merge 1 commit into
apache:masterfrom
symious:HDDS-16419
Open

symious wants to merge 1 commit into
apache:masterfrom
symious:HDDS-16419

Conversation

@symious

@symious symious commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

ContainerProtocolCalls#getBlock and #readChunk fail over between datanodes
via tryEachDatanode. For each datanode, they call
XceiverClientSpi#sendCommand(request, validators). In XceiverClientGrpc this
goes to sendCommandWithRetry, which again tries every datanode of the
pipeline. As a result:

  • every datanode is contacted N times, so a single call can take
    N × N × ozone.client.read.timeout before failing, and a read can take
    read.max.retries × N × N × read.timeout;
  • the request built for datanode A (datanodeUuid, EC replica index) may be
    sent to datanode B.

This PR makes tryEachDatanode the only failover layer for these two commands:

  • XceiverClientSpi
    • Add sendCommand(request, validators, datanode), which sends the command
      to the given datanode only. The default implementation falls back to
      sendCommand(request, validators), so Ratis and short-circuit clients are
      unchanged.
    • Add getDatanodesInOrder(blockID, cmdType). The default implementation
      returns the datanodes in Pipeline#getClosestNode order, which is the
      current behavior of tryEachDatanode.
  • XceiverClientGrpc
    • Extract the single-datanode part of sendCommandWithRetry into
      sendCommandToDatanode: send, run validators, update the GetBlock
      datanode cache. Both the existing retry loop and the new
      sendCommand(request, validators, datanode) use it.
    • getDatanodesInOrder reuses sortDatanodes, so the datanode order
      (leader / cached GetBlock datanode first, non-IN_SERVICE last,
      topology-aware or shuffled) is the same as before.
  • ContainerProtocolCalls
    • tryEachDatanode iterates over getDatanodesInOrder.
    • The per-datanode GetBlock/ReadChunk calls use the new single-datanode
      sendCommand.

Other commands (listBlock, readSmallFile, ...) still use the inner failover
and are not affected.

After this change, the worst case for a read is
read.max.retries × N × read.timeout.

Note: getBlockFromDatanode (used by ozone debug replicas chunk-info) now
really queries only the given datanode. Before this change it could silently
return the result from another datanode.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16419

How was this patch tested?

  • TestXceiverClientGrpc
    • testGetBlockRetryAlNodes / testReadChunkRetryAllNodes now assert that
      each datanode is contacted exactly once (previously N² calls), and that
      the request's datanodeUuid matches the datanode it is sent to.
    • New testReadChunkRetryNextNodeOnShortRead: a short read from the first
      datanode is retried on the next datanode.
    • Existing ordering tests (leader, topology, IN_SERVICE first, connection
      reuse after GetBlock) pass unchanged.
  • TestChunkInputStream, TestBlockInputStream pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@yandrey321

Copy link
Copy Markdown
Contributor

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants