HDDS-16382. Dedicated SCM client RPC timeout and retry for OM request critical path

[ivandika3]

What changes were proposed in this pull request?

We need reduce the following SCM client configuration so that OM read and write critical path is not blocked for too long when SCM is unresponsive

• hdds.scmclient.rpc.timeout
• ipc.client.connect.timeout
• ipc.client.connect.max.retries.on.timeouts

However, OzoneManager instantiates quite a lot of SCM client and each of them share a single configuration although they have different latency and retry requirement

Critical paths (low latency and fail fast)

• scmContainerClient (used in read critical path)
• scmBlockClient (used in write critical path)
• scmTopologyClient is also part of the scmBlockClient

SCM security clients (tolerates high latency and unbounded timeout)

• scmSecurityClient
• secretKeyClient

OM delete client (tolerates high latency and timeout due to the large payload)

• scmBlockClient

Additionally, they are all stored in the same ClientCache entry. ClientCache is a Map<SocketFactory, Client> and the standard implementation SocketFactory returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that has the following hashCode

@Override
public int hashCode() {
  return this.getClass().hashCode();
} 

So this means that even if we use different two configurations, only the first configured client will be stored in the ClientCache and the subsequent one will reuse the cache entry. Therefore, we cannot simply create a separate configuration with different timeout.

The ClientCache#getClient also documented this decision

// Construct & cache client.  The configuration is only used for timeout,
// and Clients have connection pools.  So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations.  Since the IPC is usually intended globally, not
// per-job, we choose (a). 

Since we need two different timeout, it conflicts with the ClientCache mechanism.

Therefore, we need another solution to split the two different clients into two different SocketFactory.

This patch introduces OmScmLocationClientConfig which is a separate configuration for SCM block location (write) and container location (read) clients. This configuration is prefixed by "ozone.om.scmclient.location" so it does not conflict with "hdds.scmclient" used by normal clients. The location protocol will use "ozone.om.scmclient.location" while the other non-critical protocol still uses "hdds.scmclient". The default value has been changed for this configuration is lower, PTAL.

In another note, the Hadoop client design decisions seems to be suited only for user Hadoop client (15 minutes might make sense for a single Hadoop client). However, Ozone decided to reuse the Hadoop client into our critical path without revisiting whether the default timeout makes sense in this context. So in the future, we need to check whether OM should use Hadoop RPC client for its critical path or whether we need to implement a high performance link for OM and SCM.

Code generated by: GPT 5.6 Sol (with some manual updates).

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16382

How was this patch tested?

UT. (Clean CI: https://github.com/ivandika3/ozone/actions/runs/34941367128)