NettyNioAsyncHttpClient:NPE in HandlerSubscriber.maybeRequestMore (null subscription) when channelWritabilityChanged fires during the Expect: 100-continue window

[Adiiigo]

Describe the bug

HandlerSubscriber.maybeRequestMore() calls subscription.request(...) with no null check, and channelWritabilityChanged() calls it unconditionally (unlike channelActive(), which is state-gated). subscription is assigned only in onSubscribe() and never reset to null.

For Expect: 100-continue requests, HttpStreamsClientHandler.subscribeSubscriberToStream() deliberately defers the body subscription until the 100 Continue response arrives. The HandlerSubscriber is nonetheless already in the pipeline (added by HttpStreamsHandler.unbufferedWrite()), so it keeps receiving channel events while subscription == null — for a full server round-trip. If channelWritabilityChanged fires in that window, maybeRequestMore() NPEs.

The S3 client sets Expect: 100-continue on every PutObject/UploadPart (StreamingRequestInterceptor), so every async S3 upload takes this path.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The async client should stream the body and complete the upload. During the Expect: 100-continue wait, channelWritabilityChanged must not dereference a not-yet-assigned subscription; it should no-op until onSubscribe runs, then request demand normally.

Current Behavior

Non-retryable failure of the upload:

  Caused by: java.lang.NullPointerException: Cannot invoke "org.reactivestreams.Subscription.request(long)" because "this.subscription" is null
      at ...netty.internal.nrs.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:303)
      at ...netty.internal.nrs.HandlerSubscriber.channelWritabilityChanged(HandlerSubscriber.java:157)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelWritabilityChanged(...)
      ...

Wrapped as SdkClientException: Unable to execute HTTP request: ... "this.subscription" is null and classified non-retryable, so the operation fails.

Reproduction Steps

No standalone repro attached yet. Deterministic conditions: S3AsyncClient PutObject/UploadPart (so Expect: 100-continue is set) against a server that delays the 100 Continue, with a channel writability transition forced during the wait (small write-buffer high-water mark / backpressured body). Happy to contribute a minimal repro.

Possible Solution

Null-guard the deref. (Illustrative — not yet compiled or covered by a test.) When subscription == null the method simply returns; demand is requested when onSubscribe later runs (provideSubscription → maybeStart → maybeRequestMore):

  private void maybeRequestMore() {
      if (subscription != null
              && outstandingDemand <= demandLowWatermark
              && ctx.channel().isWritable()) {
          long toRequest = demandHighWatermark - outstandingDemand;
          outstandingDemand = demandHighWatermark;
          subscription.request(toRequest);
      }
  }
  

Alternatively (or additionally), gate channelWritabilityChanged on state == RUNNING, mirroring channelActive.

Note: S3Configuration.expectContinueEnabled(false) (newer SDK versions) suppresses the header and hides the symptom, but that's a workaround — the unguarded deref in HandlerSubscriber is the actual defect and stays reachable by any path that defers the body subscription past the current event-loop task.

Additional Information/Context

Root cause (line numbers from tag 2.30.31; master has identical channelWritabilityChanged / maybeRequestMore / channelActive bodies)

1. HandlerSubscriber#maybeRequestMore — subscription.request(toRequest) guarded only by isWritable(), no null/state check.
2. HandlerSubscriber#channelWritabilityChanged — calls maybeRequestMore() unguarded (contrast channelActive, which checks state).
3. HttpStreamsHandler#unbufferedWrite — adds the HandlerSubscriber to the pipeline, then calls subscribeSubscriberToStream(...).
4. HttpStreamsClientHandler#subscribeSubscriberToStream — when is100ContinueExpected(msg), stashes the subscriber and does not subscribe; the subscribe (→ onSubscribe, which sets subscription) happens later in channelRead() on the 100 Continue.

Since subscription is set only in onSubscribe and never nulled: subscription == null ⇔ onSubscribe hasn't run ⇔ (for these requests) still awaiting 100 Continue. Any channelWritabilityChanged in that window hits the unguarded deref.

Why this isn't seen more widely

• Only PutObject/UploadPart carry Expect: 100-continue, so only uploads defer the subscription; reads never reach this state. (In our workload, 100% of occurrences are UploadPart, none on reads.)
• It also requires a writability change (channel becoming writable) inside that round-trip window — most plausibly the request-head/TLS write draining back below the low-water mark; we haven't traced the exact per-occurrence trigger. That's rare on an idle connection but frequent under high-concurrency uploads with backpressured sockets, so low-volume callers effectively never hit it.
• The same defect was reported upstream long ago (playframework/netty-reactive-streams#30) and never fixed; the code was vendored here and is still unguarded on master.

AWS Java SDK version used

2.30.31 (also on master); client: netty-nio-client; Netty 4.1.135.Final

JDK version used

21

Operating System and version

Linux

---

Additional context (caller-side use case, domain load, upload code path, observed signature): see comment below (#7271 (comment))

[Adiiigo]

Caller context and observed load

Use case

The affected deployment is AWS OpenSearch Service (managed OpenSearch) with the remote-store feature enabled (S3-backed segment durability). Under sustained high-volume ingest, every data node continuously uploads new segment files to S3 via the async S3 client — the workload that reliably exposes this NPE.

Domain shape and load (incident of manifestation, 24 h window)

Cluster: OpenSearch 3.5, ~177 data nodes (mixed r-family, hot tier only), 3 dedicated cluster-managers, single region. Bundled aws-sdk-java v2.30.31, netty 4.1.135.Final, reactive-streams 1.0.4.

Concurrent write drivers: 4 concurrent OpenSearch Ingestion (OSIS) pipelines feeding the cluster; ~150 primary shards across active indices at peak; indexing rate at incident peak ≈ 4× the 7-day pre-incident baseline.

Indicators: Incident peak that establish the scale of the caller — enough concurrent upload traffic and backpressured sockets to make the writability-flip trigger fire in bulk:

Signal	Peak value	Notes
Data-node CPU	94 – 97 %	Multi-hour saturation.
JVM heap pressure	~88 %	No OOM / no circuit breakers tripped.
IndexingLatency (max)	~70 s	Heavy end-to-end backpressure.
ThreadpoolWriteRejected (cumulative)	~55 M	Write threadpool saturated.
RemoteStorePressureService — bytes_lag vs threshold	~3.5 GB vs 1.12 GB (≈ 3.5×)	Rejecting writes on lag.
RemoteStorePressureService — time_lag vs threshold	~10.8 s vs 7.5 s (≈ 1.4×)	Same.

(These indicators are not the mechanism of the NPE — the NPE is a latent SDK race. They quantify the caller-side load under which the race actually fires.)

Client build

OpenSearch's repository-s3 plugin builds the S3 async client in S3AsyncService.buildAsyncClient(...) using the default NettyNioAsyncHttpClient (the path that carries this defect):

  // OpenSearch: plugins/repository-s3 -> S3AsyncService#buildAsyncClient (simplified)
  NettyNioAsyncHttpClient.Builder http = NettyNioAsyncHttpClient.builder()
      .connectionAcquisitionTimeout(...)
      .maxConcurrency(...)
      .tcpKeepAlive(true)
      .tlsTrustManagersProvider(...);
  
  S3AsyncClient client = S3AsyncClient.builder()
      .httpClient(http.build())
      .region(region)
      .credentialsProvider(...)
      .overrideConfiguration(o -> o.retryPolicy(...).addMetricPublisher(...))
      .build();

An opt-in CRT-client wiring exists (buildAsyncCrtHttpClient) which bypasses this path, but the default (and this deployment) uses netty-nio.

Upload code path (OpenSearch → SDK)

Segment uploads flow through OpenSearch's AsyncTransferManager:

• Single-part: AsyncTransferManager.uploadInOneChunk → s3AsyncClient.putObject(req, body)
• Multipart: AsyncTransferManager.doUploadInParts → AsyncPartsHandler.uploadPart → s3AsyncClient.uploadPart(req, body)

Both build the request body the same way, on a background stream-read executor:

  AsyncRequestBody body = AsyncRequestBody.fromInputStream(
          inputStream,
          inputStreamContainer.getContentLength(),
          streamReadExecutor);
  putObjectRespFuture = s3AsyncClient.putObject(putObjectRequest, body);   // or .uploadPart(...)  

Every one of these calls is a PutObjectRequest / UploadPartRequest, so the S3 client's StreamingRequestInterceptor unconditionally adds Expect: 100-continue on every upload — i.e. every upload takes the deferred-subscription path where the reported NPE lives.

Impact

RetryableStageHelper.retryPolicyDisallowedRetryException classifies the NPE as non-retryable, so each affected PutObject/UploadPart fails on first attempt. OpenSearch surfaces it as a failed segment upload, which feeds directly into the remote-store upload backlog and amplifies the very lag pressure that produced the writability churn — a small feedback loop.