Keep retrying failed replication DLQ writes

[taylan-oai]

Summary

Retain replication tasks when the shared dead-letter queue (DLQ) handler fails, so repeated shard lookup or write failures cannot advance acknowledgement past the task.

Problem

The shared streaming-replication DLQ handler gives a failed task three attempts to enter the DLQ. If all three fail, the next ExecutableTaskImpl.MarkPoisonPill call returns success without writing anything. The task tracker then removes the task and can acknowledge past it. The destination may therefore miss a workflow update with no DLQ entry available to replay.

The limit counts calls, including failures to find the destination shard; three attempts do not necessarily mean three storage writes.

Approach

Remove the attempt cap and its counter. Each call reports the actual shard lookup or DLQ write result, so the tracker retains failed tasks and retries during subsequent stream status updates. A successful write releases the task and allows acknowledgement progress. Existing failure logs and the replication DLQ failure metric continue to expose errors.

Validation

The new regression fails on the original code: both failure cases advance the watermark on the fourth attempt. With the fix, the full replication package passes:

• go test -p 4 -tags test_dep ./service/history/replication -count=1

The regression covers four consecutive DLQ write failures and four consecutive shard lookup failures. Each failure retains the task's acknowledgement boundary, and a later successful write releases it. Changed-package repository lint passes.

Risks, rollout, and scope

A persistent DLQ failure now holds the acknowledgement boundary until it is repaired. Retained tasks still count toward the receiver's existing limit for outstanding tasks. This change uses the current status-update retry cadence and adds no new configuration or storage format. Existing task-specific handling of unknown task types and malformed history payloads is outside this shared-handler fix.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yux0]

This PR makes sense. I continue this change in PR: #12150 to add a break glass config.