Release abandoned orchestrator tasks when an executor is retired

[wangbill (YunchuWang)]

Release abandoned orchestrator tasks when an executor is retired

Fixes the long-standing "memory leak when debugger attached" reported in
Azure/azure-functions-durable-extension#340. The issue is filed on the
extension repo but is labeled dtfx/external because the root cause is here, in DurableTask.Core.

Root cause

TaskOrchestrationContext creates a TaskCompletionSource<string> for every activity,
sub-orchestration and timer, registers it in openTasks, and the orchestrator awaits tcs.Task.
When an episode ends while the orchestrator is parked on one of those awaits, the TCS is deliberately
abandoned in a pending state. That is inherent to the replay model and is normally harmless.

It stops being harmless when a debugger is attached:

1. TaskAwaiter.OnCompletedInternal calls OutputWaitEtwEvents because Task.s_asyncDebuggingEnabled
   is true.
2. That calls Task.AddToActiveTasks(task), which inserts the task into the process-wide,
   strongly referenced static dictionary Task.s_currentActiveTasks.
3. The matching Task.RemoveFromActiveTasks only runs inside the awaiter's continuation wrapper — that
   is, only when the awaited task completes.

An abandoned task never completes, so it is rooted forever. Each rooted Task pins its continuation →
async state machine → TaskOrchestrationContext → history, inputs and outputs. Notably only the
AddToActiveTasks branch roots anything (the ETW branch does not), which is exactly why the leak is
debugger-specific.

The fix

Cancel the open tasks once the executor is guaranteed never to run again, so the abandoned awaiter
continuations actually run and unregister themselves.

TaskOrchestrationContext gains internal void ReleaseOpenTasks(), which cancels every open task.
It is idempotent, sets isReleased = true before cancelling, and snapshots openTasks.Values before
iterating — resumed user code can mutate the dictionary, e.g. CreateTimer's cancellation-token
callback calls openTasks.Remove. A new ThrowIfReleased() guard at the top of ScheduleTaskInternal,
CreateSubOrchestrationInstanceCore and CreateTimer stops orchestrator code that swallows the
cancellation from scheduling new work that would leak the same way.

TaskOrchestrationExecutor implements IDisposable. Dispose() restores the orchestrator ambient
environment (TaskOrchestrationSynchronizationContext + OrchestrationContext.IsOrchestratorThread)
because cancelling resumes orchestrator code synchronously, calls context.ReleaseOpenTasks(), and
observes this.result.Exception if faulted so nothing surfaces as an UnobservedTaskException.

TaskOrchestrationDispatcher owns executor lifetime via workItem.Cursor, so it is the only
component that knows when an executor will never run again. A new
static void ReleaseCursor(ref OrchestrationExecutionCursor? cursor) is called from a new finally on
the outer try of OnProcessWorkItemSessionAsync — this covers both the legacy Session == null path
and the session path, since workItem.Cursor is only ever assigned in OnProcessWorkItemAsync, which
both paths call. It also replaces the bare workItem.Cursor = null; at the continue-as-new site.

Results

Direct-executor harness, 2000 episodes × fan-out 100, debugger simulated by setting
Task.s_asyncDebuggingEnabled:

	Retained after full blocking GC	Entries leaked into Task.s_currentActiveTasks
Before	400.76 MB	1,006,000 (~210 KB/episode)
After	0.01 MB	0

With no debugger both are 0, confirming the debugger-specific mechanism.

End to end through the real TaskHubWorker/TaskOrchestrationDispatcher with
LocalOrchestrationService, 200 instances × 3 activities:

	Leaked orchestrator tasks
Before	4,200
After	0

The 4,200 break down as 600 each of TaskOrchestration`4.<Execute>, the user orchestrator's
<RunTask>, ScheduleTask, both ScheduleTaskToWorker overloads, ScheduleTaskInternal, plus ~600 TCS
tasks — exactly 3 abandoned awaits per instance. (A residual ~1,200 Task.Delay promises appear
identically in both runs; that is the emulator's own queue polling and is unrelated.)

Two things reviewers will care about

1. TaskOrchestrationExecutor is public and now implements IDisposable. This is both source and
binary compatible — no existing member changed and no caller is required to do anything. Be aware that
it may newly trip CA2000 ("dispose objects before losing scope") in downstream repos that construct
a TaskOrchestrationExecutor and have that analyzer enabled.

2. Extended sessions are explicitly unaffected. Release only happens when the executor is retired,
never between episodes. This is deliberate: cancelling at the end of every episode inside ExecuteCore
would break extended sessions, because HandleTaskCompletedEvent calls info.Result.SetResult(...),
which throws InvalidOperationException on an already-cancelled TCS. There is a regression test named
ExtendedSession_OpenTasksSurviveBetweenEpisodes proving open tasks still survive between episodes.

Alternatives considered and rejected

• Cancel at the end of every episode — breaks extended sessions, as above.
• A custom awaitable that avoids registration — only fixes the innermost await; user-code awaits
  (Task.WhenAll, awaiting the returned Task<T>) still register.
• Reflectively removing entries from s_currentActiveTasks — uses private API and cannot reach the
  intermediate async-method tasks.

A note on the leak-detection test

Dispose_WithAsyncDebuggingEnabled_DoesNotLeakActiveTasks deliberately measures how
Task.s_currentActiveTasks growth scales with fan-out (1 vs 25 abandoned awaits over 20 episodes)
rather than asserting an absolute count. Two reasons: the dictionary is process-wide, so the test host's
own async plumbing pollutes it; and on .NET Framework an async method's task is not an
AsyncStateMachineBox<T>, so entries cannot be attributed by type there. The test is self-validating —
it first asserts that undisposed executors DO leak proportionally, so it can never silently stop
detecting the regression.

Testing

All green in Debug (the test projects only build in Debug: tools/DurableTask.props sets
SIGN_ASSEMBLY in Release, and AssemblyInfo.cs wraps InternalsVisibleTo in #if !SIGN_ASSEMBLY).

Suite	net8.0	net48
DurableTask.Core.Tests	145/145 ✅	126 passed, 7 failed ⚠️
DurableTask.Emulator.Tests	5/5 ✅	5/5 ✅
New TaskOrchestrationExecutorTests	6/6 ✅	6/6 ✅

The 7 net48 failures are all in ContinueAsNewTraceBehaviorTests, a file this PR does not touch. I
verified they are pre-existing on main by stashing this change and re-running that class on net48 —
the same 7 fail identically at baseline.

Full solution build: 0 warnings, 0 errors.

Known follow-up (not fixed here)

azure-functions-durable-extension has its own secondary TCS leaks of the same shape that this change
does not cover, because an orchestrator parked purely on an external event has no DTFx open tasks:

• DurableOrchestrationContext.pendingExternalEvents (EventTaskCompletionSource<T>)
• TaskCommonShim.timeoutTaskCompletionSource

Those need an equivalent release hook in the extension repo.

[Copilot]

Pull request overview

This PR addresses a debugger-specific memory leak in DurableTask.Core by ensuring that “abandoned” orchestrator awaits (open activity/sub-orchestration/timer tasks) are released when an orchestration executor is retired, allowing CLR async-debug bookkeeping (Task.s_currentActiveTasks) to unregister those tasks.

Changes:

• Add TaskOrchestrationContext.ReleaseOpenTasks() and a released-context guard to prevent scheduling new work after release.
• Make TaskOrchestrationExecutor implement IDisposable and release open tasks on disposal while restoring orchestrator ambient execution context.
• Ensure TaskOrchestrationDispatcher disposes retired executors via a centralized ReleaseCursor(...) helper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
Test/DurableTask.Core.Tests/TaskOrchestrationExecutorTests.cs	Adds regression tests for executor retirement/leak behavior, but currently placed in the legacy Test/ tree (not in the active test project).
src/DurableTask.Core/TaskOrchestrationExecutor.cs	Implements IDisposable to release abandoned orchestrator continuations when the executor is retired.
src/DurableTask.Core/TaskOrchestrationDispatcher.cs	Disposes retired executors in finally at session end and when continuing-as-new, via ReleaseCursor.
src/DurableTask.Core/TaskOrchestrationContext.cs	Adds open-task cancellation (ReleaseOpenTasks) and prevents scheduling new open tasks after release (ThrowIfReleased).

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.