Reset session isolation level on pool return (fixes #96) - #4330
Reset session isolation level on pool return (fixes #96)#4330priyankatiwari08 wants to merge 9 commits into
Conversation
SqlTransaction and TransactionScope leave the underlying SQL Server session with the elevated isolation level after Commit/Rollback. sp_reset_connection does not reset it, so the next user of the pooled physical connection silently inherits it. This change tracks when a non-default isolation level was set via the TM Begin path and, on pool return, issues a 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED' batch right before sp_reset_connection (piggybacked on the same TDS header, no extra round trip). On failure the connection is doomed instead of returned to the pool. A new AppContext switch 'Switch.Microsoft.Data.SqlClient.UseLegacyIsolationLevelBehavior' preserves previous behavior for callers that depend on it (default: false). Adds IsolationLevelLeakTest under ManualTests covering the two repro scenarios plus a negative test for the kill switch.
4e8ed47 to
9f67fb1
Compare
apoorvdeshmukh
left a comment
There was a problem hiding this comment.
Can you look into this test failure?
Seems related to the changes
- IsolationLevelLeakTest: switch to LocalAppContextSwitchesHelper so the cached LocalAppContextSwitches value is forced, not just set via AppContext.SetSwitch (production code reads the cache after first use, which made the negative test order-dependent and caused the CI failure). - Helper restores the previous value on dispose, so global state does not leak to other tests. - TransactionScope test: set Enlist=true explicitly on the pooled connection string so the test does not depend on the user's local DataTestUtility.TCPConnectionString defaults. - LocalAppContextSwitchesHelper: expose UseLegacyIsolationLevelBehavior get/set + capture/restore wired through the standard reflection helpers.
apoorvdeshmukh
left a comment
There was a problem hiding this comment.
Can you look into this failure for LegacySwitch_PreservesOldLeakBehavior?
Tests are mostly failing with this one. You can put the PR into draft till this is addressed.
Is this happening only in CI or do you observe it locally under VS as well?
The leak only manifests on on-prem SQL Server: Azure SQL DB resets the session isolation level inside sp_reset_connection, so the legacy switch becomes a no-op there and the assertion (expects 'Serializable' to leak) fails on Azure CI legs.
| isoLevel != TdsEnums.TransactionManagerIsolationLevel.Unspecified && | ||
| isoLevel != TdsEnums.TransactionManagerIsolationLevel.ReadCommitted) | ||
| { | ||
| _isolationLevelDirty = true; |
There was a problem hiding this comment.
It's worth noting that Azure Synapse Analytics does things slightly differently - it only supports the SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED statement, and the default isolation level can vary based upon a database option.
I think this might also present an issue for #4335.
There was a problem hiding this comment.
Good catch, thanks. Addressed in 645939a, though with graceful degradation rather than endpoint detection.
ResetSessionIsolationLevel now:
- Clears
_isolationLevelDirtyup front, before the try, so an endpoint that rejects the statement isn't retried on every subsequent pool return. - Catches
SqlExceptionin a newwhen (!IsConnectionDoomed)clause that traces viaSqlClientEventSourceand does not doom the connection. The pre-existingcatch (Exception e) when (ADP.IsCatchableExceptionType(e))->DoomThisConnection()remains for transport-level failures.
This is safe because sp_reset_connection is carried as a header flag that the server processes before the batch body, so whatever reset the endpoint natively performs has already taken effect by the time the SET statement errors.
I went with error handling instead of detecting Synapse because there's no cached EngineEdition on the connection, so detection at deactivate time would cost an extra query. ADP.IsAzureSynapseOnDemandEndpoint exists but only matches on-demand/serverless and Fabric DW endpoints, not dedicated SQL pools.
On the default-level-varies-by-database-option point: agreed, this is a real behavioural difference. Resetting to READ COMMITTED is what sp_reset_connection does on SQL Server / Azure SQL DB, but on an endpoint whose configured default is something else we'd now be normalising to READ COMMITTED instead of that endpoint's default. Happy to gate the whole reset behind an engine-edition check if you'd prefer correctness over the extra round trip - let me know which you'd rather see.
Also worth flagging for #4335.
There was a problem hiding this comment.
I've tested against a few more unusual TDS endpoints and have three more detailed pieces of information.
First: this specifically only raises an error on a Synapse dedicated SQL pool - the serverless pools are absolutely fine. Your point about ADP.IsAzureSynapseOnDemandEndpoint is completely right. When it does throw an error, its error number 104409: "Setting IsolationLevel to [ReadCommitted/RepeatableRead/Snapshot/Serializable] is not supported".
Second: if I run ALTER DATABASE [] SET READ_COMMITTED_SNAPSHOT ON to change a Synapse database's transaction isolation level, Synapse continues to throw the same error if I run SET TRANSACTION ISOLATION LEVEL READ COMMITTED. I have no way to check what the current transaction level is, but the documentation states:
Once enabled, all transactions in this database are executed under
READ COMMITTED SNAPSHOT ISOLATIONand the settingREAD UNCOMMITTEDat the session level isn't honored.
READ COMMITTED thus isn't just a default - it's the only available option. Catching the exception should be absolutely fine, Synapse won't perform any behavioural changes as the result of the statement. Filtering to only catch exceptions where the error number is 104409 seems sensible too.
Third: connecting to the TDS endpoint for Microsoft Dataverse throws an exception when any kind of transaction isolation level is set. I don't think we'll ever hit this code path under that circumstance, and I think #4335 is also safe - the documentation for this endpoint confirms that it doesn't support transactions.
With all of the above noted, a top-level guard to avoid the extra server round-trip might be a good idea too. Synapse dedicated SQL pool endpoints end in .sql.azuresynapse.net, but don't end in -ondemand.sql.azuresynapse.net. Could we also return early if this heuristic matches?
…dy-carnival # Conflicts: # src/Microsoft.Data.SqlClient/tests/ManualTests/Microsoft.Data.SqlClient.ManualTests.csproj
- ResetSessionIsolationLevel: clear the dirty flag up front so a server that rejects the statement is not retried on every pool return, and add a non-dooming catch for SqlException so endpoints that reject SET TRANSACTION ISOLATION LEVEL READ COMMITTED (e.g. Azure Synapse dedicated SQL pools) degrade gracefully instead of dooming the connection. sp_reset_connection rides as a header flag processed before the batch body, so the reset itself has already taken effect. - LocalAppContextSwitchesHelper: the UseLegacyIsolationLevelBehavior getter read the raw cached field instead of invoking the public property, so it returned null after a reset. Use GetSwitchPropertyValue like every other switch. - LocalAppContextSwitchesTest: cover UseLegacyIsolationLevelBehavior in TestDefaultAppContextSwitchValues. - IsolationLevelLeakTest: drop the unused System.Data using (CS8019). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:3971
- ResetSessionIsolationLevel() can run while the connection is being returned to a transaction-preserving pool (EnlistedTransaction != null). In that case, issuing
SET TRANSACTION ISOLATION LEVEL READ COMMITTEDcan change the isolation level for subsequent work in the same ambient transaction, which is a behavioral regression for TransactionScope scenarios that open/close multiple connections within a single scope.
if (_isolationLevelDirty && !LocalAppContextSwitches.UseLegacyIsolationLevelBehavior)
{
ResetSessionIsolationLevel();
}
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/TransactionTest/IsolationLevelLeakTest.cs:18
- Manual tests are partitioned by the xUnit
Trait("Set", ...)filter (see build.proj TestSetFilter). Without aSettrait, these new tests may be skipped in CI when manual tests are run with-p:TestSet=....
public static class IsolationLevelLeakTest
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:4013
- PR description says: "On failure, the connection is doomed so it's destroyed instead of pooled." But the implementation explicitly does not doom the connection for non-dooming SqlExceptions (it logs and allows pooling). Please align the PR description with the intended behavior, or change the code to doom on this failure if that was the requirement.
catch (SqlException) when (!IsConnectionDoomed)
{
// The server rejected the statement but the session itself is healthy (the parser
// dooms the connection for transport-level failures, so reaching here means a plain
// T-SQL error). Not every SQL Server-compatible endpoint accepts this statement:
// Azure Synapse Analytics dedicated SQL pools, for example, only support
// SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED.
//
// sp_reset_connection still ran, because it is carried as a flag on the header of
// the batch above and is processed by the server before the batch body. The
// connection is therefore no less clean than it was under the legacy behavior, so
// degrade gracefully and let it be pooled rather than destroying it.
SqlClientEventSource.Log.TryTraceEvent(
"<sc.SqlInternalConnectionTds.ResetSessionIsolationLevel|ADV> {0}, " +
"server rejected the session isolation level reset; leaving the session " +
"isolation level unchanged.",
ObjectID);
}
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4330 +/- ##
==========================================
- Coverage 64.78% 62.89% -1.89%
==========================================
Files 288 283 -5
Lines 44418 67449 +23031
==========================================
+ Hits 28774 42422 +13648
- Misses 15644 25027 +9383
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Explains why the pool-return isolation-level scrub (dotnet#96) and the TransactionScope re-enlistment re-assert (dotnet#146) are opposite failures of the same sp_reset_connection inconsistency, and why neither fix subsumes the other. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6a5c2db9-5c3a-4cbb-9ae5-d120b6aa1988
|
Added a design note at Short version: both bugs come from the same fact —
Neither subsumes the other: this PR never runs while the scope is still open (and would write the wrong level if it did), and #4335 only fires on the re-enlist branch, which the #96 repro — no live transaction, often plain The two PRs do overlap textually (same file, same switches helper, same test folder), so they should be sequenced. Suggested order is this PR first (broader blast radius, self-contained on the deactivate path), then rebase #4335 on top. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:4013
- PR description says that if resetting the isolation level fails "the connection is doomed so it's destroyed instead of pooled", but the implementation intentionally does not doom the connection on a T-SQL
SqlException(it logs and allows pooling). Please align the PR description with the actual behavior (or adjust the code if dooming-on-any-failure is the intended contract).
catch (SqlException) when (!IsConnectionDoomed)
{
// The server rejected the statement but the session itself is healthy (the parser
// dooms the connection for transport-level failures, so reaching here means a plain
// T-SQL error). Not every SQL Server-compatible endpoint accepts this statement:
// Azure Synapse Analytics dedicated SQL pools, for example, only support
// SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED.
//
// sp_reset_connection still ran, because it is carried as a flag on the header of
// the batch above and is processed by the server before the batch body. The
// connection is therefore no less clean than it was under the legacy behavior, so
// degrade gracefully and let it be pooled rather than destroying it.
SqlClientEventSource.Log.TryTraceEvent(
"<sc.SqlInternalConnectionTds.ResetSessionIsolationLevel|ADV> {0}, " +
"server rejected the session isolation level reset; leaving the session " +
"isolation level unchanged.",
ObjectID);
}
CI partitions the ManualTests assembly by the "Set" trait. build.proj builds a TestSetFilter of (Set=1|Set=2|Set=3|Set=AE) and ANDs it into the ManualTests filter, and the pipelines run testSets: [1, 2, 3]. IsolationLevelLeakTest was added before the Set trait requirement landed on main (dotnet#4071), so it carried no Set trait. After merging main, the three tests in this file silently stopped matching any CI leg's filter and were never executed -- the PR's own regression tests were producing a false green. Add [Trait("Set", "3")], matching every other class in SQL/TransactionTest (TransactionTest, TransactionEnlistmentTest, DistributedTransactionTest). Verified: - vstest --TestCaseFilter:"Set=3" now lists all three tests; "Set=1" correctly lists none. - All three pass locally against SQL Server 2022 (on-prem, TCP). - No interference when run alongside the other Set=3 transaction and connection-pool tests (19 passed, 1 skipped, 0 failed). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e54f2587-2955-4c93-9896-a35e73feb6e5
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/LocalAppContextSwitchesTest.cs:38
TestDefaultAppContextSwitchValuesassertsUseOverallConnectTimeoutForPoolWaitlater in the test, but its cached field is not reset tonullin the initial reset block. If another test has already cached or mutated this switch viaLocalAppContextSwitchesHelper, this test can become order-dependent. Reset it alongside the other switches.
switchesHelper.EnableMultiSubnetFailoverByDefault = null;
switchesHelper.IgnoreServerProvidedFailoverPartner = null;
switchesHelper.UseLegacyFailoverAlternationOnLoginSqlErrors = null;
switchesHelper.UseLegacyIsolationLevelBehavior = null;
switchesHelper.LegacyRowVersionNullBehavior = null;
The isolation-level reset was issued from ResetConnection(), which turned a method that previously performed no network I/O into one that does a blocking write plus a blocking read. That matters because ResetConnection() has a second caller. The connection pool invokes it from PutObjectFromTransactedPool, which runs on the System.Transactions transaction-completion callback thread while holding a lock on the connection. The code immediately below that call documents why it passes probeLiveness: false there -- it deliberately declines even a non-blocking socket poll on a thread it does not own. Issuing a batch one line earlier violated that contract. Changes: - Move the ResetSessionIsolationLevel() call from ResetConnection() into Deactivate(). Deactivate() always runs on the thread that closed the connection, and every pooled return passes through it, so this covers the same cases without putting socket work on the transaction-completion thread. Verified by instrumenting the call site: all invocations across the three regression tests arrive via Deactivate() -> CloseConnection, and none via PutObjectFromTransactedPool. - Replace timeout: 0 with ConnectionOptions.ConnectTimeout. TdsParserStateObject .SetTimeoutMilliseconds maps a timeout of 0 to long.MaxValue, so the previous value allowed an unresponsive server to block SqlConnection.Close() indefinitely. ConnectTimeout matches the existing convention for internal operations in this file (see GetDTCAddress). - Correct the round-trip claim in the code comments and in the design note. PrepareResetConnection performs no I/O of its own -- it sets a flag that TdsParser.CheckResetConnection consumes at the next packet write. The legacy close path therefore sent nothing, so this is a genuine extra round trip on Close(), not a shared one. The reset still rides this batch's TDS header rather than the next user's first command, so it is not billed twice, but the batch itself is new cost. No functional change to the fix itself: all three IsolationLevelLeakTest tests and the nine sibling TransactionTest tests pass against SQL Server 2022, and the driver builds clean for net462, net8.0 and net9.0. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e54f2587-2955-4c93-9896-a35e73feb6e5
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:4006
ResetSessionIsolationLevelintends to bound the round-trip byConnect Timeout, butConnectTimeoutcan legally be0(no timeout). Passing0here will still map to an effectively-infinite timeout (as the comment notes), which meansSqlConnection.Close()can hang indefinitely on an unresponsive server. Clamp the timeout to a minimum > 0 (or substitute a sane default whenConnectTimeout==0).
// Bounded by the connection's own connect timeout. Passing 0 here would map to
// long.MaxValue in TdsParserStateObject.SetTimeoutMilliseconds, which would let
// an unresponsive server block SqlConnection.Close indefinitely.
timeout: ConnectionOptions.ConnectTimeout,
Scrubbing the session isolation level on pool return corrupts a live TransactionScope. Close() is routinely called inside a scope, and the connection is still enlisted at that point, so issuing SET TRANSACTION ISOLATION LEVEL READ COMMITTED there downgrades the level for the next connection vended into that same scope from the transacted pool. That is exactly the defect tracked by dotnet#146, and this PR was making it universal rather than Azure-only. Instrumenting the call site confirmed it: the TransactionScope test reached the reset with EnlistedTransaction non-null. The reset now runs in Activate(), before enlistment, and is skipped when the connection is already enlisted. Activate() is the only hook that is simultaneously outside any live transaction, off the System.Transactions completion thread that ResetConnection() is called on from PutObjectFromTransactedPool, and traversed by every vend. It also means the extra round trip is only paid by connections that are actually reused. _isolationLevelDirty is an instance field, so it survives pool residency, and the pending sp_reset_connection still rides this batch's TDS header. Adds TransactionScope_SecondConnectionInSameScopeKeepsIsolationLevel as a regression guard; it fails without the enlistment gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e54f2587-2955-4c93-9896-a35e73feb6e5
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:2752
- The comment says the isolation level will be "reset on deactivate", but the implementation actually resets on the next checkout via Activate() (ResetSessionIsolationLevel is called from Activate). Please update the comment to match the actual behavior so future maintenance doesn’t get misled.
// A successful Begin with a non-default isolation level mutates
// SQL Server's session transaction_isolation_level. sp_reset_connection
// will not undo this, so remember it and reset on deactivate.
if (requestType == TdsEnums.TransactionManagerRequestType.Begin &&
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Connection/SqlConnectionInternal.cs:2079
- The PR title/description says the isolation level reset happens on pool return / Deactivate(), but the code change performs the reset on pool checkout in Activate() (before enlistment) when _isolationLevelDirty is set. Please update the PR title/description (and the "Fix" section) to match the implementation, or adjust the code if Deactivate-on-return is still the intended behavior.
This issue also appears on line 2749 of the same file.
// sp_reset_connection does not reset the session transaction isolation level, so when a
// previous Begin raised it we scrub it here, on checkout.
//
// This runs on checkout rather than on pool return for two reasons:
//
// - On return the connection may still be enlisted in a live TransactionScope, because
// Close is routinely called inside the scope. Issuing SET there would downgrade the
// isolation level for any further connection vended into that same scope from the
// transacted pool, which is the defect tracked by #146.
// - ResetConnection, the other pool-return hook, is also invoked from
// PutObjectFromTransactedPool on the System.Transactions transaction-completion
// callback thread while holding a lock on the connection. That path deliberately
// avoids socket work on a thread it does not own.
//
// Activate always runs on the thread performing the checkout, and by then the previous
// transaction has ended, so neither constraint applies. It also means the cost is only
// paid by connections that are actually reused.
//
// EnlistedTransaction is non-null here only when the connection is being re-vended into
// a transaction it is already enlisted in; scrubbing then would hit the same #146
// problem, so it is skipped.
if (_isolationLevelDirty &&
!LocalAppContextSwitches.UseLegacyIsolationLevelBehavior &&
EnlistedTransaction is null &&
!IsConnectionDoomed)
{
ResetSessionIsolationLevel();
}
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/TransactionTest/IsolationLevelLeakTest.cs:20
- This new test file is missing the XML
documentation required by the repository testing guidelines (see .github/instructions/testing.instructions.md:182-186). Please add
comments to the test class and each test method (and any helper methods where helpful) so the test intent is preserved for maintainers.
// SqlTransaction / TransactionScope used to leave the pooled connection
// with the elevated session isolation level. The next user of the pooled
// connection would silently inherit it. The fix resets the session
// isolation level to READ COMMITTED when the connection is next taken out
// of the pool.
[Trait("Set", "3")]
public static class IsolationLevelLeakTest
{
Fixes #96.
Problem
SqlTransactionandTransactionScopeleave the underlying SQL Serversession with the elevated isolation level after Commit/Rollback. Because
sp_reset_connectiondoes not reset the session isolation level, the nextuser of the pooled physical connection silently inherits it.
Repro
using var c = new SqlConnection(cs); c.Open();using var tx = c.BeginTransaction(IsolationLevel.Serializable); tx.Rollback();sys.dm_exec_sessions.transaction_isolation_levelis still 4 (Serializable).Reproduced on 5.2.2 and 7.0.1 against SQL Server, on both net8.0 and net472.
Fix
SqlInternalConnectionTdswhen a non-default isolation level was set via the TMBeginpath (_isolationLevelDirty).Activate()(the pool-checkout path), before enlistment and only when the connection is not already enlisted, issueSET TRANSACTION ISOLATION LEVEL READ COMMITTED;viaTdsExecuteSQLBatch, bounded by the connection'sConnect Timeout.READ UNCOMMITTED). Transport-level failures doom the connection so it is destroyed rather than pooled.Why checkout and not pool return
Two earlier revisions put the batch on the return side — first in
ResetConnection(), then inDeactivate(). Both are wrong, for differentreasons.
ResetConnection()is also called by the pool fromChannelDbConnectionPool.PutObjectFromTransactedPool, which runs on theSystem.Transactionstransaction-completion callback thread while holding alock on the connection — a call site that deliberately declines even a
non-blocking socket poll on a thread it does not own.
ResetConnection()previously performed no network I/O at all, and that was load-bearing.
Deactivate()avoids that thread, but the connection may still be enlistedin a live
TransactionScopethere, becauseClose()is routinely called insidethe scope. Instrumenting the call site confirmed it:
Issuing
SETin that state downgrades the isolation level for the nextconnection vended into the same scope from the transacted pool — which is
exactly the defect tracked by #146, except this would make it universal rather
than Azure-only.
Activate()is subject to neither constraint: it always runs on the threadperforming the checkout, the previous transaction has ended by then, and every
vend passes through it. The
EnlistedTransaction is nullgate keeps it off there-attach path that #4335 owns.
_isolationLevelDirtyis an instance field, soit survives pool residency, and the pending
sp_reset_connectionstill ridesthis batch's TDS header.
Cost
One extra round trip on
Open(), paid only when a previousBeginraisedthe session isolation level and the connection is actually reused.
An earlier version of this description claimed there was no extra round trip.
That was wrong.
PrepareResetConnectionperforms no I/O of its own — it sets aflag that
TdsParser.CheckResetConnectionconsumes at the next packet write —so the legacy close path sent nothing at all. The queued
sp_reset_connectiondoes ride this batch's TDS header rather than the caller's first command, so the
reset is not billed twice, but the batch itself is new cost.
Kill switch
Switch.Microsoft.Data.SqlClient.UseLegacyIsolationLevelBehavior(defaultfalse) restores the previous behavior.Tests
Adds
IsolationLevelLeakTestunderManualTests/SQL/TransactionTest/:SqlTransaction_SerializableDoesNotLeakAcrossPoolTransactionScope_SerializableDoesNotLeakAcrossPoolTransactionScope_SecondConnectionInSameScopeKeepsIsolationLevel— regression guard for the Wrong isolation level with Sql Azure and TransactionScope #146 interaction; verified to fail without the enlistment gateLegacySwitch_PreservesOldLeakBehavior(negative test)Validation
Built clean for net462 / net8.0 / net9.0, 0 warnings. All four tests plus the
nine sibling
TransactionTesttests pass against a local SQL Server 2022. Withthe legacy switch enabled,
[After]isSerializable(kill-switch verified).