问：

例外

我们在一个客户实例（八个）中遇到了一个生产问题，我们无法理解（因此无法解决）。在对同一 API 的 250 次调用中，该问题发生不到 10 次。例外情况如下：

An exception occurred while iterating over the results of a query for context type '"OurNamespace.OurDbContext2"'."
""System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Transactions.TransactionException: The operation is not valid for the state of the transaction.
 ---> System.Transactions.TransactionPromotionException: Failure while attempting to promote transaction.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an open DataReader associated with this Command which must be closed first.
 ---> System.ComponentModel.Win32Exception (258): Unknown error 258
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at Microsoft.Data.SqlClient.SqlDelegatedTransaction.Promote()
ClientConnectionId:09c2251f-ce5e-4f27-9c37-77ba27e69399
Error Number:−2,State:0,Class:11
ClientConnectionId before routing:f245b876-d483-4827-bdd9-ec446c664f64

完整调用堆栈

at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
   at Microsoft.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.InitializeReader(Enumerator enumerator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1124(Closure , QueryContext )
   at OurNamespace.AspNetCore.Identity.EntityFrameworkCore.TenantSettingsStore`1.FindById(Guid tenantId)
   at OurNamespace.AspNetCore.Identity.Caching.DefaultCache`1.GetOrAdd(String key, TimeSpan duration, Func`1 get)
   at OurNamespace.AspNetCore.Identity.ConfigureMultitenantIdentityOptions.Configure(TenantIdentityOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsManager`1.Get(String name)
   at OurNamespace.AspNetCore.Identity.ExtendedUserValidator`1.get_TenantIdentityOptions()
   at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateTenant(MultitenantUserManager`1 manager, TUser user, ICollection`1 errors)
   at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateAsync(UserManager`1 manager, TUser user)
   at Microsoft.AspNetCore.Identity.UserManager`1.ValidateUserAsync(TUser user)
   at Microsoft.AspNetCore.Identity.UserManager`1.CreateAsync(TUser user)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(TUser user, Boolean sendInvitation, String additionalInvitationQueryParameters)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(CreateUserViewModel model)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.<>c__DisplayClass7_0.<<Create>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at IRM.AspNetCore.Identity.ResilientTransaction.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass33_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.ResilientTransaction.ExecuteAsync(Func`2 action, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.Create(CreatePersonViewModel model, String password, ExternalLoginInfo loginInfo, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonsController`2.Post(CreatePersonViewModel personViewModel, ICreatePersonAndUserService`2 creationService)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at OurNamespace.Serilog.AspNetCore.SerilogCallContextMiddleware.Invoke(HttpContext context, ICallContext callContext)
   at OurNamespace.AspNetCore.Identity.UI.ManageTenantMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at OurNamespace.AspNetCore.Authentication.Cookies.SameSiteCookieFixMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at OurNamespace.AspNetCore.ErrorHandling.ExceptionHandlerMiddleware.Awaited(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

我们认为会影响问题的情况

以下是一些需要注意的薄弱因素，我们认为这些薄弱因素可能会影响问题：

• 问题出在 API 上，是唯一一个针对两个 DbContext 实例的 API。
  • 因此，我们根据推荐的方法自行制定执行策略。
  • 我们使用 TransactionScope（也如上面的链接中所述）。
  • 对第一个 DbContext 实例的所有调用都成功。
  • 两个 DbContext 使用相同的数据库/连接字符串。
• 应用程序和数据库存在于不同的 Azure 订阅中（位于同一数据中心）。
  • 这是客户选择自己托管数据库的八个实例中的唯一一个。

我们使用的版本：

• .NET 核心 6.NET Core 6
• Microsoft.EntityFrameworkCore（以及 SQL Server 的相关参考）：6.0.20
• Microsoft.Data.SqlClient：5.1.1

它是一个 Linux 应用服务。我们还使用 SQL Azure 作为分布式缓存。

发生示例代码的位置

该问题似乎发生在第一次调用 OurDbContext2 上的数据库时，这可能是不同的操作，具体取决于缓存和其他逻辑（上面的调用堆栈只是一个示例）。我选择上面的调用堆栈，因为在这个（不常见的）示例中，我们通过非常简单的方法从选项中使用的数据库中获取设置：

public TenantSettings FindById(Guid tenantId)
{
    using var scope = _serviceProvider.CreateScope();
    using var context = scope.ServiceProvider.GetService<TContext>()!;
            
    var dbSet = context.Set<TenantSettings>();
    var result = dbSet.AsNoTracking().SingleOrDefault(ts => ts.TenantId == tenantId) ?? new TenantSettings { TenantId = tenantId };
            
    return result;
}

该问题不涉及任何选项或此特定操作（也请忽略我们使用 _serviceProvider.CreateScope 的反模式这一事实，因为我们对此有特定的原因），因为它也发生在其他情况下。我选择上述操作，因为我相信它非常清楚地表明该问题与“已经有一个打开的 DataReader 与此命令相关联，必须首先关闭”无关，因为我们在其自己的 DI 范围内本地创建了一个新的 DbContext 实例。

这是另一个发生这种情况的例子：

protected virtual Task<List<TRole>> GetDefaultRolesForNewUserAsync(TUser user, CancellationToken cancellationToken = default)
{
    return Roles.Where(r => r.AddForNewUser).AsNoTracking().ToListAsync(cancellationToken);
}

我们尝试过的事情

我们增加了 SQL Azure 实例的容量（从 20 个 DTU 增加到 50 个 DTU），从而减少了发生的问题。但是，从指标来看，我们看不到实例处于任何类型的负载之下。例如，我们检查了会话、DTU 使用情况、CPU、失败连接和其他指标。

我们已在连接字符串中启用了 MARS，但这完全没有影响问题。

我们已经浏览了所有代码，以确保在使用或遍历列表之前，我们没有错过任何 ToList/FirstOrDefault 来实际获取对象（因为这是“已经有一个打开的 DataReader 与此命令关联，必须首先关闭”的常见原因。

我们已处理了应用服务性能和可用性疑难解答下的许多报告。似乎没有任何SNAT端口耗尽，我们没有发现任何其他迹象表明应用服务可能存在问题。

我们最好的猜测是，这是 SQL Azure 实例的某种基础结构问题（我们的经验是“未知错误 258”通常是），但我们不知道它可能是什么。应用程序负载不重（两个不同的应用服务实例，每个实例的 RPM 约为 7-8 rpm），并且流量没有峰值。