Hi,
We have 2 Exchange 2013 servers (CU10) with a single DAG. We experienced some unexpected database failover.
The following errors occured (in this order) in Event Viewer:
Event 1001 from MSExchangeIS:
Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (Unable to apply maintenance GetNonKeyColumnValuesForPrimaryKey-norow, index corruption?) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruptionInternal(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruption(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.GetNonKeyColumnValuesForPrimaryKey(Context context, Object[] primaryKeyValues)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.DoMaintenanceDelete(Context context, Byte[] propertyBlob)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenance(Context context, LogicalOperation operation, Byte[] propertyBlob)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceChunk(Context context, Queue`1 maintenanceRecords, Int32 numberOfRecordsToApply, Boolean commitTransaction)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceToIndexNoLock(Context context, Boolean canRepopulate, Boolean canPulseTransaction, Int64 itemCount)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceToIndex(Context context, Boolean canRepopulate, Boolean canPulseTransaction, Int64 itemCount)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.UpdateIndexImpl(Context context, Object populationCallback, Int64 itemCount, Boolean& indexRepopulated, Boolean& indexRepopulatedFromAnotherIndex)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetCategoryHeaderViewLogicalIndex(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetInScopePseudoIndexes(Context context, SearchCriteria findRowCriteria, IList`1& masterIndexes)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ViewTable.GetCountOperator(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ViewTable.GetRowCount(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetRowCount(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiViewTableBase.GetRowCount(MapiContext context)
at Microsoft.Exchange.Protocols.MAPI.MapiViewTableBase.QueryPosition(MapiContext context, Int32& numerator, Int32& denominator)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.QueryPosition(MapiContext context, MapiViewTableBase view, QueryPositionResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.QueryPosition(IServerObject serverObject, QueryPositionResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopQueryPosition.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize,
Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.Execute(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>c__DisplayClass9.<DoRpc>b__6(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn,
Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1
auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion
completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request,
ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>c__DisplayClassf.<EcPoolSessionDoRpc>b__c()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc(IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion
completion)
at EcPoolSessionDoRpcRpc.EcDispatchCall(EcPoolSessionDoRpcRpc* , SafeRpcAsyncStateHandle pAsyncState, IPoolRpcServer server)
at PoolRpcServer_Wrapper.InternalExecute(PoolRpcServer_Wrapper* , SafeRpcAsyncStateHandle pAsyncState)
at Microsoft.Exchange.Rpc.ManagedExceptionAsyncCrashWrapper.Execute<class Microsoft::Exchange::Rpc::PoolRpc::SafeEcPoolSessionDoRpcRpcAsyncStateHandle>(ManagedExceptionAsyncCrashWrapper* , _RPC_ASYNC_STATE* pAsyncState)
at EcPoolSessionDoRpc_Managed(_RPC_ASYNC_STATE* pAsyncState, Void* cpxh, UInt32 ulSessionHandle, UInt32* pulFlags, UInt32 cbIn, Byte* rgbIn, UInt32* pcbOut, Byte** ppbOut, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** ppbAuxOut)
Event 1002 from MSExchangeIS:
Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: Unable to apply maintenance GetNonKeyColumnValuesForPrimaryKey-norow, index corruption?
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruptionInternal(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruption(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.GetNonKeyColumnValuesForPrimaryKey(Context context, Object[] primaryKeyValues)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.DoMaintenanceDelete(Context context, Byte[] propertyBlob)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenance(Context context, LogicalOperation operation, Byte[] propertyBlob)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceChunk(Context context, Queue`1 maintenanceRecords, Int32 numberOfRecordsToApply, Boolean commitTransaction)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceToIndexNoLock(Context context, Boolean canRepopulate, Boolean canPulseTransaction, Int64 itemCount)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenanceToIndex(Context context, Boolean canRepopulate, Boolean canPulseTransaction, Int64 itemCount)
at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.UpdateIndexImpl(Context context, Object populationCallback, Int64 itemCount, Boolean& indexRepopulated, Boolean& indexRepopulatedFromAnotherIndex)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetCategoryHeaderViewLogicalIndex(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetInScopePseudoIndexes(Context context, SearchCriteria findRowCriteria, IList`1& masterIndexes)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ViewTable.GetCountOperator(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ViewTable.GetRowCount(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.MessageViewTable.GetRowCount(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiViewTableBase.GetRowCount(MapiContext context)
at Microsoft.Exchange.Protocols.MAPI.MapiViewTableBase.QueryPosition(MapiContext context, Int32& numerator, Int32& denominator)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.QueryPosition(MapiContext context, MapiViewTableBase view, QueryPositionResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.QueryPosition(IServerObject serverObject, QueryPositionResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopQueryPosition.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize,
Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.Execute(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>c__DisplayClass9.<DoRpc>b__6(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn,
Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1
auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion
completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request,
ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>c__DisplayClassf.<EcPoolSessionDoRpc>b__c()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)).
Event 1013 from MSExchangeIS:
The mailbox with mailboxguid "03d4ed89-03b8-480f-a6b2-5f9030f18b3a" caused crash or resource outage on database (GUID="f0ad4d29-21c1-43bc-8dee-2ea075774e09"). Description: ExAssertException: MED.ExAssert.AssertInternal,MESSC.ErrorHelper.AssertRetail,MESSLI.LogicalIndex.HandleIndexCorruptionInternal,MESSLI.LogicalIndex.HandleIndexCorruption,MESSLI.LogicalIndex.GetNonKeyColumnValuesForPrimaryKey,MESSLI.LogicalIndex.DoMaintenanceDelete,MESSLI.Log.
Event 4999 from Exchange Common:
Watson report about to be sent for process id: 11696, with parameters: E12, c-RTL-AMD64, 15.00.1130.007, M.E.Store.Worker, M.E.S.Storage.LazyIndexing, M.E.S.S.L.LogicalIndex.HandleIndexCorruptionInternal, M.E.Diagnostics.ExAssertException, a762,
15.00.1130.002.
ErrorReportingEnabled: False
Event 1009 from MSExchangeFastSearch:
The indexing of mailbox database Small_db1 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException:
The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: An exception occurred processing MDB: f0ad4d29-21c1-43bc-8dee-2ea075774e09. ---> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException:
Failed to read notifications, MDB: f0ad4d29-21c1-43bc-8dee-2ea075774e09. ---> Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
Lid: 44776
Lid: 12514 Win32Error: 0x6BA
Lid: 62184
Lid: 16280 dwParam: 0x0 Msg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x0 Msg: EEInfo: ProcessID: 4848
Lid: 12696 dwParam: 0x0 Msg: EEInfo: Generation Time: 0416-12-23T17:29:55.9900000Z
Lid: 10648 dwParam: 0x0 Msg: EEInfo: Generating component: 2
Lid: 14744 dwParam: 0x0 Msg: EEInfo: Status: 1722
Lid: 9624 dwParam: 0x0 Msg: EEInfo: Detection location: 10
Lid: 13720 dwParam: 0x0 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 2
Lid: 12952 dwParam: 0x0 Msg: EEInfo: prm[0]: Long val: 3221225524
Lid: 8856 dwParam: 0x0 Msg: EEInfo: prm[1]: Unicode string: \RPC Control\LRPC-906bf00992caff8a8b
Lid: 62184
Lid: 16280 dwParam: 0x0 Msg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x0 Msg: EEInfo: ProcessID: 4848
Lid: 12696 dwParam: 0x0 Msg: EEInfo: Generation Time: 0416-12-23T17:29:55.9900000Z
Lid: 10648 dwParam: 0x0 Msg: EEInfo: Generating component: 2
Lid: 14744 dwParam: 0x0 Msg: EEInfo: Status: 1727
Lid: 9624 dwParam: 0x0 Msg: EEInfo: Detection location: 1010
Lid: 13720 dwParam: 0x0 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam: 0x0 Msg: EEInfo: NumberOfParameters: 1
Lid: 12952 dwParam: 0x0 Msg: EEInfo: prm[0]: Long val: 3221225527
Lid: 48332
Lid: 49384
Lid: 51176 StoreEc: 0x476
Lid: 40680 StoreEc: 0x476
Lid: 58316
Lid: 16354 StoreEc: 0x476
Lid: 38985 StoreEc: 0x476
Lid: 20098
Lid: 20585 StoreEc: 0x476
at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()
at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass1`1.<TranslateMapiExceptionsWithReturnValue>b__0()
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall)
at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)
at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.PollForEvents()
at Microsoft.Exchange.Search.Mdb.NotificationsFeeder.<>c__DisplayClass6.<ProcessingProcedure>b__5()
at Microsoft.Exchange.Search.Core.Common.Executable.TryRunUnderExceptionHandler(Action action, LocalizedString message)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchFeedingController.ExecuteComplete(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)
Event 126 from ExchangeStoreDB:
At '23/12/2016 18:29:55' the Exchange store database 'Small_db1' copy on this server encountered an error that caused the database to be dismounted. For more detail about the failure, consult the Event log on the server for other "ExchangeStoreDb" or "msexchangerepl" events. A successful failover restored service.
I am failing to understand what happened here. I checked the mailbox with the GUID (mentioned in Event 1013) but there seem to be no corruptions.
Any help would be very welcome. Please don't hesitate to ask if you need more information.