I have a 2 site DAG with 4 members, 2 at each site with DAC set.
Primary Site:
EXMB01
EXMB02
FSW
Failover Site:
EXMB03 - PAM
EXMB04
AFSW
After updating and rebooting one of the members in the primary active site (exmb02), it looks like the server was removed from the cluster as it was booting back up:
Event ID: 4621
Source: FailoverClustering
This node was successfully removed from the cluster.
Directly preceding that, in the Failover Cluster Manager events, I can see the following progression:
Event ID: 4620
Task Category: Cluster Evict/Destroy Cleanup
Unloading the cluster service registry hive during cluster node cleanup failed. The error code was '3221225569'. You may be unable to create or join a cluster with this machine until cleanup has been successfully completed. For manual cleanup, execute the 'Clear-ClusterNode' PowerShell cmdlet on this machine.
and then: Cluster node cleanup encounted at least one error. You may be unable to create or join a cluster with this machine until cleanup has been successfully completed. Please review and resolve preceding cluster node cleanup events. For manual cleanup, execute the 'Clear-ClusterNode' PowerShell cmdlet on this machine.
And then from all three other nodes:
Node MGR: Cluster node USTRYCO01EXMB02 has been evicted from the failover cluster.
I intend to move the PAM role back to the primary sight, I had not noticed it was there.
Curiously, if I run Cluster.exe node, I can see that all 4 nodes are listed as UP.
What do I need to do to recover EXMB02? I've read that sometimes just a reboot will clean up the cluster configuration, but how would I add the node back into the cluster? Why is it still listed as a node if it was evected? Also, what is the significance of having the PAM role in the primary data center as opposed to the failover data center?