I know there a multiple thread on this issue, but none of the solutions have helped me yet. Same as everyone else, i'm able to add one server to the DAG, but not two. It does not matter which one, the second one always fails the same way. Here is the log (same as everyone else)
add-databaseavailabiltygroupserver started on machine ENVOYTHREE.[2013-07-03T13:40:48] add-dagserver started
[2013-07-03T13:40:48] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2013-07-03T13:40:48] Option 'Identity' = 'ExchangeDag'.
[2013-07-03T13:40:48] Option 'MailboxServer' = 'ENVOYONE'.
[2013-07-03T13:40:48] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2013-07-03T13:40:48] Option 'WhatIf' = ''.
[2013-07-03T13:40:48] Process: w3wp w3wp.exe:4308.
[2013-07-03T13:40:48] User context = 'NT AUTHORITY\SYSTEM'.
[2013-07-03T13:40:48] Member of group 'Everyone'.
[2013-07-03T13:40:48] Member of group 'BUILTIN\Users'.
[2013-07-03T13:40:48] Member of group 'NT AUTHORITY\SERVICE'.
[2013-07-03T13:40:48] Member of group 'CONSOLE LOGON'.
[2013-07-03T13:40:48] Member of group 'NT AUTHORITY\Authenticated Users'.
[2013-07-03T13:40:48] Member of group 'NT AUTHORITY\This Organization'.
[2013-07-03T13:40:48] Member of group 'BUILTIN\IIS_IUSRS'.
[2013-07-03T13:40:48] Member of group 'LOCAL'.
[2013-07-03T13:40:48] Member of group 'IIS APPPOOL\MSExchangeECPAppPool'.
[2013-07-03T13:40:48] Member of group 'BUILTIN\Administrators'.
[2013-07-03T13:40:48] Updated Progress 'Validating the parameters.' 2%.
[2013-07-03T13:40:48] Working
[2013-07-03T13:40:48] Mailbox server: value passed in = ENVOYONE, mailboxServer.Name = ENVOYONE, mailboxServer.Fqdn = ENVOYONE.RedForest.Wanderingmad.com
[2013-07-03T13:40:48] LogClussvcState: clussvc is Stopped on ENVOYONE.RedForest.Wanderingmad.com. Exception (if any) = none
[2013-07-03T13:40:48] The IP addresses for the DAG are (blank means DHCP): 172.16.40.110
[2013-07-03T13:40:48] Looking up IP addresses for ExchangeDag.
[2013-07-03T13:40:51] ExchangeDag = [ 172.16.40.110 ].
[2013-07-03T13:40:51] Looking up IP addresses for envoyone.
[2013-07-03T13:40:51] envoyone = [ 172.16.40.100 ].
[2013-07-03T13:40:51] Looking up IP addresses for ENVOYONE.RedForest.Wanderingmad.com.
[2013-07-03T13:40:51] ENVOYONE.RedForest.Wanderingmad.com = [ 172.16.40.100 ].
[2013-07-03T13:40:51] DAG ExchangeDag has 1 servers:
[2013-07-03T13:40:51] DAG ExchangeDag contains server ENVOYTWO.
[2013-07-03T13:40:51] Updated Progress 'Checking if Mailbox server 'ENVOYONE' is in a database availability group.' 4%.
[2013-07-03T13:40:51] Working
[2013-07-03T13:40:51] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(ENVOYONE.RedForest.Wanderingmad.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
[2013-07-03T13:40:51] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2013-07-03T13:40:51] DumpClusterTopology: Opening remote cluster ExchangeDag.
[2013-07-03T13:40:51] Dumping the cluster by connecting to: ExchangeDag.
[2013-07-03T13:40:51] The cluster's name is: ExchangeDag.
[2013-07-03T13:40:51] Groups
[2013-07-03T13:40:51] group: Available Storage [not a CMS]
[2013-07-03T13:40:51] OwnerNode: ENVOYTWO.RedForest.Wanderingmad.com
[2013-07-03T13:40:51] State: Offline
[2013-07-03T13:40:51] group: Cluster Group [Cluster Main Group]
[2013-07-03T13:40:51] OwnerNode: ENVOYTWO.RedForest.Wanderingmad.com
[2013-07-03T13:40:51] State: Online
[2013-07-03T13:40:51] Resource: Cluster IP Address [Online, type = IP Address, PossibleOwners = ENVOYTWO ]
[2013-07-03T13:40:51] Address = [172.16.40.110]
[2013-07-03T13:40:51] EnableDhcp = [0]
[2013-07-03T13:40:51] Network = [Cluster Network 1]
[2013-07-03T13:40:51] Resource: Cluster Name [Online, type = Network Name, PossibleOwners = ENVOYTWO ]
[2013-07-03T13:40:51] NetName = [EXCHANGEDAG]
[2013-07-03T13:40:51] Nodes
[2013-07-03T13:40:51] node: ENVOYTWO.RedForest.Wanderingmad.com [ state = Up ]
[2013-07-03T13:40:51] Subnets
[2013-07-03T13:40:51] Name(Cluster Network 1), Mask(172.16.40.0/24), Role(ClusterNetworkRoleInternalAndClient)
[2013-07-03T13:40:51] NIC 172.16.40.101 on Node Envoytwo in State=Up
[2013-07-03T13:40:51] Name(Cluster Network 2), Mask(172.16.45.0/24), Role(ClusterNetworkRoleInternalUse)
[2013-07-03T13:40:51] NIC 172.16.45.20 on Node Envoytwo in State=Up
[2013-07-03T13:40:51] Opening the cluster on nodes [envoytwo].
[2013-07-03T13:40:51] Other mailbox servers in the DAG are already members of cluster 'ExchangeDag'
[2013-07-03T13:40:51] The server ENVOYONE does not belong to a cluster, and the other servers belong to ExchangeDag.
[2013-07-03T13:40:51] Successfully resolved the servers based on the stopped servers list.
[2013-07-03T13:40:51] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2013-07-03T13:40:51] The following servers are in the StoppedServers list:
[2013-07-03T13:40:51] Verifiying that the members of database availability group 'ExchangeDag' are also members of the cluster.
[2013-07-03T13:40:51] Verifying that the members of cluster 'ExchangeDag' are also members of the database availability group.
[2013-07-03T13:40:51] According to GetNodeClusterState(), the server ENVOYONE is NotConfigured.
[2013-07-03T13:40:51] The CNO is currently Online.
[2013-07-03T13:40:51] InternalValidate() done.
[2013-07-03T13:40:51] Updated Progress 'Adding server 'ENVOYONE' to database availability group 'ExchangeDag'.' 6%.
[2013-07-03T13:40:51] Working
[2013-07-03T13:40:51] Updated Progress 'Adding server 'ENVOYONE' to the cluster.' 8%.
[2013-07-03T13:40:51] Working
[2013-07-03T13:47:09] The following log entry comes from a different process that's running on machine 'ENVOYTWO.RedForest.Wanderingmad.com'. BEGIN
[2013-07-03T13:47:09] [2013-07-03T13:40:51] Opening a local AmCluster handle.
[2013-07-03T13:40:51] Updated Progress 'Adding server 'envoyone' to database availability group 'ExchangeDag'.' 2%.
[2013-07-03T13:40:51] Working
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = ExchangeDag, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = ENVOYONE.RedForest.Wanderingmad.com, dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:40:51] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:43:52] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 100, szObjectName = ENVOYONE, dwStatus = 0x5b4 )
[2013-07-03T13:43:52] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = ENVOYONE, dwStatus = 0x0 )
[2013-07-03T13:47:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = , dwStatus = 0x0 )
[2013-07-03T13:47:09] The preceding log entry comes from a different process running on computer 'ENVOYTWO.RedForest.Wanderingmad.com'. END
[2013-07-03T13:47:09] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-07-03_13-40-48.909_add-databaseavailabiltygroupserver.log".
[2013-07-03T13:47:09] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. ---> System.ComponentModel.Win32Exception: This operation returned because the timeout period expired
--- End of inner exception stack trace ---
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
--- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
--- End of stack trace on server (ENVOYTWO.RedForest.Wanderingmad.com) ---
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2013-07-03T13:47:09] Updated Progress 'Done!' 100%.
[2013-07-03T13:47:09] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().
Steps i've done:
1. Disabled chimney offload on Nics
2. I've re-installed servers multiple times, both the O/S and exchange 2013 Cu1
3. I had a DAG built earlier, but the vm was deleted so I had to start from scratch. The new DAG is named something different than previous ones
4. get-databaseavailabilitygroupnetwork comes up with no problem
5. validate cluster comes back clean, only with a TTL error
6. Machine are not cloned, and both exist under hyper-v 2012 on separate servers
7. firewall is off on both machines
8. there is no A/V or 3rd part virus software on server
9 Both computers can talk to eachother
10. When I try to create the cluster manually it times out.
11. No network teaming
12. Loaded under server 2012, not 2008r2