A client recently came to me and wanted us to flatten his hyper-converged appliance, and replace the 3rd party storage solution with Server 2016’s Storage Spaces Direct. With all hardware being on the HCL for Server 2016 and SSD, they can contact Microsoft for any support requirements under their EA.
Although they recognized the value of RDMA network infrastructure, there was absolutely no budget for additional hardware during this fiscal year, the solution needed to be implemented with the existing Intel-based 10GbE infrastructure. This greatly simplified the configuration of the converged network, as there were no SET team or QoS configs required, and the resulting PowerShell script was less involved.
For this environment, following the existing configuration, we’ll be using a LACP NIC team and VLAN tagging.
A full ConfigMgr environment is place for server and workstation imaging, however we wanted something much more basic for the 3 S2D nodes. On a member server on the same subnet as the HCA, I added the WDS role. In WDS, I added the Server 2016 ISO and imported the driver store from the existing nodes…It’s not the prettiest, but it will at least provide functioning NIC and Storage drivers…we’ll upgrade what’s important later. There should be a blog here on how to do this, but if there’s not and you’re interested, let us know in the comments and we’ll be sure to write one up 😊
After migrating VMs off the existing cluster, I ensured that IPMI access (aka Dell DRAC, HP ILO) was functioning to each of the nodes and shut them down. We then had someone in the datacenter remove the 9 data drives from the front of the appliance.
I booted up the first machine, and sent an F12 command to PXE boot. WDS delivered down the image, and I was able to log in and launch Windows Update.
While the first machine was patching, I imaged the other two. The units run 60GB Direct on Motherboard SSDs, and Server 2016 installs in six and a half minutes.
Once the three nodes were imaged and updated, I installed the latest NIC/Storage drivers on each, configured my firewall rules for all network profiles, then joined the domain. At this point I don’t want any unnecessary policies being applied, so the computer accounts in AD were left in the Computers container. The nodes the needed to be shut down and have the drives re-installed. Along with the existing 9 SSDs, 6 more were added, and the nodes were powered back on.
After a stop to get coffee and check on the weather, I got back to my desk and made sure the three nodes were pingable, then logged into each. In order for the drives to be added into the storage pool, they must first be cleaned. Diskpart – Clean works great…just make sure the data is all good and backed up first. After the drives are clean, we’ll do the rest of the set up in PowerShell.
Here’s everything that was done to build the S2D cluster from this point on, which I ran connected to Node 1 in PowerShell. Yes, I could have made this script significantly smaller by using ForEach instead of ScriptBlock, but it just sort of evolved this way…
#Role Installation
Install-WindowsFeature -Name File-Services
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools
Install-WindowsFeature -Name Hyper-V -IncludeAllSubFeature -IncludeManagementTools -Restart
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {
Install-WindowsFeature -Name File-Services
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools
Install-WindowsFeature -Name Hyper-V -IncludeAllSubFeature -IncludeManagementTools -Restart
}
Invoke-Command -Computername SVR-S2D-03 -ScriptBlock {
Install-WindowsFeature -Name File-Services
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools
Install-WindowsFeature -Name Hyper-V -IncludeAllSubFeature -IncludeManagementTools -Restart
}
#Reboot Nodes
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {Restart-Computer -Force}
Invoke-Command -Computername SVR-S2D-03 -ScriptBlock {Restart-Computer -Force}
Restart-Computer -Force
#Build NIC Team
New-NetlbfoTeam PRODVLAN “Ethernet", "Ethernet 3" -TeamingMode LACP –verbose
#Get the Status of the Network Adapters
Get-NetAdapter | Sort Name
#Create the new Hyper-V Vswitch VSW01
new-vmswitch "VSW01" -MinimumBandwidthMode Weight -NetAdapterName "PRODVLAN" -verbose
#Check the Bindings
Get-NetadapterBinding | where {$_.DisplayName –like “Hyper-V*”}
#Check the Adapter Settings
Get-NetAdapter | sort name
#Create the Converged Adapters
Add-VMNetworkAdapter –ManagementOS –Name “LM” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “HB” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –SwitchName “VSW01” –verbose
#Review the NIC Configuration Again
Get-NetAdapter | Sort name
#Rename the HOST NIC
Rename-NetAdapter –Name “vEthernet (VSW01)” –NewName “vEthernet (Host)” –verbose
#Review the NIC Configuration Again
Get-NetAdapter | Sort name
#Set the weighting on the NIC's
Set-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –MinimumBandwidthWeight 40
Set-VMNetworkAdapter –ManagementOS –Name “LM” –MinimumBandwidthWeight 30
Set-VMNetworkAdapter –ManagementOS –Name “HB” –MinimumBandwidthWeight 20
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "HB" -Access -VLanID 257
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LM" -Access -VLanID 258
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CLUSTERCSV" -Access -VLanID 259
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {
New-NetlbfoTeam PRODVLAN “Ethernet 3", "Ethernet 4" -TeamingMode LACP –verbose
new-vmswitch "VSW01" -MinimumBandwidthMode Weight -NetAdapterName "PRODVLAN" -verbose
Add-VMNetworkAdapter –ManagementOS –Name “LM” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “HB” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –SwitchName “VSW01” –verbose
Rename-NetAdapter –Name “vEthernet (VSW01)” –NewName “vEthernet (Host)” –verbose
Set-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –MinimumBandwidthWeight 40
Set-VMNetworkAdapter –ManagementOS –Name “LM” –MinimumBandwidthWeight 30
Set-VMNetworkAdapter –ManagementOS –Name “HB” –MinimumBandwidthWeight 20
Set-VMNetworkAdapter –ManagementOS –Name “VSW01” –MinimumBandwidthWeight 10
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "HB" -Access -VLanID 257
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LM" -Access -VLanID 258
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CLUSTERCSV" -Access -VLanID 259
}
Invoke-Command -Computername SVR-S2D-03 -ScriptBlock {
New-NetlbfoTeam PRODVLAN “Ethernet", "Ethernet 2" -TeamingMode LACP –verbose
new-vmswitch "VSW01" -MinimumBandwidthMode Weight -NetAdapterName "PRODVLAN" -verbose
Add-VMNetworkAdapter –ManagementOS –Name “LM” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “HB” –SwitchName “VSW01” –verbose
Add-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –SwitchName “VSW01” –verbose
Rename-NetAdapter –Name “vEthernet (VSW01)” –NewName “vEthernet (Host)” –verbose
Set-VMNetworkAdapter –ManagementOS –Name “CLUSTERCSV” –MinimumBandwidthWeight 40
Set-VMNetworkAdapter –ManagementOS –Name “LM” –MinimumBandwidthWeight 30
Set-VMNetworkAdapter –ManagementOS –Name “HB” –MinimumBandwidthWeight 20
Set-VMNetworkAdapter –ManagementOS –Name “VSW01” –MinimumBandwidthWeight 10
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "HB" -Access -VLanID 257
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LM" -Access -VLanID 258
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CLUSTERCSV" -Access -VLanID 259
}
#Configure IPs for Converged Network
New-NetIPAddress -IPAddress 10.10.1.120 -PrefixLength 18 -InterfaceAlias "vEthernet (Host)" -DefaultGateway 10.10.1.1
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Host)" -ServerAddresses 10.10.1.2,10.10.1.1
New-NetIPAddress -IPAddress 10.10.1.130 -PrefixLength 18 -InterfaceAlias "vEthernet (HB)"
New-NetIPAddress -IPAddress 10.11.1.140 -PrefixLength 18 -InterfaceAlias "vEthernet (LM)"
New-NetIPAddress -IPAddress 172.16.11.10 -PrefixLength 24 -InterfaceAlias "vEthernet (CLUSTERCSV)"
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {
New-NetIPAddress -IPAddress 10.10.1.121 -PrefixLength 18 -InterfaceAlias "vEthernet (Host)" -DefaultGateway 10.10.1.1
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Host)" -ServerAddresses 10.10.1.2,10.10.1.1
New-NetIPAddress -IPAddress 10.10.1.131 -PrefixLength 18 -InterfaceAlias "vEthernet (HB)"
New-NetIPAddress -IPAddress 10.10.1.141 -PrefixLength 18 -InterfaceAlias "vEthernet (LM)"
New-NetIPAddress -IPAddress 172.16.11.11 -PrefixLength 24 -InterfaceAlias "vEthernet (CLUSTERCSV)"
}
Invoke-Command -Computername SVR-S2D-03 -ScriptBlock {
New-NetIPAddress -IPAddress 10.10.1.122 -PrefixLength 18 -InterfaceAlias "vEthernet (Host)" -DefaultGateway 10.10.1.1
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Host)" -ServerAddresses 10.10.1.2,10.10.1.1
New-NetIPAddress -IPAddress 10.10.1.132 -PrefixLength 18 -InterfaceAlias "vEthernet (HB)"
New-NetIPAddress -IPAddress 10.10.1.142 -PrefixLength 18 -InterfaceAlias "vEthernet (LM)"
New-NetIPAddress -IPAddress 172.16.11.12 -PrefixLength 24 -InterfaceAlias "vEthernet (CLUSTERCSV)"
}
#Get Status of NICs
Get-NetAdapter | Sort Name
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {Get-NetAdapter | Sort Name}
Invoke-Command -Computername SVR-S2D-03 -ScriptBlock {Get-NetAdapter | Sort Name}
#Build Cluster
Test-Cluster -Node SVR-S2D-01,SVR-S2D-02,SVR-S2D-03 -Include "Storage Spaces Direct", "Inventory", "System Configuration", "Network"
New-Cluster -Name S2DCluster -Node SVR-S2D-01,SVR-S2D-02,SVR-S2D-03 -NoStorage -StaticAddress 10.10.1.119
#Configure Cluster Cloud Witness
Set-ClusterQuorum -CloudWitness -AccountName s2dwitnessazurestorageaccountnotsharedwithotherservices -AccessKey CrAzYLoNgAlPhAnUmErIcEnCrYpTiOnKeY -Endpoint core.windows.net
#Validate cluster for S2D
Get-StorageSubsystem
#Make sure that all data drives show OperationalStatus=OK
Get-PhysicalDisk | ft
#Make sure that all data drives show CanPool=True, OperationalStatus=True
Invoke-Command -Computername SVR-S2D-02 -ScriptBlock {
Get-StorageSubsystem
Get-PhysicalDisk | ft
}
#Enable S2D
Enable-ClusterS2D -PoolFriendlyName S2DPool -Confirm:$false
#Provision Cluster Shared Volumes
New-Volume -StoragePoolFriendlyName S2DPool -FriendlyName MirrorDisk1 -FileSystem CSVFS_REFS -Size 500GB -PhysicalDiskRedundancy 2
If you’re doing this in your lab, it’s important to mention that you don’t just hit F5 and run this entire script. Select a few lines, run each step on other nodes before executing it on the first, and address any issues that the checks identify…The first time around I forgot to build the team using LACP, and my converged adapters were failing to communicate.
Should you get part way through the script and fail, it is best to just reset everything, fix your issue, then proceed from scratch. There are scripts on the storage team’s technet blog that will blow everything storage related away, so you can start anew.
If you want to blow away the failover cluster and converged network, check our previous blogs on clustering for a script.
Hope this helps!
É
