As promised this is a bonus post that will take us into a bit deeper dive on the core Networking stack of Windows Server 2016. Specifically, the new SET Team and RDMA Configurations required for Storage Spaces Direct.
Configuring Datacenter Bridging
The following Windows PowerShell script provides an example of how to enable and configure Data Center Bridging (DCB) for SMB Direct on S2DNode1:
# # Turn on DCB Install-WindowsFeature Data-Center-Bridging # # Set a policy for SMB-Direct New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3 # # Turn on Flow Control for SMB Enable-NetQosFlowControl -Priority 3 # # Make sure flow control is off for other traffic Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7 # # Apply policy to the target adapters Enable-NetAdapterQos -Name "Ethernet 3", “Ethernet 4” # # Give SMB Direct 30% of the bandwidth minimum New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 30 -Algorithm ETS
You can complete the installation of the DataCenter Bridging on the remaining nodes by running the following PowerShell command.
Invoke-Command -ComputerName S2DNode2 -ScriptBlock {
#
# Turn on DCB
Install-WindowsFeature Data-Center-Bridging
#
# Set a policy for SMB-Direct
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
#
# Turn on Flow Control for SMB
Enable-NetQosFlowControl -Priority 3
#
# Make sure flow control is off for other traffic
Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7
#
# Apply policy to the target adapters
Enable-NetAdapterQos -Name "Ethernet 3","Ethernet 4"
#
# Give SMB Direct 30% of the bandwidth minimum
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 30 -Algorithm ETS
}
Configuring the SET Team
A switch-embedded team must be created at the time the Hyper-V switch is created.
When creating the Hyper-V switch using the New-VMSwitch PowerShell cmdlet, the “EnableEmbeddedTeaming” option must be selected.
Below is a screen shot of the network adapter configuration prior to building out the networking stack.
Base Networking Configuration
Note: You must ensure that you purchase a supported network adapter for your configuration. RDMA over Converged Ethernet is what you want to look for and we personally recommend the: ConnectX®-3 Pro EN Single/Dual-Port Adapters 10/40/56GbE Adapters w/ PCI Express 3.0. More information can be found at Mellanox’s website: http://www.mellanox.com/page/products_dyn?product_family=162&mtag=connectx_3_pro_en_card
To build the SET Team you can run the following PowerShell command
New-VMSwitch -Name TeamedvSwitch02 -NetAdapterName "Ethernet 3","Ethernet 4" -EnableEmbeddedTeaming $true
You can complete the installation of the SET Team on the remaining nodes by running the following PowerShell command.
Invoke-Command -ComputerName S2DNode2 -ScriptBlock {
New-VMSwitch -Name TeamedvSwitch02 -NetAdapterName "Ethernet 3","Ethernet 4" -EnableEmbeddedTeaming $true
}
The result of the creation of the new SET Team looks like this in the Networking Control Panel.
SET Team created
Note: Remember that all configurations of the SET Teams are done through SCVMM or PowerShell. When you look at Server Manager you will see that NIC Teaming is disabled because this is only a view for LBFO Teaming.
Creating RDMA Enabled Virtual Adapters
Now that a SET Team has been created we need to create the RDMA enabled Virtual Adapters. This is what S2D will use its core network communications.
Below is a screen shot of RDMA Enabled Virtual Adapters post-configuration. I thought I would show you the finished product before showing the script to create them.
That is right under the screen shot.
RDMA Virtual Adapters Created
Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_4 -managementOS Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_3 -managementOS Enable-NetAdapterRDMA "vEthernet (SMB_3)","vEthernet (SMB_4)"
You can complete the installation of the RDMA enabled Virtual Adapters on the remaining nodes by running the following PowerShell command.
Invoke-Command -ComputerName S2DNode2 -ScriptBlock {
Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_3 -managementOS
Add-VMNetworkAdapter -SwitchName TeamedvSwitch02 -Name SMB_4 -managementOS
Enable-NetAdapterRDMA "vEthernet (SMB_3)","vEthernet (SMB_4)"
}
Verifying RDMA Enabled Virtual Adapters
To verify that the RDMA Enabled Virtual Adapters have been created properly you can run the following PowerShell command. The below PowerShell commands will allow us to see if the configurations have been enabled or not on our Virtual Network Adapters.
Get-NetOffloadGlobalSetting | select networkdirect Get-NetAdapterRdma Get-NetAdapterHardwareInfo
Verifying the RDMA Configurations
Viewing RDMA Performance Counters
There are specific performance counters that can be viewed for RDMA. Let’s have a look at those.
RDMA Performance Counters
Lastly, we need to ensure that the nodes are listening on the RDMA Interfaces by using Netstat.
netstat.exe -xan | ? {$_ -match "445"}
Viewing SMB Direct Connections using Netstat
Hopefully this has helped fill in some of the blanks for you on the Networking Configuration with RDMA and SET Teams used for Storage Spaces Direct.
In Part 6 of this Blog Series we will actually build our Failover CLuster and enable Storage Spaces Direct.
Thanks,
Cristal
Cristal Kawula
Cristal Kawula is the co-founder of MVPDays Community Roadshow and #MVPHour live Twitter Chat. She was also a member of the S2D Technical Advisory board and is the President of TriCon Elite Consulting. Cristal is also only the 2nd woman in the world to receive the prestigious Veeam Vanguard award.
BLOG: http://www.checkyourlogs.net
Twitter: @supercristal1
