Hey Networking fans, I’m back again this time with a really cool little trick to capture RoCE v2 traffic and view it using WireShark. Now I know what all of you are thinking… How can you capture traffic that is offloaded from one Nic to another NIc directly to the OS. Heck if you watch Task Manager you don’t even see any traffic.

Well there is a trick and here is what is required:

  1. You need to get yourself a custom build of WireShark from Mellanox. https://community.mellanox.com/docs/DOC-2362
  2. You will also need to use IBDump.exe which is located at C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\IBDump.exe

Then you just need to make some simple configuration changes to your newly installed version of WireShark.

These next few steps below are courtesy Mellanox.

Attached is a preliminary version of Wireshark with InfiniBand support for RoCEv2. In time, it will be added to the Wireshark upstream.

After the installation, click Edit -> Preferences -> Protocols -> InfiniBand, and perform the following adjustments:

1. Type the UDP port as 4791 (RoCEv2 default).

2. Make sure the two check boxes on top are cleared.

Example:


Following the steps above, you can open any *.pcapng file to see the InfiniBand parsing of the RoCE traffic.

Here is a parsing example of the BTH header of a CNP packet, OpCode 0x81 (RoCEv2 Congestion Management Ack):


Here is a look at my system….

Then just go onto your Host System with your Mellanox Adapter and run the following:

C:\Program Files\Mellanox\MLNX_VPI\IB\Tools>ibdump.exe

Initiating resources …

searching for IB devices in host

Port active_mtu=1024

MR was registered with addr=000002685AF13E30, lkey=0x40024e16, rkey=0x40024e16, flags=0x8

————————————————

Device : “ibv_device0”

Physical port : 1

Link layer : Ethernet

Dump file : sniffer.pcap

Sniffer WQEs (max burst size) : 4096

————————————————

Input: guid 7033ed0003078a24, port_num 1, direction 1, qpn 0xaa

Output: reg_id 0

Input: guid 7033ed0003078a24, port_num 1, direction 2, qpn 0xaa

Output: reg_id 0x1

Ready to capture (Press ^c to stop):

Captured: 9483 packets, 7825031 bytes

Interrupted (signal 2) – exiting …

Captured: 9486 packets, 7826645 bytes

-D- Input: guid 0, reg_id 0x1

-D- Input: guid 0, reg_id 0

C:\Program Files\Mellanox\MLNX_VPI\IB\Tools>

Last just open it up in our Mellanox Version of WireShark. Let’s see what this looks like shall we:

And there you have it we are able to sniff RoCE / RDMA Traffic we just need the right tools.

Hope you enjoy and Happy learning,

Dave