1 How to get best performance with NICs on Intel platforms
2 ========================================================
4 This document is a step-by-step guide for getting high performance from DPDK applications on Intel platforms.
7 Hardware and Memory Requirements
8 --------------------------------
10 For best performance use an Intel Xeon class server system such as Ivy Bridge, Haswell or newer.
12 Ensure that each memory channel has at least one memory DIMM inserted, and that the memory size for each is at least 4GB.
13 **Note**: this has one of the most direct effects on performance.
15 You can check the memory configuration using ``dmidecode`` as follows::
17 dmidecode -t memory | grep Locator
37 The sample output above shows a total of 8 channels, from ``A`` to ``H``, where each channel has 2 DIMMs.
39 You can also use ``dmidecode`` to determine the memory frequency::
41 dmidecode -t memory | grep Speed
44 Configured Clock Speed: 2134 MHz
46 Configured Clock Speed: Unknown
48 Configured Clock Speed: 2134 MHz
52 Configured Clock Speed: 2134 MHz
54 Configured Clock Speed: Unknown
56 Configured Clock Speed: 2134 MHz
58 Configured Clock Speed: Unknown
60 The output shows a speed of 2133 MHz (DDR4) and Unknown (not existing).
61 This aligns with the previous output which showed that each channel has one memory bar.
64 Network Interface Card Requirements
65 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
67 Use a `DPDK supported <https://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE.
69 Make sure each NIC has been flashed the latest version of NVM/firmware.
71 Use PCIe Gen3 slots, such as Gen3 ``x8`` or Gen3 ``x16`` because PCIe Gen2 slots don't provide enough bandwidth
72 for 2 x 10GbE and above.
73 You can use ``lspci`` to check the speed of a PCI slot using something like the following::
75 lspci -s 03:00.1 -vv | grep LnkSta
77 LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- ...
78 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ ...
80 When inserting NICs into PCI slots always check the caption, such as CPU0 or CPU1 to indicate which socket it is connected to.
82 Care should be take with NUMA.
83 If you are using 2 or more ports from different NICs, it is best to ensure that these NICs are on the same CPU socket.
84 An example of how to determine this is shown further below.
90 The following are some recommendations on BIOS settings. Different platforms will have different BIOS naming
91 so the following is mainly for reference:
93 #. Establish the steady state for the system, consider reviewing BIOS settings desired for best performance characteristic e.g. optimize for performance or energy efficiency.
95 #. Match the BIOS settings to the needs of the application you are testing.
97 #. Typically, **Performance** as the CPU Power and Performance policy is a reasonable starting point.
99 #. Consider using Turbo Boost to increase the frequency on cores.
101 #. Disable all virtualization options when you test the physical function of the NIC, and turn on VT-d if you wants to use VFIO.
104 Linux boot command line
105 ~~~~~~~~~~~~~~~~~~~~~~~
107 The following are some recommendations on GRUB boot settings:
109 #. Use the default grub file as a starting point.
111 #. Reserve 1G huge pages via grub configurations. For example to reserve 8 huge pages of 1G size::
113 default_hugepagesz=1G hugepagesz=1G hugepages=8
115 #. Isolate CPU cores which will be used for DPDK. For example::
117 isolcpus=2,3,4,5,6,7,8
119 #. If it wants to use VFIO, use the following additional grub parameters::
121 iommu=pt intel_iommu=on
124 Configurations before running DPDK
125 ----------------------------------
127 1. Reserve huge pages.
128 See the earlier section on :ref:`linux_gsg_hugepages` for more details.
130 .. code-block:: console
132 # Get the hugepage size.
133 awk '/Hugepagesize/ {print $2}' /proc/meminfo
135 # Get the total huge page numbers.
136 awk '/HugePages_Total/ {print $2} ' /proc/meminfo
138 # Unmount the hugepages.
139 umount `awk '/hugetlbfs/ {print $2}' /proc/mounts`
141 # Create the hugepage mount folder.
144 # Mount to the specific folder.
145 mount -t hugetlbfs nodev /mnt/huge
147 2. Check the CPU layout using the DPDK ``cpu_layout`` utility:
149 .. code-block:: console
153 usertools/cpu_layout.py
155 Or run ``lscpu`` to check the cores on each socket.
157 3. Check your NIC id and related socket id:
159 .. code-block:: console
161 # List all the NICs with PCI address and device IDs.
164 For example suppose your output was as follows::
166 82:00.0 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583]
167 82:00.1 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583]
168 85:00.0 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583]
169 85:00.1 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583]
171 Check the PCI device related numa node id:
173 .. code-block:: console
175 cat /sys/bus/pci/devices/0000\:xx\:00.x/numa_node
177 Usually ``0x:00.x`` is on socket 0 and ``8x:00.x`` is on socket 1.
178 **Note**: To get the best performance, ensure that the core and NICs are in the same socket.
179 In the example above ``85:00.0`` is on socket 1 and should be used by cores on socket 1 for the best performance.
181 4. Check which kernel drivers needs to be loaded and whether there is a need to unbind the network ports from their kernel drivers.
182 More details about DPDK setup and Linux kernel requirements see :ref:`linux_gsg_compiling_dpdk` and :ref:`linux_gsg_linux_drivers`.