1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2014-2018 Chelsio Communications.
8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
10 has support for the latest Linux and FreeBSD operating systems.
12 More information can be found at `Chelsio Communications Official Website
13 <http://www.chelsio.com>`_.
18 CXGBE PMD has support for:
20 - Multiple queues for TX and RX
21 - Receiver Side Steering (RSS)
22 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
23 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
28 - Port hardware statistics
34 The Chelsio Terminator series of devices provide two/four ports but
35 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
36 itself as a PCI driver that allocates one Ethernet device per detected
39 For this reason, one cannot whitelist/blacklist a single port without
40 whitelisting/blacklisting the other ports on the same device.
42 Supported Chelsio T5 NICs
43 -------------------------
46 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
47 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
48 - Other T5 NICs: T522-CR
50 Supported Chelsio T6 NICs
51 -------------------------
53 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
54 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
59 - Requires firmware version **1.17.14.0** and higher. Visit
60 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
61 bundled with the latest Chelsio Unified Wire package.
63 For Linux, installing and loading the latest cxgb4 kernel driver from the
64 Chelsio Unified Wire package should get you the latest firmware. More
65 information can be obtained from the User Guide that is bundled with the
66 Chelsio Unified Wire package.
68 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
69 package must be manually flashed via cxgbetool available in FreeBSD source
72 Instructions on how to manually flash the firmware are given in section
73 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
76 Pre-Installation Configuration
77 ------------------------------
82 The following options can be modified in the ``.config`` file. Please note that
83 enabling debugging options may affect system performance.
85 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
87 Toggle compilation of librte_pmd_cxgbe driver.
89 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
91 Toggle display of generic debugging messages.
93 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
95 Toggle display of registers related run-time check messages.
97 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
99 Toggle display of firmware mailbox related run-time check messages.
101 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
103 Toggle display of transmission data path run-time check messages.
105 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
107 Toggle display of receiving data path run-time check messages.
109 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
111 Toggle behaviour to prefer Throughput or Latency.
113 .. _driver-compilation:
115 Driver compilation and testing
116 ------------------------------
118 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
124 .. _linux-installation:
129 Steps to manually install the latest firmware from the downloaded Chelsio
130 Unified Wire package for Linux operating system are as follows:
132 #. Load the kernel module:
134 .. code-block:: console
138 #. Use ifconfig to get the interface name assigned to Chelsio card:
140 .. code-block:: console
142 ifconfig -a | grep "00:07:43"
146 .. code-block:: console
148 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
149 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
153 .. code-block:: console
155 cd <path_to_uwire>/tools/cxgbtool
158 #. Use cxgbtool to load the firmware config file onto the card:
160 .. code-block:: console
162 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
164 #. Use cxgbtool to load the firmware image onto the card:
166 .. code-block:: console
168 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
170 #. Unload and reload the kernel module:
172 .. code-block:: console
177 #. Verify with ethtool:
179 .. code-block:: console
181 ethtool -i p1p1 | grep "firmware"
185 .. code-block:: console
187 firmware-version: 1.17.14.0, TP 0.1.4.9
192 This section demonstrates how to launch **testpmd** with Chelsio
193 devices managed by librte_pmd_cxgbe in Linux operating system.
195 #. Load the kernel module:
197 .. code-block:: console
201 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
203 .. code-block:: console
209 .. code-block:: console
211 cxgb4 0000:02:00.4 p1p1: renamed from eth0
212 cxgb4 0000:02:00.4 p1p2: renamed from eth1
216 Both the interfaces of a Chelsio 2-port adapter are bound to the
217 same PCI bus address.
219 #. Unload the kernel module:
221 .. code-block:: console
223 modprobe -ar cxgb4 csiostor
227 Follow instructions available in the document
228 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
233 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
237 .. code-block:: console
240 EAL: PCI device 0000:02:00.4 on NUMA socket -1
241 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
242 EAL: PCI memory mapped at 0x7fd7c0200000
243 EAL: PCI memory mapped at 0x7fd77cdfd000
244 EAL: PCI memory mapped at 0x7fd7c10b7000
245 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
246 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
247 Interactive-mode selected
248 Configuring Port 0 (socket 0)
249 Port 0: 00:07:43:2D:EA:C0
250 Configuring Port 1 (socket 0)
251 Port 1: 00:07:43:2D:EA:C8
252 Checking link statuses...
253 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
254 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
255 Port 0 Link Up - speed 10000 Mbps - full-duplex
256 Port 1 Link Up - speed 10000 Mbps - full-duplex
262 Flow control pause TX/RX is disabled by default and can be enabled via
263 testpmd. Refer section :ref:`flow-control` for more details.
268 .. _freebsd-installation:
273 Steps to manually install the latest firmware from the downloaded Chelsio
274 Unified Wire package for FreeBSD operating system are as follows:
276 #. Load the kernel module:
278 .. code-block:: console
282 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
284 .. code-block:: console
290 .. code-block:: console
292 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
293 cxl0: <port 0> on t5nex0
294 cxl1: <port 1> on t5nex0
295 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
297 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
299 #. Install cxgbetool from FreeBSD source repository:
301 .. code-block:: console
303 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
306 #. Use cxgbetool to load the firmware image onto the card:
308 .. code-block:: console
310 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
312 #. Unload and reload the kernel module:
314 .. code-block:: console
319 #. Verify with sysctl:
321 .. code-block:: console
323 sysctl -a | grep "t5nex" | grep "firmware"
327 .. code-block:: console
329 dev.t5nex.0.firmware_version: 1.17.14.0
334 This section demonstrates how to launch **testpmd** with Chelsio
335 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
337 #. Change to DPDK source directory where the target has been compiled in
338 section :ref:`driver-compilation`:
340 .. code-block:: console
342 cd <DPDK-source-directory>
344 #. Copy the contigmem kernel module to /boot/kernel directory:
346 .. code-block:: console
348 cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
350 #. Add the following lines to /boot/loader.conf:
352 .. code-block:: console
354 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
355 hw.contigmem.num_buffers=2
356 hw.contigmem.buffer_size=1073741824
357 # load contigmem module during boot process
360 The above lines load the contigmem kernel module during boot process and
361 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
362 This is to avoid issues with potential memory fragmentation during later
363 system up time, which may result in failure of allocating the contiguous
364 memory required for the contigmem kernel module.
366 #. Restart the system and ensure the contigmem module is loaded successfully:
368 .. code-block:: console
371 kldstat | grep "contigmem"
375 .. code-block:: console
377 2 1 0xffffffff817f1000 3118 contigmem.ko
379 #. Repeat step 1 to ensure that you are in the DPDK source directory.
381 #. Load the cxgbe kernel module:
383 .. code-block:: console
387 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
389 .. code-block:: console
391 pciconf -l | grep "t5nex"
395 .. code-block:: console
397 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
399 In the above example, the t5nex0 is bound to 2:0:4 bus address.
403 Both the interfaces of a Chelsio 2-port adapter are bound to the
404 same PCI bus address.
406 #. Unload the kernel module:
408 .. code-block:: console
412 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
414 .. code-block:: console
416 kenv hw.nic_uio.bdfs="2:0:4"
418 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
423 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
425 #. Load nic_uio kernel driver:
427 .. code-block:: console
429 kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
431 #. Start testpmd with basic parameters:
433 .. code-block:: console
435 ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
439 .. code-block:: console
442 EAL: PCI device 0000:02:00.4 on NUMA socket 0
443 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
444 EAL: PCI memory mapped at 0x8007ec000
445 EAL: PCI memory mapped at 0x842800000
446 EAL: PCI memory mapped at 0x80086c000
447 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
448 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
449 Interactive-mode selected
450 Configuring Port 0 (socket 0)
451 Port 0: 00:07:43:2D:EA:C0
452 Configuring Port 1 (socket 0)
453 Port 1: 00:07:43:2D:EA:C8
454 Checking link statuses...
455 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
456 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
457 Port 0 Link Up - speed 10000 Mbps - full-duplex
458 Port 1 Link Up - speed 10000 Mbps - full-duplex
464 Flow control pause TX/RX is disabled by default and can be enabled via
465 testpmd. Refer section :ref:`flow-control` for more details.
467 Sample Application Notes
468 ------------------------
472 Enable/Disable Flow Control
473 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
475 Flow control pause TX/RX is disabled by default and can be enabled via
478 .. code-block:: console
480 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
481 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
483 To disable again, run:
485 .. code-block:: console
487 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
488 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
493 There are two ways to enable sending and receiving of jumbo frames via testpmd.
494 One method involves using the **mtu** command, which changes the mtu of an
495 individual port without having to stop the selected port. Another method
496 involves stopping all the ports first and then running **max-pkt-len** command
497 to configure the mtu of all the ports with a single command.
499 - To configure each port individually, run the mtu command as follows:
501 .. code-block:: console
503 testpmd> port config mtu 0 9000
504 testpmd> port config mtu 1 9000
506 - To configure all the ports at once, stop all the ports first and run the
507 max-pkt-len command as follows:
509 .. code-block:: console
511 testpmd> port stop all
512 testpmd> port config all max-pkt-len 9000