1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2014-2018 Chelsio Communications.
8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
10 has support for the latest Linux and FreeBSD operating systems.
12 CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions
13 and has support for the latest Linux operating systems.
15 More information can be found at `Chelsio Communications Official Website
16 <http://www.chelsio.com>`_.
21 CXGBE and CXGBEVF PMD has support for:
23 - Multiple queues for TX and RX
24 - Receiver Side Steering (RSS)
25 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
26 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
31 - Port hardware statistics
38 The Chelsio Terminator series of devices provide two/four ports but
39 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
40 itself as a PCI driver that allocates one Ethernet device per detected
43 For this reason, one cannot whitelist/blacklist a single port without
44 whitelisting/blacklisting the other ports on the same device.
48 Supported Chelsio T5 NICs
49 -------------------------
52 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
53 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
54 - Other T5 NICs: T522-CR
58 Supported Chelsio T6 NICs
59 -------------------------
61 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
62 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
64 Supported SR-IOV Chelsio NICs
65 -----------------------------
67 SR-IOV virtual functions are supported on all the Chelsio NICs listed
68 in :ref:`t5-nics` and :ref:`t6-nics`.
73 - Requires firmware version **1.17.14.0** and higher. Visit
74 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
75 bundled with the latest Chelsio Unified Wire package.
77 For Linux, installing and loading the latest cxgb4 kernel driver from the
78 Chelsio Unified Wire package should get you the latest firmware. More
79 information can be obtained from the User Guide that is bundled with the
80 Chelsio Unified Wire package.
82 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
83 package must be manually flashed via cxgbetool available in FreeBSD source
86 Instructions on how to manually flash the firmware are given in section
87 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
90 Pre-Installation Configuration
91 ------------------------------
96 The following options can be modified in the ``.config`` file. Please note that
97 enabling debugging options may affect system performance.
99 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
101 Toggle compilation of librte_pmd_cxgbe driver.
105 This controls compilation of both CXGBE and CXGBEVF PMD.
107 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
109 Toggle display of generic debugging messages.
111 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
113 Toggle display of registers related run-time check messages.
115 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
117 Toggle display of firmware mailbox related run-time check messages.
119 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
121 Toggle display of transmission data path run-time check messages.
123 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
125 Toggle display of receiving data path run-time check messages.
127 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
129 Toggle behaviour to prefer Throughput or Latency.
134 The following ``devargs`` options can be enabled at runtime. They must
135 be passed as part of EAL arguments. For example,
137 .. code-block:: console
139 testpmd -w 02:00.4,keep_ovlan=1 -- -i
141 - ``keep_ovlan`` (default **0**)
143 Toggle behaviour to keep/strip outer VLAN in Q-in-Q packets. If
144 enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise,
145 the outer VLAN tag is stripped in Q-in-Q packets.
147 - ``force_link_up`` (default **0**)
149 When set to 1, CXGBEVF PMD always forces link as up for all VFs on
150 underlying Chelsio NICs. This enables multiple VFs on the same NIC
151 to send traffic to each other even when the physical link is down.
153 .. _driver-compilation:
155 Driver compilation and testing
156 ------------------------------
158 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
164 .. _linux-installation:
169 Steps to manually install the latest firmware from the downloaded Chelsio
170 Unified Wire package for Linux operating system are as follows:
172 #. Load the kernel module:
174 .. code-block:: console
178 #. Use ifconfig to get the interface name assigned to Chelsio card:
180 .. code-block:: console
182 ifconfig -a | grep "00:07:43"
186 .. code-block:: console
188 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
189 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
193 .. code-block:: console
195 cd <path_to_uwire>/tools/cxgbtool
198 #. Use cxgbtool to load the firmware config file onto the card:
200 .. code-block:: console
202 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
204 #. Use cxgbtool to load the firmware image onto the card:
206 .. code-block:: console
208 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
210 #. Unload and reload the kernel module:
212 .. code-block:: console
217 #. Verify with ethtool:
219 .. code-block:: console
221 ethtool -i p1p1 | grep "firmware"
225 .. code-block:: console
227 firmware-version: 1.17.14.0, TP 0.1.4.9
232 This section demonstrates how to launch **testpmd** with Chelsio
233 devices managed by librte_pmd_cxgbe in Linux operating system.
235 #. Load the kernel module:
237 .. code-block:: console
241 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
243 .. code-block:: console
249 .. code-block:: console
251 cxgb4 0000:02:00.4 p1p1: renamed from eth0
252 cxgb4 0000:02:00.4 p1p2: renamed from eth1
256 Both the interfaces of a Chelsio 2-port adapter are bound to the
257 same PCI bus address.
259 #. Unload the kernel module:
261 .. code-block:: console
263 modprobe -ar cxgb4 csiostor
267 Follow instructions available in the document
268 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
273 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
277 .. code-block:: console
280 EAL: PCI device 0000:02:00.4 on NUMA socket -1
281 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
282 EAL: PCI memory mapped at 0x7fd7c0200000
283 EAL: PCI memory mapped at 0x7fd77cdfd000
284 EAL: PCI memory mapped at 0x7fd7c10b7000
285 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
286 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
287 Interactive-mode selected
288 Configuring Port 0 (socket 0)
289 Port 0: 00:07:43:2D:EA:C0
290 Configuring Port 1 (socket 0)
291 Port 1: 00:07:43:2D:EA:C8
292 Checking link statuses...
293 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
294 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
295 Port 0 Link Up - speed 10000 Mbps - full-duplex
296 Port 1 Link Up - speed 10000 Mbps - full-duplex
302 Flow control pause TX/RX is disabled by default and can be enabled via
303 testpmd. Refer section :ref:`flow-control` for more details.
305 Configuring SR-IOV Virtual Functions
306 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
308 This section demonstrates how to enable SR-IOV virtual functions
309 on Chelsio NICs and demonstrates how to run testpmd with SR-IOV
312 #. Load the kernel module:
314 .. code-block:: console
318 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
320 .. code-block:: console
326 .. code-block:: console
328 cxgb4 0000:02:00.4 p1p1: renamed from eth0
329 cxgb4 0000:02:00.4 p1p2: renamed from eth1
333 Both the interfaces of a Chelsio 2-port adapter are bound to the
334 same PCI bus address.
336 #. Use ifconfig to get the interface name assigned to Chelsio card:
338 .. code-block:: console
340 ifconfig -a | grep "00:07:43"
344 .. code-block:: console
346 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
347 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
349 #. Bring up the interfaces:
351 .. code-block:: console
356 #. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for
357 SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3.
358 To instantiate one SR-IOV VF on each PF0 and PF1:
360 .. code-block:: console
362 echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
363 echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs
365 #. Get the PCI bus addresses of the virtual functions:
367 .. code-block:: console
369 lspci | grep -i "Chelsio" | grep -i "VF"
373 .. code-block:: console
375 02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
376 02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
380 Follow instructions available in the document
381 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
382 to bind virtual functions and run testpmd.
386 .. code-block:: console
389 EAL: PCI device 0000:02:01.0 on NUMA socket 0
390 EAL: probe driver: 1425:5803 net_cxgbevf
391 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
392 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
393 PMD: rte_cxgbe_pmd: Chelsio rev 0
394 PMD: rte_cxgbe_pmd: No bootstrap loaded
395 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
396 PMD: rte_cxgbe_pmd: 0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP
397 EAL: PCI device 0000:02:01.1 on NUMA socket 0
398 EAL: probe driver: 1425:5803 net_cxgbevf
399 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
400 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
401 PMD: rte_cxgbe_pmd: Chelsio rev 0
402 PMD: rte_cxgbe_pmd: No bootstrap loaded
403 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
404 PMD: rte_cxgbe_pmd: 0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP
405 Configuring Port 0 (socket 0)
406 Port 0: 06:44:29:44:40:00
407 Configuring Port 1 (socket 0)
408 Port 1: 06:44:29:44:40:10
409 Checking link statuses...
416 .. _freebsd-installation:
421 Steps to manually install the latest firmware from the downloaded Chelsio
422 Unified Wire package for FreeBSD operating system are as follows:
424 #. Load the kernel module:
426 .. code-block:: console
430 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
432 .. code-block:: console
438 .. code-block:: console
440 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
441 cxl0: <port 0> on t5nex0
442 cxl1: <port 1> on t5nex0
443 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
445 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
447 #. Install cxgbetool from FreeBSD source repository:
449 .. code-block:: console
451 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
454 #. Use cxgbetool to load the firmware image onto the card:
456 .. code-block:: console
458 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
460 #. Unload and reload the kernel module:
462 .. code-block:: console
467 #. Verify with sysctl:
469 .. code-block:: console
471 sysctl -a | grep "t5nex" | grep "firmware"
475 .. code-block:: console
477 dev.t5nex.0.firmware_version: 1.17.14.0
482 This section demonstrates how to launch **testpmd** with Chelsio
483 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
485 #. Change to DPDK source directory where the target has been compiled in
486 section :ref:`driver-compilation`:
488 .. code-block:: console
490 cd <DPDK-source-directory>
492 #. Copy the contigmem kernel module to /boot/kernel directory:
494 .. code-block:: console
496 cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
498 #. Add the following lines to /boot/loader.conf:
500 .. code-block:: console
502 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
503 hw.contigmem.num_buffers=2
504 hw.contigmem.buffer_size=1073741824
505 # load contigmem module during boot process
508 The above lines load the contigmem kernel module during boot process and
509 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
510 This is to avoid issues with potential memory fragmentation during later
511 system up time, which may result in failure of allocating the contiguous
512 memory required for the contigmem kernel module.
514 #. Restart the system and ensure the contigmem module is loaded successfully:
516 .. code-block:: console
519 kldstat | grep "contigmem"
523 .. code-block:: console
525 2 1 0xffffffff817f1000 3118 contigmem.ko
527 #. Repeat step 1 to ensure that you are in the DPDK source directory.
529 #. Load the cxgbe kernel module:
531 .. code-block:: console
535 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
537 .. code-block:: console
539 pciconf -l | grep "t5nex"
543 .. code-block:: console
545 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
547 In the above example, the t5nex0 is bound to 2:0:4 bus address.
551 Both the interfaces of a Chelsio 2-port adapter are bound to the
552 same PCI bus address.
554 #. Unload the kernel module:
556 .. code-block:: console
560 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
562 .. code-block:: console
564 kenv hw.nic_uio.bdfs="2:0:4"
566 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
571 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
573 #. Load nic_uio kernel driver:
575 .. code-block:: console
577 kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
579 #. Start testpmd with basic parameters:
581 .. code-block:: console
583 ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
587 .. code-block:: console
590 EAL: PCI device 0000:02:00.4 on NUMA socket 0
591 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
592 EAL: PCI memory mapped at 0x8007ec000
593 EAL: PCI memory mapped at 0x842800000
594 EAL: PCI memory mapped at 0x80086c000
595 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
596 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
597 Interactive-mode selected
598 Configuring Port 0 (socket 0)
599 Port 0: 00:07:43:2D:EA:C0
600 Configuring Port 1 (socket 0)
601 Port 1: 00:07:43:2D:EA:C8
602 Checking link statuses...
603 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
604 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
605 Port 0 Link Up - speed 10000 Mbps - full-duplex
606 Port 1 Link Up - speed 10000 Mbps - full-duplex
612 Flow control pause TX/RX is disabled by default and can be enabled via
613 testpmd. Refer section :ref:`flow-control` for more details.
615 Sample Application Notes
616 ------------------------
620 Enable/Disable Flow Control
621 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
623 Flow control pause TX/RX is disabled by default and can be enabled via
626 .. code-block:: console
628 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
629 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
631 To disable again, run:
633 .. code-block:: console
635 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
636 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
641 There are two ways to enable sending and receiving of jumbo frames via testpmd.
642 One method involves using the **mtu** command, which changes the mtu of an
643 individual port without having to stop the selected port. Another method
644 involves stopping all the ports first and then running **max-pkt-len** command
645 to configure the mtu of all the ports with a single command.
647 - To configure each port individually, run the mtu command as follows:
649 .. code-block:: console
651 testpmd> port config mtu 0 9000
652 testpmd> port config mtu 1 9000
654 - To configure all the ports at once, stop all the ports first and run the
655 max-pkt-len command as follows:
657 .. code-block:: console
659 testpmd> port stop all
660 testpmd> port config all max-pkt-len 9000