1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2014-2018 Chelsio Communications.
8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
10 has support for the latest Linux and FreeBSD operating systems.
12 CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions
13 and has support for the latest Linux operating systems.
15 More information can be found at `Chelsio Communications Official Website
16 <http://www.chelsio.com>`_.
21 CXGBE and CXGBEVF PMD has support for:
23 - Multiple queues for TX and RX
24 - Receiver Side Steering (RSS)
25 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
26 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
31 - Port hardware statistics
33 - Flow API - Support for both Wildcard (LE-TCAM) and Exact (HASH) match filters.
38 The Chelsio Terminator series of devices provide two/four ports but
39 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
40 itself as a PCI driver that allocates one Ethernet device per detected
43 For this reason, one cannot whitelist/blacklist a single port without
44 whitelisting/blacklisting the other ports on the same device.
48 Supported Chelsio T5 NICs
49 -------------------------
52 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
53 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
54 - Other T5 NICs: T522-CR
58 Supported Chelsio T6 NICs
59 -------------------------
61 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
62 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
64 Supported SR-IOV Chelsio NICs
65 -----------------------------
67 SR-IOV virtual functions are supported on all the Chelsio NICs listed
68 in :ref:`t5-nics` and :ref:`t6-nics`.
73 - Requires firmware version **1.17.14.0** and higher. Visit
74 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
75 bundled with the latest Chelsio Unified Wire package.
77 For Linux, installing and loading the latest cxgb4 kernel driver from the
78 Chelsio Unified Wire package should get you the latest firmware. More
79 information can be obtained from the User Guide that is bundled with the
80 Chelsio Unified Wire package.
82 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
83 package must be manually flashed via cxgbetool available in FreeBSD source
86 Instructions on how to manually flash the firmware are given in section
87 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
90 Pre-Installation Configuration
91 ------------------------------
96 The following options can be modified in the ``.config`` file. Please note that
97 enabling debugging options may affect system performance.
99 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
101 Toggle compilation of librte_pmd_cxgbe driver.
105 This controls compilation of both CXGBE and CXGBEVF PMD.
107 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
109 Toggle behavior to prefer Throughput or Latency.
114 The following ``devargs`` options can be enabled at runtime. They must
115 be passed as part of EAL arguments. For example,
117 .. code-block:: console
119 testpmd -w 02:00.4,keep_ovlan=1 -- -i
121 - ``keep_ovlan`` (default **0**)
123 Toggle behavior to keep/strip outer VLAN in Q-in-Q packets. If
124 enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise,
125 the outer VLAN tag is stripped in Q-in-Q packets.
127 - ``force_link_up`` (default **0**)
129 When set to 1, CXGBEVF PMD always forces link as up for all VFs on
130 underlying Chelsio NICs. This enables multiple VFs on the same NIC
131 to send traffic to each other even when the physical link is down.
133 .. _driver-compilation:
135 Driver compilation and testing
136 ------------------------------
138 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
144 .. _linux-installation:
149 Steps to manually install the latest firmware from the downloaded Chelsio
150 Unified Wire package for Linux operating system are as follows:
152 #. Load the kernel module:
154 .. code-block:: console
158 #. Use ifconfig to get the interface name assigned to Chelsio card:
160 .. code-block:: console
162 ifconfig -a | grep "00:07:43"
166 .. code-block:: console
168 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
169 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
173 .. code-block:: console
175 cd <path_to_uwire>/tools/cxgbtool
178 #. Use cxgbtool to load the firmware config file onto the card:
180 .. code-block:: console
182 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
184 #. Use cxgbtool to load the firmware image onto the card:
186 .. code-block:: console
188 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
190 #. Unload and reload the kernel module:
192 .. code-block:: console
197 #. Verify with ethtool:
199 .. code-block:: console
201 ethtool -i p1p1 | grep "firmware"
205 .. code-block:: console
207 firmware-version: 1.17.14.0, TP 0.1.4.9
212 This section demonstrates how to launch **testpmd** with Chelsio
213 devices managed by librte_pmd_cxgbe in Linux operating system.
215 #. Load the kernel module:
217 .. code-block:: console
221 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
223 .. code-block:: console
229 .. code-block:: console
231 cxgb4 0000:02:00.4 p1p1: renamed from eth0
232 cxgb4 0000:02:00.4 p1p2: renamed from eth1
236 Both the interfaces of a Chelsio 2-port adapter are bound to the
237 same PCI bus address.
239 #. Unload the kernel module:
241 .. code-block:: console
243 modprobe -ar cxgb4 csiostor
247 Follow instructions available in the document
248 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
253 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
257 .. code-block:: console
260 EAL: PCI device 0000:02:00.4 on NUMA socket -1
261 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
262 EAL: PCI memory mapped at 0x7fd7c0200000
263 EAL: PCI memory mapped at 0x7fd77cdfd000
264 EAL: PCI memory mapped at 0x7fd7c10b7000
265 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
266 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
267 Interactive-mode selected
268 Configuring Port 0 (socket 0)
269 Port 0: 00:07:43:2D:EA:C0
270 Configuring Port 1 (socket 0)
271 Port 1: 00:07:43:2D:EA:C8
272 Checking link statuses...
273 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
274 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
275 Port 0 Link Up - speed 10000 Mbps - full-duplex
276 Port 1 Link Up - speed 10000 Mbps - full-duplex
282 Flow control pause TX/RX is disabled by default and can be enabled via
283 testpmd. Refer section :ref:`flow-control` for more details.
285 Configuring SR-IOV Virtual Functions
286 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
288 This section demonstrates how to enable SR-IOV virtual functions
289 on Chelsio NICs and demonstrates how to run testpmd with SR-IOV
292 #. Load the kernel module:
294 .. code-block:: console
298 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
300 .. code-block:: console
306 .. code-block:: console
308 cxgb4 0000:02:00.4 p1p1: renamed from eth0
309 cxgb4 0000:02:00.4 p1p2: renamed from eth1
313 Both the interfaces of a Chelsio 2-port adapter are bound to the
314 same PCI bus address.
316 #. Use ifconfig to get the interface name assigned to Chelsio card:
318 .. code-block:: console
320 ifconfig -a | grep "00:07:43"
324 .. code-block:: console
326 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
327 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
329 #. Bring up the interfaces:
331 .. code-block:: console
336 #. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for
337 SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3.
338 To instantiate one SR-IOV VF on each PF0 and PF1:
340 .. code-block:: console
342 echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
343 echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs
345 #. Get the PCI bus addresses of the virtual functions:
347 .. code-block:: console
349 lspci | grep -i "Chelsio" | grep -i "VF"
353 .. code-block:: console
355 02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
356 02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
360 Follow instructions available in the document
361 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
362 to bind virtual functions and run testpmd.
366 .. code-block:: console
369 EAL: PCI device 0000:02:01.0 on NUMA socket 0
370 EAL: probe driver: 1425:5803 net_cxgbevf
371 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
372 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
373 PMD: rte_cxgbe_pmd: Chelsio rev 0
374 PMD: rte_cxgbe_pmd: No bootstrap loaded
375 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
376 PMD: rte_cxgbe_pmd: 0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP
377 EAL: PCI device 0000:02:01.1 on NUMA socket 0
378 EAL: probe driver: 1425:5803 net_cxgbevf
379 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
380 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
381 PMD: rte_cxgbe_pmd: Chelsio rev 0
382 PMD: rte_cxgbe_pmd: No bootstrap loaded
383 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
384 PMD: rte_cxgbe_pmd: 0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP
385 Configuring Port 0 (socket 0)
386 Port 0: 06:44:29:44:40:00
387 Configuring Port 1 (socket 0)
388 Port 1: 06:44:29:44:40:10
389 Checking link statuses...
396 .. _freebsd-installation:
401 Steps to manually install the latest firmware from the downloaded Chelsio
402 Unified Wire package for FreeBSD operating system are as follows:
404 #. Load the kernel module:
406 .. code-block:: console
410 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
412 .. code-block:: console
418 .. code-block:: console
420 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
421 cxl0: <port 0> on t5nex0
422 cxl1: <port 1> on t5nex0
423 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
425 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
427 #. Install cxgbetool from FreeBSD source repository:
429 .. code-block:: console
431 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
434 #. Use cxgbetool to load the firmware image onto the card:
436 .. code-block:: console
438 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
440 #. Unload and reload the kernel module:
442 .. code-block:: console
447 #. Verify with sysctl:
449 .. code-block:: console
451 sysctl -a | grep "t5nex" | grep "firmware"
455 .. code-block:: console
457 dev.t5nex.0.firmware_version: 1.17.14.0
462 This section demonstrates how to launch **testpmd** with Chelsio
463 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
465 #. Change to DPDK source directory where the target has been compiled in
466 section :ref:`driver-compilation`:
468 .. code-block:: console
470 cd <DPDK-source-directory>
472 #. Copy the contigmem kernel module to /boot/kernel directory:
474 .. code-block:: console
476 cp x86_64-native-freebsd-clang/kmod/contigmem.ko /boot/kernel/
478 #. Add the following lines to /boot/loader.conf:
480 .. code-block:: console
482 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
483 hw.contigmem.num_buffers=2
484 hw.contigmem.buffer_size=1073741824
485 # load contigmem module during boot process
488 The above lines load the contigmem kernel module during boot process and
489 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
490 This is to avoid issues with potential memory fragmentation during later
491 system up time, which may result in failure of allocating the contiguous
492 memory required for the contigmem kernel module.
494 #. Restart the system and ensure the contigmem module is loaded successfully:
496 .. code-block:: console
499 kldstat | grep "contigmem"
503 .. code-block:: console
505 2 1 0xffffffff817f1000 3118 contigmem.ko
507 #. Repeat step 1 to ensure that you are in the DPDK source directory.
509 #. Load the cxgbe kernel module:
511 .. code-block:: console
515 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
517 .. code-block:: console
519 pciconf -l | grep "t5nex"
523 .. code-block:: console
525 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
527 In the above example, the t5nex0 is bound to 2:0:4 bus address.
531 Both the interfaces of a Chelsio 2-port adapter are bound to the
532 same PCI bus address.
534 #. Unload the kernel module:
536 .. code-block:: console
540 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
542 .. code-block:: console
544 kenv hw.nic_uio.bdfs="2:0:4"
546 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
551 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
553 #. Load nic_uio kernel driver:
555 .. code-block:: console
557 kldload ./x86_64-native-freebsd-clang/kmod/nic_uio.ko
559 #. Start testpmd with basic parameters:
561 .. code-block:: console
563 ./x86_64-native-freebsd-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
567 .. code-block:: console
570 EAL: PCI device 0000:02:00.4 on NUMA socket 0
571 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
572 EAL: PCI memory mapped at 0x8007ec000
573 EAL: PCI memory mapped at 0x842800000
574 EAL: PCI memory mapped at 0x80086c000
575 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
576 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
577 Interactive-mode selected
578 Configuring Port 0 (socket 0)
579 Port 0: 00:07:43:2D:EA:C0
580 Configuring Port 1 (socket 0)
581 Port 1: 00:07:43:2D:EA:C8
582 Checking link statuses...
583 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
584 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
585 Port 0 Link Up - speed 10000 Mbps - full-duplex
586 Port 1 Link Up - speed 10000 Mbps - full-duplex
592 Flow control pause TX/RX is disabled by default and can be enabled via
593 testpmd. Refer section :ref:`flow-control` for more details.
595 Sample Application Notes
596 ------------------------
600 Enable/Disable Flow Control
601 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
603 Flow control pause TX/RX is disabled by default and can be enabled via
606 .. code-block:: console
608 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
609 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
611 To disable again, run:
613 .. code-block:: console
615 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
616 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
621 There are two ways to enable sending and receiving of jumbo frames via testpmd.
622 One method involves using the **mtu** command, which changes the mtu of an
623 individual port without having to stop the selected port. Another method
624 involves stopping all the ports first and then running **max-pkt-len** command
625 to configure the mtu of all the ports with a single command.
627 - To configure each port individually, run the mtu command as follows:
629 .. code-block:: console
631 testpmd> port config mtu 0 9000
632 testpmd> port config mtu 1 9000
634 - To configure all the ports at once, stop all the ports first and run the
635 max-pkt-len command as follows:
637 .. code-block:: console
639 testpmd> port stop all
640 testpmd> port config all max-pkt-len 9000