1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2014-2018 Chelsio Communications.
8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
10 has support for the latest Linux and FreeBSD operating systems.
12 CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions
13 and has support for the latest Linux operating systems.
15 More information can be found at `Chelsio Communications Official Website
16 <http://www.chelsio.com>`_.
21 CXGBE and CXGBEVF PMD has support for:
23 - Multiple queues for TX and RX
24 - Receiver Side Steering (RSS)
25 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
26 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
31 - Port hardware statistics
33 - Flow API - Support for both Wildcard (LE-TCAM) and Exact (HASH) match filters.
38 The Chelsio Terminator series of devices provide two/four ports but
39 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
40 itself as a PCI driver that allocates one Ethernet device per detected
43 For this reason, one cannot whitelist/blacklist a single port without
44 whitelisting/blacklisting the other ports on the same device.
48 Supported Chelsio T5 NICs
49 -------------------------
52 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
53 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
54 - Other T5 NICs: T522-CR
58 Supported Chelsio T6 NICs
59 -------------------------
61 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
62 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
64 Supported SR-IOV Chelsio NICs
65 -----------------------------
67 SR-IOV virtual functions are supported on all the Chelsio NICs listed
68 in :ref:`t5-nics` and :ref:`t6-nics`.
73 - Requires firmware version **1.17.14.0** and higher. Visit
74 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
75 bundled with the latest Chelsio Unified Wire package.
77 For Linux, installing and loading the latest cxgb4 kernel driver from the
78 Chelsio Unified Wire package should get you the latest firmware. More
79 information can be obtained from the User Guide that is bundled with the
80 Chelsio Unified Wire package.
82 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
83 package must be manually flashed via cxgbetool available in FreeBSD source
86 Instructions on how to manually flash the firmware are given in section
87 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
90 Pre-Installation Configuration
91 ------------------------------
96 The following options can be modified in the ``.config`` file. Please note that
97 enabling debugging options may affect system performance.
99 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
101 Toggle compilation of librte_pmd_cxgbe driver.
105 This controls compilation of both CXGBE and CXGBEVF PMD.
107 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
109 Toggle behavior to prefer Throughput or Latency.
114 The following ``devargs`` options can be enabled at runtime. They must
115 be passed as part of EAL arguments. For example,
117 .. code-block:: console
119 testpmd -w 02:00.4,keep_ovlan=1 -- -i
121 Common Runtime Options
122 ^^^^^^^^^^^^^^^^^^^^^^
124 - ``keep_ovlan`` (default **0**)
126 Toggle behavior to keep/strip outer VLAN in Q-in-Q packets. If
127 enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise,
128 the outer VLAN tag is stripped in Q-in-Q packets.
130 CXGBE VF Only Runtime Options
131 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
133 - ``force_link_up`` (default **0**)
135 When set to 1, CXGBEVF PMD always forces link as up for all VFs on
136 underlying Chelsio NICs. This enables multiple VFs on the same NIC
137 to send traffic to each other even when the physical link is down.
139 .. _driver-compilation:
141 Driver compilation and testing
142 ------------------------------
144 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
150 .. _linux-installation:
155 Steps to manually install the latest firmware from the downloaded Chelsio
156 Unified Wire package for Linux operating system are as follows:
158 #. Load the kernel module:
160 .. code-block:: console
164 #. Use ifconfig to get the interface name assigned to Chelsio card:
166 .. code-block:: console
168 ifconfig -a | grep "00:07:43"
172 .. code-block:: console
174 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
175 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
179 .. code-block:: console
181 cd <path_to_uwire>/tools/cxgbtool
184 #. Use cxgbtool to load the firmware config file onto the card:
186 .. code-block:: console
188 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
190 #. Use cxgbtool to load the firmware image onto the card:
192 .. code-block:: console
194 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
196 #. Unload and reload the kernel module:
198 .. code-block:: console
203 #. Verify with ethtool:
205 .. code-block:: console
207 ethtool -i p1p1 | grep "firmware"
211 .. code-block:: console
213 firmware-version: 1.17.14.0, TP 0.1.4.9
218 This section demonstrates how to launch **testpmd** with Chelsio
219 devices managed by librte_pmd_cxgbe in Linux operating system.
221 #. Load the kernel module:
223 .. code-block:: console
227 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
229 .. code-block:: console
235 .. code-block:: console
237 cxgb4 0000:02:00.4 p1p1: renamed from eth0
238 cxgb4 0000:02:00.4 p1p2: renamed from eth1
242 Both the interfaces of a Chelsio 2-port adapter are bound to the
243 same PCI bus address.
245 #. Unload the kernel module:
247 .. code-block:: console
249 modprobe -ar cxgb4 csiostor
253 Follow instructions available in the document
254 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
259 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
263 .. code-block:: console
266 EAL: PCI device 0000:02:00.4 on NUMA socket -1
267 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
268 EAL: PCI memory mapped at 0x7fd7c0200000
269 EAL: PCI memory mapped at 0x7fd77cdfd000
270 EAL: PCI memory mapped at 0x7fd7c10b7000
271 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
272 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
273 Interactive-mode selected
274 Configuring Port 0 (socket 0)
275 Port 0: 00:07:43:2D:EA:C0
276 Configuring Port 1 (socket 0)
277 Port 1: 00:07:43:2D:EA:C8
278 Checking link statuses...
279 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
280 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
281 Port 0 Link Up - speed 10000 Mbps - full-duplex
282 Port 1 Link Up - speed 10000 Mbps - full-duplex
288 Flow control pause TX/RX is disabled by default and can be enabled via
289 testpmd. Refer section :ref:`flow-control` for more details.
291 Configuring SR-IOV Virtual Functions
292 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
294 This section demonstrates how to enable SR-IOV virtual functions
295 on Chelsio NICs and demonstrates how to run testpmd with SR-IOV
298 #. Load the kernel module:
300 .. code-block:: console
304 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
306 .. code-block:: console
312 .. code-block:: console
314 cxgb4 0000:02:00.4 p1p1: renamed from eth0
315 cxgb4 0000:02:00.4 p1p2: renamed from eth1
319 Both the interfaces of a Chelsio 2-port adapter are bound to the
320 same PCI bus address.
322 #. Use ifconfig to get the interface name assigned to Chelsio card:
324 .. code-block:: console
326 ifconfig -a | grep "00:07:43"
330 .. code-block:: console
332 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
333 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
335 #. Bring up the interfaces:
337 .. code-block:: console
342 #. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for
343 SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3.
344 To instantiate one SR-IOV VF on each PF0 and PF1:
346 .. code-block:: console
348 echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
349 echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs
351 #. Get the PCI bus addresses of the virtual functions:
353 .. code-block:: console
355 lspci | grep -i "Chelsio" | grep -i "VF"
359 .. code-block:: console
361 02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
362 02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
366 Follow instructions available in the document
367 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
368 to bind virtual functions and run testpmd.
372 .. code-block:: console
375 EAL: PCI device 0000:02:01.0 on NUMA socket 0
376 EAL: probe driver: 1425:5803 net_cxgbevf
377 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
378 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
379 PMD: rte_cxgbe_pmd: Chelsio rev 0
380 PMD: rte_cxgbe_pmd: No bootstrap loaded
381 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
382 PMD: rte_cxgbe_pmd: 0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP
383 EAL: PCI device 0000:02:01.1 on NUMA socket 0
384 EAL: probe driver: 1425:5803 net_cxgbevf
385 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
386 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
387 PMD: rte_cxgbe_pmd: Chelsio rev 0
388 PMD: rte_cxgbe_pmd: No bootstrap loaded
389 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
390 PMD: rte_cxgbe_pmd: 0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP
391 Configuring Port 0 (socket 0)
392 Port 0: 06:44:29:44:40:00
393 Configuring Port 1 (socket 0)
394 Port 1: 06:44:29:44:40:10
395 Checking link statuses...
402 .. _freebsd-installation:
407 Steps to manually install the latest firmware from the downloaded Chelsio
408 Unified Wire package for FreeBSD operating system are as follows:
410 #. Load the kernel module:
412 .. code-block:: console
416 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
418 .. code-block:: console
424 .. code-block:: console
426 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
427 cxl0: <port 0> on t5nex0
428 cxl1: <port 1> on t5nex0
429 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
431 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
433 #. Install cxgbetool from FreeBSD source repository:
435 .. code-block:: console
437 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
440 #. Use cxgbetool to load the firmware image onto the card:
442 .. code-block:: console
444 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
446 #. Unload and reload the kernel module:
448 .. code-block:: console
453 #. Verify with sysctl:
455 .. code-block:: console
457 sysctl -a | grep "t5nex" | grep "firmware"
461 .. code-block:: console
463 dev.t5nex.0.firmware_version: 1.17.14.0
468 This section demonstrates how to launch **testpmd** with Chelsio
469 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
471 #. Change to DPDK source directory where the target has been compiled in
472 section :ref:`driver-compilation`:
474 .. code-block:: console
476 cd <DPDK-source-directory>
478 #. Copy the contigmem kernel module to /boot/kernel directory:
480 .. code-block:: console
482 cp x86_64-native-freebsd-clang/kmod/contigmem.ko /boot/kernel/
484 #. Add the following lines to /boot/loader.conf:
486 .. code-block:: console
488 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
489 hw.contigmem.num_buffers=2
490 hw.contigmem.buffer_size=1073741824
491 # load contigmem module during boot process
494 The above lines load the contigmem kernel module during boot process and
495 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
496 This is to avoid issues with potential memory fragmentation during later
497 system up time, which may result in failure of allocating the contiguous
498 memory required for the contigmem kernel module.
500 #. Restart the system and ensure the contigmem module is loaded successfully:
502 .. code-block:: console
505 kldstat | grep "contigmem"
509 .. code-block:: console
511 2 1 0xffffffff817f1000 3118 contigmem.ko
513 #. Repeat step 1 to ensure that you are in the DPDK source directory.
515 #. Load the cxgbe kernel module:
517 .. code-block:: console
521 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
523 .. code-block:: console
525 pciconf -l | grep "t5nex"
529 .. code-block:: console
531 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
533 In the above example, the t5nex0 is bound to 2:0:4 bus address.
537 Both the interfaces of a Chelsio 2-port adapter are bound to the
538 same PCI bus address.
540 #. Unload the kernel module:
542 .. code-block:: console
546 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
548 .. code-block:: console
550 kenv hw.nic_uio.bdfs="2:0:4"
552 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
557 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
559 #. Load nic_uio kernel driver:
561 .. code-block:: console
563 kldload ./x86_64-native-freebsd-clang/kmod/nic_uio.ko
565 #. Start testpmd with basic parameters:
567 .. code-block:: console
569 ./x86_64-native-freebsd-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
573 .. code-block:: console
576 EAL: PCI device 0000:02:00.4 on NUMA socket 0
577 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
578 EAL: PCI memory mapped at 0x8007ec000
579 EAL: PCI memory mapped at 0x842800000
580 EAL: PCI memory mapped at 0x80086c000
581 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
582 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
583 Interactive-mode selected
584 Configuring Port 0 (socket 0)
585 Port 0: 00:07:43:2D:EA:C0
586 Configuring Port 1 (socket 0)
587 Port 1: 00:07:43:2D:EA:C8
588 Checking link statuses...
589 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
590 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
591 Port 0 Link Up - speed 10000 Mbps - full-duplex
592 Port 1 Link Up - speed 10000 Mbps - full-duplex
598 Flow control pause TX/RX is disabled by default and can be enabled via
599 testpmd. Refer section :ref:`flow-control` for more details.
601 Sample Application Notes
602 ------------------------
606 Enable/Disable Flow Control
607 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
609 Flow control pause TX/RX is disabled by default and can be enabled via
612 .. code-block:: console
614 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
615 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
617 To disable again, run:
619 .. code-block:: console
621 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
622 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
627 There are two ways to enable sending and receiving of jumbo frames via testpmd.
628 One method involves using the **mtu** command, which changes the mtu of an
629 individual port without having to stop the selected port. Another method
630 involves stopping all the ports first and then running **max-pkt-len** command
631 to configure the mtu of all the ports with a single command.
633 - To configure each port individually, run the mtu command as follows:
635 .. code-block:: console
637 testpmd> port config mtu 0 9000
638 testpmd> port config mtu 1 9000
640 - To configure all the ports at once, stop all the ports first and run the
641 max-pkt-len command as follows:
643 .. code-block:: console
645 testpmd> port stop all
646 testpmd> port config all max-pkt-len 9000