1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2014-2018 Chelsio Communications.
8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
10 has support for the latest Linux and FreeBSD operating systems.
12 CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions
13 and has support for the latest Linux operating systems.
15 More information can be found at `Chelsio Communications Official Website
16 <http://www.chelsio.com>`_.
21 CXGBE and CXGBEVF PMD has support for:
23 - Multiple queues for TX and RX
24 - Receiver Side Steering (RSS)
25 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
26 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
31 - Port hardware statistics
37 The Chelsio Terminator series of devices provide two/four ports but
38 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
39 itself as a PCI driver that allocates one Ethernet device per detected
42 For this reason, one cannot whitelist/blacklist a single port without
43 whitelisting/blacklisting the other ports on the same device.
47 Supported Chelsio T5 NICs
48 -------------------------
51 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
52 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
53 - Other T5 NICs: T522-CR
57 Supported Chelsio T6 NICs
58 -------------------------
60 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
61 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
63 Supported SR-IOV Chelsio NICs
64 -----------------------------
66 SR-IOV virtual functions are supported on all the Chelsio NICs listed
67 in :ref:`t5-nics` and :ref:`t6-nics`.
72 - Requires firmware version **1.17.14.0** and higher. Visit
73 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
74 bundled with the latest Chelsio Unified Wire package.
76 For Linux, installing and loading the latest cxgb4 kernel driver from the
77 Chelsio Unified Wire package should get you the latest firmware. More
78 information can be obtained from the User Guide that is bundled with the
79 Chelsio Unified Wire package.
81 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
82 package must be manually flashed via cxgbetool available in FreeBSD source
85 Instructions on how to manually flash the firmware are given in section
86 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
89 Pre-Installation Configuration
90 ------------------------------
95 The following options can be modified in the ``.config`` file. Please note that
96 enabling debugging options may affect system performance.
98 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
100 Toggle compilation of librte_pmd_cxgbe driver.
104 This controls compilation of both CXGBE and CXGBEVF PMD.
106 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
108 Toggle display of generic debugging messages.
110 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
112 Toggle display of registers related run-time check messages.
114 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
116 Toggle display of firmware mailbox related run-time check messages.
118 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
120 Toggle display of transmission data path run-time check messages.
122 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
124 Toggle display of receiving data path run-time check messages.
126 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
128 Toggle behaviour to prefer Throughput or Latency.
133 The following ``devargs`` options can be enabled at runtime. They must
134 be passed as part of EAL arguments. For example,
136 .. code-block:: console
138 testpmd -w 02:00.4,keep_ovlan=1 -- -i
140 - ``keep_ovlan`` (default **0**)
142 Toggle behaviour to keep/strip outer VLAN in Q-in-Q packets. If
143 enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise,
144 the outer VLAN tag is stripped in Q-in-Q packets.
146 - ``force_link_up`` (default **0**)
148 When set to 1, CXGBEVF PMD always forces link as up for all VFs on
149 underlying Chelsio NICs. This enables multiple VFs on the same NIC
150 to send traffic to each other even when the physical link is down.
152 .. _driver-compilation:
154 Driver compilation and testing
155 ------------------------------
157 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
163 .. _linux-installation:
168 Steps to manually install the latest firmware from the downloaded Chelsio
169 Unified Wire package for Linux operating system are as follows:
171 #. Load the kernel module:
173 .. code-block:: console
177 #. Use ifconfig to get the interface name assigned to Chelsio card:
179 .. code-block:: console
181 ifconfig -a | grep "00:07:43"
185 .. code-block:: console
187 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
188 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
192 .. code-block:: console
194 cd <path_to_uwire>/tools/cxgbtool
197 #. Use cxgbtool to load the firmware config file onto the card:
199 .. code-block:: console
201 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
203 #. Use cxgbtool to load the firmware image onto the card:
205 .. code-block:: console
207 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
209 #. Unload and reload the kernel module:
211 .. code-block:: console
216 #. Verify with ethtool:
218 .. code-block:: console
220 ethtool -i p1p1 | grep "firmware"
224 .. code-block:: console
226 firmware-version: 1.17.14.0, TP 0.1.4.9
231 This section demonstrates how to launch **testpmd** with Chelsio
232 devices managed by librte_pmd_cxgbe in Linux operating system.
234 #. Load the kernel module:
236 .. code-block:: console
240 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
242 .. code-block:: console
248 .. code-block:: console
250 cxgb4 0000:02:00.4 p1p1: renamed from eth0
251 cxgb4 0000:02:00.4 p1p2: renamed from eth1
255 Both the interfaces of a Chelsio 2-port adapter are bound to the
256 same PCI bus address.
258 #. Unload the kernel module:
260 .. code-block:: console
262 modprobe -ar cxgb4 csiostor
266 Follow instructions available in the document
267 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
272 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
276 .. code-block:: console
279 EAL: PCI device 0000:02:00.4 on NUMA socket -1
280 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
281 EAL: PCI memory mapped at 0x7fd7c0200000
282 EAL: PCI memory mapped at 0x7fd77cdfd000
283 EAL: PCI memory mapped at 0x7fd7c10b7000
284 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
285 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
286 Interactive-mode selected
287 Configuring Port 0 (socket 0)
288 Port 0: 00:07:43:2D:EA:C0
289 Configuring Port 1 (socket 0)
290 Port 1: 00:07:43:2D:EA:C8
291 Checking link statuses...
292 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
293 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
294 Port 0 Link Up - speed 10000 Mbps - full-duplex
295 Port 1 Link Up - speed 10000 Mbps - full-duplex
301 Flow control pause TX/RX is disabled by default and can be enabled via
302 testpmd. Refer section :ref:`flow-control` for more details.
304 Configuring SR-IOV Virtual Functions
305 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
307 This section demonstrates how to enable SR-IOV virtual functions
308 on Chelsio NICs and demonstrates how to run testpmd with SR-IOV
311 #. Load the kernel module:
313 .. code-block:: console
317 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
319 .. code-block:: console
325 .. code-block:: console
327 cxgb4 0000:02:00.4 p1p1: renamed from eth0
328 cxgb4 0000:02:00.4 p1p2: renamed from eth1
332 Both the interfaces of a Chelsio 2-port adapter are bound to the
333 same PCI bus address.
335 #. Use ifconfig to get the interface name assigned to Chelsio card:
337 .. code-block:: console
339 ifconfig -a | grep "00:07:43"
343 .. code-block:: console
345 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
346 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
348 #. Bring up the interfaces:
350 .. code-block:: console
355 #. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for
356 SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3.
357 To instantiate one SR-IOV VF on each PF0 and PF1:
359 .. code-block:: console
361 echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
362 echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs
364 #. Get the PCI bus addresses of the virtual functions:
366 .. code-block:: console
368 lspci | grep -i "Chelsio" | grep -i "VF"
372 .. code-block:: console
374 02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
375 02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
379 Follow instructions available in the document
380 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
381 to bind virtual functions and run testpmd.
385 .. code-block:: console
388 EAL: PCI device 0000:02:01.0 on NUMA socket 0
389 EAL: probe driver: 1425:5803 net_cxgbevf
390 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
391 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
392 PMD: rte_cxgbe_pmd: Chelsio rev 0
393 PMD: rte_cxgbe_pmd: No bootstrap loaded
394 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
395 PMD: rte_cxgbe_pmd: 0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP
396 EAL: PCI device 0000:02:01.1 on NUMA socket 0
397 EAL: probe driver: 1425:5803 net_cxgbevf
398 PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
399 PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
400 PMD: rte_cxgbe_pmd: Chelsio rev 0
401 PMD: rte_cxgbe_pmd: No bootstrap loaded
402 PMD: rte_cxgbe_pmd: No Expansion ROM loaded
403 PMD: rte_cxgbe_pmd: 0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP
404 Configuring Port 0 (socket 0)
405 Port 0: 06:44:29:44:40:00
406 Configuring Port 1 (socket 0)
407 Port 1: 06:44:29:44:40:10
408 Checking link statuses...
415 .. _freebsd-installation:
420 Steps to manually install the latest firmware from the downloaded Chelsio
421 Unified Wire package for FreeBSD operating system are as follows:
423 #. Load the kernel module:
425 .. code-block:: console
429 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
431 .. code-block:: console
437 .. code-block:: console
439 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
440 cxl0: <port 0> on t5nex0
441 cxl1: <port 1> on t5nex0
442 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
444 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
446 #. Install cxgbetool from FreeBSD source repository:
448 .. code-block:: console
450 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
453 #. Use cxgbetool to load the firmware image onto the card:
455 .. code-block:: console
457 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
459 #. Unload and reload the kernel module:
461 .. code-block:: console
466 #. Verify with sysctl:
468 .. code-block:: console
470 sysctl -a | grep "t5nex" | grep "firmware"
474 .. code-block:: console
476 dev.t5nex.0.firmware_version: 1.17.14.0
481 This section demonstrates how to launch **testpmd** with Chelsio
482 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
484 #. Change to DPDK source directory where the target has been compiled in
485 section :ref:`driver-compilation`:
487 .. code-block:: console
489 cd <DPDK-source-directory>
491 #. Copy the contigmem kernel module to /boot/kernel directory:
493 .. code-block:: console
495 cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
497 #. Add the following lines to /boot/loader.conf:
499 .. code-block:: console
501 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
502 hw.contigmem.num_buffers=2
503 hw.contigmem.buffer_size=1073741824
504 # load contigmem module during boot process
507 The above lines load the contigmem kernel module during boot process and
508 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
509 This is to avoid issues with potential memory fragmentation during later
510 system up time, which may result in failure of allocating the contiguous
511 memory required for the contigmem kernel module.
513 #. Restart the system and ensure the contigmem module is loaded successfully:
515 .. code-block:: console
518 kldstat | grep "contigmem"
522 .. code-block:: console
524 2 1 0xffffffff817f1000 3118 contigmem.ko
526 #. Repeat step 1 to ensure that you are in the DPDK source directory.
528 #. Load the cxgbe kernel module:
530 .. code-block:: console
534 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
536 .. code-block:: console
538 pciconf -l | grep "t5nex"
542 .. code-block:: console
544 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
546 In the above example, the t5nex0 is bound to 2:0:4 bus address.
550 Both the interfaces of a Chelsio 2-port adapter are bound to the
551 same PCI bus address.
553 #. Unload the kernel module:
555 .. code-block:: console
559 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
561 .. code-block:: console
563 kenv hw.nic_uio.bdfs="2:0:4"
565 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
570 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
572 #. Load nic_uio kernel driver:
574 .. code-block:: console
576 kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
578 #. Start testpmd with basic parameters:
580 .. code-block:: console
582 ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
586 .. code-block:: console
589 EAL: PCI device 0000:02:00.4 on NUMA socket 0
590 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
591 EAL: PCI memory mapped at 0x8007ec000
592 EAL: PCI memory mapped at 0x842800000
593 EAL: PCI memory mapped at 0x80086c000
594 PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
595 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
596 Interactive-mode selected
597 Configuring Port 0 (socket 0)
598 Port 0: 00:07:43:2D:EA:C0
599 Configuring Port 1 (socket 0)
600 Port 1: 00:07:43:2D:EA:C8
601 Checking link statuses...
602 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
603 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
604 Port 0 Link Up - speed 10000 Mbps - full-duplex
605 Port 1 Link Up - speed 10000 Mbps - full-duplex
611 Flow control pause TX/RX is disabled by default and can be enabled via
612 testpmd. Refer section :ref:`flow-control` for more details.
614 Sample Application Notes
615 ------------------------
619 Enable/Disable Flow Control
620 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
622 Flow control pause TX/RX is disabled by default and can be enabled via
625 .. code-block:: console
627 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
628 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
630 To disable again, run:
632 .. code-block:: console
634 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
635 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
640 There are two ways to enable sending and receiving of jumbo frames via testpmd.
641 One method involves using the **mtu** command, which changes the mtu of an
642 individual port without having to stop the selected port. Another method
643 involves stopping all the ports first and then running **max-pkt-len** command
644 to configure the mtu of all the ports with a single command.
646 - To configure each port individually, run the mtu command as follows:
648 .. code-block:: console
650 testpmd> port config mtu 0 9000
651 testpmd> port config mtu 1 9000
653 - To configure all the ports at once, stop all the ports first and run the
654 max-pkt-len command as follows:
656 .. code-block:: console
658 testpmd> port stop all
659 testpmd> port config all max-pkt-len 9000