2 Copyright 2015-2017 Chelsio Communications.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Chelsio Communications nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 CXGBE Poll Mode Driver
32 ======================
34 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
35 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
36 has support for the latest Linux and FreeBSD operating systems.
38 More information can be found at `Chelsio Communications Official Website
39 <http://www.chelsio.com>`_.
44 CXGBE PMD has support for:
46 - Multiple queues for TX and RX
47 - Receiver Side Steering (RSS)
48 Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
49 For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
54 - Port hardware statistics
60 The Chelsio Terminator series of devices provide two/four ports but
61 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
62 itself as a PCI driver that allocates one Ethernet device per detected
65 For this reason, one cannot whitelist/blacklist a single port without
66 whitelisting/blacklisting the other ports on the same device.
68 Supported Chelsio T5 NICs
69 -------------------------
72 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
73 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
74 - Other T5 NICs: T522-CR
76 Supported Chelsio T6 NICs
77 -------------------------
79 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
80 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
85 - Requires firmware version **1.16.43.0** and higher. Visit
86 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
87 bundled with the latest Chelsio Unified Wire package.
89 For Linux, installing and loading the latest cxgb4 kernel driver from the
90 Chelsio Unified Wire package should get you the latest firmware. More
91 information can be obtained from the User Guide that is bundled with the
92 Chelsio Unified Wire package.
94 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
95 package must be manually flashed via cxgbetool available in FreeBSD source
98 Instructions on how to manually flash the firmware are given in section
99 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
102 Pre-Installation Configuration
103 ------------------------------
108 The following options can be modified in the ``.config`` file. Please note that
109 enabling debugging options may affect system performance.
111 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
113 Toggle compilation of librte_pmd_cxgbe driver.
115 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
117 Toggle display of generic debugging messages.
119 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
121 Toggle display of registers related run-time check messages.
123 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
125 Toggle display of firmware mailbox related run-time check messages.
127 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
129 Toggle display of transmission data path run-time check messages.
131 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
133 Toggle display of receiving data path run-time check messages.
135 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
137 Toggle behaviour to prefer Throughput or Latency.
139 .. _driver-compilation:
141 Driver compilation and testing
142 ------------------------------
144 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
150 .. _linux-installation:
155 Steps to manually install the latest firmware from the downloaded Chelsio
156 Unified Wire package for Linux operating system are as follows:
158 #. Load the kernel module:
160 .. code-block:: console
164 #. Use ifconfig to get the interface name assigned to Chelsio card:
166 .. code-block:: console
168 ifconfig -a | grep "00:07:43"
172 .. code-block:: console
174 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
175 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
179 .. code-block:: console
181 cd <path_to_uwire>/tools/cxgbtool
184 #. Use cxgbtool to load the firmware config file onto the card:
186 .. code-block:: console
188 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
190 #. Use cxgbtool to load the firmware image onto the card:
192 .. code-block:: console
194 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
196 #. Unload and reload the kernel module:
198 .. code-block:: console
203 #. Verify with ethtool:
205 .. code-block:: console
207 ethtool -i p1p1 | grep "firmware"
211 .. code-block:: console
213 firmware-version: 1.16.43.0, TP 0.1.4.9
218 This section demonstrates how to launch **testpmd** with Chelsio
219 devices managed by librte_pmd_cxgbe in Linux operating system.
221 #. Load the kernel module:
223 .. code-block:: console
227 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
229 .. code-block:: console
235 .. code-block:: console
237 cxgb4 0000:02:00.4 p1p1: renamed from eth0
238 cxgb4 0000:02:00.4 p1p2: renamed from eth1
242 Both the interfaces of a Chelsio 2-port adapter are bound to the
243 same PCI bus address.
245 #. Unload the kernel module:
247 .. code-block:: console
249 modprobe -ar cxgb4 csiostor
253 Follow instructions available in the document
254 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
259 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
263 .. code-block:: console
266 EAL: PCI device 0000:02:00.4 on NUMA socket -1
267 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
268 EAL: PCI memory mapped at 0x7fd7c0200000
269 EAL: PCI memory mapped at 0x7fd77cdfd000
270 EAL: PCI memory mapped at 0x7fd7c10b7000
271 PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9
272 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
273 Interactive-mode selected
274 Configuring Port 0 (socket 0)
275 Port 0: 00:07:43:2D:EA:C0
276 Configuring Port 1 (socket 0)
277 Port 1: 00:07:43:2D:EA:C8
278 Checking link statuses...
279 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
280 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
281 Port 0 Link Up - speed 10000 Mbps - full-duplex
282 Port 1 Link Up - speed 10000 Mbps - full-duplex
288 Flow control pause TX/RX is disabled by default and can be enabled via
289 testpmd. Refer section :ref:`flow-control` for more details.
294 .. _freebsd-installation:
299 Steps to manually install the latest firmware from the downloaded Chelsio
300 Unified Wire package for FreeBSD operating system are as follows:
302 #. Load the kernel module:
304 .. code-block:: console
308 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
310 .. code-block:: console
316 .. code-block:: console
318 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
319 cxl0: <port 0> on t5nex0
320 cxl1: <port 1> on t5nex0
321 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
323 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
325 #. Install cxgbetool from FreeBSD source repository:
327 .. code-block:: console
329 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
332 #. Use cxgbetool to load the firmware image onto the card:
334 .. code-block:: console
336 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
338 #. Unload and reload the kernel module:
340 .. code-block:: console
345 #. Verify with sysctl:
347 .. code-block:: console
349 sysctl -a | grep "t5nex" | grep "firmware"
353 .. code-block:: console
355 dev.t5nex.0.firmware_version: 1.16.43.0
360 This section demonstrates how to launch **testpmd** with Chelsio
361 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
363 #. Change to DPDK source directory where the target has been compiled in
364 section :ref:`driver-compilation`:
366 .. code-block:: console
368 cd <DPDK-source-directory>
370 #. Copy the contigmem kernel module to /boot/kernel directory:
372 .. code-block:: console
374 cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
376 #. Add the following lines to /boot/loader.conf:
378 .. code-block:: console
380 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
381 hw.contigmem.num_buffers=2
382 hw.contigmem.buffer_size=1073741824
383 # load contigmem module during boot process
386 The above lines load the contigmem kernel module during boot process and
387 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
388 This is to avoid issues with potential memory fragmentation during later
389 system up time, which may result in failure of allocating the contiguous
390 memory required for the contigmem kernel module.
392 #. Restart the system and ensure the contigmem module is loaded successfully:
394 .. code-block:: console
397 kldstat | grep "contigmem"
401 .. code-block:: console
403 2 1 0xffffffff817f1000 3118 contigmem.ko
405 #. Repeat step 1 to ensure that you are in the DPDK source directory.
407 #. Load the cxgbe kernel module:
409 .. code-block:: console
413 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
415 .. code-block:: console
417 pciconf -l | grep "t5nex"
421 .. code-block:: console
423 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
425 In the above example, the t5nex0 is bound to 2:0:4 bus address.
429 Both the interfaces of a Chelsio 2-port adapter are bound to the
430 same PCI bus address.
432 #. Unload the kernel module:
434 .. code-block:: console
438 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
440 .. code-block:: console
442 kenv hw.nic_uio.bdfs="2:0:4"
444 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
449 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
451 #. Load nic_uio kernel driver:
453 .. code-block:: console
455 kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
457 #. Start testpmd with basic parameters:
459 .. code-block:: console
461 ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
465 .. code-block:: console
468 EAL: PCI device 0000:02:00.4 on NUMA socket 0
469 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
470 EAL: PCI memory mapped at 0x8007ec000
471 EAL: PCI memory mapped at 0x842800000
472 EAL: PCI memory mapped at 0x80086c000
473 PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9
474 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
475 Interactive-mode selected
476 Configuring Port 0 (socket 0)
477 Port 0: 00:07:43:2D:EA:C0
478 Configuring Port 1 (socket 0)
479 Port 1: 00:07:43:2D:EA:C8
480 Checking link statuses...
481 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
482 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
483 Port 0 Link Up - speed 10000 Mbps - full-duplex
484 Port 1 Link Up - speed 10000 Mbps - full-duplex
490 Flow control pause TX/RX is disabled by default and can be enabled via
491 testpmd. Refer section :ref:`flow-control` for more details.
493 Sample Application Notes
494 ------------------------
498 Enable/Disable Flow Control
499 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
501 Flow control pause TX/RX is disabled by default and can be enabled via
504 .. code-block:: console
506 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
507 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
509 To disable again, run:
511 .. code-block:: console
513 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
514 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
519 There are two ways to enable sending and receiving of jumbo frames via testpmd.
520 One method involves using the **mtu** command, which changes the mtu of an
521 individual port without having to stop the selected port. Another method
522 involves stopping all the ports first and then running **max-pkt-len** command
523 to configure the mtu of all the ports with a single command.
525 - To configure each port individually, run the mtu command as follows:
527 .. code-block:: console
529 testpmd> port config mtu 0 9000
530 testpmd> port config mtu 1 9000
532 - To configure all the ports at once, stop all the ports first and run the
533 max-pkt-len command as follows:
535 .. code-block:: console
537 testpmd> port stop all
538 testpmd> port config all max-pkt-len 9000