2 Copyright 2015 Chelsio Communications.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Chelsio Communications nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 CXGBE Poll Mode Driver
32 ======================
34 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
35 for **Chelsio T5** 10/40 Gbps family of adapters. CXGBE PMD has support
36 for the latest Linux and FreeBSD operating systems.
38 More information can be found at `Chelsio Communications Official Website
39 <http://www.chelsio.com>`_.
44 CXGBE PMD has support for:
46 - Multiple queues for TX and RX
47 - Receiver Side Steering (RSS)
52 - Port hardware statistics
58 The Chelsio T5 devices provide two/four ports but expose a single PCI bus
59 address, thus, librte_pmd_cxgbe registers itself as a
60 PCI driver that allocates one Ethernet device per detected port.
62 For this reason, one cannot whitelist/blacklist a single port without
63 whitelisting/blacklisting the other ports on the same device.
65 Supported Chelsio T5 NICs
66 -------------------------
69 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
70 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
71 - Other T5 NICs: T522-CR
76 - Requires firmware version **1.13.32.0** and higher. Visit
77 `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
78 bundled with the latest Chelsio Unified Wire package.
80 For Linux, installing and loading the latest cxgb4 kernel driver from the
81 Chelsio Unified Wire package should get you the latest firmware. More
82 information can be obtained from the User Guide that is bundled with the
83 Chelsio Unified Wire package.
85 For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
86 package must be manually flashed via cxgbetool available in FreeBSD source
89 Instructions on how to manually flash the firmware are given in section
90 :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
93 Pre-Installation Configuration
94 ------------------------------
99 The following options can be modified in the ``.config`` file. Please note that
100 enabling debugging options may affect system performance.
102 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
104 Toggle compilation of librte_pmd_cxgbe driver.
106 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
108 Toggle display of generic debugging messages.
110 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
112 Toggle display of registers related run-time check messages.
114 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
116 Toggle display of firmware mailbox related run-time check messages.
118 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
120 Toggle display of transmission data path run-time check messages.
122 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
124 Toggle display of receiving data path run-time check messages.
126 .. _driver-compilation:
128 Driver compilation and testing
129 ------------------------------
131 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
137 .. _linux-installation:
142 Steps to manually install the latest firmware from the downloaded Chelsio
143 Unified Wire package for Linux operating system are as follows:
145 #. Load the kernel module:
147 .. code-block:: console
151 #. Use ifconfig to get the interface name assigned to Chelsio card:
153 .. code-block:: console
155 ifconfig -a | grep "00:07:43"
159 .. code-block:: console
161 p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0
162 p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8
166 .. code-block:: console
168 cd <path_to_uwire>/tools/cxgbtool
171 #. Use cxgbtool to load the firmware config file onto the card:
173 .. code-block:: console
175 cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
177 #. Use cxgbtool to load the firmware image onto the card:
179 .. code-block:: console
181 cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
183 #. Unload and reload the kernel module:
185 .. code-block:: console
190 #. Verify with ethtool:
192 .. code-block:: console
194 ethtool -i p1p1 | grep "firmware"
198 .. code-block:: console
200 firmware-version: 1.13.32.0, TP 0.1.4.8
205 This section demonstrates how to launch **testpmd** with Chelsio T5
206 devices managed by librte_pmd_cxgbe in Linux operating system.
208 #. Load the kernel module:
210 .. code-block:: console
214 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
216 .. code-block:: console
222 .. code-block:: console
224 cxgb4 0000:02:00.4 p1p1: renamed from eth0
225 cxgb4 0000:02:00.4 p1p2: renamed from eth1
229 Both the interfaces of a Chelsio T5 2-port adapter are bound to the
230 same PCI bus address.
232 #. Unload the kernel module:
234 .. code-block:: console
236 modprobe -ar cxgb4 csiostor
240 Follow instructions available in the document
241 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
246 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
250 .. code-block:: console
253 EAL: PCI device 0000:02:00.4 on NUMA socket -1
254 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
255 EAL: PCI memory mapped at 0x7fd7c0200000
256 EAL: PCI memory mapped at 0x7fd77cdfd000
257 EAL: PCI memory mapped at 0x7fd7c10b7000
258 PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
259 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
260 Interactive-mode selected
261 Configuring Port 0 (socket 0)
262 Port 0: 00:07:43:2D:EA:C0
263 Configuring Port 1 (socket 0)
264 Port 1: 00:07:43:2D:EA:C8
265 Checking link statuses...
266 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
267 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
268 Port 0 Link Up - speed 10000 Mbps - full-duplex
269 Port 1 Link Up - speed 10000 Mbps - full-duplex
275 Flow control pause TX/RX is disabled by default and can be enabled via
276 testpmd. Refer section :ref:`flow-control` for more details.
281 .. _freebsd-installation:
286 Steps to manually install the latest firmware from the downloaded Chelsio
287 Unified Wire package for FreeBSD operating system are as follows:
289 #. Load the kernel module:
291 .. code-block:: console
295 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
297 .. code-block:: console
303 .. code-block:: console
305 t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
306 cxl0: <port 0> on t5nex0
307 cxl1: <port 1> on t5nex0
308 t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
310 In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
312 #. Install cxgbetool from FreeBSD source repository:
314 .. code-block:: console
316 cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
319 #. Use cxgbetool to load the firmware image onto the card:
321 .. code-block:: console
323 cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
325 #. Unload and reload the kernel module:
327 .. code-block:: console
332 #. Verify with sysctl:
334 .. code-block:: console
336 sysctl -a | grep "t5nex" | grep "firmware"
340 .. code-block:: console
342 dev.t5nex.0.firmware_version: 1.13.32.0
347 This section demonstrates how to launch **testpmd** with Chelsio T5
348 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
350 #. Change to DPDK source directory where the target has been compiled in
351 section :ref:`driver-compilation`:
353 .. code-block:: console
355 cd <DPDK-source-directory>
357 #. Copy the contigmem kernel module to /boot/kernel directory:
359 .. code-block:: console
361 cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
363 #. Add the following lines to /boot/loader.conf:
365 .. code-block:: console
367 # reserve 2 x 1G blocks of contiguous memory using contigmem driver
368 hw.contigmem.num_buffers=2
369 hw.contigmem.buffer_size=1073741824
370 # load contigmem module during boot process
373 The above lines load the contigmem kernel module during boot process and
374 allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
375 This is to avoid issues with potential memory fragmentation during later
376 system up time, which may result in failure of allocating the contiguous
377 memory required for the contigmem kernel module.
379 #. Restart the system and ensure the contigmem module is loaded successfully:
381 .. code-block:: console
384 kldstat | grep "contigmem"
388 .. code-block:: console
390 2 1 0xffffffff817f1000 3118 contigmem.ko
392 #. Repeat step 1 to ensure that you are in the DPDK source directory.
394 #. Load the cxgbe kernel module:
396 .. code-block:: console
400 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
402 .. code-block:: console
404 pciconf -l | grep "t5nex"
408 .. code-block:: console
410 t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
412 In the above example, the t5nex0 is bound to 2:0:4 bus address.
416 Both the interfaces of a Chelsio T5 2-port adapter are bound to the
417 same PCI bus address.
419 #. Unload the kernel module:
421 .. code-block:: console
425 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
427 .. code-block:: console
429 kenv hw.nic_uio.bdfs="2:0:4"
431 This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
436 Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
438 #. Load nic_uio kernel driver:
440 .. code-block:: console
442 kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
444 #. Start testpmd with basic parameters:
446 .. code-block:: console
448 ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
452 .. code-block:: console
455 EAL: PCI device 0000:02:00.4 on NUMA socket 0
456 EAL: probe driver: 1425:5401 rte_cxgbe_pmd
457 EAL: PCI memory mapped at 0x8007ec000
458 EAL: PCI memory mapped at 0x842800000
459 EAL: PCI memory mapped at 0x80086c000
460 PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
461 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
462 Interactive-mode selected
463 Configuring Port 0 (socket 0)
464 Port 0: 00:07:43:2D:EA:C0
465 Configuring Port 1 (socket 0)
466 Port 1: 00:07:43:2D:EA:C8
467 Checking link statuses...
468 PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
469 PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
470 Port 0 Link Up - speed 10000 Mbps - full-duplex
471 Port 1 Link Up - speed 10000 Mbps - full-duplex
477 Flow control pause TX/RX is disabled by default and can be enabled via
478 testpmd. Refer section :ref:`flow-control` for more details.
480 Sample Application Notes
481 ------------------------
485 Enable/Disable Flow Control
486 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
488 Flow control pause TX/RX is disabled by default and can be enabled via
491 .. code-block:: console
493 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
494 testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
496 To disable again, run:
498 .. code-block:: console
500 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
501 testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
506 There are two ways to enable sending and receiving of jumbo frames via testpmd.
507 One method involves using the **mtu** command, which changes the mtu of an
508 individual port without having to stop the selected port. Another method
509 involves stopping all the ports first and then running **max-pkt-len** command
510 to configure the mtu of all the ports with a single command.
512 - To configure each port individually, run the mtu command as follows:
514 .. code-block:: console
516 testpmd> port config mtu 0 9000
517 testpmd> port config mtu 1 9000
519 - To configure all the ports at once, stop all the ports first and run the
520 max-pkt-len command as follows:
522 .. code-block:: console
524 testpmd> port stop all
525 testpmd> port config all max-pkt-len 9000