X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fnics%2Fcxgbe.rst;h=8651a7be602ee017070d3a21b0f9f27be3b9746b;hb=78214fb8821fab0669c1c48f00fc4773e5a9eb98;hp=7557db07b6b053b7e3e98307335899d8cb168620;hpb=652bfba0ae2b78afefcb4fc889ea4cc723a8c702;p=dpdk.git diff --git a/doc/guides/nics/cxgbe.rst b/doc/guides/nics/cxgbe.rst index 7557db07b6..8651a7be60 100644 --- a/doc/guides/nics/cxgbe.rst +++ b/doc/guides/nics/cxgbe.rst @@ -1,5 +1,5 @@ .. BSD LICENSE - Copyright 2015 Chelsio Communications. + Copyright 2015-2017 Chelsio Communications. All rights reserved. Redistribution and use in source and binary forms, with or without @@ -32,7 +32,8 @@ CXGBE Poll Mode Driver ====================== The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support -for **Chelsio T5** 10/40 Gbps family of adapters. +for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD +has support for the latest Linux and FreeBSD operating systems. More information can be found at `Chelsio Communications Official Website `_. @@ -49,13 +50,15 @@ CXGBE PMD has support for: - Promiscuous mode - All multicast mode - Port hardware statistics +- Jumbo frames Limitations ----------- -The Chelsio T5 devices provide two/four ports but expose a single PCI bus -address, thus, librte_pmd_cxgbe registers itself as a -PCI driver that allocates one Ethernet device per detected port. +The Chelsio Terminator series of devices provide two/four ports but +expose a single PCI bus address, thus, librte_pmd_cxgbe registers +itself as a PCI driver that allocates one Ethernet device per detected +port. For this reason, one cannot whitelist/blacklist a single port without whitelisting/blacklisting the other ports on the same device. @@ -68,15 +71,31 @@ Supported Chelsio T5 NICs - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR - Other T5 NICs: T522-CR +Supported Chelsio T6 NICs +------------------------- + +- 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR +- 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR + Prerequisites ------------- -- Requires firmware version **1.13.32.0** and higher. Visit +- Requires firmware version **1.16.43.0** and higher. Visit `Chelsio Download Center `_ to get latest firmware - bundled with the latest Chelsio Unified Wire package. Installing and loading - the latest cxgb4 kernel driver from the Chelsio Unified Wire package should - get you the latest firmware. More information can be obtained from the User - Guide that is bundled with the Chelsio Unified Wire package. + bundled with the latest Chelsio Unified Wire package. + + For Linux, installing and loading the latest cxgb4 kernel driver from the + Chelsio Unified Wire package should get you the latest firmware. More + information can be obtained from the User Guide that is bundled with the + Chelsio Unified Wire package. + + For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire + package must be manually flashed via cxgbetool available in FreeBSD source + repository. + + Instructions on how to manually flash the firmware are given in section + :ref:`linux-installation` for Linux and section :ref:`freebsd-installation` + for FreeBSD. Pre-Installation Configuration ------------------------------ @@ -111,22 +130,28 @@ enabling debugging options may affect system performance. Toggle display of receiving data path run-time check messages. -Driver Compilation -~~~~~~~~~~~~~~~~~~ +- ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**) -To compile CXGBE PMD for Linux x86_64 gcc target, run the following "make" -command: + Toggle behaviour to prefer Throughput or Latency. -.. code-block:: console +.. _driver-compilation: + +Driver compilation and testing +------------------------------ - cd - make config T=x86_64-native-linuxapp-gcc install +Refer to the document :ref:`compiling and testing a PMD for a NIC ` +for details. -Installation ------------- +Linux +----- -Steps to manually install the latest firmware from the downloaded source files -are as follows: +.. _linux-installation: + +Linux Installation +~~~~~~~~~~~~~~~~~~ + +Steps to manually install the latest firmware from the downloaded Chelsio +Unified Wire package for Linux operating system are as follows: #. Load the kernel module: @@ -166,6 +191,13 @@ are as follows: cxgbtool p1p1 loadfw /src/network/firmware/t5fw-*.bin +#. Unload and reload the kernel module: + + .. code-block:: console + + modprobe -r cxgb4 + modprobe cxgb4 + #. Verify with ethtool: .. code-block:: console @@ -176,13 +208,13 @@ are as follows: .. code-block:: console - firmware-version: 1.13.32.0, TP 0.1.4.8 + firmware-version: 1.16.43.0, TP 0.1.4.9 -Sample Application Notes -------------------------- +Running testpmd +~~~~~~~~~~~~~~~ -This section demonstrates how to launch **testpmd** with Chelsio T5 -devices managed by librte_pmd_cxgbe. +This section demonstrates how to launch **testpmd** with Chelsio +devices managed by librte_pmd_cxgbe in Linux operating system. #. Load the kernel module: @@ -205,7 +237,7 @@ devices managed by librte_pmd_cxgbe. .. note:: - Both the interfaces of a Chelsio T5 2-port adapter are bound to the + Both the interfaces of a Chelsio 2-port adapter are bound to the same PCI bus address. #. Unload the kernel module: @@ -214,64 +246,229 @@ devices managed by librte_pmd_cxgbe. modprobe -ar cxgb4 csiostor -#. Request huge pages: +#. Running testpmd + + Follow instructions available in the document + :ref:`compiling and testing a PMD for a NIC ` + to run testpmd. + + .. note:: + + Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs. + + Example output: + + .. code-block:: console + + [...] + EAL: PCI device 0000:02:00.4 on NUMA socket -1 + EAL: probe driver: 1425:5401 rte_cxgbe_pmd + EAL: PCI memory mapped at 0x7fd7c0200000 + EAL: PCI memory mapped at 0x7fd77cdfd000 + EAL: PCI memory mapped at 0x7fd7c10b7000 + PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9 + PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter + Interactive-mode selected + Configuring Port 0 (socket 0) + Port 0: 00:07:43:2D:EA:C0 + Configuring Port 1 (socket 0) + Port 1: 00:07:43:2D:EA:C8 + Checking link statuses... + PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted + PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted + Port 0 Link Up - speed 10000 Mbps - full-duplex + Port 1 Link Up - speed 10000 Mbps - full-duplex + Done + testpmd> + + .. note:: + + Flow control pause TX/RX is disabled by default and can be enabled via + testpmd. Refer section :ref:`flow-control` for more details. + +FreeBSD +------- + +.. _freebsd-installation: + +FreeBSD Installation +~~~~~~~~~~~~~~~~~~~~ + +Steps to manually install the latest firmware from the downloaded Chelsio +Unified Wire package for FreeBSD operating system are as follows: + +#. Load the kernel module: + + .. code-block:: console + + kldload if_cxgbe + +#. Use dmesg to get the t5nex instance assigned to the Chelsio card: + + .. code-block:: console + + dmesg | grep "t5nex" + + Example output: .. code-block:: console - echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages + t5nex0: irq 16 at device 0.4 on pci2 + cxl0: on t5nex0 + cxl1: on t5nex0 + t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq + + In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance. -#. Load igb_uio or vfio-pci driver: +#. Install cxgbetool from FreeBSD source repository: .. code-block:: console - insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko + cd /tools/tools/cxgbetool/ + make && make install - or +#. Use cxgbetool to load the firmware image onto the card: .. code-block:: console - modprobe vfio-pci + cxgbetool t5nex0 loadfw /src/network/firmware/t5fw-*.bin -#. Bind the Chelsio T5 adapters to igb_uio or vfio-pci loaded in the previous - step: +#. Unload and reload the kernel module: .. code-block:: console - ./tools/dpdk_nic_bind.py --bind igb_uio 0000:02:00.4 + kldunload if_cxgbe + kldload if_cxgbe - or +#. Verify with sysctl: - Setup VFIO permissions for regular users and then bind to vfio-pci: + .. code-block:: console + + sysctl -a | grep "t5nex" | grep "firmware" + + Example output: .. code-block:: console - sudo chmod a+x /dev/vfio + dev.t5nex.0.firmware_version: 1.16.43.0 + +Running testpmd +~~~~~~~~~~~~~~~ - sudo chmod 0666 /dev/vfio/* +This section demonstrates how to launch **testpmd** with Chelsio +devices managed by librte_pmd_cxgbe in FreeBSD operating system. - ./tools/dpdk_nic_bind.py --bind vfio-pci 0000:02:00.4 +#. Change to DPDK source directory where the target has been compiled in + section :ref:`driver-compilation`: + + .. code-block:: console + + cd + +#. Copy the contigmem kernel module to /boot/kernel directory: + + .. code-block:: console + + cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/ + +#. Add the following lines to /boot/loader.conf: + + .. code-block:: console + + # reserve 2 x 1G blocks of contiguous memory using contigmem driver + hw.contigmem.num_buffers=2 + hw.contigmem.buffer_size=1073741824 + # load contigmem module during boot process + contigmem_load="YES" + + The above lines load the contigmem kernel module during boot process and + allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on. + This is to avoid issues with potential memory fragmentation during later + system up time, which may result in failure of allocating the contiguous + memory required for the contigmem kernel module. + +#. Restart the system and ensure the contigmem module is loaded successfully: + + .. code-block:: console + + reboot + kldstat | grep "contigmem" + + Example output: + + .. code-block:: console + + 2 1 0xffffffff817f1000 3118 contigmem.ko + +#. Repeat step 1 to ensure that you are in the DPDK source directory. + +#. Load the cxgbe kernel module: + + .. code-block:: console + + kldload if_cxgbe + +#. Get the PCI bus addresses of the interfaces bound to t5nex driver: + + .. code-block:: console + + pciconf -l | grep "t5nex" + + Example output: + + .. code-block:: console + + t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 + + In the above example, the t5nex0 is bound to 2:0:4 bus address. .. note:: - CXGBE PMD currently only supports the binding of PF4 for Chelsio T5 NICs. + Both the interfaces of a Chelsio 2-port adapter are bound to the + same PCI bus address. + +#. Unload the kernel module: + + .. code-block:: console + + kldunload if_cxgbe + +#. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter: + + .. code-block:: console + + kenv hw.nic_uio.bdfs="2:0:4" + + This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in + the next step. + + .. note:: + + Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs. + +#. Load nic_uio kernel driver: + + .. code-block:: console + + kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko #. Start testpmd with basic parameters: .. code-block:: console - ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i + ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i Example output: .. code-block:: console [...] - EAL: PCI device 0000:02:00.4 on NUMA socket -1 + EAL: PCI device 0000:02:00.4 on NUMA socket 0 EAL: probe driver: 1425:5401 rte_cxgbe_pmd - EAL: PCI memory mapped at 0x7fd7c0200000 - EAL: PCI memory mapped at 0x7fd77cdfd000 - EAL: PCI memory mapped at 0x7fd7c10b7000 - PMD: rte_cxgbe_pmd: fw: 1.13.33.0, TP: 0.1.4.8 + EAL: PCI memory mapped at 0x8007ec000 + EAL: PCI memory mapped at 0x842800000 + EAL: PCI memory mapped at 0x80086c000 + PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter Interactive-mode selected Configuring Port 0 (socket 0) @@ -289,16 +486,51 @@ devices managed by librte_pmd_cxgbe. .. note:: Flow control pause TX/RX is disabled by default and can be enabled via - testpmd as follows: + testpmd. Refer section :ref:`flow-control` for more details. - .. code-block:: console +Sample Application Notes +------------------------ - testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0 - testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1 +.. _flow-control: - To disable again, use: +Enable/Disable Flow Control +~~~~~~~~~~~~~~~~~~~~~~~~~~~ - .. code-block:: console +Flow control pause TX/RX is disabled by default and can be enabled via +testpmd as follows: + +.. code-block:: console + + testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0 + testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1 + +To disable again, run: + +.. code-block:: console + + testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0 + testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1 + +Jumbo Mode +~~~~~~~~~~ + +There are two ways to enable sending and receiving of jumbo frames via testpmd. +One method involves using the **mtu** command, which changes the mtu of an +individual port without having to stop the selected port. Another method +involves stopping all the ports first and then running **max-pkt-len** command +to configure the mtu of all the ports with a single command. + +- To configure each port individually, run the mtu command as follows: + + .. code-block:: console + + testpmd> port config mtu 0 9000 + testpmd> port config mtu 1 9000 + +- To configure all the ports at once, stop all the ports first and run the + max-pkt-len command as follows: + + .. code-block:: console - testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0 - testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1 + testpmd> port stop all + testpmd> port config all max-pkt-len 9000