X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fnics%2Fthunderx.rst;h=53eaec72a796a2cd27189d3479455bdf8203edbc;hb=a3f5252e5cbdabe3f65c6077c4e208d6100f3f74;hp=acc05a4ae985f9f539747a09151dbff6c215018b;hpb=b72a77685fe3038af8b84ba1b32724f263482bf1;p=dpdk.git diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index acc05a4ae9..53eaec72a7 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -1,32 +1,5 @@ -.. BSD LICENSE - Copyright (C) Cavium networks Ltd. 2016. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Cavium networks nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2016 Cavium, Inc ThunderX NICVF Poll Mode Driver =============================== @@ -35,7 +8,7 @@ The ThunderX NICVF PMD (**librte_pmd_thunderx_nicvf**) provides poll mode driver support for the inbuilt NIC found in the **Cavium ThunderX** SoC family as well as their virtual functions (VF) in SR-IOV context. -More information can be found at `Cavium Networks Official Website +More information can be found at `Cavium, Inc Official Website `_. Features @@ -56,11 +29,14 @@ Features of the ThunderX PMD are: - VLAN stripping - SR-IOV VF - NUMA support +- Multi queue set support (up to 96 queues (12 queue sets)) per port +- Skip data bytes Supported ThunderX SoCs ----------------------- - CN88xx - CN81xx +- CN83xx Prerequisites ------------- @@ -75,92 +51,136 @@ Config File Options The following options can be modified in the ``config`` file. Please note that enabling debugging options may affect system performance. -- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``n``) +- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``y``) - By default it is enabled only for defconfig_arm64-thunderx-* config. Toggle compilation of the ``librte_pmd_thunderx_nicvf`` driver. -- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT`` (default ``n``) - - Toggle display of initialization related messages. - - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX`` (default ``n``) - Toggle display of receive fast path run-time message + Toggle asserts of receive fast path. - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX`` (default ``n``) - Toggle display of transmit fast path run-time message + Toggle asserts of transmit fast path. -- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER`` (default ``n``) +Driver compilation and testing +------------------------------ - Toggle display of generic debugging messages +Refer to the document :ref:`compiling and testing a PMD for a NIC ` +for details. -- ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX`` (default ``n``) +To compile the ThunderX NICVF PMD for Linux arm64 gcc, +use arm64-thunderx-linux-gcc as target. - Toggle display of PF mailbox related run-time check messages +Linux +----- -Driver Compilation -~~~~~~~~~~~~~~~~~~ +SR-IOV: Prerequisites and sample Application Notes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To compile the ThunderX NICVF PMD for Linux arm64 gcc target, run the -following “make” command: +Current ThunderX NIC PF/VF kernel modules maps each physical Ethernet port +automatically to virtual function (VF) and presented them as PCIe-like SR-IOV device. +This section provides instructions to configure SR-IOV with Linux OS. -.. code-block:: console +#. Verify PF devices capabilities using ``lspci``: - cd - make config T=arm64-thunderx-linuxapp-gcc install + .. code-block:: console -Linux ------ + lspci -vvv -.. _thunderx_testpmd_example: + Example output: -Running testpmd -~~~~~~~~~~~~~~~ + .. code-block:: console -This section demonstrates how to launch ``testpmd`` with ThunderX NIC VF device -managed by ``librte_pmd_thunderx_nicvf`` in the Linux operating system. + 0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01) + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) + ... + Kernel driver in use: thunder-nic + ... -#. Load ``vfio-pci`` driver: + .. note:: - .. code-block:: console + Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting. - modprobe vfio-pci +#. Verify VF devices capabilities and drivers using ``lspci``: - .. _thunderx_vfio_noiommu: + .. code-block:: console -#. Enable **VFIO-NOIOMMU** mode (optional): + lspci -vvv + + Example output: .. code-block:: console - echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + 0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01) + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... + + 0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01) + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... .. note:: - **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise. - See also :ref:`SR-IOV: Prerequisites and sample Application Notes `. + Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting. -#. Bind the ThunderX NIC VF device to ``vfio-pci`` loaded in the previous step: +#. Pass VF device to VM context (PCIe Passthrough): - Setup VFIO permissions for regular users and then bind to ``vfio-pci``: + The VF devices may be passed through to the guest VM using qemu or + virt-manager or virsh etc. + + Example qemu guest launch command: .. code-block:: console - ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2 + sudo qemu-system-aarch64 -name vm1 \ + -machine virt,gic_version=3,accel=kvm,usb=off \ + -cpu host -m 4096 \ + -smp 4,sockets=1,cores=8,threads=1 \ + -nographic -nodefaults \ + -kernel \ + -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \ + -device vfio-pci,host=0002:01:00.1 \ + -drive file=,if=none,id=disk1,format=raw \ + -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \ + -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \ + -device virtio-net-device,netdev=net0 \ + -serial stdio \ + -mem-path /dev/huge -#. Start ``testpmd`` with basic parameters: +#. Enable **VFIO-NOIOMMU** mode (optional): .. code-block:: console - ./arm64-thunderx-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0002:01:00.2 \ - -- -i --disable-hw-vlan-filter --crc-strip --no-flush-rx \ - --port-topology=loop + echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + + .. note:: + + **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise. + +#. Running testpmd: + + Follow instructions available in the document + :ref:`compiling and testing a PMD for a NIC ` + to run testpmd. Example output: .. code-block:: console + ./arm64-thunderx-linux-gcc/app/testpmd -l 0-3 -n 4 -w 0002:01:00.2 \ + -- -i --no-flush-rx \ + --port-topology=loop + ... PMD: rte_nicvf_pmd_init(): librte_pmd_thunderx nicvf version 1.0 @@ -187,140 +207,126 @@ managed by ``librte_pmd_thunderx_nicvf`` in the Linux operating system. Done testpmd> -.. _thunderx_sriov_example: +Multiple Queue Set per DPDK port configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -SR-IOV: Prerequisites and sample Application Notes -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Current ThunderX NIC PF/VF kernel modules maps each physical Ethernet port -automatically to virtual function (VF) and presented them as PCIe-like SR-IOV device. -This section provides instructions to configure SR-IOV with Linux OS. +There are two types of VFs: -#. Verify PF devices capabilities using ``lspci``: +- Primary VF +- Secondary VF - .. code-block:: console +Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. +When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF. +Each secondary VF adds 8 additional queues to the queue set. - lspci -vvv +During PMD driver initialization, the primary VF's are enumerated by checking the +specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set). +They are at the beginning of VF list (the remain ones are secondary VF's). - Example output: +The primary VFs are used as master queue sets. Secondary VFs provide +additional queue sets for primary ones. If a port is configured for more then +8 queues than it will request for additional queues from secondary VFs. - .. code-block:: console +Secondary VFs cannot be shared between primary VFs. - 0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) - ... - Kernel driver in use: thunder-nic - ... +Primary VFs are present on the beginning of the 'Network devices using kernel +driver' list, secondary VFs are on the remaining on the remaining part of the list. .. note:: - Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting. - -#. Verify VF devices capabilities and drivers using ``lspci``: - - .. code-block:: console - - lspci -vvv - - Example output: - - .. code-block:: console - - 0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... - - 0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... + The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. + We need to bind separately each specific queue set device with the ``usertools/dpdk-devbind.py`` utility. .. note:: - Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting. - -#. Verify PF/VF bind using ``dpdk-devbind.py``: + Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to + this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. - .. code-block:: console - ./tools/dpdk-devbind.py --status +Example device binding +~~~~~~~~~~~~~~~~~~~~~~ - Example output: +If a system has three interfaces, a total of 18 VF devices will be created +on a non-NUMA machine. - .. code-block:: console - - ... - 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci - 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci - 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci - ... + .. note:: -#. Load ``vfio-pci`` driver: + NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port. .. code-block:: console - modprobe vfio-pci - -#. Bind VF devices to ``vfio-pci`` using ``dpdk-devbind.py``: + # usertools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + ============================================ + + + Network devices using kernel driver + =================================== + 0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic + 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + + Other network devices + ===================== + 0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic + + +We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs +and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3). +We will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2). + + +#. Bind two primary VFs to the ``vfio-pci`` driver: .. code-block:: console - ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.1 - ./tools/dpdk-devbind.py --bind vfio-pci 0002:01:00.2 + usertools/dpdk-devbind.py -b vfio-pci 0002:01:00.2 + usertools/dpdk-devbind.py -b vfio-pci 0002:01:00.3 -#. Verify VF bind using ``dpdk-devbind.py``: +#. Bind four primary VFs to the ``vfio-pci`` driver: .. code-block:: console - ./tools/dpdk-devbind.py --status + usertools/dpdk-devbind.py -b vfio-pci 0002:01:01.7 + usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.0 + usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.1 + usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.2 - Example output: +The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage. - .. code-block:: console - ... - 0002:01:00.1 'Device 0011' drv=vfio-pci unused= - 0002:01:00.2 'Device 0011' drv=vfio-pci unused= - ... - 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci - ... - -#. Pass VF device to VM context (PCIe Passthrough): +Module params +-------------- - The VF devices may be passed through to the guest VM using qemu or - virt-manager or virsh etc. - ``librte_pmd_thunderx_nicvf`` or ``thunder-nicvf`` should be used to bind - the VF devices in the guest VM in :ref:`VFIO-NOIOMMU ` mode. - - Example qemu guest launch command: +skip_data_bytes +~~~~~~~~~~~~~~~ +This feature is used to create a hole between HEADROOM and actual data. Size of hole is specified +in bytes as module param("skip_data_bytes") to pmd. +This scheme is useful when application would like to insert vlan header without disturbing HEADROOM. +Example: .. code-block:: console - sudo qemu-system-aarch64 -name vm1 \ - -machine virt,gic_version=3,accel=kvm,usb=off \ - -cpu host -m 4096 \ - -smp 4,sockets=1,cores=8,threads=1 \ - -nographic -nodefaults \ - -kernel \ - -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \ - -device vfio-pci,host=0002:01:00.1 \ - -drive file=,if=none,id=disk1,format=raw \ - -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \ - -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \ - -device virtio-net-device,netdev=net0 \ - -serial stdio \ - -mem-path /dev/huge - -#. Refer to section :ref:`Running testpmd ` for instruction - how to launch ``testpmd`` application. + -w 0002:01:00.2,skip_data_bytes=8 Limitations ----------- @@ -329,8 +335,7 @@ CRC striping ~~~~~~~~~~~~ The ThunderX SoC family NICs strip the CRC for every packets coming into the -host interface. So, CRC will be stripped even when the -``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``. +host interface irrespective of the offload configuration. Maximum packet length ~~~~~~~~~~~~~~~~~~~~~ @@ -347,9 +352,7 @@ The ThunderX SoC family NICs support up to 12 segments per packet when working in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the frame size does not fit in the maximum number of segments. -Limited VFs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +skip_data_bytes +~~~~~~~~~~~~~~~ -The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues -for RX/TX respectively. Current driver implementation has one to one mapping -between physical port and VF hence only limited VFs can be used. +Maximum limit of skip_data_bytes is 128 bytes and number of bytes should be multiple of 8.