X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;ds=sidebyside;f=doc%2Fguides%2Fnics%2Fena.rst;h=bec97c3326a72a5d9017bcc921b5d0e30df5458d;hb=9eb974221f44cf210fe5d883bdccfcb48de75fc6;hp=c2d6b85a2929d0ac1eb3aec4e5e1cef5132d9850;hpb=35b09d76f89e7d5a4f38a2926cf6915028ed1e56;p=dpdk.git diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst index c2d6b85a29..bec97c3326 100644 --- a/doc/guides/nics/ena.rst +++ b/doc/guides/nics/ena.rst @@ -1,34 +1,7 @@ -.. BSD LICENSE - - Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates. +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates. All rights reserved. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Amazon.com, Inc. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - ENA Poll Mode Driver ==================== @@ -113,10 +86,6 @@ Configuration information * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion of the ENA PMD driver in the DPDK compilation. - - * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug - logging of device initialization within the ENA PMD driver. - * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug logging of RX logic within the ENA PMD driver. @@ -126,6 +95,14 @@ Configuration information * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver. +**Runtime Configuration Parameters** + + * **large_llq_hdr** (default 0) + + Enables or disables usage of large LLQ headers. This option will have + effect only if the device also supports large LLQ headers. Otherwise, the + default value will be used. + **ENA Configuration Parameters** * **Number of Queues** @@ -168,84 +145,121 @@ section of :ref:`the DPDK documentation ` or refer to *DPDK Release N Supported features ------------------ +* MTU configuration * Jumbo frames up to 9K -* Port Hardware Statistics * IPv4/TCP/UDP checksum offload * TSO offload * Multiple receive and transmit queues -* RSS +* RSS hash +* RSS indirection table configuration * Low Latency Queue for Tx - -Unsupported features --------------------- - -The features supported by the device and not yet supported by this PMD include: - -* Asynchronous Event Notification Queue (AENQ) +* Basic and extended statistics +* LSC event notification +* Watchdog (requires handling of timers in the application) +* Device reset upon failure Prerequisites ------------- #. Prepare the system as recommended by DPDK suite. This includes environment - variables, hugepages configuration, tool-chains and configuration + variables, hugepages configuration, tool-chains and configuration. + +#. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver. + + (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature + reduces the latency of the packets by pushing the header directly through + the PCI to the device, before the DMA is even triggered. For proper work + kernel PCI driver must support write combining (WC). In mainline version of + ``igb_uio`` (in DPDK repo) it must be enabled by loading module with + ``wc_activate=1`` flag (example below). However, mainline's vfio-pci + driver in kernel doesn't have WC support yet (planed to be added). + If vfio-pci used user should be either turn off ENAv2 (to avoid performance + impact) or recompile vfio-pci driver with patch provided in + `amzn-github `_. + +#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command + ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1`` + respectively. + +#. For ``vfio-pci`` users only: + Please make sure that ``IOMMU`` is enabled in your system, + or use ``vfio`` driver in ``noiommu`` mode:: -#. Insert igb_uio kernel module using the command 'modprobe igb_uio' + echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode -#. Bind the intended ENA device to igb_uio module + To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag + ``CONFIG_VFIO_NOIOMMU``. +#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. At this point the system should be ready to run DPDK applications. Once the -application runs to completion, the ENA can be detached from igb_uio if necessary. +application runs to completion, the ENA can be detached from attached module if +necessary. -Usage example -------------- +**Note about usage on \*.metal instances** + +On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64 +hosts. -This section demonstrates how to launch **testpmd** with Amazon ENA -devices managed by librte_pmd_ena. +* x86_64 (e.g. c5.metal, i3.metal): + IOMMU should be disabled by default. In that situation, the ``igb_uio`` can + be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please + see above). -#. Load the kernel modules: + When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this + feature, while ``vfio-pci`` should work without any changes. + To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file + ``/etc/default/grub`` with the below extra boot arguments:: - .. code-block:: console + iommu=1 intel_iommu=on - modprobe uio - insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko + Then, make the changes live by executing as a root:: - .. note:: + # grub2-mkconfig > /boot/grub2/grub.cfg - Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module + Finally, reboot should result in IOMMU being enabled. -#. Mount and request huge pages: +* arm64 (a1.metal): + IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't + supporting SMMU, which is implementation of IOMMU for arm64 architecture and + ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those + hosts, one must disable IOMMU. This can be done by updating + ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot + argument:: - .. code-block:: console + iommu.passthrough=1 - mount -t hugetlbfs nodev /mnt/hugepages - echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + Then, make the changes live by executing as a root:: -#. Bind UIO driver to ENA device (using provided by DPDK binding tool): + # grub2-mkconfig > /boot/grub2/grub.cfg - .. code-block:: console + Finally, reboot should result in IOMMU being disabled. + Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be + working in no-IOMMU mode (please see above). - ./usertools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1 +Usage example +------------- -#. Start testpmd with basic parameters: +Follow instructions available in the document +:ref:`compiling and testing a PMD for a NIC ` to launch +**testpmd** with Amazon ENA devices managed by librte_pmd_ena. - .. code-block:: console +Example output: - ./x86_64-native-linuxapp-gcc/app/testpmd -l 0-3 -n 4 -- -i +.. code-block:: console - Example output: + [...] + EAL: PCI device 0000:00:06.0 on NUMA socket -1 + EAL: Invalid NUMA socket, default to 0 + EAL: probe driver: 1d0f:ec20 net_ena - .. code-block:: console + Interactive-mode selected + testpmd: create a new mbuf pool : n=171456, size=2176, socket=0 + testpmd: preferred mempool ops selected: ring_mp_mc + Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. + Configuring Port 0 (socket 0) + Port 0: 00:00:00:11:00:01 + Checking link statuses... - [...] - EAL: PCI device 0000:02:00.1 on NUMA socket -1 - EAL: probe driver: 1d0f:ec20 rte_ena_pmd - EAL: PCI memory mapped at 0x7f9b6c400000 - PMD: eth_ena_dev_init(): Initializing 0:2:0.1 - Interactive-mode selected - Configuring Port 0 (socket 0) - Port 0: 00:00:00:11:00:01 - Checking link statuses... - Port 0 Link Up - speed 10000 Mbps - full-duplex - Done - testpmd> + Done + testpmd>