-.. BSD LICENSE
-
- Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates.
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
All rights reserved.
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
-
- * Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- * Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in
- the documentation and/or other materials provided with the
- distribution.
- * Neither the name of Amazon.com, Inc. nor the names of its
- contributors may be used to endorse or promote products derived
- from this software without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
ENA Poll Mode Driver
====================
* **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
of the ENA PMD driver in the DPDK compilation.
-
- * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug
- logging of device initialization within the ENA PMD driver.
-
* **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
logging of RX logic within the ENA PMD driver.
* **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
+**Runtime Configuration Parameters**
+
+ * **large_llq_hdr** (default 0)
+
+ Enables or disables usage of large LLQ headers. This option will have
+ effect only if the device also supports large LLQ headers. Otherwise, the
+ default value will be used.
+
**ENA Configuration Parameters**
* **Number of Queues**
Supported features
------------------
+* MTU configuration
* Jumbo frames up to 9K
-* Port Hardware Statistics
* IPv4/TCP/UDP checksum offload
* TSO offload
* Multiple receive and transmit queues
-* RSS
+* RSS hash
+* RSS indirection table configuration
* Low Latency Queue for Tx
-
-Unsupported features
---------------------
-
-The features supported by the device and not yet supported by this PMD include:
-
-* Asynchronous Event Notification Queue (AENQ)
+* Basic and extended statistics
+* LSC event notification
+* Watchdog (requires handling of timers in the application)
+* Device reset upon failure
Prerequisites
-------------
#. Prepare the system as recommended by DPDK suite. This includes environment
- variables, hugepages configuration, tool-chains and configuration
+ variables, hugepages configuration, tool-chains and configuration.
+
+#. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
+
+ (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
+ reduces the latency of the packets by pushing the header directly through
+ the PCI to the device, before the DMA is even triggered. For proper work
+ kernel PCI driver must support write combining (WC). In mainline version of
+ ``igb_uio`` (in DPDK repo) it must be enabled by loading module with
+ ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
+ driver in kernel doesn't have WC support yet (planed to be added).
+ If vfio-pci used user should be either turn off ENAv2 (to avoid performance
+ impact) or recompile vfio-pci driver with patch provided in
+ `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
+
+#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
+ ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
+ respectively.
+
+#. For ``vfio-pci`` users only:
+ Please make sure that ``IOMMU`` is enabled in your system,
+ or use ``vfio`` driver in ``noiommu`` mode::
-#. Insert igb_uio kernel module using the command 'modprobe igb_uio'
+ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
-#. Bind the intended ENA device to igb_uio module
+ To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
+ ``CONFIG_VFIO_NOIOMMU``.
+#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
At this point the system should be ready to run DPDK applications. Once the
-application runs to completion, the ENA can be detached from igb_uio if necessary.
+application runs to completion, the ENA can be detached from attached module if
+necessary.
-Usage example
--------------
+**Note about usage on \*.metal instances**
+
+On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
+hosts.
-This section demonstrates how to launch **testpmd** with Amazon ENA
-devices managed by librte_pmd_ena.
+* x86_64 (e.g. c5.metal, i3.metal):
+ IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
+ be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
+ see above).
-#. Load the kernel modules:
+ When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
+ feature, while ``vfio-pci`` should work without any changes.
+ To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
+ ``/etc/default/grub`` with the below extra boot arguments::
- .. code-block:: console
+ iommu=1 intel_iommu=on
- modprobe uio
- insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
+ Then, make the changes live by executing as a root::
- .. note::
+ # grub2-mkconfig > /boot/grub2/grub.cfg
- Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module
+ Finally, reboot should result in IOMMU being enabled.
-#. Mount and request huge pages:
+* arm64 (a1.metal):
+ IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
+ supporting SMMU, which is implementation of IOMMU for arm64 architecture and
+ ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
+ hosts, one must disable IOMMU. This can be done by updating
+ ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
+ argument::
- .. code-block:: console
+ iommu.passthrough=1
- mount -t hugetlbfs nodev /mnt/hugepages
- echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+ Then, make the changes live by executing as a root::
-#. Bind UIO driver to ENA device (using provided by DPDK binding tool):
+ # grub2-mkconfig > /boot/grub2/grub.cfg
- .. code-block:: console
+ Finally, reboot should result in IOMMU being disabled.
+ Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
+ working in no-IOMMU mode (please see above).
- ./usertools/dpdk-devbind.py --bind=igb_uio 0000:02:00.1
+Usage example
+-------------
-#. Start testpmd with basic parameters:
+Follow instructions available in the document
+:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
+**testpmd** with Amazon ENA devices managed by librte_pmd_ena.
- .. code-block:: console
+Example output:
- ./x86_64-native-linuxapp-gcc/app/testpmd -l 0-3 -n 4 -- -i
+.. code-block:: console
- Example output:
+ [...]
+ EAL: PCI device 0000:00:06.0 on NUMA socket -1
+ EAL: Invalid NUMA socket, default to 0
+ EAL: probe driver: 1d0f:ec20 net_ena
- .. code-block:: console
+ Interactive-mode selected
+ testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
+ testpmd: preferred mempool ops selected: ring_mp_mc
+ Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
+ Configuring Port 0 (socket 0)
+ Port 0: 00:00:00:11:00:01
+ Checking link statuses...
- [...]
- EAL: PCI device 0000:02:00.1 on NUMA socket -1
- EAL: probe driver: 1d0f:ec20 rte_ena_pmd
- EAL: PCI memory mapped at 0x7f9b6c400000
- PMD: eth_ena_dev_init(): Initializing 0:2:0.1
- Interactive-mode selected
- Configuring Port 0 (socket 0)
- Port 0: 00:00:00:11:00:01
- Checking link statuses...
- Port 0 Link Up - speed 10000 Mbps - full-duplex
- Done
- testpmd>
+ Done
+ testpmd>
git.droids-corp.org / dpdk.git / blobdiff