-.. BSD LICENSE
-
- Copyright (c) 2015-2018 Amazon.com, Inc. or its affiliates.
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
All rights reserved.
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
-
- * Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- * Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in
- the documentation and/or other materials provided with the
- distribution.
- * Neither the name of Amazon.com, Inc. nor the names of its
- contributors may be used to endorse or promote products derived
- from this software without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
ENA Poll Mode Driver
====================
* **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
+**Runtime Configuration Parameters**
+
+ * **large_llq_hdr** (default 0)
+
+ Enables or disables usage of large LLQ headers. This option will have
+ effect only if the device also supports large LLQ headers. Otherwise, the
+ default value will be used.
+
**ENA Configuration Parameters**
* **Number of Queues**
#. Prepare the system as recommended by DPDK suite. This includes environment
variables, hugepages configuration, tool-chains and configuration.
-#. ENA PMD can operate with ``vfio-pci`` or ``igb_uio`` driver.
+#. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
+
+ (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
+ reduces the latency of the packets by pushing the header directly through
+ the PCI to the device, before the DMA is even triggered. For proper work
+ kernel PCI driver must support write combining (WC). In mainline version of
+ ``igb_uio`` (in DPDK repo) it must be enabled by loading module with
+ ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
+ driver in kernel doesn't have WC support yet (planed to be added).
+ If vfio-pci used user should be either turn off ENAv2 (to avoid performance
+ impact) or recompile vfio-pci driver with patch provided in
+ `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
- ``modprobe vfio-pci`` or ``modprobe igb_uio`` respectively.
+ ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
+ respectively.
#. For ``vfio-pci`` users only:
Please make sure that ``IOMMU`` is enabled in your system,
echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
-#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
+ To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
+ ``CONFIG_VFIO_NOIOMMU``.
+#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
At this point the system should be ready to run DPDK applications. Once the
-application runs to completion, the ENA can be detached from igb_uio if necessary.
+application runs to completion, the ENA can be detached from attached module if
+necessary.
+
+**Note about usage on \*.metal instances**
+
+On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
+hosts.
+
+* x86_64 (e.g. c5.metal, i3.metal):
+ IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
+ be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
+ see above).
+
+ When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
+ feature, while ``vfio-pci`` should work without any changes.
+ To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
+ ``/etc/default/grub`` with the below extra boot arguments::
+
+ iommu=1 intel_iommu=on
+
+ Then, make the changes live by executing as a root::
+
+ # grub2-mkconfig > /boot/grub2/grub.cfg
+
+ Finally, reboot should result in IOMMU being enabled.
+
+* arm64 (a1.metal):
+ IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
+ supporting SMMU, which is implementation of IOMMU for arm64 architecture and
+ ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
+ hosts, one must disable IOMMU. This can be done by updating
+ ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
+ argument::
+
+ iommu.passthrough=1
+
+ Then, make the changes live by executing as a root::
+
+ # grub2-mkconfig > /boot/grub2/grub.cfg
+
+ Finally, reboot should result in IOMMU being disabled.
+ Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
+ working in no-IOMMU mode (please see above).
Usage example
-------------
git.droids-corp.org / dpdk.git / blobdiff