1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
8 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
9 Network Adapter (ENA) family.
14 The ENA driver exposes a lightweight management interface with a
15 minimal set of memory mapped registers and an extendable command set
16 through an Admin Queue.
18 The driver supports a wide range of ENA adapters, is link-speed
19 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
20 etc.), and it negotiates and supports an extendable feature set.
22 ENA adapters allow high speed and low overhead Ethernet traffic
23 processing by providing a dedicated Tx/Rx queue pair per CPU core.
25 The ENA driver supports industry standard TCP/IP offload features such
26 as checksum offload and TCP transmit segmentation offload (TSO).
28 Receive-side scaling (RSS) is supported for multi-core scaling.
30 Some of the ENA devices support a working mode called Low-latency
31 Queue (LLQ), which saves several more microseconds.
36 ENA management interface is exposed by means of:
39 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
41 ENA device memory-mapped PCIe space for registers (MMIO registers)
42 are accessed only during driver initialization and are not involved
43 in further normal device operation.
45 AQ is used for submitting management commands, and the
46 results/responses are reported asynchronously through ACQ.
48 ENA introduces a very small set of management commands with room for
49 vendor-specific extensions. Most of the management operations are
50 framed in a generic Get/Set feature command.
52 The following admin queue commands are supported:
54 * Create I/O submission queue
55 * Create I/O completion queue
56 * Destroy I/O submission queue
57 * Destroy I/O completion queue
62 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
68 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
69 SQ correspondingly). Each SQ has a completion queue (CQ) associated
72 The SQs and CQs are implemented as descriptor rings in contiguous
75 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
77 The driver supports multi-queue for both Tx and Rx.
79 Configuration information
80 -------------------------
82 **DPDK Configuration Parameters**
84 The following configuration options are available for the ENA PMD:
86 * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
87 of the ENA PMD driver in the DPDK compilation.
89 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
90 logging of RX logic within the ENA PMD driver.
92 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
93 logging of TX logic within the ENA PMD driver.
95 * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
96 logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
98 **Runtime Configuration Parameters**
100 * **large_llq_hdr** (default 0)
102 Enables or disables usage of large LLQ headers. This option will have
103 effect only if the device also supports large LLQ headers. Otherwise, the
104 default value will be used.
106 **ENA Configuration Parameters**
108 * **Number of Queues**
110 This is the requested number of queues upon initialization, however, the actual
111 number of receive and transmit queues to be created will be the minimum between
112 the maximal number supported by the device and number of queues requested.
116 This is the requested size of receive/transmit queues, while the actual size
117 will be the minimum between the requested size and the maximal receive/transmit
118 supported by the device.
123 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
124 instructions on how to build DPDK.
126 By default the ENA PMD library will be built into the DPDK library.
128 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
129 documentation that comes with DPDK suite <linux_gsg>`.
131 Supported ENA adapters
132 ----------------------
134 Current ENA PMD supports the following ENA adapters including:
136 * ``1d0f:ec20`` - ENA VF
137 * ``1d0f:ec21`` - ENA VF with LLQ support
139 Supported Operating Systems
140 ---------------------------
142 Any Linux distribution fulfilling the conditions described in ``System Requirements``
143 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
149 * Jumbo frames up to 9K
150 * IPv4/TCP/UDP checksum offload
152 * Multiple receive and transmit queues
154 * RSS indirection table configuration
155 * Low Latency Queue for Tx
156 * Basic and extended statistics
157 * LSC event notification
158 * Watchdog (requires handling of timers in the application)
159 * Device reset upon failure
164 #. Prepare the system as recommended by DPDK suite. This includes environment
165 variables, hugepages configuration, tool-chains and configuration.
167 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
169 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
170 reduces the latency of the packets by pushing the header directly through
171 the PCI to the device, before the DMA is even triggered. For proper work
172 kernel PCI driver must support write combining (WC). In mainline version of
173 ``igb_uio`` (in DPDK repo) it must be enabled by loading module with
174 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
175 driver in kernel doesn't have WC support yet (planed to be added).
176 If vfio-pci used user should be either turn off ENAv2 (to avoid performance
177 impact) or recompile vfio-pci driver with patch provided in
178 `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
180 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
181 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
184 #. For ``vfio-pci`` users only:
185 Please make sure that ``IOMMU`` is enabled in your system,
186 or use ``vfio`` driver in ``noiommu`` mode::
188 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
190 To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
191 ``CONFIG_VFIO_NOIOMMU``.
193 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
195 At this point the system should be ready to run DPDK applications. Once the
196 application runs to completion, the ENA can be detached from attached module if
199 **Note about usage on \*.metal instances**
201 On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
204 * x86_64 (e.g. c5.metal, i3.metal):
205 IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
206 be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
209 When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
210 feature, while ``vfio-pci`` should work without any changes.
211 To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
212 ``/etc/default/grub`` with the below extra boot arguments::
214 iommu=1 intel_iommu=on
216 Then, make the changes live by executing as a root::
218 # grub2-mkconfig > /boot/grub2/grub.cfg
220 Finally, reboot should result in IOMMU being enabled.
223 IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
224 supporting SMMU, which is implementation of IOMMU for arm64 architecture and
225 ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
226 hosts, one must disable IOMMU. This can be done by updating
227 ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
232 Then, make the changes live by executing as a root::
234 # grub2-mkconfig > /boot/grub2/grub.cfg
236 Finally, reboot should result in IOMMU being disabled.
237 Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
238 working in no-IOMMU mode (please see above).
243 Follow instructions available in the document
244 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
245 **testpmd** with Amazon ENA devices managed by librte_pmd_ena.
249 .. code-block:: console
252 EAL: PCI device 0000:00:06.0 on NUMA socket -1
253 EAL: Invalid NUMA socket, default to 0
254 EAL: probe driver: 1d0f:ec20 net_ena
256 Interactive-mode selected
257 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
258 testpmd: preferred mempool ops selected: ring_mp_mc
259 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
260 Configuring Port 0 (socket 0)
261 Port 0: 00:00:00:11:00:01
262 Checking link statuses...