1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
8 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
9 Network Adapter (ENA) family.
14 The ENA driver exposes a lightweight management interface with a
15 minimal set of memory mapped registers and an extendable command set
16 through an Admin Queue.
18 The driver supports a wide range of ENA adapters, is link-speed
19 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
20 etc.), and it negotiates and supports an extendable feature set.
22 ENA adapters allow high speed and low overhead Ethernet traffic
23 processing by providing a dedicated Tx/Rx queue pair per CPU core.
25 The ENA driver supports industry standard TCP/IP offload features such
26 as checksum offload and TCP transmit segmentation offload (TSO).
28 Receive-side scaling (RSS) is supported for multi-core scaling.
30 Some of the ENA devices support a working mode called Low-latency
31 Queue (LLQ), which saves several more microseconds.
36 ENA management interface is exposed by means of:
39 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
41 ENA device memory-mapped PCIe space for registers (MMIO registers)
42 are accessed only during driver initialization and are not involved
43 in further normal device operation.
45 AQ is used for submitting management commands, and the
46 results/responses are reported asynchronously through ACQ.
48 ENA introduces a very small set of management commands with room for
49 vendor-specific extensions. Most of the management operations are
50 framed in a generic Get/Set feature command.
52 The following admin queue commands are supported:
54 * Create I/O submission queue
55 * Create I/O completion queue
56 * Destroy I/O submission queue
57 * Destroy I/O completion queue
62 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
68 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
69 SQ correspondingly). Each SQ has a completion queue (CQ) associated
72 The SQs and CQs are implemented as descriptor rings in contiguous
75 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
77 The driver supports multi-queue for both Tx and Rx.
79 Configuration information
80 -------------------------
82 **Runtime Configuration Parameters**
84 * **large_llq_hdr** (default 0)
86 Enables or disables usage of large LLQ headers. This option will have
87 effect only if the device also supports large LLQ headers. Otherwise, the
88 default value will be used.
90 * **miss_txc_to** (default 5)
92 Number of seconds after which the Tx packet will be considered missing.
93 If the missing packets number will exceed dynamically calculated threshold,
94 the driver will trigger the device reset which should be handled by the
95 application. Checking for missing Tx completions happens in the driver's
96 timer service. Setting this parameter to 0 disables this feature. Maximum
97 allowed value is 60 seconds.
99 * **enable_llq** (default 1)
101 Determines whenever the driver should use the LLQ (if it's available) or
104 **NOTE: On the 6th generation AWS instances disabling LLQ may lead to a
105 huge performance degradation. In general disabling LLQ is highly not
108 **ENA Configuration Parameters**
110 * **Number of Queues**
112 This is the requested number of queues upon initialization, however, the actual
113 number of receive and transmit queues to be created will be the minimum between
114 the maximal number supported by the device and number of queues requested.
118 This is the requested size of receive/transmit queues, while the actual size
119 will be the minimum between the requested size and the maximal receive/transmit
120 supported by the device.
125 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
126 instructions on how to build DPDK.
128 By default the ENA PMD library will be built into the DPDK library.
130 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
131 documentation that comes with DPDK suite <linux_gsg>`.
133 Supported ENA adapters
134 ----------------------
136 Current ENA PMD supports the following ENA adapters including:
138 * ``1d0f:ec20`` - ENA VF
139 * ``1d0f:ec21`` - ENA VF RSERV0
141 Supported Operating Systems
142 ---------------------------
144 Any Linux distribution fulfilling the conditions described in ``System Requirements``
145 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
151 * Jumbo frames up to 9K
152 * IPv4/TCP/UDP checksum offload
154 * Multiple receive and transmit queues
156 * RSS indirection table configuration
157 * Low Latency Queue for Tx
158 * Basic and extended statistics
159 * LSC event notification
160 * Watchdog (requires handling of timers in the application)
161 * Device reset upon failure
167 #. Prepare the system as recommended by DPDK suite. This includes environment
168 variables, hugepages configuration, tool-chains and configuration.
170 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
172 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
173 reduces the latency of the packets by pushing the header directly through
174 the PCI to the device, before the DMA is even triggered. For proper work
175 kernel PCI driver must support write combining (WC).
176 In DPDK ``igb_uio`` it must be enabled by loading module with
177 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
178 driver in kernel doesn't have WC support yet (planed to be added).
179 If vfio-pci is used user should follow `AWS ENA PMD documentation
180 <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/README.md>`_.
182 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
183 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
186 #. For ``vfio-pci`` users only:
187 Please make sure that ``IOMMU`` is enabled in your system,
188 or use ``vfio`` driver in ``noiommu`` mode::
190 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
192 To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
193 ``CONFIG_VFIO_NOIOMMU``.
195 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
197 At this point the system should be ready to run DPDK applications. Once the
198 application runs to completion, the ENA can be detached from attached module if
201 **Rx interrupts support**
203 ENA PMD supports Rx interrupts, which can be used to wake up lcores waiting for
204 input. Please note that it won't work with ``igb_uio``, so to use this feature,
205 the ``vfio-pci`` should be used.
207 ENA handles admin interrupts and AENQ notifications on separate interrupt.
208 There is possibility that there won't be enough event file descriptors to
209 handle both admin and Rx interrupts. In that situation the Rx interrupt request
212 **Note about usage on \*.metal instances**
214 On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
217 * x86_64 (e.g. c5.metal, i3.metal):
218 IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
219 be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
222 When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
223 feature, while ``vfio-pci`` should work without any changes.
224 To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
225 ``/etc/default/grub`` with the below extra boot arguments::
227 iommu=1 intel_iommu=on
229 Then, make the changes live by executing as a root::
231 # grub2-mkconfig > /boot/grub2/grub.cfg
233 Finally, reboot should result in IOMMU being enabled.
236 IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
237 supporting SMMU, which is implementation of IOMMU for arm64 architecture and
238 ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
239 hosts, one must disable IOMMU. This can be done by updating
240 ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
245 Then, make the changes live by executing as a root::
247 # grub2-mkconfig > /boot/grub2/grub.cfg
249 Finally, reboot should result in IOMMU being disabled.
250 Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
251 working in no-IOMMU mode (please see above).
256 Follow instructions available in the document
257 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
258 **testpmd** with Amazon ENA devices managed by librte_net_ena.
262 .. code-block:: console
265 EAL: PCI device 0000:00:06.0 on NUMA socket -1
266 EAL: Device 0000:00:06.0 is not NUMA-aware, defaulting socket to 0
267 EAL: probe driver: 1d0f:ec20 net_ena
269 Interactive-mode selected
270 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
271 testpmd: preferred mempool ops selected: ring_mp_mc
272 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
273 Configuring Port 0 (socket 0)
274 Port 0: 00:00:00:11:00:01
275 Checking link statuses...