1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
8 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
9 Network Adapter (ENA) family.
14 The ENA driver exposes a lightweight management interface with a
15 minimal set of memory mapped registers and an extendable command set
16 through an Admin Queue.
18 The driver supports a wide range of ENA adapters, is link-speed
19 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
20 etc.), and it negotiates and supports an extendable feature set.
22 ENA adapters allow high speed and low overhead Ethernet traffic
23 processing by providing a dedicated Tx/Rx queue pair per CPU core.
25 The ENA driver supports industry standard TCP/IP offload features such
26 as checksum offload and TCP transmit segmentation offload (TSO).
28 Receive-side scaling (RSS) is supported for multi-core scaling.
30 Some of the ENA devices support a working mode called Low-latency
31 Queue (LLQ), which saves several more microseconds.
36 ENA management interface is exposed by means of:
39 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
41 ENA device memory-mapped PCIe space for registers (MMIO registers)
42 are accessed only during driver initialization and are not involved
43 in further normal device operation.
45 AQ is used for submitting management commands, and the
46 results/responses are reported asynchronously through ACQ.
48 ENA introduces a very small set of management commands with room for
49 vendor-specific extensions. Most of the management operations are
50 framed in a generic Get/Set feature command.
52 The following admin queue commands are supported:
54 * Create I/O submission queue
55 * Create I/O completion queue
56 * Destroy I/O submission queue
57 * Destroy I/O completion queue
62 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
68 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
69 SQ correspondingly). Each SQ has a completion queue (CQ) associated
72 The SQs and CQs are implemented as descriptor rings in contiguous
75 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
77 The driver supports multi-queue for both Tx and Rx.
79 Configuration information
80 -------------------------
82 **Runtime Configuration Parameters**
84 * **large_llq_hdr** (default 0)
86 Enables or disables usage of large LLQ headers. This option will have
87 effect only if the device also supports large LLQ headers. Otherwise, the
88 default value will be used.
90 **ENA Configuration Parameters**
92 * **Number of Queues**
94 This is the requested number of queues upon initialization, however, the actual
95 number of receive and transmit queues to be created will be the minimum between
96 the maximal number supported by the device and number of queues requested.
100 This is the requested size of receive/transmit queues, while the actual size
101 will be the minimum between the requested size and the maximal receive/transmit
102 supported by the device.
107 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
108 instructions on how to build DPDK.
110 By default the ENA PMD library will be built into the DPDK library.
112 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
113 documentation that comes with DPDK suite <linux_gsg>`.
115 Supported ENA adapters
116 ----------------------
118 Current ENA PMD supports the following ENA adapters including:
120 * ``1d0f:ec20`` - ENA VF
121 * ``1d0f:ec21`` - ENA VF RSERV0
123 Supported Operating Systems
124 ---------------------------
126 Any Linux distribution fulfilling the conditions described in ``System Requirements``
127 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
133 * Jumbo frames up to 9K
134 * IPv4/TCP/UDP checksum offload
136 * Multiple receive and transmit queues
138 * RSS indirection table configuration
139 * Low Latency Queue for Tx
140 * Basic and extended statistics
141 * LSC event notification
142 * Watchdog (requires handling of timers in the application)
143 * Device reset upon failure
149 #. Prepare the system as recommended by DPDK suite. This includes environment
150 variables, hugepages configuration, tool-chains and configuration.
152 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
154 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
155 reduces the latency of the packets by pushing the header directly through
156 the PCI to the device, before the DMA is even triggered. For proper work
157 kernel PCI driver must support write combining (WC).
158 In DPDK ``igb_uio`` it must be enabled by loading module with
159 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
160 driver in kernel doesn't have WC support yet (planed to be added).
161 If vfio-pci used user should be either turn off ENAv2 (to avoid performance
162 impact) or recompile vfio-pci driver with patch provided in
163 `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
165 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
166 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
169 #. For ``vfio-pci`` users only:
170 Please make sure that ``IOMMU`` is enabled in your system,
171 or use ``vfio`` driver in ``noiommu`` mode::
173 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
175 To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
176 ``CONFIG_VFIO_NOIOMMU``.
178 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
180 At this point the system should be ready to run DPDK applications. Once the
181 application runs to completion, the ENA can be detached from attached module if
184 **Rx interrupts support**
186 ENA PMD supports Rx interrupts, which can be used to wake up lcores waiting for
187 input. Please note that it won't work with ``igb_uio``, so to use this feature,
188 the ``vfio-pci`` should be used.
190 ENA handles admin interrupts and AENQ notifications on separate interrupt.
191 There is possibility that there won't be enough event file descriptors to
192 handle both admin and Rx interrupts. In that situation the Rx interrupt request
195 **Note about usage on \*.metal instances**
197 On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
200 * x86_64 (e.g. c5.metal, i3.metal):
201 IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
202 be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
205 When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
206 feature, while ``vfio-pci`` should work without any changes.
207 To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
208 ``/etc/default/grub`` with the below extra boot arguments::
210 iommu=1 intel_iommu=on
212 Then, make the changes live by executing as a root::
214 # grub2-mkconfig > /boot/grub2/grub.cfg
216 Finally, reboot should result in IOMMU being enabled.
219 IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
220 supporting SMMU, which is implementation of IOMMU for arm64 architecture and
221 ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
222 hosts, one must disable IOMMU. This can be done by updating
223 ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
228 Then, make the changes live by executing as a root::
230 # grub2-mkconfig > /boot/grub2/grub.cfg
232 Finally, reboot should result in IOMMU being disabled.
233 Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
234 working in no-IOMMU mode (please see above).
239 Follow instructions available in the document
240 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
241 **testpmd** with Amazon ENA devices managed by librte_net_ena.
245 .. code-block:: console
248 EAL: PCI device 0000:00:06.0 on NUMA socket -1
249 EAL: Device 0000:00:06.0 is not NUMA-aware, defaulting socket to 0
250 EAL: probe driver: 1d0f:ec20 net_ena
252 Interactive-mode selected
253 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
254 testpmd: preferred mempool ops selected: ring_mp_mc
255 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
256 Configuring Port 0 (socket 0)
257 Port 0: 00:00:00:11:00:01
258 Checking link statuses...