1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates.
8 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
9 Network Adapter (ENA) family.
14 The ENA driver exposes a lightweight management interface with a
15 minimal set of memory mapped registers and an extendable command set
16 through an Admin Queue.
18 The driver supports a wide range of ENA adapters, is link-speed
19 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
20 etc.), and it negotiates and supports an extendable feature set.
22 ENA adapters allow high speed and low overhead Ethernet traffic
23 processing by providing a dedicated Tx/Rx queue pair per CPU core.
25 The ENA driver supports industry standard TCP/IP offload features such
26 as checksum offload and TCP transmit segmentation offload (TSO).
28 Receive-side scaling (RSS) is supported for multi-core scaling.
30 Some of the ENA devices support a working mode called Low-latency
31 Queue (LLQ), which saves several more microseconds.
36 ENA management interface is exposed by means of:
39 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
41 ENA device memory-mapped PCIe space for registers (MMIO registers)
42 are accessed only during driver initialization and are not involved
43 in further normal device operation.
45 AQ is used for submitting management commands, and the
46 results/responses are reported asynchronously through ACQ.
48 ENA introduces a very small set of management commands with room for
49 vendor-specific extensions. Most of the management operations are
50 framed in a generic Get/Set feature command.
52 The following admin queue commands are supported:
54 * Create I/O submission queue
55 * Create I/O completion queue
56 * Destroy I/O submission queue
57 * Destroy I/O completion queue
62 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
68 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
69 SQ correspondingly). Each SQ has a completion queue (CQ) associated
72 The SQs and CQs are implemented as descriptor rings in contiguous
75 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
77 The driver supports multi-queue for both Tx and Rx.
79 Configuration information
80 -------------------------
82 **Runtime Configuration Parameters**
84 * **large_llq_hdr** (default 0)
86 Enables or disables usage of large LLQ headers. This option will have
87 effect only if the device also supports large LLQ headers. Otherwise, the
88 default value will be used.
90 * **miss_txc_to** (default 5)
92 Number of seconds after which the Tx packet will be considered missing.
93 If the missing packets number will exceed dynamically calculated threshold,
94 the driver will trigger the device reset which should be handled by the
95 application. Checking for missing Tx completions happens in the driver's
96 timer service. Setting this parameter to 0 disables this feature. Maximum
97 allowed value is 60 seconds.
99 **ENA Configuration Parameters**
101 * **Number of Queues**
103 This is the requested number of queues upon initialization, however, the actual
104 number of receive and transmit queues to be created will be the minimum between
105 the maximal number supported by the device and number of queues requested.
109 This is the requested size of receive/transmit queues, while the actual size
110 will be the minimum between the requested size and the maximal receive/transmit
111 supported by the device.
116 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
117 instructions on how to build DPDK.
119 By default the ENA PMD library will be built into the DPDK library.
121 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
122 documentation that comes with DPDK suite <linux_gsg>`.
124 Supported ENA adapters
125 ----------------------
127 Current ENA PMD supports the following ENA adapters including:
129 * ``1d0f:ec20`` - ENA VF
130 * ``1d0f:ec21`` - ENA VF RSERV0
132 Supported Operating Systems
133 ---------------------------
135 Any Linux distribution fulfilling the conditions described in ``System Requirements``
136 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
142 * Jumbo frames up to 9K
143 * IPv4/TCP/UDP checksum offload
145 * Multiple receive and transmit queues
147 * RSS indirection table configuration
148 * Low Latency Queue for Tx
149 * Basic and extended statistics
150 * LSC event notification
151 * Watchdog (requires handling of timers in the application)
152 * Device reset upon failure
158 #. Prepare the system as recommended by DPDK suite. This includes environment
159 variables, hugepages configuration, tool-chains and configuration.
161 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
163 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
164 reduces the latency of the packets by pushing the header directly through
165 the PCI to the device, before the DMA is even triggered. For proper work
166 kernel PCI driver must support write combining (WC).
167 In DPDK ``igb_uio`` it must be enabled by loading module with
168 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
169 driver in kernel doesn't have WC support yet (planed to be added).
170 If vfio-pci is used user should follow `AWS ENA PMD documentation
171 <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/README.md>`_.
173 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
174 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
177 #. For ``vfio-pci`` users only:
178 Please make sure that ``IOMMU`` is enabled in your system,
179 or use ``vfio`` driver in ``noiommu`` mode::
181 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
183 To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
184 ``CONFIG_VFIO_NOIOMMU``.
186 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
188 At this point the system should be ready to run DPDK applications. Once the
189 application runs to completion, the ENA can be detached from attached module if
192 **Rx interrupts support**
194 ENA PMD supports Rx interrupts, which can be used to wake up lcores waiting for
195 input. Please note that it won't work with ``igb_uio``, so to use this feature,
196 the ``vfio-pci`` should be used.
198 ENA handles admin interrupts and AENQ notifications on separate interrupt.
199 There is possibility that there won't be enough event file descriptors to
200 handle both admin and Rx interrupts. In that situation the Rx interrupt request
203 **Note about usage on \*.metal instances**
205 On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
208 * x86_64 (e.g. c5.metal, i3.metal):
209 IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
210 be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
213 When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
214 feature, while ``vfio-pci`` should work without any changes.
215 To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
216 ``/etc/default/grub`` with the below extra boot arguments::
218 iommu=1 intel_iommu=on
220 Then, make the changes live by executing as a root::
222 # grub2-mkconfig > /boot/grub2/grub.cfg
224 Finally, reboot should result in IOMMU being enabled.
227 IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
228 supporting SMMU, which is implementation of IOMMU for arm64 architecture and
229 ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
230 hosts, one must disable IOMMU. This can be done by updating
231 ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
236 Then, make the changes live by executing as a root::
238 # grub2-mkconfig > /boot/grub2/grub.cfg
240 Finally, reboot should result in IOMMU being disabled.
241 Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
242 working in no-IOMMU mode (please see above).
247 Follow instructions available in the document
248 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
249 **testpmd** with Amazon ENA devices managed by librte_net_ena.
253 .. code-block:: console
256 EAL: PCI device 0000:00:06.0 on NUMA socket -1
257 EAL: Device 0000:00:06.0 is not NUMA-aware, defaulting socket to 0
258 EAL: probe driver: 1d0f:ec20 net_ena
260 Interactive-mode selected
261 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
262 testpmd: preferred mempool ops selected: ring_mp_mc
263 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
264 Configuring Port 0 (socket 0)
265 Port 0: 00:00:00:11:00:01
266 Checking link statuses...