1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright (c) 2015-2019 Amazon.com, Inc. or its affiliates.
8 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
9 Network Adapter (ENA) family.
14 The ENA driver exposes a lightweight management interface with a
15 minimal set of memory mapped registers and an extendable command set
16 through an Admin Queue.
18 The driver supports a wide range of ENA adapters, is link-speed
19 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
20 etc.), and it negotiates and supports an extendable feature set.
22 ENA adapters allow high speed and low overhead Ethernet traffic
23 processing by providing a dedicated Tx/Rx queue pair per CPU core.
25 The ENA driver supports industry standard TCP/IP offload features such
26 as checksum offload and TCP transmit segmentation offload (TSO).
28 Receive-side scaling (RSS) is supported for multi-core scaling.
30 Some of the ENA devices support a working mode called Low-latency
31 Queue (LLQ), which saves several more microseconds.
36 ENA management interface is exposed by means of:
39 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
41 ENA device memory-mapped PCIe space for registers (MMIO registers)
42 are accessed only during driver initialization and are not involved
43 in further normal device operation.
45 AQ is used for submitting management commands, and the
46 results/responses are reported asynchronously through ACQ.
48 ENA introduces a very small set of management commands with room for
49 vendor-specific extensions. Most of the management operations are
50 framed in a generic Get/Set feature command.
52 The following admin queue commands are supported:
54 * Create I/O submission queue
55 * Create I/O completion queue
56 * Destroy I/O submission queue
57 * Destroy I/O completion queue
62 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
68 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
69 SQ correspondingly). Each SQ has a completion queue (CQ) associated
72 The SQs and CQs are implemented as descriptor rings in contiguous
75 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
77 The driver supports multi-queue for both Tx and Rx.
79 Configuration information
80 -------------------------
82 **DPDK Configuration Parameters**
84 The following configuration options are available for the ENA PMD:
86 * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
87 of the ENA PMD driver in the DPDK compilation.
89 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
90 logging of RX logic within the ENA PMD driver.
92 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
93 logging of TX logic within the ENA PMD driver.
95 * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
96 logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
98 **ENA Configuration Parameters**
100 * **Number of Queues**
102 This is the requested number of queues upon initialization, however, the actual
103 number of receive and transmit queues to be created will be the minimum between
104 the maximal number supported by the device and number of queues requested.
108 This is the requested size of receive/transmit queues, while the actual size
109 will be the minimum between the requested size and the maximal receive/transmit
110 supported by the device.
115 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
116 instructions on how to build DPDK.
118 By default the ENA PMD library will be built into the DPDK library.
120 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
121 documentation that comes with DPDK suite <linux_gsg>`.
123 Supported ENA adapters
124 ----------------------
126 Current ENA PMD supports the following ENA adapters including:
128 * ``1d0f:ec20`` - ENA VF
129 * ``1d0f:ec21`` - ENA VF with LLQ support
131 Supported Operating Systems
132 ---------------------------
134 Any Linux distribution fulfilling the conditions described in ``System Requirements``
135 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
141 * Jumbo frames up to 9K
142 * IPv4/TCP/UDP checksum offload
144 * Multiple receive and transmit queues
146 * RSS indirection table configuration
147 * Low Latency Queue for Tx
148 * Basic and extended statistics
149 * LSC event notification
150 * Watchdog (requires handling of timers in the application)
151 * Device reset upon failure
156 #. Prepare the system as recommended by DPDK suite. This includes environment
157 variables, hugepages configuration, tool-chains and configuration.
159 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
161 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
162 reduces the latency of the packets by pushing the header directly through
163 the PCI to the device, before the DMA is even triggered. For proper work
164 kernel PCI driver must support write combining (WC). In mainline version of
165 ``igb_uio`` (in DPDK repo) it must be enabled by loading module with
166 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
167 driver in kernel doesn't have WC support yet (planed to be added).
168 If vfio-pci used user should be either turn off ENAv2 (to avoid performance
169 impact) or recompile vfio-pci driver with patch provided in
170 `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
172 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
173 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
176 #. For ``vfio-pci`` users only:
177 Please make sure that ``IOMMU`` is enabled in your system,
178 or use ``vfio`` driver in ``noiommu`` mode::
180 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
182 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
185 At this point the system should be ready to run DPDK applications. Once the
186 application runs to completion, the ENA can be detached from igb_uio if necessary.
191 Follow instructions available in the document
192 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
193 **testpmd** with Amazon ENA devices managed by librte_pmd_ena.
197 .. code-block:: console
200 EAL: PCI device 0000:00:06.0 on NUMA socket -1
201 EAL: Invalid NUMA socket, default to 0
202 EAL: probe driver: 1d0f:ec20 net_ena
204 Interactive-mode selected
205 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
206 testpmd: preferred mempool ops selected: ring_mp_mc
207 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
208 Configuring Port 0 (socket 0)
209 Port 0: 00:00:00:11:00:01
210 Checking link statuses...