3 Copyright (c) 2015-2018 Amazon.com, Inc. or its affiliates.
6 Redistribution and use in source and binary forms, with or without
7 modification, are permitted provided that the following conditions
10 * Redistributions of source code must retain the above copyright
11 notice, this list of conditions and the following disclaimer.
12 * Redistributions in binary form must reproduce the above copyright
13 notice, this list of conditions and the following disclaimer in
14 the documentation and/or other materials provided with the
16 * Neither the name of Amazon.com, Inc. nor the names of its
17 contributors may be used to endorse or promote products derived
18 from this software without specific prior written permission.
20 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
22 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
23 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
25 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
26 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
27 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
28 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
29 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
30 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
35 The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic
36 Network Adapter (ENA) family.
41 The ENA driver exposes a lightweight management interface with a
42 minimal set of memory mapped registers and an extendable command set
43 through an Admin Queue.
45 The driver supports a wide range of ENA adapters, is link-speed
46 independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE,
47 etc.), and it negotiates and supports an extendable feature set.
49 ENA adapters allow high speed and low overhead Ethernet traffic
50 processing by providing a dedicated Tx/Rx queue pair per CPU core.
52 The ENA driver supports industry standard TCP/IP offload features such
53 as checksum offload and TCP transmit segmentation offload (TSO).
55 Receive-side scaling (RSS) is supported for multi-core scaling.
57 Some of the ENA devices support a working mode called Low-latency
58 Queue (LLQ), which saves several more microseconds.
63 ENA management interface is exposed by means of:
66 * Admin Queue (AQ) and Admin Completion Queue (ACQ)
68 ENA device memory-mapped PCIe space for registers (MMIO registers)
69 are accessed only during driver initialization and are not involved
70 in further normal device operation.
72 AQ is used for submitting management commands, and the
73 results/responses are reported asynchronously through ACQ.
75 ENA introduces a very small set of management commands with room for
76 vendor-specific extensions. Most of the management operations are
77 framed in a generic Get/Set feature command.
79 The following admin queue commands are supported:
81 * Create I/O submission queue
82 * Create I/O completion queue
83 * Destroy I/O submission queue
84 * Destroy I/O completion queue
89 Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature
95 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
96 SQ correspondingly). Each SQ has a completion queue (CQ) associated
99 The SQs and CQs are implemented as descriptor rings in contiguous
102 Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor
104 The driver supports multi-queue for both Tx and Rx.
106 Configuration information
107 -------------------------
109 **DPDK Configuration Parameters**
111 The following configuration options are available for the ENA PMD:
113 * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion
114 of the ENA PMD driver in the DPDK compilation.
116 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug
117 logging of RX logic within the ENA PMD driver.
119 * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug
120 logging of TX logic within the ENA PMD driver.
122 * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug
123 logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver.
125 **ENA Configuration Parameters**
127 * **Number of Queues**
129 This is the requested number of queues upon initialization, however, the actual
130 number of receive and transmit queues to be created will be the minimum between
131 the maximal number supported by the device and number of queues requested.
135 This is the requested size of receive/transmit queues, while the actual size
136 will be the minimum between the requested size and the maximal receive/transmit
137 supported by the device.
142 See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for
143 instructions on how to build DPDK.
145 By default the ENA PMD library will be built into the DPDK library.
147 For configuring and using UIO and VFIO frameworks, please also refer :ref:`the
148 documentation that comes with DPDK suite <linux_gsg>`.
150 Supported ENA adapters
151 ----------------------
153 Current ENA PMD supports the following ENA adapters including:
155 * ``1d0f:ec20`` - ENA VF
156 * ``1d0f:ec21`` - ENA VF with LLQ support
158 Supported Operating Systems
159 ---------------------------
161 Any Linux distribution fulfilling the conditions described in ``System Requirements``
162 section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*.
168 * Jumbo frames up to 9K
169 * IPv4/TCP/UDP checksum offload
171 * Multiple receive and transmit queues
173 * RSS indirection table configuration
174 * Low Latency Queue for Tx
175 * Basic and extended statistics
176 * LSC event notification
177 * Watchdog (requires handling of timers in the application)
178 * Device reset upon failure
183 #. Prepare the system as recommended by DPDK suite. This includes environment
184 variables, hugepages configuration, tool-chains and configuration.
186 #. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver.
188 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature
189 reduces the latency of the packets by pushing the header directly through
190 the PCI to the device, before the DMA is even triggered. For proper work
191 kernel PCI driver must support write combining (WC). In mainline version of
192 ``igb_uio`` (in DPDK repo) it must be enabled by loding module with
193 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci
194 driver in kernel doesn't have WC support yet (planed to be added).
195 If vfio-pci used user should be either turn off ENAv2 (to avoid performance
196 impact) or recompile vfio-pci driver with patch provided in
197 `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_.
199 #. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command
200 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1``
203 #. For ``vfio-pci`` users only:
204 Please make sure that ``IOMMU`` is enabled in your system,
205 or use ``vfio`` driver in ``noiommu`` mode::
207 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
209 #. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
212 At this point the system should be ready to run DPDK applications. Once the
213 application runs to completion, the ENA can be detached from igb_uio if necessary.
218 Follow instructions available in the document
219 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch
220 **testpmd** with Amazon ENA devices managed by librte_pmd_ena.
224 .. code-block:: console
227 EAL: PCI device 0000:00:06.0 on NUMA socket -1
228 EAL: Invalid NUMA socket, default to 0
229 EAL: probe driver: 1d0f:ec20 net_ena
231 Interactive-mode selected
232 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
233 testpmd: preferred mempool ops selected: ring_mp_mc
234 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
235 Configuring Port 0 (socket 0)
236 Port 0: 00:00:00:11:00:01
237 Checking link statuses...