2 Copyright (C) NXP. 2016.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of NXP nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 DPAA2 Poll Mode Driver
32 ======================
34 The DPAA2 NIC PMD (**librte_pmd_dpaa2**) provides poll mode driver
35 support for the inbuilt NIC found in the **NXP DPAA2** SoC family.
37 More information can be found at `NXP Official Website
38 <http://www.nxp.com/products/microcontrollers-and-processors/arm-processors/qoriq-arm-processors:QORIQ-ARM>`_.
40 NXP DPAA2 (Data Path Acceleration Architecture Gen2)
41 ----------------------------------------------------
43 This section provides an overview of the NXP DPAA2 architecture
44 and how it is integrated into the DPDK.
49 - Overview of DPAA2 objects
50 - DPAA2 driver architecture overview
57 Reference: `FSL MC BUS in Linux Kernel <https://www.kernel.org/doc/readme/drivers-staging-fsl-mc-README.txt>`_.
59 DPAA2 is a hardware architecture designed for high-speed network
60 packet processing. DPAA2 consists of sophisticated mechanisms for
61 processing Ethernet packets, queue management, buffer management,
62 autonomous L2 switching, virtual Ethernet bridging, and accelerator
63 (e.g. crypto) sharing.
65 A DPAA2 hardware component called the Management Complex (or MC) manages the
66 DPAA2 hardware resources. The MC provides an object-based abstraction for
67 software drivers to use the DPAA2 hardware.
69 The MC uses DPAA2 hardware resources such as queues, buffer pools, and
70 network ports to create functional objects/devices such as network
71 interfaces, an L2 switch, or accelerator instances.
73 The MC provides memory-mapped I/O command interfaces (MC portals)
74 which DPAA2 software drivers use to operate on DPAA2 objects:
76 The diagram below shows an overview of the DPAA2 resource management
79 .. code-block:: console
81 +--------------------------------------+
85 +-----------------------------|--------+
87 | (create,discover,connect
91 +------------------------| mc portal |-+
93 | +- - - - - - - - - - - - -V- - -+ |
95 | | Management Complex (MC) | |
97 | +- - - - - - - - - - - - - - - -+ |
100 | Resources Objects |
101 | --------- ------- |
103 | -buffer pools -DPMCP |
104 | -Eth MACs/ports -DPIO |
105 | -network interface -DPNI |
107 | -queue portals -DPBP |
111 +--------------------------------------+
113 The MC mediates operations such as create, discover,
114 connect, configuration, and destroy. Fast-path operations
115 on data, such as packet transmit/receive, are not mediated by
116 the MC and are done directly using memory mapped regions in
119 Overview of DPAA2 Objects
120 ~~~~~~~~~~~~~~~~~~~~~~~~~
122 The section provides a brief overview of some key DPAA2 objects.
123 A simple scenario is described illustrating the objects involved
124 in creating a network interfaces.
126 DPRC (Datapath Resource Container)
128 A DPRC is a container object that holds all the other
129 types of DPAA2 objects. In the example diagram below there
130 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
133 .. code-block:: console
135 +---------------------------------------------------------+
138 | +-------+ +-------+ +-------+ +-------+ +-------+ |
139 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |
140 | +-------+ +-------+ +-------+ +---+---+ +---+---+ |
141 | | DPMCP | | DPIO | |
142 | +-------+ +-------+ |
146 +---------------------------------------------------------+
148 From the point of view of an OS, a DPRC behaves similar to a plug and
149 play bus, like PCI. DPRC commands can be used to enumerate the contents
150 of the DPRC, discover the hardware objects present (including mappable
151 regions and interrupts).
153 .. code-block:: console
157 +--+--------+-------+-------+-------+
159 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1
163 Hardware objects can be created and destroyed dynamically, providing
164 the ability to hot plug/unplug objects in and out of the DPRC.
166 A DPRC has a mappable MMIO region (an MC portal) that can be used
167 to send MC commands. It has an interrupt for status events (like
170 All objects in a container share the same hardware "isolation context".
171 This means that with respect to an IOMMU the isolation granularity
172 is at the DPRC (container) level, not at the individual object
175 DPRCs can be defined statically and populated with objects
176 via a config file passed to the MC when firmware starts
177 it. There is also a Linux user space tool called "restool"
178 that can be used to create/destroy containers and objects
181 DPAA2 Objects for an Ethernet Network Interface
182 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
185 queuing mechanisms, configuration mechanisms, buffer management,
186 physical ports, and interrupts. DPAA2 uses a more granular approach
187 utilizing multiple hardware objects. Each object provides specialized
188 functions. Groups of these objects are used by software to provide
189 Ethernet network interface functionality. This approach provides
190 efficient use of finite hardware resources, flexibility, and
191 performance advantages.
193 The diagram below shows the objects needed for a simple
194 network interface configuration on a system with 2 CPUs.
196 .. code-block:: console
219 Below the objects are described. For each object a brief description
220 is provided along with a summary of the kinds of operations the object
221 supports and a summary of key resources of the object (MMIO regions
224 DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
225 hardware device that connects to an Ethernet PHY and allows
226 physical transmission and reception of Ethernet frames.
229 - IRQs: DPNI link change
230 - commands: set link up/down, link config, get stats, IRQ config, enable, reset
232 DPNI (Datapath Network Interface): contains TX/RX queues,
233 network interface configuration, and RX buffer pool configuration
234 mechanisms. The TX/RX queues are in memory and are identified by
239 - commands: port config, offload config, queue config, parse/classify config, IRQ config, enable, reset
241 DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
242 packets and do hardware buffer pool management operations. The DPAA2
243 architecture separates the mechanism to access queues (the DPIO object)
244 from the queues themselves. The DPIO provides an MMIO interface to
245 enqueue/dequeue packets. To enqueue something a descriptor is written
246 to the DPIO MMIO region, which includes the target queue number.
247 There will typically be one DPIO assigned to each CPU. This allows all
248 CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are
249 expected to be shared by different DPAA2 drivers.
251 - MMIO regions: queue operations, buffer management
252 - IRQs: data availability, congestion notification, buffer pool depletion
253 - commands: IRQ config, enable, reset
255 DPBP (Datapath Buffer Pool): represents a hardware buffer
260 - commands: enable, reset
262 DPMCP (Datapath MC Portal): provides an MC command portal.
263 Used by drivers to send commands to the MC to manage
266 - MMIO regions: MC command portal
267 - IRQs: command completion
268 - commands: IRQ config, enable, reset
273 Some objects have explicit relationships that must
278 - DPNI <--> L2-switch-port
280 A DPNI must be connected to something such as a DPMAC,
281 another DPNI, or L2 switch port. The DPNI connection
282 is made via a DPRC command.
284 .. code-block:: console
294 A network interface requires a 'buffer pool' (DPBP object) which provides
295 a list of pointers to memory where received Ethernet data is to be copied.
296 The Ethernet driver configures the DPBPs associated with the network
302 All interrupts generated by DPAA2 objects are message
303 interrupts. At the hardware level message interrupts
304 generated by devices will normally have 3 components--
305 1) a non-spoofable 'device-id' expressed on the hardware
306 bus, 2) an address, 3) a data value.
308 In the case of DPAA2 devices/objects, all objects in the
309 same container/DPRC share the same 'device-id'.
310 For ARM-based SoC this is the same as the stream ID.
313 DPAA2 DPDK - Poll Mode Driver Overview
314 --------------------------------------
316 This section provides an overview of the drivers for
317 DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
318 drivers and 2) functional object drivers (such as Ethernet).
320 As described previously, a DPRC is a container that holds the other
321 types of DPAA2 objects. It is functionally similar to a plug-and-play
324 Each object in the DPRC is a Linux "device" and is bound to a driver.
325 The diagram below shows the dpaa2 drivers involved in a networking
326 scenario and the objects bound to each driver. A brief description
327 of each driver follows.
329 .. code-block: console
335 +------------+ +------------+
336 | Ethernet |.......| Mempool |
337 . . . . . . . . . | (DPNI) | | (DPBP) |
338 . +---+---+----+ +-----+------+
344 . . . . . . . . . . .| DPIO driver| .
349 +----+------+-------+ +-----+----- | .
351 | VFIO fslmc-bus |....................|.....................
354 +-------------------+ |
356 ========================== HARDWARE =====|=======================
364 =========================================|========================
367 A brief description of each driver is provided below.
372 The DPAA2 bus driver is a rte_bus driver which scans the fsl-mc bus.
373 Key functions include:
375 - Reading the container and setting up vfio group
376 - Scanning and parsing the various MC objects and adding them to
377 their respective device list.
379 Additionally, it also provides the object driver for generic MC objects.
384 The DPIO driver is bound to DPIO objects and provides services that allow
385 other drivers such as the Ethernet driver to enqueue and dequeue data for
386 their respective objects.
387 Key services include:
389 - Data availability notifications
390 - Hardware queuing operations (enqueue and dequeue of data)
391 - Hardware buffer pool management
393 To transmit a packet the Ethernet driver puts data on a queue and
394 invokes a DPIO API. For receive, the Ethernet driver registers
395 a data availability notification callback. To dequeue a packet
398 There is typically one DPIO object per physical CPU for optimum
399 performance, allowing different CPUs to simultaneously enqueue
402 The DPIO driver operates on behalf of all DPAA2 drivers
403 active -- Ethernet, crypto, compression, etc.
405 DPBP based Mempool driver
406 ~~~~~~~~~~~~~~~~~~~~~~~~~
408 The DPBP driver is bound to a DPBP objects and provides sevices to
409 create a hardware offloaded packet buffer mempool.
413 The Ethernet driver is bound to a DPNI and implements the kernel
414 interfaces needed to connect the DPAA2 network interface to
417 Each DPNI corresponds to a DPDK network interface.
422 Features of the DPAA2 PMD are:
424 - Multiple queues for TX and RX
425 - Receive Side Scaling (RSS)
426 - Packet type information
441 There are three main pre-requisities for executing DPAA2 PMD on a DPAA2
444 1. **ARM 64 Tool Chain**
446 For example, the `*aarch64* Linaro Toolchain <https://releases.linaro.org/components/toolchain/binaries/4.9-2017.01/aarch64-linux-gnu>`_.
450 It can be obtained from `NXP's Github hosting <https://github.com/qoriq-open-source/linux>`_.
452 3. **Rootfile system**
454 Any *aarch64* supporting filesystem can be used. For example,
455 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
456 from `here <http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
458 As an alternative method, DPAA2 PMD can also be executed using images provided
459 as part of SDK from NXP. The SDK includes all the above prerequisites necessary
460 to bring up a DPAA2 board.
462 The following dependencies are not part of DPDK and must be installed
467 NXP Linux software development kit (SDK) includes support for family
468 of QorIQ® ARM-Architecture-based system on chip (SoC) processors
469 and corresponding boards.
471 It includes the Linux board support packages (BSPs) for NXP SoCs,
472 a fully operational tool chain, kernel and board specific modules.
474 SDK and related information can be obtained from: `NXP QorIQ SDK <http://www.nxp.com/products/software-and-tools/run-time-software/linux-sdk/linux-sdk-for-qoriq-processors:SDKLINUX>`_.
476 - **DPDK Extra Scripts**
478 DPAA2 based resources can be configured easily with the help of ready scripts
479 as provided in the DPDK Extra repository.
481 `DPDK Extras Scripts <https://github.com/qoriq-open-source/dpdk-extras>`_.
483 Currently supported by DPDK:
486 - MC Firmware version **10.0.0** and higher.
487 - Supported architectures: **arm64 LE**.
489 - Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
493 Some part of fslmc bus code (mc flib - object library) routines are
494 dual licensed (BSD & GPLv2).
496 Pre-Installation Configuration
497 ------------------------------
502 The following options can be modified in the ``config`` file.
503 Please note that enabling debugging options may affect system performance.
505 - ``CONFIG_RTE_LIBRTE_FSLMC_BUS`` (default ``n``)
507 By default it is enabled only for defconfig_arm64-dpaa2-* config.
508 Toggle compilation of the ``librte_bus_fslmc`` driver.
510 - ``CONFIG_RTE_LIBRTE_DPAA2_PMD`` (default ``n``)
512 By default it is enabled only for defconfig_arm64-dpaa2-* config.
513 Toggle compilation of the ``librte_pmd_dpaa2`` driver.
515 - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_DRIVER`` (default ``n``)
517 Toggle display of generic debugging messages
519 - ``CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA`` (default ``y``)
521 Toggle to use physical address vs virtual address for hardware accelerators.
523 - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_INIT`` (default ``n``)
525 Toggle display of initialization related messages.
527 - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_RX`` (default ``n``)
529 Toggle display of receive fast path run-time message
531 - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX`` (default ``n``)
533 Toggle display of transmit fast path run-time message
535 - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE`` (default ``n``)
537 Toggle display of transmit fast path buffer free run-time message
539 Driver compilation and testing
540 ------------------------------
542 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
547 Follow instructions available in the document
548 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
553 .. code-block:: console
555 ./arm64-dpaa2-linuxapp-gcc/testpmd -c 0xff -n 1 \
556 -- -i --portmask=0x3 --nb-cores=1 --no-flush-rx
559 EAL: Registered [pci] bus.
560 EAL: Registered [fslmc] bus.
561 EAL: Detected 8 lcore(s)
562 EAL: Probing VFIO support...
563 EAL: VFIO support initialized
565 PMD: DPAA2: Processing Container = dprc.2
566 EAL: fslmc: DPRC contains = 51 devices
567 EAL: fslmc: Bus scan completed
569 Configuring Port 0 (socket 0)
570 Port 0: 00:00:00:00:00:01
571 Configuring Port 1 (socket 0)
572 Port 1: 00:00:00:00:00:02
574 Checking link statuses...
575 Port 0 Link Up - speed 10000 Mbps - full-duplex
576 Port 1 Link Up - speed 10000 Mbps - full-duplex
585 DPAA2 drivers for DPDK can only work on NXP SoCs as listed in the
586 ``Supported DPAA2 SoCs``.
588 Maximum packet length
589 ~~~~~~~~~~~~~~~~~~~~~
591 The DPAA2 SoC family support a maximum of a 10240 jumbo frame. The value
592 is fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len``
593 member of ``struct rte_eth_conf`` is set to a value lower than 10240, frames
594 up to 10240 bytes can still reach the host interface.