1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2019 Intel Corporation.
4 .. include:: <isonum.txt>
6 Packet copying using Intel\ |reg| QuickData Technology
7 ======================================================
12 This sample is intended as a demonstration of the basic components of a DPDK
13 forwarding application and example of how to use IOAT driver API to make
16 Also while forwarding, the MAC addresses are affected as follows:
18 * The source MAC address is replaced by the TX port MAC address
20 * The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID
22 This application can be used to compare performance of using software packet
23 copy with copy done using a DMA device for different sizes of packets.
24 The example will print out statistics each second. The stats shows
25 received/send packets and packets dropped or failed to copy.
27 Compiling the Application
28 -------------------------
30 To compile the sample application see :doc:`compiling`.
32 The application is located in the ``ioat`` sub-directory.
35 Running the Application
36 -----------------------
38 In order to run the hardware copy application, the copying device
39 needs to be bound to user-space IO driver.
41 Refer to the "IOAT Rawdev Driver" chapter in the "Rawdev Drivers" document
42 for information on using the driver.
44 The application requires a number of command line options:
46 .. code-block:: console
48 ./<build_dir>/examples/dpdk-ioat [EAL options] -- [-p MASK] [-q NQ] [-s RS] [-c <sw|hw>]
53 * p MASK: A hexadecimal bitmask of the ports to configure (default is all)
55 * q NQ: Number of Rx queues used per port equivalent to CBDMA channels
56 per port (default is 1)
58 * c CT: Performed packet copy type: software (sw) or hardware using
59 DMA (hw) (default is hw)
61 * s RS: Size of IOAT rawdev ring for hardware copy mode or rte_ring for
62 software copy mode (default is 2048)
64 * --[no-]mac-updating: Whether MAC address of packets should be changed
65 or not (default is mac-updating)
67 The application can be launched in various configurations depending on
68 provided parameters. The app can use up to 2 lcores: one of them receives
69 incoming traffic and makes a copy of each packet. The second lcore then
70 updates MAC address and sends the copy. If one lcore per port is used,
71 both operations are done sequentially. For each configuration an additional
72 lcore is needed since the main lcore does not handle traffic but is
73 responsible for configuration, statistics printing and safe shutdown of
74 all ports and devices.
76 The application can use a maximum of 8 ports.
78 To run the application in a Linux environment with 3 lcores (the main lcore,
79 plus two forwarding cores), a single port (port 0), software copying and MAC
80 updating issue the command:
82 .. code-block:: console
84 $ ./<build_dir>/examples/dpdk-ioat -l 0-2 -n 2 -- -p 0x1 --mac-updating -c sw
86 To run the application in a Linux environment with 2 lcores (the main lcore,
87 plus one forwarding core), 2 ports (ports 0 and 1), hardware copying and no MAC
88 updating issue the command:
90 .. code-block:: console
92 $ ./<build_dir>/examples/dpdk-ioat -l 0-1 -n 1 -- -p 0x3 --no-mac-updating -c hw
94 Refer to the *DPDK Getting Started Guide* for general information on
95 running applications and the Environment Abstraction Layer (EAL) options.
100 The following sections provide an explanation of the main components of the
103 All DPDK library functions used in the sample code are prefixed with
104 ``rte_`` and are explained in detail in the *DPDK API Documentation*.
110 The ``main()`` function performs the initialization and calls the execution
111 threads for each lcore.
113 The first task is to initialize the Environment Abstraction Layer (EAL).
114 The ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
115 function. The value returned is the number of parsed arguments:
117 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
119 :start-after: Init EAL. 8<
120 :end-before: >8 End of init EAL.
124 The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers)
125 used by the application:
127 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
129 :start-after: Allocates mempool to hold the mbufs. 8<
130 :end-before: >8 End of allocates mempool to hold the mbufs.
133 Mbufs are the packet buffer structure used by DPDK. They are explained in
134 detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*.
136 The ``main()`` function also initializes the ports:
138 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
140 :start-after: Initialize each port. 8<
141 :end-before: >8 End of initializing each port.
144 Each port is configured using ``port_init()`` function. The Ethernet
145 ports are configured with local settings using the ``rte_eth_dev_configure()``
146 function and the ``port_conf`` struct. The RSS is enabled so that
147 multiple Rx queues could be used for packet receiving and copying by
148 multiple CBDMA channels per port:
150 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
152 :start-after: Configuring port to use RSS for multiple RX queues. 8<
153 :end-before: >8 End of configuring port to use RSS for multiple RX queues.
156 For this example the ports are set up with the number of Rx queues provided
157 with -q option and 1 Tx queue using the ``rte_eth_rx_queue_setup()``
158 and ``rte_eth_tx_queue_setup()`` functions.
160 The Ethernet port is then started:
162 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
164 :start-after: Start device. 8<
165 :end-before: >8 End of starting device.
169 Finally the Rx port is set in promiscuous mode:
171 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
173 :start-after: RX port is set in promiscuous mode. 8<
174 :end-before: >8 End of RX port is set in promiscuous mode.
178 After that each port application assigns resources needed.
180 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
182 :start-after: Assigning each port resources. 8<
183 :end-before: >8 End of assigning each port resources.
186 Depending on mode set (whether copy should be done by software or by hardware)
187 special structures are assigned to each port. If software copy was chosen,
188 application have to assign ring structures for packet exchanging between lcores
191 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
193 :start-after: Assign ring structures for packet exchanging. 8<
194 :end-before: >8 End of assigning ring structures for packet exchanging.
198 When using hardware copy each Rx queue of the port is assigned an
199 IOAT device (``assign_rawdevs()``) using IOAT Rawdev Driver API
202 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
204 :start-after: Using IOAT rawdev API functions. 8<
205 :end-before: >8 End of using IOAT rawdev API functions.
209 The initialization of hardware device is done by ``rte_rawdev_configure()``
210 function using ``rte_rawdev_info`` struct. After configuration the device is
211 started using ``rte_rawdev_start()`` function. Each of the above operations
212 is done in ``configure_rawdev_queue()``.
214 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
216 :start-after: Configuration of device. 8<
217 :end-before: >8 End of configuration of device.
220 If initialization is successful, memory for hardware device
221 statistics is allocated.
223 Finally ``main()`` function starts all packet handling lcores and starts
224 printing stats in a loop on the main lcore. The application can be
225 interrupted and closed using ``Ctrl-C``. The main lcore waits for
226 all worker lcores to finish, deallocates resources and exits.
228 The processing lcores launching function are described below.
230 The Lcores Launching Functions
231 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
233 As described above, ``main()`` function invokes ``start_forwarding_cores()``
234 function in order to start processing for each lcore:
236 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
238 :start-after: Start processing for each lcore. 8<
239 :end-before: >8 End of starting to processfor each lcore.
242 The function launches Rx/Tx processing functions on configured lcores
243 using ``rte_eal_remote_launch()``. The configured ports, their number
244 and number of assigned lcores are stored in user-defined
245 ``rxtx_transmission_config`` struct:
247 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
249 :start-after: Configuring ports and number of assigned lcores in struct. 8<
250 :end-before: >8 End of configuration of ports and number of assigned lcores.
253 The structure is initialized in 'main()' function with the values
254 corresponding to ports and lcores configuration provided by the user.
256 The Lcores Processing Functions
257 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
259 For receiving packets on each port, the ``ioat_rx_port()`` function is used.
260 The function receives packets on each configured Rx queue. Depending on the
261 mode the user chose, it will enqueue packets to IOAT rawdev channels and
262 then invoke copy process (hardware copy), or perform software copy of each
263 packet using ``pktmbuf_sw_copy()`` function and enqueue them to an rte_ring:
265 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
267 :start-after: Receive packets on one port and enqueue to IOAT rawdev or rte_ring. 8<
268 :end-before: >8 End of receive packets on one port and enqueue to IOAT rawdev or rte_ring.
271 The packets are received in burst mode using ``rte_eth_rx_burst()``
272 function. When using hardware copy mode the packets are enqueued in
273 copying device's buffer using ``ioat_enqueue_packets()`` which calls
274 ``rte_ioat_enqueue_copy()``. When all received packets are in the
275 buffer the copy operations are started by calling ``rte_ioat_perform_ops()``.
276 Function ``rte_ioat_enqueue_copy()`` operates on physical address of
277 the packet. Structure ``rte_mbuf`` contains only physical address to
278 start of the data buffer (``buf_iova``). Thus the address is adjusted
279 by ``addr_offset`` value in order to get the address of ``rearm_data``
280 member of ``rte_mbuf``. That way both the packet data and metadata can
281 be copied in a single operation. This method can be used because the mbufs
282 are direct mbufs allocated by the apps. If another app uses external buffers,
283 or indirect mbufs, then multiple copy operations must be used.
285 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
287 :start-after: Receive packets on one port and enqueue to IOAT rawdev or rte_ring. 8<
288 :end-before: >8 End of receive packets on one port and enqueue to IOAT rawdev or rte_ring.
292 All completed copies are processed by ``ioat_tx_port()`` function. When using
293 hardware copy mode the function invokes ``rte_ioat_completed_ops()``
294 on each assigned IOAT channel to gather copied packets. If software copy
295 mode is used the function dequeues copied packets from the rte_ring. Then each
296 packet MAC address is changed if it was enabled. After that copies are sent
297 in burst mode using `` rte_eth_tx_burst()``.
300 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
302 :start-after: Transmit packets from IOAT rawdev/rte_ring for one port. 8<
303 :end-before: >8 End of transmitting packets from IOAT.
306 The Packet Copying Functions
307 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
309 In order to perform packet copy there is a user-defined function
310 ``pktmbuf_sw_copy()`` used. It copies a whole packet by copying
311 metadata from source packet to new mbuf, and then copying a data
312 chunk of source packet. Both memory copies are done using
315 .. literalinclude:: ../../../examples/ioat/ioatfwd.c
317 :start-after: Perform packet copy there is a user-defined function. 8<
318 :end-before: >8 End of perform packet copy there is a user-defined function.
321 The metadata in this example is copied from ``rearm_data`` member of
322 ``rte_mbuf`` struct up to ``cacheline1``.
324 In order to understand why software packet copying is done as shown
325 above please refer to the "Mbuf Library" section of the
326 *DPDK Programmer's Guide*.