1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2010-2014 Intel Corporation.
6 Multi-process Sample Application
7 ================================
9 This chapter describes the example applications for multi-processing that are included in the DPDK.
14 Building the Sample Applications
15 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
16 The multi-process example applications are built in the same way as other sample applications,
17 and as documented in the *DPDK Getting Started Guide*.
20 To compile the sample application see :doc:`compiling`.
22 The applications are located in the ``multi_process`` sub-directory.
26 If just a specific multi-process application needs to be built,
27 the final make command can be run just in that application's directory,
28 rather than at the top-level multi-process directory.
30 Basic Multi-process Example
31 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
33 The examples/simple_mp folder in the DPDK release contains a basic example application to demonstrate how
34 two DPDK processes can work together using queues and memory pools to share information.
36 Running the Application
37 ^^^^^^^^^^^^^^^^^^^^^^^
39 To run the application, start one copy of the simple_mp binary in one terminal,
40 passing at least two cores in the coremask/corelist, as follows:
42 .. code-block:: console
44 ./build/simple_mp -l 0-1 -n 4 --proc-type=primary
46 For the first DPDK process run, the proc-type flag can be omitted or set to auto,
47 since all DPDK processes will default to being a primary instance,
48 meaning they have control over the hugepage shared memory regions.
49 The process should start successfully and display a command prompt as follows:
51 .. code-block:: console
53 $ ./build/simple_mp -l 0-1 -n 4 --proc-type=primary
54 EAL: coremask set to 3
55 EAL: Detected lcore 0 on socket 0
56 EAL: Detected lcore 1 on socket 0
57 EAL: Detected lcore 2 on socket 0
58 EAL: Detected lcore 3 on socket 0
61 EAL: Requesting 2 pages of size 1073741824
62 EAL: Requesting 768 pages of size 2097152
63 EAL: Ask a virtual area of 0x40000000 bytes
64 EAL: Virtual area found at 0x7ff200000000 (size = 0x40000000)
67 EAL: Master core 0 is ready (tid=54e41820)
68 EAL: Core 1 is ready (tid=53b32700)
74 To run the secondary process to communicate with the primary process,
75 again run the same binary setting at least two cores in the coremask/corelist:
77 .. code-block:: console
79 ./build/simple_mp -l 2-3 -n 4 --proc-type=secondary
81 When running a secondary process such as that shown above, the proc-type parameter can again be specified as auto.
82 However, omitting the parameter altogether will cause the process to try and start as a primary rather than secondary process.
84 Once the process type is specified correctly,
85 the process starts up, displaying largely similar status messages to the primary instance as it initializes.
86 Once again, you will be presented with a command prompt.
88 Once both processes are running, messages can be sent between them using the send command.
89 At any stage, either process can be terminated using the quit command.
91 .. code-block:: console
93 EAL: Master core 10 is ready (tid=b5f89820) EAL: Master core 8 is ready (tid=864a3820)
94 EAL: Core 11 is ready (tid=84ffe700) EAL: Core 9 is ready (tid=85995700)
95 Starting core 11 Starting core 9
96 simple_mp > send hello_secondary simple_mp > core 9: Received 'hello_secondary'
97 simple_mp > core 11: Received 'hello_primary' simple_mp > send hello_primary
98 simple_mp > quit simple_mp > quit
102 If the primary instance is terminated, the secondary instance must also be shut-down and restarted after the primary.
103 This is necessary because the primary instance will clear and reset the shared memory regions on startup,
104 invalidating the secondary process's pointers.
105 The secondary process can be stopped and restarted without affecting the primary process.
107 How the Application Works
108 ^^^^^^^^^^^^^^^^^^^^^^^^^
110 The core of this example application is based on using two queues and a single memory pool in shared memory.
111 These three objects are created at startup by the primary process,
112 since the secondary process cannot create objects in memory as it cannot reserve memory zones,
113 and the secondary process then uses lookup functions to attach to these objects as it starts up.
117 if (rte_eal_process_type() == RTE_PROC_PRIMARY){
118 send_ring = rte_ring_create(_PRI_2_SEC, ring_size, SOCKET0, flags);
119 recv_ring = rte_ring_create(_SEC_2_PRI, ring_size, SOCKET0, flags);
120 message_pool = rte_mempool_create(_MSG_POOL, pool_size, string_size, pool_cache, priv_data_sz, NULL, NULL, NULL, NULL, SOCKET0, flags);
122 recv_ring = rte_ring_lookup(_PRI_2_SEC);
123 send_ring = rte_ring_lookup(_SEC_2_PRI);
124 message_pool = rte_mempool_lookup(_MSG_POOL);
127 Note, however, that the named ring structure used as send_ring in the primary process is the recv_ring in the secondary process.
129 Once the rings and memory pools are all available in both the primary and secondary processes,
130 the application simply dedicates two threads to sending and receiving messages respectively.
131 The receive thread simply dequeues any messages on the receive ring, prints them,
132 and frees the buffer space used by the messages back to the memory pool.
133 The send thread makes use of the command-prompt library to interactively request user input for messages to send.
134 Once a send command is issued by the user, a buffer is allocated from the memory pool, filled in with the message contents,
135 then enqueued on the appropriate rte_ring.
137 Symmetric Multi-process Example
138 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
140 The second example of DPDK multi-process support demonstrates how a set of processes can run in parallel,
141 with each process performing the same set of packet- processing operations.
142 (Since each process is identical in functionality to the others,
143 we refer to this as symmetric multi-processing, to differentiate it from asymmetric multi- processing -
144 such as a client-server mode of operation seen in the next example,
145 where different processes perform different tasks, yet co-operate to form a packet-processing system.)
146 The following diagram shows the data-flow through the application, using two processes.
148 .. _figure_sym_multi_proc_app:
150 .. figure:: img/sym_multi_proc_app.*
152 Example Data Flow in a Symmetric Multi-process Application
155 As the diagram shows, each process reads packets from each of the network ports in use.
156 RSS is used to distribute incoming packets on each port to different hardware RX queues.
157 Each process reads a different RX queue on each port and so does not contend with any other process for that queue access.
158 Similarly, each process writes outgoing packets to a different TX queue on each port.
160 Running the Application
161 ^^^^^^^^^^^^^^^^^^^^^^^
163 As with the simple_mp example, the first instance of the symmetric_mp process must be run as the primary instance,
164 though with a number of other application- specific parameters also provided after the EAL arguments.
165 These additional parameters are:
167 * -p <portmask>, where portmask is a hexadecimal bitmask of what ports on the system are to be used.
168 For example: -p 3 to use ports 0 and 1 only.
170 * --num-procs <N>, where N is the total number of symmetric_mp instances that will be run side-by-side to perform packet processing.
171 This parameter is used to configure the appropriate number of receive queues on each network port.
173 * --proc-id <n>, where n is a numeric value in the range 0 <= n < N (number of processes, specified above).
174 This identifies which symmetric_mp instance is being run, so that each process can read a unique receive queue on each network port.
176 The secondary symmetric_mp instances must also have these parameters specified,
177 and the first two must be the same as those passed to the primary instance, or errors result.
179 For example, to run a set of four symmetric_mp instances, running on lcores 1-4,
180 all performing level-2 forwarding of packets between ports 0 and 1,
181 the following commands can be used (assuming run as root):
183 .. code-block:: console
185 # ./build/symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=0
186 # ./build/symmetric_mp -l 2 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=1
187 # ./build/symmetric_mp -l 3 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=2
188 # ./build/symmetric_mp -l 4 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=3
192 In the above example, the process type can be explicitly specified as primary or secondary, rather than auto.
193 When using auto, the first process run creates all the memory structures needed for all processes -
194 irrespective of whether it has a proc-id of 0, 1, 2 or 3.
198 For the symmetric multi-process example, since all processes work in the same manner,
199 once the hugepage shared memory and the network ports are initialized,
200 it is not necessary to restart all processes if the primary instance dies.
201 Instead, that process can be restarted as a secondary,
202 by explicitly setting the proc-type to secondary on the command line.
203 (All subsequent instances launched will also need this explicitly specified,
204 as auto-detection will detect no primary processes running and therefore attempt to re-initialize shared memory.)
206 How the Application Works
207 ^^^^^^^^^^^^^^^^^^^^^^^^^
209 The initialization calls in both the primary and secondary instances are the same for the most part,
210 calling the rte_eal_init(), 1 G and 10 G driver initialization and then probing devices.
211 Thereafter, the initialization done depends on whether the process is configured as a primary or secondary instance.
213 In the primary instance, a memory pool is created for the packet mbufs and the network ports to be used are initialized -
214 the number of RX and TX queues per port being determined by the num-procs parameter passed on the command-line.
215 The structures for the initialized network ports are stored in shared memory and
216 therefore will be accessible by the secondary process as it initializes.
221 rte_exit(EXIT_FAILURE, "Application must use an even number of ports\n");
223 for(i = 0; i < num_ports; i++){
224 if(proc_type == RTE_PROC_PRIMARY)
225 if (smp_port_init(ports[i], mp, (uint16_t)num_procs) < 0)
226 rte_exit(EXIT_FAILURE, "Error initializing ports\n");
229 In the secondary instance, rather than initializing the network ports, the port information exported by the primary process is used,
230 giving the secondary process access to the hardware and software rings for each network port.
231 Similarly, the memory pool of mbufs is accessed by doing a lookup for it by name:
235 mp = (proc_type == RTE_PROC_SECONDARY) ? rte_mempool_lookup(_SMP_MBUF_POOL) : rte_mempool_create(_SMP_MBUF_POOL, NB_MBUFS, MBUF_SIZE, ... )
237 Once this initialization is complete, the main loop of each process, both primary and secondary,
238 is exactly the same - each process reads from each port using the queue corresponding to its proc-id parameter,
239 and writes to the corresponding transmit queue on the output port.
241 Client-Server Multi-process Example
242 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
244 The third example multi-process application included with the DPDK shows how one can
245 use a client-server type multi-process design to do packet processing.
246 In this example, a single server process performs the packet reception from the ports being used and
247 distributes these packets using round-robin ordering among a set of client processes,
248 which perform the actual packet processing.
249 In this case, the client applications just perform level-2 forwarding of packets by sending each packet out on a different network port.
251 The following diagram shows the data-flow through the application, using two client processes.
253 .. _figure_client_svr_sym_multi_proc_app:
255 .. figure:: img/client_svr_sym_multi_proc_app.*
257 Example Data Flow in a Client-Server Symmetric Multi-process Application
260 Running the Application
261 ^^^^^^^^^^^^^^^^^^^^^^^
263 The server process must be run initially as the primary process to set up all memory structures for use by the clients.
264 In addition to the EAL parameters, the application- specific parameters are:
266 * -p <portmask >, where portmask is a hexadecimal bitmask of what ports on the system are to be used.
267 For example: -p 3 to use ports 0 and 1 only.
269 * -n <num-clients>, where the num-clients parameter is the number of client processes that will process the packets received
270 by the server application.
274 In the server process, a single thread, the master thread, that is, the lowest numbered lcore in the coremask/corelist, performs all packet I/O.
275 If a coremask/corelist is specified with more than a single lcore bit set in it,
276 an additional lcore will be used for a thread to periodically print packet count statistics.
278 Since the server application stores configuration data in shared memory, including the network ports to be used,
279 the only application parameter needed by a client process is its client instance ID.
280 Therefore, to run a server application on lcore 1 (with lcore 2 printing statistics) along with two client processes running on lcores 3 and 4,
281 the following commands could be used:
283 .. code-block:: console
285 # ./mp_server/build/mp_server -l 1-2 -n 4 -- -p 3 -n 2
286 # ./mp_client/build/mp_client -l 3 -n 4 --proc-type=auto -- -n 0
287 # ./mp_client/build/mp_client -l 4 -n 4 --proc-type=auto -- -n 1
291 If the server application dies and needs to be restarted, all client applications also need to be restarted,
292 as there is no support in the server application for it to run as a secondary process.
293 Any client processes that need restarting can be restarted without affecting the server process.
295 How the Application Works
296 ^^^^^^^^^^^^^^^^^^^^^^^^^
298 The server process performs the network port and data structure initialization much as the symmetric multi-process application does when run as primary.
299 One additional enhancement in this sample application is that the server process stores its port configuration data in a memory zone in hugepage shared memory.
300 This eliminates the need for the client processes to have the portmask parameter passed into them on the command line,
301 as is done for the symmetric multi-process application, and therefore eliminates mismatched parameters as a potential source of errors.
303 In the same way that the server process is designed to be run as a primary process instance only,
304 the client processes are designed to be run as secondary instances only.
305 They have no code to attempt to create shared memory objects.
306 Instead, handles to all needed rings and memory pools are obtained via calls to rte_ring_lookup() and rte_mempool_lookup().
307 The network ports for use by the processes are obtained by loading the network port drivers and probing the PCI bus,
308 which will, as in the symmetric multi-process example,
309 automatically get access to the network ports using the settings already configured by the primary/server process.
311 Once all applications are initialized, the server operates by reading packets from each network port in turn and
312 distributing those packets to the client queues (software rings, one for each client process) in round-robin order.
313 On the client side, the packets are read from the rings in as big of bursts as possible, then routed out to a different network port.
314 The routing used is very simple. All packets received on the first NIC port are transmitted back out on the second port and vice versa.
315 Similarly, packets are routed between the 3rd and 4th network ports and so on.
316 The sending of packets is done by writing the packets directly to the network ports; they are not transferred back via the server process.
318 In both the server and the client processes, outgoing packets are buffered before being sent,
319 so as to allow the sending of multiple packets in a single burst to improve efficiency.
320 For example, the client process will buffer packets to send,
321 until either the buffer is full or until we receive no further packets from the server.