1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2017 Intel Corporation.
3 Copyright(c) 2018 Arm Limited.
8 The DPDK Event device library is an abstraction that provides the application
9 with features to schedule events. This is achieved using the PMD architecture
10 similar to the ethdev or cryptodev APIs, which may already be familiar to the
13 The eventdev framework introduces the event driven programming model. In a
14 polling model, lcores poll ethdev ports and associated Rx queues directly
15 to look for a packet. By contrast in an event driven model, lcores call the
16 scheduler that selects packets for them based on programmer-specified criteria.
17 The Eventdev library adds support for an event driven programming model, which
18 offers applications automatic multicore scaling, dynamic load balancing,
19 pipelining, packet ingress order maintenance and synchronization services to
20 simplify application packet processing.
22 By introducing an event driven programming model, DPDK can support both polling
23 and event driven programming models for packet processing, and applications are
24 free to choose whatever model (or combination of the two) best suits their
27 Step-by-step instructions of the eventdev design is available in the `API
28 Walk-through`_ section later in this document.
33 The eventdev API represents each event with a generic struct, which contains a
34 payload and metadata required for scheduling by an eventdev. The
35 ``rte_event`` struct is a 16 byte C structure, defined in
36 ``libs/librte_eventdev/rte_eventdev.h``.
41 The rte_event structure contains the following metadata fields, which the
42 application fills in to have the event scheduled as required:
44 * ``flow_id`` - The targeted flow identifier for the enq/deq operation.
45 * ``event_type`` - The source of this event, e.g. RTE_EVENT_TYPE_ETHDEV or CPU.
46 * ``sub_event_type`` - Distinguishes events inside the application, that have
47 the same event_type (see above)
48 * ``op`` - This field takes one of the RTE_EVENT_OP_* values, and tells the
49 eventdev about the status of the event - valid values are NEW, FORWARD or
51 * ``sched_type`` - Represents the type of scheduling that should be performed
52 on this event, valid values are the RTE_SCHED_TYPE_ORDERED, ATOMIC and
54 * ``queue_id`` - The identifier for the event queue that the event is sent to.
55 * ``priority`` - The priority of this event, see RTE_EVENT_DEV_PRIORITY.
60 The rte_event struct contains a union for payload, allowing flexibility in what
61 the actual event being scheduled is. The payload is a union of the following:
65 * ``struct rte_mbuf *mbuf``
67 These three items in a union occupy the same 64 bits at the end of the rte_event
68 structure. The application can utilize the 64 bits directly by accessing the
69 u64 variable, while the event_ptr and mbuf are provided as convenience
70 variables. For example the mbuf pointer in the union can used to schedule a
76 An event queue is a queue containing events that are scheduled by the event
77 device. An event queue contains events of different flows associated with
78 scheduling types, such as atomic, ordered, or parallel.
80 Queue All Types Capable
81 ^^^^^^^^^^^^^^^^^^^^^^^
83 If RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES capability bit is set in the event device,
84 then events of any type may be sent to any queue. Otherwise, the queues only
85 support events of the type that it was created with.
87 Queue All Types Incapable
88 ^^^^^^^^^^^^^^^^^^^^^^^^^
90 In this case, each stage has a specified scheduling type. The application
91 configures each queue for a specific type of scheduling, and just enqueues all
92 events to the eventdev. An example of a PMD of this type is the eventdev
95 The Eventdev API supports the following scheduling types per queue:
101 Atomic, Ordered and Parallel are load-balanced scheduling types: the output
102 of the queue can be spread out over multiple CPU cores.
104 Atomic scheduling on a queue ensures that a single flow is not present on two
105 different CPU cores at the same time. Ordered allows sending all flows to any
106 core, but the scheduler must ensure that on egress the packets are returned to
107 ingress order on downstream queue enqueue. Parallel allows sending all flows
108 to all CPU cores, without any re-ordering guarantees.
113 There is a SINGLE_LINK flag which allows an application to indicate that only
114 one port will be connected to a queue. Queues configured with the single-link
115 flag follow a FIFO like structure, maintaining ordering but it is only capable
116 of being linked to a single port (see below for port and queue linking details).
122 Ports are the points of contact between worker cores and the eventdev. The
123 general use-case will see one CPU core using one port to enqueue and dequeue
124 events from an eventdev. Ports are linked to queues in order to retrieve events
125 from those queues (more details in `Linking Queues and Ports`_ below).
131 This section will introduce the reader to the eventdev API, showing how to
132 create and configure an eventdev and use it for a two-stage atomic pipeline
133 with one core each for RX and TX. RX and TX cores are shown here for
134 illustration, refer to Eventdev Adapter documentation for further details.
135 The diagram below shows the final state of the application after this
138 .. _figure_eventdev-usage1:
140 .. figure:: img/eventdev_usage.*
142 Sample eventdev usage, with RX, two atomic stages and a single-link to TX.
145 A high level overview of the setup steps are:
147 * rte_event_dev_configure()
148 * rte_event_queue_setup()
149 * rte_event_port_setup()
150 * rte_event_port_link()
151 * rte_event_dev_start()
157 The eventdev library uses vdev options to add devices to the DPDK application.
158 The ``--vdev`` EAL option allows adding eventdev instances to your DPDK
159 application, using the name of the eventdev PMD as an argument.
161 For example, to create an instance of the software eventdev scheduler, the
162 following vdev arguments should be provided to the application EAL command line:
164 .. code-block:: console
166 ./dpdk_application --vdev="event_sw0"
168 In the following code, we configure eventdev instance with 3 queues
169 and 6 ports as follows. The 3 queues consist of 2 Atomic and 1 Single-Link,
170 while the 6 ports consist of 4 workers, 1 RX and 1 TX.
174 const struct rte_event_dev_config config = {
175 .nb_event_queues = 3,
177 .nb_events_limit = 4096,
178 .nb_event_queue_flows = 1024,
179 .nb_event_port_dequeue_depth = 128,
180 .nb_event_port_enqueue_depth = 128,
182 int err = rte_event_dev_configure(dev_id, &config);
184 The remainder of this walk-through assumes that dev_id is 0.
189 Once the eventdev itself is configured, the next step is to configure queues.
190 This is done by setting the appropriate values in a queue_conf structure, and
191 calling the setup function. Repeat this step for each queue, starting from
192 0 and ending at ``nb_event_queues - 1`` from the event_dev config above.
196 struct rte_event_queue_conf atomic_conf = {
197 .schedule_type = RTE_SCHED_TYPE_ATOMIC,
198 .priority = RTE_EVENT_DEV_PRIORITY_NORMAL,
199 .nb_atomic_flows = 1024,
200 .nb_atomic_order_sequences = 1024,
202 struct rte_event_queue_conf single_link_conf = {
203 .event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK,
208 int single_link_q = 2;
209 int err = rte_event_queue_setup(dev_id, atomic_q_1, &atomic_conf);
210 int err = rte_event_queue_setup(dev_id, atomic_q_2, &atomic_conf);
211 int err = rte_event_queue_setup(dev_id, single_link_q, &single_link_conf);
213 As shown above, queue IDs are as follows:
215 * id 0, atomic queue #1
216 * id 1, atomic queue #2
217 * id 2, single-link queue
219 These queues are used for the remainder of this walk-through.
224 Once queues are set up successfully, create the ports as required.
228 struct rte_event_port_conf rx_conf = {
229 .dequeue_depth = 128,
230 .enqueue_depth = 128,
231 .new_event_threshold = 1024,
233 struct rte_event_port_conf worker_conf = {
236 .new_event_threshold = 4096,
238 struct rte_event_port_conf tx_conf = {
239 .dequeue_depth = 128,
240 .enqueue_depth = 128,
241 .new_event_threshold = 4096,
246 int err = rte_event_port_setup(dev_id, rx_port_id, &rx_conf);
248 for (worker_port_id = 1; worker_port_id <= 4; worker_port_id++) {
249 int err = rte_event_port_setup(dev_id, worker_port_id, &worker_conf);
253 int err = rte_event_port_setup(dev_id, tx_port_id, &tx_conf);
258 * ports 1,2,3,4: Workers
261 These ports are used for the remainder of this walk-through.
263 Linking Queues and Ports
264 ~~~~~~~~~~~~~~~~~~~~~~~~
266 The final step is to "wire up" the ports to the queues. After this, the
267 eventdev is capable of scheduling events, and when cores request work to do,
268 the correct events are provided to that core. Note that the RX core takes input
269 from e.g.: a NIC so it is not linked to any eventdev queues.
271 Linking all workers to atomic queues, and the TX core to the single-link queue
272 can be achieved like this:
276 uint8_t rx_port_id = 0;
277 uint8_t tx_port_id = 5;
278 uint8_t atomic_qs[] = {0, 1};
279 uint8_t single_link_q = 2;
280 uint8_t priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
283 for (worker_port_id = 1; worker_port_id <= 4; worker_port_id++) {
284 int links_made = rte_event_port_link(dev_id, worker_port_id, atomic_qs, NULL, 2);
286 int links_made = rte_event_port_link(dev_id, tx_port_id, &single_link_q, &priority, 1);
288 Starting the EventDev
289 ~~~~~~~~~~~~~~~~~~~~~
291 A single function call tells the eventdev instance to start processing
292 events. Note that all queues must be linked to for the instance to start, as
293 if any queue is not linked to, enqueuing to that queue will cause the
294 application to backpressure and eventually stall due to no space in the
299 int err = rte_event_dev_start(dev_id);
303 EventDev needs to be started before starting the event producers such
304 as event_eth_rx_adapter, event_timer_adapter and event_crypto_adapter.
306 Ingress of New Events
307 ~~~~~~~~~~~~~~~~~~~~~
309 Now that the eventdev is set up, and ready to receive events, the RX core must
310 enqueue some events into the system for it to schedule. The events to be
311 scheduled are ordinary DPDK packets, received from an eth_rx_burst() as normal.
312 The following code shows how those packets can be enqueued into the eventdev:
316 const uint16_t nb_rx = rte_eth_rx_burst(eth_port, 0, mbufs, BATCH_SIZE);
318 for (i = 0; i < nb_rx; i++) {
319 ev[i].flow_id = mbufs[i]->hash.rss;
320 ev[i].op = RTE_EVENT_OP_NEW;
321 ev[i].sched_type = RTE_SCHED_TYPE_ATOMIC;
322 ev[i].queue_id = atomic_q_1;
323 ev[i].event_type = RTE_EVENT_TYPE_ETHDEV;
324 ev[i].sub_event_type = 0;
325 ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
326 ev[i].mbuf = mbufs[i];
329 const int nb_tx = rte_event_enqueue_burst(dev_id, rx_port_id, ev, nb_rx);
330 if (nb_tx != nb_rx) {
331 for(i = nb_tx; i < nb_rx; i++)
332 rte_pktmbuf_free(mbufs[i]);
338 Now that the RX core has injected events, there is work to be done by the
339 workers. Note that each worker will dequeue as many events as it can in a burst,
340 process each one individually, and then burst the packets back into the
343 The worker can lookup the events source from ``event.queue_id``, which should
344 indicate to the worker what workload needs to be performed on the event.
345 Once done, the worker can update the ``event.queue_id`` to a new value, to send
346 the event to the next stage in the pipeline.
351 struct rte_event events[BATCH_SIZE];
352 uint16_t nb_rx = rte_event_dequeue_burst(dev_id, worker_port_id, events, BATCH_SIZE, timeout);
354 for (i = 0; i < nb_rx; i++) {
355 /* process mbuf using events[i].queue_id as pipeline stage */
356 struct rte_mbuf *mbuf = events[i].mbuf;
357 /* Send event to next stage in pipeline */
358 events[i].queue_id++;
361 uint16_t nb_tx = rte_event_enqueue_burst(dev_id, worker_port_id, events, nb_rx);
367 Finally, when the packet is ready for egress or needs to be dropped, we need
368 to inform the eventdev that the packet is no longer being handled by the
369 application. This can be done by calling dequeue() or dequeue_burst(), which
370 indicates that the previous burst of packets is no longer in use by the
373 An event driven worker thread has following typical workflow on fastpath:
378 rte_event_dequeue_burst(...);
380 rte_event_enqueue_burst(...);
387 The eventdev library allows an application to easily schedule events as it
388 requires, either using a run-to-completion or pipeline processing model. The
389 queues and ports abstract the logical functionality of an eventdev, providing
390 the application with a generic method to schedule events. With the flexible
391 PMD infrastructure applications benefit of improvements in existing eventdevs
392 and additions of new ones without modification.