2 Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Intel Corporation nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
36 The ring allows the management of queues.
37 Instead of having a linked list of infinite size, the rte_ring has the following properties:
41 * Maximum size is fixed, the pointers are stored in a table
43 * Lockless implementation
45 * Multi-consumer or single-consumer dequeue
47 * Multi-producer or single-producer enqueue
49 * Bulk dequeue - Dequeues the specified count of objects if successful; otherwise fails
51 * Bulk enqueue - Enqueues the specified count of objects if successful; otherwise fails
53 * Burst dequeue - Dequeue the maximum available objects if the specified count cannot be fulfilled
55 * Burst enqueue - Enqueue the maximum available objects if the specified count cannot be fulfilled
57 The advantages of this data structure over a linked list queue are as follows:
59 * Faster; only requires a single Compare-And-Swap instruction of sizeof(void \*) instead of several double-Compare-And-Swap instructions.
61 * Simpler than a full lockless queue.
63 * Adapted to bulk enqueue/dequeue operations.
64 As pointers are stored in a table, a dequeue of several objects will not produce as many cache misses as in a linked queue.
65 Also, a bulk dequeue of many objects does not cost more than a dequeue of a simple object.
71 * Having many rings costs more in terms of memory than a linked list queue. An empty ring contains at least N pointers.
73 A simplified representation of a Ring is shown in with consumer and producer head and tail pointers to objects stored in the data structure.
77 **Figure 4. Ring Structure**
79 .. image5_png has been replaced
83 References for Ring Implementation in FreeBSD*
84 ----------------------------------------------
86 The following code was added in FreeBSD 8.0, and is used in some network device drivers (at least in Intel drivers):
88 * `bufring.h in FreeBSD <http://svn.freebsd.org/viewvc/base/release/8.0.0/sys/sys/buf_ring.h?revision=199625&view=markup>`_
90 * `bufring.c in FreeBSD <http://svn.freebsd.org/viewvc/base/release/8.0.0/sys/kern/subr_bufring.c?revision=199625&view=markup>`_
92 Lockless Ring Buffer in Linux*
93 ------------------------------
95 The following is a link describing the `Linux Lockless Ring Buffer Design <http://lwn.net/Articles/340400/>`_.
103 A ring is identified by a unique name.
104 It is not possible to create two rings with the same name (rte_ring_create() returns NULL if this is attempted).
109 The ring can have a high water mark (threshold).
110 Once an enqueue operation reaches the high water mark, the producer is notified, if the water mark is configured.
112 This mechanism can be used, for example, to exert a back pressure on I/O to inform the LAN to PAUSE.
117 When debug is enabled (CONFIG_RTE_LIBRTE_RING_DEBUG is set),
118 the library stores some per-ring statistic counters about the number of enqueues/dequeues.
119 These statistics are per-core to avoid concurrent accesses or atomic operations.
124 Use cases for the Ring library include:
126 * Communication between applications in the DPDK
128 * Used by memory pool allocator
130 Anatomy of a Ring Buffer
131 ------------------------
133 This section explains how a ring buffer operates.
134 The ring structure is composed of two head and tail couples; one is used by producers and one is used by the consumers.
135 The figures of the following sections refer to them as prod_head, prod_tail, cons_head and cons_tail.
137 Each figure represents a simplified state of the ring, which is a circular buffer.
138 The content of the function local variables is represented on the top of the figure,
139 and the content of ring structure is represented on the bottom of the figure.
141 Single Producer Enqueue
142 ~~~~~~~~~~~~~~~~~~~~~~~
144 This section explains what occurs when a producer adds an object to the ring.
145 In this example, only the producer head and tail (prod_head and prod_tail) are modified,
146 and there is only one producer.
148 The initial state is to have a prod_head and prod_tail pointing at the same location.
153 First, *ring->prod_head* and ring->cons_tail are copied in local variables.
154 The prod_next local variable points to the next element of the table, or several elements after in case of bulk enqueue.
156 If there is not enough room in the ring (this is detected by checking cons_tail), it returns an error.
158 .. image6_png has been replaced
165 The second step is to modify *ring->prod_head* in ring structure to point to the same location as prod_next.
167 A pointer to the added object is copied in the ring (obj4).
169 .. image7_png has been replaced
176 Once the object is added in the ring, ring->prod_tail in the ring structure is modified to point to the same location as *ring->prod_head*.
177 The enqueue operation is finished.
179 .. image8_png has been replaced
183 Single Consumer Dequeue
184 ~~~~~~~~~~~~~~~~~~~~~~~
186 This section explains what occurs when a consumer dequeues an object from the ring.
187 In this example, only the consumer head and tail (cons_head and cons_tail) are modified and there is only one consumer.
189 The initial state is to have a cons_head and cons_tail pointing at the same location.
194 First, ring->cons_head and ring->prod_tail are copied in local variables.
195 The cons_next local variable points to the next element of the table, or several elements after in the case of bulk dequeue.
197 If there are not enough objects in the ring (this is detected by checking prod_tail), it returns an error.
199 .. image9_png has been replaced
206 The second step is to modify ring->cons_head in the ring structure to point to the same location as cons_next.
208 The pointer to the dequeued object (obj1) is copied in the pointer given by the user.
210 .. image10_png has been replaced
217 Finally, ring->cons_tail in the ring structure is modified to point to the same location as ring->cons_head.
218 The dequeue operation is finished.
220 .. image11_png has been replaced
224 Multiple Producers Enqueue
225 ~~~~~~~~~~~~~~~~~~~~~~~~~~
227 This section explains what occurs when two producers concurrently add an object to the ring.
228 In this example, only the producer head and tail (prod_head and prod_tail) are modified.
230 The initial state is to have a prod_head and prod_tail pointing at the same location.
232 MC Enqueue First Step
233 ^^^^^^^^^^^^^^^^^^^^^
235 On both cores, *ring->prod_head* and ring->cons_tail are copied in local variables.
236 The prod_next local variable points to the next element of the table,
237 or several elements after in the case of bulk enqueue.
239 If there is not enough room in the ring (this is detected by checking cons_tail), it returns an error.
241 .. image12_png has been replaced
245 MC Enqueue Second Step
246 ^^^^^^^^^^^^^^^^^^^^^^
248 The second step is to modify ring->prod_head in the ring structure to point to the same location as prod_next.
249 This operation is done using a Compare And Swap (CAS) instruction, which does the following operations atomically:
251 * If ring->prod_head is different to local variable prod_head,
252 the CAS operation fails, and the code restarts at first step.
254 * Otherwise, ring->prod_head is set to local prod_next,
255 the CAS operation is successful, and processing continues.
257 In the figure, the operation succeeded on core 1, and step one restarted on core 2.
259 .. image13_png has been replaced
263 MC Enqueue Third Step
264 ^^^^^^^^^^^^^^^^^^^^^
266 The CAS operation is retried on core 2 with success.
268 The core 1 updates one element of the ring(obj4), and the core 2 updates another one (obj5).
270 .. image14_png has been replaced
274 MC Enqueue Fourth Step
275 ^^^^^^^^^^^^^^^^^^^^^^
277 Each core now wants to update ring->prod_tail.
278 A core can only update it if ring->prod_tail is equal to the prod_head local variable.
279 This is only true on core 1. The operation is finished on core 1.
281 .. image15_png has been replaced
288 Once ring->prod_tail is updated by core 1, core 2 is allowed to update it too.
289 The operation is also finished on core 2.
291 .. image16_png has been replaced
295 Modulo 32-bit Indexes
296 ~~~~~~~~~~~~~~~~~~~~~
298 In the preceding figures, the prod_head, prod_tail, cons_head and cons_tail indexes are represented by arrows.
299 In the actual implementation, these values are not between 0 and size(ring)-1 as would be assumed.
300 The indexes are between 0 and 2^32 -1, and we mask their value when we access the pointer table (the ring itself).
301 32-bit modulo also implies that operations on indexes (such as, add/subtract) will automatically do 2^32 modulo
302 if the result overflows the 32-bit number range.
304 The following are two examples that help to explain how indexes are used in a ring.
308 To simplify the explanation, operations with modulo 16-bit are used instead of modulo 32-bit.
309 In addition, the four indexes are defined as unsigned 16-bit integers,
310 as opposed to unsigned 32-bit integers in the more realistic case.
312 .. image17_png has been replaced
316 This ring contains 11000 entries.
318 .. image18_png has been replaced
322 This ring contains 12536 entries.
326 For ease of understanding, we use modulo 65536 operations in the above examples.
327 In real execution cases, this is redundant for low efficiency, but is done automatically when the result overflows.
329 The code always maintains a distance between producer and consumer between 0 and size(ring)-1.
330 Thanks to this property, we can do subtractions between 2 index values in a modulo-32bit base:
331 that's why the overflow of the indexes is not a problem.
333 At any time, entries and free_entries are between 0 and size(ring)-1,
334 even if only the first term of subtraction has overflowed:
338 uint32_t entries = (prod_tail - cons_head);
339 uint32_t free_entries = (mask + cons_tail -prod_head);
344 * `bufring.h in FreeBSD <http://svn.freebsd.org/viewvc/base/release/8.0.0/sys/sys/buf_ring.h?revision=199625&view=markup>`_ (version 8)
346 * `bufring.c in FreeBSD <http://svn.freebsd.org/viewvc/base/release/8.0.0/sys/kern/subr_bufring.c?revision=199625&view=markup>`_ (version 8)
348 * `Linux Lockless Ring Buffer Design <http://lwn.net/Articles/340400/>`_
350 .. |ring1| image:: img/ring1.svg
352 .. |ring-enqueue1| image:: img/ring-enqueue1.svg
354 .. |ring-enqueue2| image:: img/ring-enqueue2.svg
356 .. |ring-enqueue3| image:: img/ring-enqueue3.svg
358 .. |ring-dequeue1| image:: img/ring-dequeue1.svg
360 .. |ring-dequeue2| image:: img/ring-dequeue2.svg
362 .. |ring-dequeue3| image:: img/ring-dequeue3.svg
364 .. |ring-mp-enqueue1| image:: img/ring-mp-enqueue1.svg
366 .. |ring-mp-enqueue2| image:: img/ring-mp-enqueue2.svg
368 .. |ring-mp-enqueue3| image:: img/ring-mp-enqueue3.svg
370 .. |ring-mp-enqueue4| image:: img/ring-mp-enqueue4.svg
372 .. |ring-mp-enqueue5| image:: img/ring-mp-enqueue5.svg
374 .. |ring-modulo1| image:: img/ring-modulo1.svg
376 .. |ring-modulo2| image:: img/ring-modulo2.svg