X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fprog_guide%2Fmulti_proc_support.rst;h=a84083b96c8afdc0b062c8933403cb79d5140b70;hb=08b1d50543dea1a9e817f67f4b97ea22347e2a09;hp=371d028db698b4846772e533da2140a36f958c83;hpb=e22266669e86e2768910aa83929413d45d133695;p=dpdk.git diff --git a/doc/guides/prog_guide/multi_proc_support.rst b/doc/guides/prog_guide/multi_proc_support.rst index 371d028db6..a84083b96c 100644 --- a/doc/guides/prog_guide/multi_proc_support.rst +++ b/doc/guides/prog_guide/multi_proc_support.rst @@ -63,6 +63,10 @@ and point to the same objects, in both processes. Refer to `Multi-process Limitations`_ for details of how Linux kernel Address-Space Layout Randomization (ASLR) can affect memory sharing. + If the primary process was run with ``--legacy-mem`` or + ``--single-file-segments`` switch, secondary processes must be run with the + same switch specified. Otherwise, memory corruption may occur. + .. _figure_multi_process_memory: .. figure:: img/multi_process_memory.* @@ -116,8 +120,13 @@ The rte part of the filenames of each of the above is configurable using the fil In addition to specifying the file-prefix parameter, any DPDK applications that are to be run side-by-side must explicitly limit their memory use. -This is done by passing the -m flag to each process to specify how much hugepage memory, in megabytes, -each process can use (or passing ``--socket-mem`` to specify how much hugepage memory on each socket each process can use). +This is less of a problem on Linux, as by default, applications will not +allocate more memory than they need. However if ``--legacy-mem`` is used, DPDK +will attempt to preallocate all memory it can get to, and memory use must be +explicitly limited. This is done by passing the ``-m`` flag to each process to +specify how much hugepage memory, in megabytes, each process can use (or passing +``--socket-mem`` to specify how much hugepage memory on each socket each process +can use). .. note:: @@ -144,8 +153,10 @@ There are a number of limitations to what can be done when running DPDK multi-pr Some of these are documented below: * The multi-process feature requires that the exact same hugepage memory mappings be present in all applications. - The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping, - so it may be necessary to disable this feature in order to reliably run multi-process applications. + This makes secondary process startup process generally unreliable. Disabling + Linux security feature - Address-Space Layout Randomization (ASLR) may + help getting more consistent mappings, but not necessarily more reliable - + if the mappings are wrong, they will be consistently wrong! .. warning:: @@ -165,7 +176,7 @@ Some of these are documented below: * The use of function pointers between multiple processes running based of different compiled binaries is not supported, since the location of a given function in one process may be different to its location in a second. - This prevents the librte_hash library from behaving properly as in a multi-threaded instance, + This prevents the librte_hash library from behaving properly as in a multi-process instance, since it uses a pointer to the hash function internally. To work around this issue, it is recommended that multi-process applications perform the hash calculations by directly calling @@ -209,8 +220,8 @@ way communication mechanism, with the requester expecting a response from the other side. Both messages and requests will trigger a named callback on the receiver side. -These callbacks will be called from within a dedicated IPC thread that is not -part of EAL lcore threads. +These callbacks will be called from within a dedicated IPC or interrupt thread +that are not part of EAL lcore threads. Registering for incoming messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -252,9 +263,9 @@ To send a request, a message descriptor ``rte_mp_msg`` must be populated. Additionally, a ``timespec`` value must be specified as a timeout, after which IPC will stop waiting and return. -For synchronous synchronous requests, the ``rte_mp_reply`` descriptor must also -be created. This is where the responses will be stored. The list of fields that -will be populated by IPC are as follows: +For synchronous requests, the ``rte_mp_reply`` descriptor must also be created. +This is where the responses will be stored. +The list of fields that will be populated by IPC are as follows: * ``nb_sent`` - number indicating how many requests were sent (i.e. how many peer processes were active at the time of the request). @@ -262,13 +273,20 @@ will be populated by IPC are as follows: those peer processes that were active at the time of request, how many have replied) * ``msgs`` - pointer to where all of the responses are stored. The order in - which responses appear is undefined. Whendoing sycnrhonous requests, this + which responses appear is undefined. When doing synchronous requests, this memory must be freed by the requestor after request completes! For asynchronous requests, a function pointer to the callback function must be provided instead. This callback will be called when the request either has timed out, or will have received a response to all the messages that were sent. +.. warning:: + + When an asynchronous request times out, the callback will be called not by + a dedicated IPC thread, but rather from EAL interrupt thread. Because of + this, it may not be possible for DPDK to trigger another interrupt-based + event (such as an alarm) while handling asynchronous IPC callback. + When the callback is called, the original request descriptor will be provided (so that it would be possible to determine for which sent message this is a callback to), along with a response descriptor like the one described above. @@ -291,6 +309,13 @@ If a response is required, a new ``rte_mp_msg`` message descriptor must be constructed and sent via ``rte_mp_reply()`` function, along with ``peer`` pointer. The resulting response will then be delivered to the correct requestor. +.. warning:: + Simply returning a value when processing a request callback will not send a + response to the request - it must always be explicitly sent even in case + of errors. Implementation of error signalling rests with the application, + there is no built-in way to indicate success or error for a request. Failing + to do so will cause the requestor to time out while waiting on a response. + Misc considerations ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -300,6 +325,17 @@ supported. However, since sending messages (not requests) does not involve an IPC thread, sending messages while processing another message or request is supported. +Since the memory sybsystem uses IPC internally, memory allocations and IPC must +not be mixed: it is not safe to use IPC inside a memory-related callback, nor is +it safe to allocate/free memory inside IPC callbacks. Attempting to do so may +lead to a deadlock. + +Asynchronous request callbacks may be triggered either from IPC thread or from +interrupt thread, depending on whether the request has timed out. It is +therefore suggested to avoid waiting for interrupt-based events (such as alarms) +inside asynchronous IPC request callbacks. This limitation does not apply to +messages or synchronous requests. + If callbacks spend a long time processing the incoming requests, the requestor might time out, so setting the right timeout value on the requestor side is imperative.