Secondary processes should run alongside primary process with same DPDK version.
Secondary processes which requires access to physical devices in Primary process, must
- be passed with the same whitelist and blacklist options.
+ be passed with the same allow and block options.
To support these two process types, and other multi-process setups described later,
two additional command-line parameters are available to the EAL:
Refer to `Multi-process Limitations`_ for details of
how Linux kernel Address-Space Layout Randomization (ASLR) can affect memory sharing.
+ If the primary process was run with ``--legacy-mem`` or
+ ``--single-file-segments`` switch, secondary processes must be run with the
+ same switch specified. Otherwise, memory corruption may occur.
+
.. _figure_multi_process_memory:
.. figure:: img/multi_process_memory.*
The EAL also supports an auto-detection mode (set by EAL ``--proc-type=auto`` flag ),
-whereby an DPDK process is started as a secondary instance if a primary instance is already running.
+whereby a DPDK process is started as a secondary instance if a primary instance is already running.
Deployment Models
-----------------
In addition to specifying the file-prefix parameter,
any DPDK applications that are to be run side-by-side must explicitly limit their memory use.
-This is done by passing the -m flag to each process to specify how much hugepage memory, in megabytes,
-each process can use (or passing ``--socket-mem`` to specify how much hugepage memory on each socket each process can use).
+This is less of a problem on Linux, as by default, applications will not
+allocate more memory than they need. However if ``--legacy-mem`` is used, DPDK
+will attempt to preallocate all memory it can get to, and memory use must be
+explicitly limited. This is done by passing the ``-m`` flag to each process to
+specify how much hugepage memory, in megabytes, each process can use (or passing
+``--socket-mem`` to specify how much hugepage memory on each socket each process
+can use).
.. note::
Independent DPDK instances running side-by-side on a single machine cannot share any network ports.
- Any network ports being used by one process should be blacklisted in every other process.
+ Any network ports being used by one process should be blocked by every other process.
Running Multiple Independent Groups of DPDK Applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some of these are documented below:
* The multi-process feature requires that the exact same hugepage memory mappings be present in all applications.
- The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping,
- so it may be necessary to disable this feature in order to reliably run multi-process applications.
+ This makes secondary process startup process generally unreliable. Disabling
+ Linux security feature - Address-Space Layout Randomization (ASLR) may
+ help getting more consistent mappings, but not necessarily more reliable -
+ if the mappings are wrong, they will be consistently wrong!
.. warning::
* The use of function pointers between multiple processes running based of different compiled binaries is not supported,
since the location of a given function in one process may be different to its location in a second.
- This prevents the librte_hash library from behaving properly as in a multi-threaded instance,
+ This prevents the librte_hash library from behaving properly as in a multi-process instance,
since it uses a pointer to the hash function internally.
To work around this issue, it is recommended that multi-process applications perform the hash calculations by directly calling
other side.
Both messages and requests will trigger a named callback on the receiver side.
-These callbacks will be called from within a dedicated IPC thread that is not
-part of EAL lcore threads.
+These callbacks will be called from within a dedicated IPC or interrupt thread
+that are not part of EAL lcore threads.
Registering for incoming messages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additionally, a ``timespec`` value must be specified as a timeout, after which
IPC will stop waiting and return.
-For synchronous synchronous requests, the ``rte_mp_reply`` descriptor must also
-be created. This is where the responses will be stored. The list of fields that
-will be populated by IPC are as follows:
+For synchronous requests, the ``rte_mp_reply`` descriptor must also be created.
+This is where the responses will be stored.
+The list of fields that will be populated by IPC are as follows:
* ``nb_sent`` - number indicating how many requests were sent (i.e. how many
peer processes were active at the time of the request).
those peer processes that were active at the time of request, how many have
replied)
* ``msgs`` - pointer to where all of the responses are stored. The order in
- which responses appear is undefined. Whendoing sycnrhonous requests, this
+ which responses appear is undefined. When doing synchronous requests, this
memory must be freed by the requestor after request completes!
For asynchronous requests, a function pointer to the callback function must be
provided instead. This callback will be called when the request either has timed
out, or will have received a response to all the messages that were sent.
+.. warning::
+
+ When an asynchronous request times out, the callback will be called not by
+ a dedicated IPC thread, but rather from EAL interrupt thread. Because of
+ this, it may not be possible for DPDK to trigger another interrupt-based
+ event (such as an alarm) while handling asynchronous IPC callback.
+
When the callback is called, the original request descriptor will be provided
(so that it would be possible to determine for which sent message this is a
callback to), along with a response descriptor like the one described above.
constructed and sent via ``rte_mp_reply()`` function, along with ``peer``
pointer. The resulting response will then be delivered to the correct requestor.
+.. warning::
+ Simply returning a value when processing a request callback will not send a
+ response to the request - it must always be explicitly sent even in case
+ of errors. Implementation of error signalling rests with the application,
+ there is no built-in way to indicate success or error for a request. Failing
+ to do so will cause the requestor to time out while waiting on a response.
+
Misc considerations
~~~~~~~~~~~~~~~~~~~~~~~~
IPC thread, sending messages while processing another message or request is
supported.
+Since the memory sybsystem uses IPC internally, memory allocations and IPC must
+not be mixed: it is not safe to use IPC inside a memory-related callback, nor is
+it safe to allocate/free memory inside IPC callbacks. Attempting to do so may
+lead to a deadlock.
+
+Asynchronous request callbacks may be triggered either from IPC thread or from
+interrupt thread, depending on whether the request has timed out. It is
+therefore suggested to avoid waiting for interrupt-based events (such as alarms)
+inside asynchronous IPC request callbacks. This limitation does not apply to
+messages or synchronous requests.
+
If callbacks spend a long time processing the incoming requests, the requestor
might time out, so setting the right timeout value on the requestor side is
imperative.
git.droids-corp.org / dpdk.git / blobdiff