which remain functional, although they stop receiving unicast packets as
long as they share the same MAC address.
+The :ref:`flow_isolated_mode` is supported.
+
Compiling librte_pmd_mlx4 causes DPDK to be linked against libibverbs.
Configuration
Toggle compilation of librte_pmd_mlx4 itself.
-- ``CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS`` (default **n**)
+- ``CONFIG_RTE_IBVERBS_LINK_DLOPEN`` (default **n**)
Build PMD with additional code to make it loadable without hard
dependencies on **libibverbs** nor **libmlx4**, which may not be installed
This option has no performance impact.
+- ``CONFIG_RTE_IBVERBS_LINK_STATIC`` (default **n**)
+
+ Embed static flavor of the dependencies **libibverbs** and **libmlx4**
+ in the PMD shared library or the executable static binary.
+
- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG`` (default **n**)
Toggle debugging code and stricter compilation flags. Enabling this option
adds additional run-time checks and debugging messages at the cost of
lower performance.
+This option is available in meson:
+
+- ``ibverbs_link`` can be ``static``, ``shared``, or ``dlopen``.
+
Environment variables
~~~~~~~~~~~~~~~~~~~~~
A list of directories in which to search for the rdma-core "glue" plug-in,
separated by colons or semi-colons.
- Only matters when compiled with ``CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS``
+ Only matters when compiled with ``CONFIG_RTE_IBVERBS_LINK_DLOPEN``
enabled and most useful when ``CONFIG_RTE_EAL_PMD_PATH`` is also set,
since ``LD_LIBRARY_PATH`` has no effect in this case.
times for additional ports. All ports are probed by default if left
unspecified.
+- ``mr_ext_memseg_en`` parameter [int]
+
+ A nonzero value enables extending memseg when registering DMA memory. If
+ enabled, the number of entries in MR (Memory Region) lookup table on datapath
+ is minimized and it benefits performance. On the other hand, it worsens memory
+ utilization because registered memory is pinned by kernel driver. Even if a
+ page in the extended chunk is freed, that doesn't become reusable until the
+ entire memory is freed.
+
+ Enabled by default.
+
Kernel module parameters
~~~~~~~~~~~~~~~~~~~~~~~~
Limitations
-----------
+- For secondary process:
+
+ - Forked secondary process not supported.
+ - External memory unregistered in EAL memseg list cannot be used for DMA
+ unless such memory has been registered by ``mlx4_mr_update_ext_mp()`` in
+ primary process and remapped to the same virtual address in secondary
+ process. If the external memory is registered by primary process but has
+ different virtual address in secondary process, unexpected error may happen.
+
- CRC stripping is supported by default and always reported as "true".
The ability to enable/disable CRC stripping requires OFED version
4.3-1.5.0.0 and above or rdma-core version v18 and above.
- Minimal Linux kernel version: 4.14.
- Minimal RDMA core version: v15 (see `RDMA core installation documentation`_).
+- Starting with rdma-core v21, static libraries can be built::
+
+ cd build
+ CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
+ ninja
+
.. _`RDMA core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md
+If rdma-core libraries are built but not installed, DPDK makefile can link them,
+thanks to these environment variables:
+
+ - ``EXTRA_CFLAGS=-I/path/to/rdma-core/build/include``
+ - ``EXTRA_LDFLAGS=-L/path/to/rdma-core/build/lib``
+ - ``PKG_CONFIG_PATH=/path/to/rdma-core/build/lib/pkgconfig``
+
.. _Mellanox_OFED_as_a_fallback:
Mellanox OFED as a fallback
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- `Mellanox OFED`_ version: **4.3**.
+- `Mellanox OFED`_ version: **4.4, 4.5, 4.6**.
- firmware version: **2.42.5000** and above.
.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
2. Install the required libraries and kernel modules either by installing
only the required set, or by installing the entire Mellanox OFED:
- For bare metal use:
-
- .. code-block:: console
+ For bare metal use::
./mlnxofedinstall --dpdk --upstream-libs
- For SR-IOV hypervisors use:
-
- .. code-block:: console
+ For SR-IOV hypervisors use::
./mlnxofedinstall --dpdk --upstream-libs --enable-sriov --hypervisor
- For SR-IOV virtual machine use:
-
- .. code-block:: console
+ For SR-IOV virtual machine use::
./mlnxofedinstall --dpdk --upstream-libs --guest
-3. Verify the firmware is the correct one:
-
- .. code-block:: console
+3. Verify the firmware is the correct one::
ibv_devinfo
-4. Set all ports links to Ethernet, follow instructions on the screen:
-
- .. code-block:: console
+4. Set all ports links to Ethernet, follow instructions on the screen::
connectx_port_config
Quick Start Guide
-----------------
-1. Set all ports links to Ethernet
-
- .. code-block:: console
+1. Set all ports links to Ethernet::
PCI=<NIC PCI address>
echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port0"
.. _QSG_2:
2. In case of bare metal or hypervisor, configure optimized steering mode
- by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``:
-
- .. code-block:: console
+ by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``::
options mlx4_core log_num_mgm_entry_size=-7
If VLAN filtering is used, set log_num_mgm_entry_size=-1.
Performance degradation can occur on this case.
-3. Restart the driver:
-
- .. code-block:: console
+3. Restart the driver::
/etc/init.d/openibd restart
- or:
-
- .. code-block:: console
+ or::
service openibd restart
Performance tuning
------------------
-1. Verify the optimized steering mode is configured:
-
- .. code-block:: console
+1. Verify the optimized steering mode is configured::
cat /sys/module/mlx4_core/parameters/log_num_mgm_entry_size
2. Use the CPU near local NUMA node to which the PCIe adapter is connected,
for better performance. For VMs, verify that the right CPU
- and NUMA node are pinned according to the above. Run:
-
- .. code-block:: console
+ and NUMA node are pinned according to the above. Run::
lstopo-no-graphics
This in order to forward packets from one to the other without
NUMA performance penalty.
-4. Disable pause frames:
-
- .. code-block:: console
+4. Disable pause frames::
ethtool -A <netdev> rx off tx off
to set the PCI max read request parameter to 1K. This can be
done in the following way:
- To query the read request size use:
-
- .. code-block:: console
+ To query the read request size use::
setpci -s <NIC PCI address> 68.w
- If the output is different than 3XXX, set it by:
-
- .. code-block:: console
+ If the output is different than 3XXX, set it by::
setpci -s <NIC PCI address> 68.w=3XXX
This section demonstrates how to launch **testpmd** with Mellanox ConnectX-3
devices managed by librte_pmd_mlx4.
-#. Load the kernel modules:
-
- .. code-block:: console
+#. Load the kernel modules::
modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib
Alternatively if MLNX_OFED is fully installed, the following script can
- be run:
-
- .. code-block:: console
+ be run::
/etc/init.d/openibd restart
not have to be loaded.
#. Make sure Ethernet interfaces are in working order and linked to kernel
- verbs. Related sysfs entries should be present:
-
- .. code-block:: console
+ verbs. Related sysfs entries should be present::
ls -d /sys/class/net/*/device/infiniband_verbs/uverbs* | cut -d / -f 5
- Example output:
-
- .. code-block:: console
+ Example output::
eth2
eth3
eth4
eth5
-#. Optionally, retrieve their PCI bus addresses for whitelisting:
-
- .. code-block:: console
+#. Optionally, retrieve their PCI bus addresses for whitelisting::
{
for intf in eth2 eth3 eth4 eth5;
} |
sed -n 's,.*/\(.*\),-w \1,p'
- Example output:
-
- .. code-block:: console
+ Example output::
-w 0000:83:00.0
-w 0000:83:00.0
There are only two distinct PCI bus addresses because the Mellanox
ConnectX-3 adapters installed on this system are dual port.
-#. Request huge pages:
-
- .. code-block:: console
+#. Request huge pages::
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages
-#. Start testpmd with basic parameters:
-
- .. code-block:: console
+#. Start testpmd with basic parameters::
testpmd -l 8-15 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i
- Example output:
-
- .. code-block:: console
+ Example output::
[...]
EAL: PCI device 0000:83:00.0 on NUMA socket 1
git.droids-corp.org / dpdk.git / blobdiff