-.. BSD LICENSE
+.. SPDX-License-Identifier: BSD-3-Clause
Copyright 2012 6WIND S.A.
- Copyright 2015 Mellanox
-
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
-
- * Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- * Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in
- the documentation and/or other materials provided with the
- distribution.
- * Neither the name of 6WIND S.A. nor the names of its
- contributors may be used to endorse or promote products derived
- from this software without specific prior written permission.
-
- THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ Copyright 2015 Mellanox Technologies, Ltd
MLX4 poll mode driver library
=============================
Compiling librte_pmd_mlx4 causes DPDK to be linked against libibverbs.
-Features
---------
-
-- Multi arch support: x86_64 and POWER8.
-- Link state information is provided.
-- RX interrupts.
-
Configuration
-------------
Toggle compilation of librte_pmd_mlx4 itself.
+- ``CONFIG_RTE_IBVERBS_LINK_DLOPEN`` (default **n**)
+
+ Build PMD with additional code to make it loadable without hard
+ dependencies on **libibverbs** nor **libmlx4**, which may not be installed
+ on the target system.
+
+ In this mode, their presence is still required for it to run properly,
+ however their absence won't prevent a DPDK application from starting (with
+ ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as
+ missing with ``ldd(1)``.
+
+ It works by moving these dependencies to a purpose-built rdma-core "glue"
+ plug-in which must either be installed in a directory whose name is based
+ on ``CONFIG_RTE_EAL_PMD_PATH`` suffixed with ``-glue`` if set, or in a
+ standard location for the dynamic linker (e.g. ``/lib``) if left to the
+ default empty string (``""``).
+
+ This option has no performance impact.
+
+- ``CONFIG_RTE_IBVERBS_LINK_STATIC`` (default **n**)
+
+ Embed static flavour of the dependencies **libibverbs** and **libmlx4**
+ in the PMD shared library or the executable static binary.
+
- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG`` (default **n**)
Toggle debugging code and stricter compilation flags. Enabling this option
adds additional run-time checks and debugging messages at the cost of
lower performance.
-- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS`` (default **n**)
-
- Mellanox OFED versions earlier than 4.2 may return false errors from
- Verbs object destruction APIs after the device is plugged out.
- Enabling this option replaces assertion checks that cause the program
- to abort with harmless debugging messages as a workaround.
- Relevant only when CONFIG_RTE_LIBRTE_MLX4_DEBUG is enabled.
+Environment variables
+~~~~~~~~~~~~~~~~~~~~~
-- ``CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE`` (default **8**)
+- ``MLX4_GLUE_PATH``
- Maximum number of cached memory pools (MPs) per TX queue. Each MP from
- which buffers are to be transmitted must be associated to memory regions
- (MRs). This is a slow operation that must be cached.
+ A list of directories in which to search for the rdma-core "glue" plug-in,
+ separated by colons or semi-colons.
- This value is always 1 for RX queues since they use a single MP.
+ Only matters when compiled with ``CONFIG_RTE_IBVERBS_LINK_DLOPEN``
+ enabled and most useful when ``CONFIG_RTE_EAL_PMD_PATH`` is also set,
+ since ``LD_LIBRARY_PATH`` has no effect in this case.
Run-time configuration
~~~~~~~~~~~~~~~~~~~~~~
following limitation: VLAN filtering is not supported with this mode.
This is the recommended mode in case VLAN filter is not needed.
+Limitations
+-----------
+
+- CRC stripping is supported by default and always reported as "true".
+ The ability to enable/disable CRC stripping requires OFED version
+ 4.3-1.5.0.0 and above or rdma-core version v18 and above.
+
+- TSO (Transmit Segmentation Offload) is supported in OFED version
+ 4.4 and above.
+
Prerequisites
-------------
allocations and initialization. The following dependencies are not part of
DPDK and must be installed separately:
-- **libibverbs**
+- **libibverbs** (provided by rdma-core package)
User space verbs framework used by librte_pmd_mlx4. This library provides
a generic interface between the kernel and low-level user space drivers
resources allocations) to be managed by the kernel and fast operations to
never leave user space.
-- **libmlx4**
+- **libmlx4** (provided by rdma-core package)
Low-level user space driver library for Mellanox ConnectX-3 devices,
it is automatically loaded by libibverbs.
This library basically implements send/receive calls to the hardware
queues.
-- **Kernel modules** (mlnx-ofed-kernel)
+- **Kernel modules**
They provide the kernel-side verbs API and low level device drivers that
manage actual hardware initialization and resources sharing with user
Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
licensed.
-Currently supported by DPDK:
+Depending on system constraints and user preferences either RDMA core library
+with a recent enough Linux kernel release (recommended) or Mellanox OFED,
+which provides compatibility with older releases.
-- Mellanox OFED **4.1**.
-- Firmware version **2.36.5000** and above.
+Current RDMA core package and Linux kernel (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Getting Mellanox OFED
-~~~~~~~~~~~~~~~~~~~~~
+- Minimal Linux kernel version: 4.14.
+- Minimal RDMA core version: v15 (see `RDMA core installation documentation`_).
+
+- Starting with rdma-core v21, static libraries can be built::
+
+ cd build
+ CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
+ ninja
+
+.. _`RDMA core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md
-While these libraries and kernel modules are available on OpenFabrics
-Alliance's `website <https://www.openfabrics.org/>`_ and provided by package
-managers on most distributions, this PMD requires Ethernet extensions that
-may not be supported at the moment (this is a work in progress).
+If rdma-core libraries are built but not installed, DPDK makefile can link them,
+thanks to these environment variables:
-`Mellanox OFED
-<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_
-includes the necessary support and should be used in the meantime. For DPDK,
-only libibverbs, libmlx4, mlnx-ofed-kernel packages and firmware updates are
-required from that distribution.
+ - ``EXTRA_CFLAGS=-I/path/to/rdma-core/build/include``
+ - ``EXTRA_LDFLAGS=-L/path/to/rdma-core/build/lib``
+ - ``PKG_CONFIG_PATH=/path/to/rdma-core/build/lib/pkgconfig``
+
+.. _Mellanox_OFED_as_a_fallback:
+
+Mellanox OFED as a fallback
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- `Mellanox OFED`_ version: **4.4, 4.5**.
+- firmware version: **2.42.5000** and above.
+
+.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
.. note::
this DPDK release was developed and tested against is strongly
recommended. Please check the `prerequisites`_.
-Supported NICs
---------------
-
-* Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2*40G)
-
-Quick Start Guide
------------------
+Installing Mellanox OFED
+^^^^^^^^^^^^^^^^^^^^^^^^
-1. Download latest Mellanox OFED. For more info check the `prerequisites`_.
+1. Download latest Mellanox OFED.
2. Install the required libraries and kernel modules either by installing
only the required set, or by installing the entire Mellanox OFED:
.. code-block:: console
- ./mlnxofedinstall
+ ./mlnxofedinstall --dpdk --upstream-libs
For SR-IOV hypervisors use:
.. code-block:: console
- ./mlnxofedinstall --enable-sriov -hypervisor
+ ./mlnxofedinstall --dpdk --upstream-libs --enable-sriov --hypervisor
For SR-IOV virtual machine use:
.. code-block:: console
- ./mlnxofedinstall --guest
+ ./mlnxofedinstall --dpdk --upstream-libs --guest
3. Verify the firmware is the correct one:
connectx_port_config
- Or in the manual way:
+5. Continue with :ref:`section 2 of the Quick Start Guide <QSG_2>`.
+
+Supported NICs
+--------------
+
+* Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2*40G)
+
+.. _qsg:
+
+Quick Start Guide
+-----------------
+
+1. Set all ports links to Ethernet
.. code-block:: console
echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port0"
echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port1"
-5. In case of bare metal or hypervisor, configure optimized steering mode
+ .. note::
+
+ If using Mellanox OFED one can permanently set the port link
+ to Ethernet using connectx_port_config tool provided by it.
+ :ref:`Mellanox_OFED_as_a_fallback`:
+
+.. _QSG_2:
+
+2. In case of bare metal or hypervisor, configure optimized steering mode
by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``:
.. code-block:: console
If VLAN filtering is used, set log_num_mgm_entry_size=-1.
Performance degradation can occur on this case.
-6. Restart the driver:
+3. Restart the driver:
.. code-block:: console
service openibd restart
-7. Compile DPDK and you are ready to go. See instructions on
+4. Compile DPDK and you are ready to go. See instructions on
:ref:`Development Kit Build System <Development_Kit_Build_System>`
Performance tuning
The XXX can be different on different systems. Make sure to configure
according to the setpci output.
+6. To minimize overhead of searching Memory Regions:
+
+ - '--socket-mem' is recommended to pin memory by predictable amount.
+ - Configure per-lcore cache when creating Mempools for packet buffer.
+ - Refrain from dynamically allocating/freeing memory in run-time.
+
Usage example
-------------
git.droids-corp.org / dpdk.git / blobdiff