X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fnics%2Fmlx4.rst;h=ed920e91ad51013eb4cfeb9c0b2c0d6e6537e750;hb=b94c709decbe39582c68cf3605745b60a4f7985a;hp=cab45dfe84b355813bde03886f5335fa17a472c4;hpb=c2b3dba84ad319003df04a5a9fa163a953224e83;p=dpdk.git diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index cab45dfe84..ed920e91ad 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2012 6WIND S.A. - Copyright 2015 Mellanox - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Copyright 2015 Mellanox Technologies, Ltd MLX4 poll mode driver library ============================= @@ -42,11 +16,6 @@ the `Mellanox community `_. There is also a `section dedicated to this poll mode driver `_. -.. note:: - - Due to external dependencies, this driver is disabled by default. It must - be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX4_PMD=y`` and - recompiling DPDK. Implementation details ---------------------- @@ -72,6 +41,8 @@ This capability allows the PMD to coexist with kernel network interfaces which remain functional, although they stop receiving unicast packets as long as they share the same MAC address. +The :ref:`flow_isolated_mode` is supported. + Compiling librte_pmd_mlx4 causes DPDK to be linked against libibverbs. Configuration @@ -80,25 +51,28 @@ Configuration Compilation options ~~~~~~~~~~~~~~~~~~~ -These options can be modified in the ``.config`` file. +The ibverbs libraries can be linked with this PMD in a number of ways, +configured by the ``ibverbs_link`` build option: -- ``CONFIG_RTE_LIBRTE_MLX4_PMD`` (default **n**) +- ``shared`` (default): the PMD depends on some .so files. - Toggle compilation of librte_pmd_mlx4 itself. +- ``dlopen``: Split the dependencies glue in a separate library + loaded when needed by dlopen. + It make dependencies on libibverbs and libmlx4 optional, + and has no performance impact. -- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG`` (default **n**) +- ``static``: Embed static flavor of the dependencies libibverbs and libmlx4 + in the PMD shared library or the executable static binary. - Toggle debugging code and stricter compilation flags. Enabling this option - adds additional run-time checks and debugging messages at the cost of - lower performance. -- ``CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE`` (default **8**) +Environment variables +~~~~~~~~~~~~~~~~~~~~~ - Maximum number of cached memory pools (MPs) per TX queue. Each MP from - which buffers are to be transmitted must be associated to memory regions - (MRs). This is a slow operation that must be cached. +- ``MLX4_GLUE_PATH`` + + A list of directories in which to search for the rdma-core "glue" plug-in, + separated by colons or semi-colons. - This value is always 1 for RX queues since they use a single MP. Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ @@ -115,6 +89,17 @@ Run-time configuration times for additional ports. All ports are probed by default if left unspecified. +- ``mr_ext_memseg_en`` parameter [int] + + A nonzero value enables extending memseg when registering DMA memory. If + enabled, the number of entries in MR (Memory Region) lookup table on datapath + is minimized and it benefits performance. On the other hand, it worsens memory + utilization because registered memory is pinned by kernel driver. Even if a + page in the extended chunk is freed, that doesn't become reusable until the + entire memory is freed. + + Enabled by default. + Kernel module parameters ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -138,6 +123,25 @@ below. following limitation: VLAN filtering is not supported with this mode. This is the recommended mode in case VLAN filter is not needed. +Limitations +----------- + +- For secondary process: + + - Forked secondary process not supported. + - External memory unregistered in EAL memseg list cannot be used for DMA + unless such memory has been registered by ``mlx4_mr_update_ext_mp()`` in + primary process and remapped to the same virtual address in secondary + process. If the external memory is registered by primary process but has + different virtual address in secondary process, unexpected error may happen. + +- CRC stripping is supported by default and always reported as "true". + The ability to enable/disable CRC stripping requires OFED version + 4.3-1.5.0.0 and above or rdma-core version v18 and above. + +- TSO (Transmit Segmentation Offload) is supported in OFED version + 4.4 and above. + Prerequisites ------------- @@ -199,6 +203,12 @@ Current RDMA core package and Linux kernel (recommended) - Minimal Linux kernel version: 4.14. - Minimal RDMA core version: v15 (see `RDMA core installation documentation`_). +- Starting with rdma-core v21, static libraries can be built:: + + cd build + CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. + ninja + .. _`RDMA core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md .. _Mellanox_OFED_as_a_fallback: @@ -206,7 +216,7 @@ Current RDMA core package and Linux kernel (recommended) Mellanox OFED as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- `Mellanox OFED`_ version: **4.2**. +- `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. .. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers @@ -225,51 +235,34 @@ Installing Mellanox OFED 2. Install the required libraries and kernel modules either by installing only the required set, or by installing the entire Mellanox OFED: - For bare metal use: - - .. code-block:: console + For bare metal use:: ./mlnxofedinstall --dpdk --upstream-libs - For SR-IOV hypervisors use: - - .. code-block:: console + For SR-IOV hypervisors use:: ./mlnxofedinstall --dpdk --upstream-libs --enable-sriov --hypervisor - For SR-IOV virtual machine use: - - .. code-block:: console + For SR-IOV virtual machine use:: ./mlnxofedinstall --dpdk --upstream-libs --guest -3. Verify the firmware is the correct one: - - .. code-block:: console +3. Verify the firmware is the correct one:: ibv_devinfo -4. Set all ports links to Ethernet, follow instructions on the screen: - - .. code-block:: console +4. Set all ports links to Ethernet, follow instructions on the screen:: connectx_port_config 5. Continue with :ref:`section 2 of the Quick Start Guide `. -Supported NICs --------------- - -* Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2*40G) - .. _qsg: Quick Start Guide ----------------- -1. Set all ports links to Ethernet - - .. code-block:: console +1. Set all ports links to Ethernet:: PCI= echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port0" @@ -284,9 +277,7 @@ Quick Start Guide .. _QSG_2: 2. In case of bare metal or hypervisor, configure optimized steering mode - by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``: - - .. code-block:: console + by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``:: options mlx4_core log_num_mgm_entry_size=-7 @@ -295,35 +286,27 @@ Quick Start Guide If VLAN filtering is used, set log_num_mgm_entry_size=-1. Performance degradation can occur on this case. -3. Restart the driver: - - .. code-block:: console +3. Restart the driver:: /etc/init.d/openibd restart - or: - - .. code-block:: console + or:: service openibd restart -4. Compile DPDK and you are ready to go. See instructions on - :ref:`Development Kit Build System ` +4. Install DPDK and you are ready to go. + See :doc:`compilation instructions <../linux_gsg/build_dpdk>`. Performance tuning ------------------ -1. Verify the optimized steering mode is configured: - - .. code-block:: console +1. Verify the optimized steering mode is configured:: cat /sys/module/mlx4_core/parameters/log_num_mgm_entry_size 2. Use the CPU near local NUMA node to which the PCIe adapter is connected, for better performance. For VMs, verify that the right CPU - and NUMA node are pinned according to the above. Run: - - .. code-block:: console + and NUMA node are pinned according to the above. Run:: lstopo-no-graphics @@ -335,9 +318,7 @@ Performance tuning This in order to forward packets from one to the other without NUMA performance penalty. -4. Disable pause frames: - - .. code-block:: console +4. Disable pause frames:: ethtool -A rx off tx off @@ -351,37 +332,35 @@ Performance tuning to set the PCI max read request parameter to 1K. This can be done in the following way: - To query the read request size use: - - .. code-block:: console + To query the read request size use:: setpci -s 68.w - If the output is different than 3XXX, set it by: - - .. code-block:: console + If the output is different than 3XXX, set it by:: setpci -s 68.w=3XXX The XXX can be different on different systems. Make sure to configure according to the setpci output. +6. To minimize overhead of searching Memory Regions: + + - '--socket-mem' is recommended to pin memory by predictable amount. + - Configure per-lcore cache when creating Mempools for packet buffer. + - Refrain from dynamically allocating/freeing memory in run-time. + Usage example ------------- This section demonstrates how to launch **testpmd** with Mellanox ConnectX-3 devices managed by librte_pmd_mlx4. -#. Load the kernel modules: - - .. code-block:: console +#. Load the kernel modules:: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib Alternatively if MLNX_OFED is fully installed, the following script can - be run: - - .. code-block:: console + be run:: /etc/init.d/openibd restart @@ -391,24 +370,18 @@ devices managed by librte_pmd_mlx4. not have to be loaded. #. Make sure Ethernet interfaces are in working order and linked to kernel - verbs. Related sysfs entries should be present: - - .. code-block:: console + verbs. Related sysfs entries should be present:: ls -d /sys/class/net/*/device/infiniband_verbs/uverbs* | cut -d / -f 5 - Example output: - - .. code-block:: console + Example output:: eth2 eth3 eth4 eth5 -#. Optionally, retrieve their PCI bus addresses for whitelisting: - - .. code-block:: console +#. Optionally, retrieve their PCI bus addresses for whitelisting:: { for intf in eth2 eth3 eth4 eth5; @@ -418,9 +391,7 @@ devices managed by librte_pmd_mlx4. } | sed -n 's,.*/\(.*\),-w \1,p' - Example output: - - .. code-block:: console + Example output:: -w 0000:83:00.0 -w 0000:83:00.0 @@ -432,21 +403,15 @@ devices managed by librte_pmd_mlx4. There are only two distinct PCI bus addresses because the Mellanox ConnectX-3 adapters installed on this system are dual port. -#. Request huge pages: - - .. code-block:: console +#. Request huge pages:: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages -#. Start testpmd with basic parameters: - - .. code-block:: console +#. Start testpmd with basic parameters:: testpmd -l 8-15 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i - Example output: - - .. code-block:: console + Example output:: [...] EAL: PCI device 0000:83:00.0 on NUMA socket 1