.. BSD LICENSE
Copyright 2012-2015 6WIND S.A.
+ Copyright 2015 Mellanox
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
=============================
The MLX4 poll mode driver library (**librte_pmd_mlx4**) implements support
-for **Mellanox ConnectX-3 EN** 10/40 Gbps adapters as well as their virtual
-functions (VF) in SR-IOV context.
+for **Mellanox ConnectX-3** and **Mellanox ConnectX-3 Pro** 10/40 Gbps adapters
+as well as their virtual functions (VF) in SR-IOV context.
Information and documentation about this family of adapters can be found on
the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by
be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX4_PMD=y`` and
recompiling DPDK.
-.. warning::
-
- ``CONFIG_RTE_BUILD_COMBINE_LIBS`` with ``CONFIG_RTE_BUILD_SHARED_LIB``
- is not supported and thus the compilation will fail with this configuration.
-
Implementation details
----------------------
Compiling librte_pmd_mlx4 causes DPDK to be linked against libibverbs.
-Features and limitations
-------------------------
+Features
+--------
+- Multi arch support: x86_64 and POWER8.
- RSS, also known as RCA, is supported. In this mode the number of
configured RX queues must be a power of two.
- VLAN filtering is supported.
- Scattered packets are supported for TX and RX.
- Inner L3/L4 (IP, TCP and UDP) TX/RX checksum offloading and validation.
- Outer L3 (IP) TX/RX checksum offloading and validation for VXLAN frames.
+- Secondary process TX is supported.
+- RX interrupts.
-.. break
+Limitations
+-----------
- RSS hash key cannot be modified.
+- RSS RETA cannot be configured
+- RSS always includes L3 (IPv4/IPv6) and L4 (UDP/TCP). They cannot be
+ dissociated.
- Hardware counters are not implemented (they are software counters).
+- Secondary process RX is not supported.
Configuration
-------------
adds additional run-time checks and debugging messages at the cost of
lower performance.
+- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS`` (default **n**)
+
+ Mellanox OFED versions earlier than 4.2 may return false errors from
+ Verbs object destruction APIs after the device is plugged out.
+ Enabling this option replaces assertion checks that cause the program
+ to abort with harmless debugging messages as a workaround.
+ Relevant only when CONFIG_RTE_LIBRTE_MLX4_DEBUG is enabled.
+
- ``CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N`` (default **4**)
Number of scatter/gather elements (SGEs) per work request (WR). Lowering
- **ethtool** operations on related kernel interfaces also affect the PMD.
+- ``port`` parameter [int]
+
+ This parameter provides a physical port to probe and can be specified multiple
+ times for additional ports. All ports are probed by default if left
+ unspecified.
+
Kernel module parameters
~~~~~~~~~~~~~~~~~~~~~~~~
Currently supported by DPDK:
-- Mellanox OFED **3.0**.
-- Firmware version **2.34.5000** and higher.
+- Mellanox OFED **4.1**.
+- Firmware version **2.36.5000** and above.
Getting Mellanox OFED
~~~~~~~~~~~~~~~~~~~~~
this DPDK release was developed and tested against is strongly
recommended. Please check the `prerequisites`_.
+Supported NICs
+--------------
+
+* Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2*40G)
+
+Quick Start Guide
+-----------------
+
+1. Download latest Mellanox OFED. For more info check the `prerequisites`_.
+
+2. Install the required libraries and kernel modules either by installing
+ only the required set, or by installing the entire Mellanox OFED:
+
+ For bare metal use:
+
+ .. code-block:: console
+
+ ./mlnxofedinstall
+
+ For SR-IOV hypervisors use:
+
+ .. code-block:: console
+
+ ./mlnxofedinstall --enable-sriov -hypervisor
+
+ For SR-IOV virtual machine use:
+
+ .. code-block:: console
+
+ ./mlnxofedinstall --guest
+
+3. Verify the firmware is the correct one:
+
+ .. code-block:: console
+
+ ibv_devinfo
+
+4. Set all ports links to Ethernet, follow instructions on the screen:
+
+ .. code-block:: console
+
+ connectx_port_config
+
+ Or in the manual way:
+
+ .. code-block:: console
+
+ PCI=<NIC PCI address>
+ echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port0"
+ echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port1"
+
+5. In case of bare metal or hypervisor, configure optimized steering mode
+ by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``:
+
+ .. code-block:: console
+
+ options mlx4_core log_num_mgm_entry_size=-7
+
+ .. note::
+
+ If VLAN filtering is used, set log_num_mgm_entry_size=-1.
+ Performance degradation can occur on this case.
+
+6. Restart the driver:
+
+ .. code-block:: console
+
+ /etc/init.d/openibd restart
+
+ or:
+
+ .. code-block:: console
+
+ service openibd restart
+
+7. Compile DPDK and you are ready to go. See instructions on
+ :ref:`Development Kit Build System <Development_Kit_Build_System>`
+
+Performance tuning
+------------------
+
+1. Verify the optimized steering mode is configured:
+
+ .. code-block:: console
+
+ cat /sys/module/mlx4_core/parameters/log_num_mgm_entry_size
+
+2. Use environment variable MLX4_INLINE_RECV_SIZE=64 to get maximum
+ performance for 64B messages.
+
+3. Use the CPU near local NUMA node to which the PCIe adapter is connected,
+ for better performance. For VMs, verify that the right CPU
+ and NUMA node are pinned according to the above. Run:
+
+ .. code-block:: console
+
+ lstopo-no-graphics
+
+ to identify the NUMA node to which the PCIe adapter is connected.
+
+4. If more than one adapter is used, and root complex capabilities allow
+ to put both adapters on the same NUMA node without PCI bandwidth degradation,
+ it is recommended to locate both adapters on the same NUMA node.
+ This in order to forward packets from one to the other without
+ NUMA performance penalty.
+
+5. Disable pause frames:
+
+ .. code-block:: console
+
+ ethtool -A <netdev> rx off tx off
+
+6. Verify IO non-posted prefetch is disabled by default. This can be checked
+ via the BIOS configuration. Please contact you server provider for more
+ information about the settings.
+
+.. note::
+
+ On some machines, depends on the machine integrator, it is beneficial
+ to set the PCI max read request parameter to 1K. This can be
+ done in the following way:
+
+ To query the read request size use:
+
+ .. code-block:: console
+
+ setpci -s <NIC PCI address> 68.w
+
+ If the output is different than 3XXX, set it by:
+
+ .. code-block:: console
+
+ setpci -s <NIC PCI address> 68.w=3XXX
+
+ The XXX can be different on different systems. Make sure to configure
+ according to the setpci output.
+
Usage example
-------------
modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib
+ Alternatively if MLNX_OFED is fully installed, the following script can
+ be run:
+
+ .. code-block:: console
+
+ /etc/init.d/openibd restart
+
.. note::
User space I/O kernel modules (uio and igb_uio) are not used and do
.. code-block:: console
- testpmd -c 0xff00 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i
+ testpmd -l 8-15 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i
Example output:
git.droids-corp.org / dpdk.git / blobdiff