X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;ds=sidebyside;f=doc%2Fguides%2Fnics%2Fmlx4.rst;h=f8885b23fc54c01f401177e1dd7185726e9ea0f2;hb=bad555df14626c84c3eee208fdff67c83b855a9d;hp=c33aa389e7f38f890932eba09358e6266e92169c;hpb=71d6dac903d65d018c7de03c947042db5d00a70f;p=dpdk.git diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index c33aa389e7..f8885b23fc 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -1,5 +1,6 @@ .. BSD LICENSE Copyright 2012-2015 6WIND S.A. + Copyright 2015 Mellanox Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions @@ -31,8 +32,8 @@ MLX4 poll mode driver library ============================= The MLX4 poll mode driver library (**librte_pmd_mlx4**) implements support -for **Mellanox ConnectX-3 EN** 10/40 Gbps adapters as well as their virtual -functions (VF) in SR-IOV context. +for **Mellanox ConnectX-3** and **Mellanox ConnectX-3 Pro** 10/40 Gbps adapters +as well as their virtual functions (VF) in SR-IOV context. Information and documentation about this family of adapters can be found on the `Mellanox website `_. Help is also provided by @@ -73,9 +74,10 @@ long as they share the same MAC address. Compiling librte_pmd_mlx4 causes DPDK to be linked against libibverbs. -Features and limitations ------------------------- +Features +-------- +- Multi arch support: x86_64 and POWER8. - RSS, also known as RCA, is supported. In this mode the number of configured RX queues must be a power of two. - VLAN filtering is supported. @@ -86,11 +88,18 @@ Features and limitations - Scattered packets are supported for TX and RX. - Inner L3/L4 (IP, TCP and UDP) TX/RX checksum offloading and validation. - Outer L3 (IP) TX/RX checksum offloading and validation for VXLAN frames. +- Secondary process TX is supported. +- RX interrupts. -.. break +Limitations +----------- - RSS hash key cannot be modified. +- RSS RETA cannot be configured +- RSS always includes L3 (IPv4/IPv6) and L4 (UDP/TCP). They cannot be + dissociated. - Hardware counters are not implemented (they are software counters). +- Secondary process RX is not supported. Configuration ------------- @@ -110,6 +119,14 @@ These options can be modified in the ``.config`` file. adds additional run-time checks and debugging messages at the cost of lower performance. +- ``CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS`` (default **n**) + + Mellanox OFED versions earlier than 4.2 may return false errors from + Verbs object destruction APIs after the device is plugged out. + Enabling this option replaces assertion checks that cause the program + to abort with harmless debugging messages as a workaround. + Relevant only when CONFIG_RTE_LIBRTE_MLX4_DEBUG is enabled. + - ``CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N`` (default **4**) Number of scatter/gather elements (SGEs) per work request (WR). Lowering @@ -156,6 +173,12 @@ Run-time configuration - **ethtool** operations on related kernel interfaces also affect the PMD. +- ``port`` parameter [int] + + This parameter provides a physical port to probe and can be specified multiple + times for additional ports. All ports are probed by default if left + unspecified. + Kernel module parameters ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -232,8 +255,8 @@ DPDK and must be installed separately: Currently supported by DPDK: -- Mellanox OFED **3.0**. -- Firmware version **2.34.5000** and higher. +- Mellanox OFED **4.1**. +- Firmware version **2.36.5000** and above. Getting Mellanox OFED ~~~~~~~~~~~~~~~~~~~~~ @@ -255,6 +278,143 @@ required from that distribution. this DPDK release was developed and tested against is strongly recommended. Please check the `prerequisites`_. +Supported NICs +-------------- + +* Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2*40G) + +Quick Start Guide +----------------- + +1. Download latest Mellanox OFED. For more info check the `prerequisites`_. + +2. Install the required libraries and kernel modules either by installing + only the required set, or by installing the entire Mellanox OFED: + + For bare metal use: + + .. code-block:: console + + ./mlnxofedinstall + + For SR-IOV hypervisors use: + + .. code-block:: console + + ./mlnxofedinstall --enable-sriov -hypervisor + + For SR-IOV virtual machine use: + + .. code-block:: console + + ./mlnxofedinstall --guest + +3. Verify the firmware is the correct one: + + .. code-block:: console + + ibv_devinfo + +4. Set all ports links to Ethernet, follow instructions on the screen: + + .. code-block:: console + + connectx_port_config + + Or in the manual way: + + .. code-block:: console + + PCI= + echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port0" + echo eth > "/sys/bus/pci/devices/$PCI/mlx4_port1" + +5. In case of bare metal or hypervisor, configure optimized steering mode + by adding the following line to ``/etc/modprobe.d/mlx4_core.conf``: + + .. code-block:: console + + options mlx4_core log_num_mgm_entry_size=-7 + + .. note:: + + If VLAN filtering is used, set log_num_mgm_entry_size=-1. + Performance degradation can occur on this case. + +6. Restart the driver: + + .. code-block:: console + + /etc/init.d/openibd restart + + or: + + .. code-block:: console + + service openibd restart + +7. Compile DPDK and you are ready to go. See instructions on + :ref:`Development Kit Build System ` + +Performance tuning +------------------ + +1. Verify the optimized steering mode is configured: + + .. code-block:: console + + cat /sys/module/mlx4_core/parameters/log_num_mgm_entry_size + +2. Use environment variable MLX4_INLINE_RECV_SIZE=64 to get maximum + performance for 64B messages. + +3. Use the CPU near local NUMA node to which the PCIe adapter is connected, + for better performance. For VMs, verify that the right CPU + and NUMA node are pinned according to the above. Run: + + .. code-block:: console + + lstopo-no-graphics + + to identify the NUMA node to which the PCIe adapter is connected. + +4. If more than one adapter is used, and root complex capabilities allow + to put both adapters on the same NUMA node without PCI bandwidth degradation, + it is recommended to locate both adapters on the same NUMA node. + This in order to forward packets from one to the other without + NUMA performance penalty. + +5. Disable pause frames: + + .. code-block:: console + + ethtool -A rx off tx off + +6. Verify IO non-posted prefetch is disabled by default. This can be checked + via the BIOS configuration. Please contact you server provider for more + information about the settings. + +.. note:: + + On some machines, depends on the machine integrator, it is beneficial + to set the PCI max read request parameter to 1K. This can be + done in the following way: + + To query the read request size use: + + .. code-block:: console + + setpci -s 68.w + + If the output is different than 3XXX, set it by: + + .. code-block:: console + + setpci -s 68.w=3XXX + + The XXX can be different on different systems. Make sure to configure + according to the setpci output. + Usage example ------------- @@ -267,6 +427,13 @@ devices managed by librte_pmd_mlx4. modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib + Alternatively if MLNX_OFED is fully installed, the following script can + be run: + + .. code-block:: console + + /etc/init.d/openibd restart + .. note:: User space I/O kernel modules (uio and igb_uio) are not used and do @@ -324,7 +491,7 @@ devices managed by librte_pmd_mlx4. .. code-block:: console - testpmd -c 0xff00 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i + testpmd -l 8-15 -n 4 -w 0000:83:00.0 -w 0000:84:00.0 -- --rxq=2 --txq=2 -i Example output: