.. include:: <isonum.txt>
-MLX5 poll mode driver
-=====================
+MLX5 Ethernet Poll Mode Driver
+==============================
-The MLX5 poll mode driver library (**librte_net_mlx5**) provides support
+The mlx5 Ethernet poll mode driver library (**librte_net_mlx5**) provides support
for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox
ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox
ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families
of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF)
in SR-IOV context.
-Information and documentation about these adapters can be found on the
-`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the
-`Mellanox community <http://community.mellanox.com/welcome>`__.
-
-There is also a `section dedicated to this poll mode driver
-<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__.
-
Design
------
kernel support), librte_net_mlx5 relies heavily on system calls for control
operations such as querying/updating the MTU and flow control parameters.
-For security reasons and robustness, this driver only deals with virtual
-memory addresses. The way resources allocations are handled by the kernel,
-combined with hardware specifications that allow to handle virtual memory
-addresses directly, ensure that DPDK applications cannot access random
-physical memory (or memory that does not belong to the current process).
-
This capability allows the PMD to coexist with kernel network interfaces
which remain functional, although they stop receiving unicast packets as
long as they share the same MAC address.
more) can operate on the same network interfaces that owned by the DPDK
application.
-The PMD can use libibverbs and libmlx5 to access the device firmware
-or directly the hardware components.
-There are different levels of objects and bypassing abilities
-to get the best performances:
-
-- Verbs is a complete high-level generic API
-- Direct Verbs is a device-specific API
-- DevX allows to access firmware objects
-- Direct Rules manages flow steering at low-level hardware layer
-
-Enabling librte_net_mlx5 causes DPDK applications to be linked against
-libibverbs.
+See :doc:`../../platform/mlx5` guide for more design details.
Features
--------
- Multi arch support: x86_64, POWER8, ARMv8, i686.
- Multiple TX and RX queues.
+- Shared Rx queue.
+- Rx queue delay drop.
+- Support steering for external Rx queue created outside the PMD.
- Support for scattered TX frames.
- Advanced support for scattered Rx frames with tunable buffer attributes.
- IPv4, IPv6, TCPv4, TCPv6, UDPv4 and UDPv6 RSS on any number of queues.
- Matching on IPv4 Internet Header Length (IHL).
- Matching on GTP extension header with raw encap/decap action.
- Matching on Geneve TLV option header with raw encap/decap action.
+- Matching on ESP header SPI field.
- RSS support in sample action.
- E-Switch mirroring and jump.
- E-Switch mirroring and modify.
- For secondary process:
- Forked secondary process not supported.
+ - MPRQ is not supported. Callback to free externally attached MPRQ buffer is set
+ in a primary process, but has a different virtual address in a secondary process.
+ Calling a function at the wrong address leads to a segmentation fault.
- External memory unregistered in EAL memseg list cannot be used for DMA
unless such memory has been registered by ``mlx5_mr_update_ext_mp()`` in
primary process and remapped to the same virtual address in secondary
process. If the external memory is registered by primary process but has
different virtual address in secondary process, unexpected error may happen.
+- Shared Rx queue:
+
+ - Counters of received packets and bytes number of devices in same share group are same.
+ - Counters of received packets and bytes number of queues in same group and queue ID are same.
+
- When using Verbs flow engine (``dv_flow_en`` = 0), flow pattern without any
specific VLAN will match for VLAN packets as well:
Last reserved 8-bits matching is only supported When using DV flow
engine (``dv_flow_en`` = 1).
+ For ConnectX-5, the UDP destination port must be the standard one (4789).
Group zero's behavior may differ which depends on FW.
Matching value equals 0 (value & mask) is not supported.
extension header type = 0x85).
- Match on GTP extension header is not supported in group 0.
+- Flex item:
+
+ - Hardware support: BlueField-2.
+ - Flex item is supported on PF only.
+ - Hardware limits ``header_length_mask_width`` up to 6 bits.
+ - Firmware supports 8 global sample fields.
+ Each flex item allocates non-shared sample fields from that pool.
+ - Supported flex item can have 1 input link - ``eth`` or ``udp``
+ and up to 2 output links - ``ipv4`` or ``ipv6``.
+ - Flex item fields (``next_header``, ``next_protocol``, ``samples``)
+ do not participate in RSS hash functions.
+ - In flex item configuration, ``next_header.field_base`` value
+ must be byte aligned (multiple of 8).
+
- No Tx metadata go to the E-Switch steering domain for the Flow group 0.
The flows within group 0 and set metadata action are rejected by hardware.
no MPRQ feature or vectorized code can be engaged.
- When Multi-Packet Rx queue is configured (``mprq_en``), a Rx packet can be
- externally attached to a user-provided mbuf with having EXT_ATTACHED_MBUF in
+ externally attached to a user-provided mbuf with having RTE_MBUF_F_EXTERNAL in
ol_flags. As the mempool for the external buffer is managed by PMD, all the
Rx mbufs must be freed before the device is closed. Otherwise, the mempool of
the external buffers will be freed by PMD and the application which still
holds the external buffers may be corrupted.
+ User-managed mempools with external pinned data buffers
+ cannot be used in conjunction with MPRQ
+ since packets may be already attached to PMD-managed external buffers.
- If Multi-Packet Rx queue is configured (``mprq_en``) and Rx CQE compression is
enabled (``rxq_cqe_comp_en``) at the same time, RSS hash result is not fully
- supported. Some Rx packets may not have PKT_RX_RSS_HASH.
+ supported. Some Rx packets may not have RTE_MBUF_F_RX_RSS_HASH.
- IPv6 Multicast messages are not supported on VM, while promiscuous mode
and allmulticast mode are both set to off.
- CRC:
- - ``DEV_RX_OFFLOAD_KEEP_CRC`` cannot be supported with decapsulation
+ - ``RTE_ETH_RX_OFFLOAD_KEEP_CRC`` cannot be supported with decapsulation
for some NICs (such as ConnectX-6 Dx, ConnectX-6 Lx, and BlueField-2).
The capability bit ``scatter_fcs_w_decap_disable`` shows NIC support.
sample actions list.
- For E-Switch mirroring flow, supports ``RAW ENCAP``, ``Port ID``,
``VXLAN ENCAP``, ``NVGRE ENCAP`` in the sample actions list.
+ - For ConnectX-5 trusted device, the application metadata with SET_TAG index 0
+ is not supported before ``RTE_FLOW_ACTION_TYPE_SAMPLE`` action.
- Modify Field flow:
- Modification of the 802.1Q Tag, VXLAN Network or GENEVE Network ID's is not supported.
- Encapsulation levels are not supported, can modify outermost header fields only.
- Offsets must be 32-bits aligned, cannot skip past the boundary of a field.
+ - If the field type is ``RTE_FLOW_FIELD_MAC_TYPE``
+ and packet contains one or more VLAN headers,
+ the meaningful type field following the last VLAN header
+ is used as modify field operation argument.
+ The modify field action is not intended to modify VLAN headers type field,
+ dedicated VLAN push and pop actions should be used instead.
- IPv6 header item 'proto' field, indicating the next header protocol, should
not be set as extension header.
- yellow: NULL or END.
- RED: DROP / END.
- The only supported meter policy actions:
- - green: QUEUE, RSS, PORT_ID, JUMP, DROP, MARK and SET_TAG.
- - yellow: QUEUE, RSS, PORT_ID, JUMP, DROP, MARK and SET_TAG.
+ - green: QUEUE, RSS, PORT_ID, REPRESENTED_PORT, JUMP, DROP, MARK, METER and SET_TAG.
+ - yellow: QUEUE, RSS, PORT_ID, REPRESENTED_PORT, JUMP, DROP, MARK, METER and SET_TAG.
- RED: must be DROP.
- Policy actions of RSS for green and yellow should have the same configuration except queues.
+ - Policy with RSS/queue action is not supported when ``dv_xmeta_en`` enabled.
+ - If green action is METER, yellow action must be the same METER action or NULL.
- meter profile packet mode is supported.
- meter profiles of RFC2697, RFC2698 and RFC4115 are supported.
+ - RFC4115 implementation is following MEF, meaning yellow traffic may reclaim unused green bandwidth when green token bucket is full.
- Integrity:
- In order to achieve best insertion rate, application should manage the flows per lcore.
- Better to disable memory reclaim by setting ``reclaim_mem_mode`` to 0 to accelerate the flow object allocation and release with cache.
-Statistics
-----------
+- HW hashed bonding
-MLX5 supports various methods to report statistics:
+ - TXQ affinity subjects to HW hash once enabled.
-Port statistics can be queried using ``rte_eth_stats_get()``. The received and sent statistics are through SW only and counts the number of packets received or sent successfully by the PMD. The imissed counter is the amount of packets that could not be delivered to SW because a queue was full. Packets not received due to congestion in the bus or on the NIC can be queried via the rx_discards_phy xstats counter.
+- Bonding under socket direct mode
-Extended statistics can be queried using ``rte_eth_xstats_get()``. The extended statistics expose a wider set of counters counted by the device. The extended port statistics counts the number of packets received or sent successfully by the port. As Mellanox NICs are using the :ref:`Bifurcated Linux Driver <linux_gsg_linux_drivers>` those counters counts also packet received or sent by the Linux kernel. The counters with ``_phy`` suffix counts the total events on the physical port, therefore not valid for VF.
+ - Needs OFED 5.4+.
-Finally per-flow statistics can by queried using ``rte_flow_query`` when attaching a count action for specific flow. The flow counter counts the number of packets received successfully by the port and match the specific flow.
+- Timestamps:
-Configuration
--------------
-
-Compilation options
-~~~~~~~~~~~~~~~~~~~
-
-The ibverbs libraries can be linked with this PMD in a number of ways,
-configured by the ``ibverbs_link`` build option:
+ - CQE timestamp field width is limited by hardware to 63 bits, MSB is zero.
+ - In the free-running mode the timestamp counter is reset on power on
+ and 63-bit value provides over 1800 years of uptime till overflow.
+ - In the real-time mode
+ (configurable with ``REAL_TIME_CLOCK_ENABLE`` firmware settings),
+ the timestamp presents the nanoseconds elapsed since 01-Jan-1970,
+ hardware timestamp overflow will happen on 19-Jan-2038
+ (0x80000000 seconds since 01-Jan-1970).
+ - The send scheduling is based on timestamps
+ from the reference "Clock Queue" completions,
+ the scheduled send timestamps should not be specified with non-zero MSB.
-- ``shared`` (default): the PMD depends on some .so files.
+ - HW steering:
-- ``dlopen``: Split the dependencies glue in a separate library
- loaded when needed by dlopen.
- It make dependencies on libibverbs and libmlx4 optional,
- and has no performance impact.
+ - WQE based high scaling and safer flow insertion/destruction.
+ - Set ``dv_flow_en`` to 2 in order to enable HW steering.
+ - Async queue-based ``rte_flow_q`` APIs supported only.
-- ``static``: Embed static flavor of the dependencies libibverbs and libmlx4
- in the PMD shared library or the executable static binary.
+- Match on GRE header supports the following fields:
-Environment variables
-~~~~~~~~~~~~~~~~~~~~~
+ - c_rsvd0_v: C bit, K bit, S bit
+ - protocol type
+ - checksum
+ - key
+ - sequence
-- ``MLX5_GLUE_PATH``
+ Matching on checksum and sequence needs OFED 5.6+.
- A list of directories in which to search for the rdma-core "glue" plug-in,
- separated by colons or semi-colons.
+- The NIC egress flow rules on representor port are not supported.
-- ``MLX5_SHUT_UP_BF``
- Configures HW Tx doorbell register as IO-mapped.
-
- By default, the HW Tx doorbell is configured as a write-combining register.
- The register would be flushed to HW usually when the write-combining buffer
- becomes full, but it depends on CPU design.
+Statistics
+----------
- Except for vectorized Tx burst routines, a write memory barrier is enforced
- after updating the register so that the update can be immediately visible to
- HW.
+MLX5 supports various methods to report statistics:
- When vectorized Tx burst is called, the barrier is set only if the burst size
- is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental
- variable will bring better latency even though the maximum throughput can
- slightly decline.
+Port statistics can be queried using ``rte_eth_stats_get()``. The received and sent statistics are through SW only and counts the number of packets received or sent successfully by the PMD. The imissed counter is the amount of packets that could not be delivered to SW because a queue was full. Packets not received due to congestion in the bus or on the NIC can be queried via the rx_discards_phy xstats counter.
-Run-time configuration
-~~~~~~~~~~~~~~~~~~~~~~
+Extended statistics can be queried using ``rte_eth_xstats_get()``. The extended statistics expose a wider set of counters counted by the device. The extended port statistics counts the number of packets received or sent successfully by the port. As Mellanox NICs are using the :ref:`Bifurcated Linux Driver <linux_gsg_linux_drivers>` those counters counts also packet received or sent by the Linux kernel. The counters with ``_phy`` suffix counts the total events on the physical port, therefore not valid for VF.
-- librte_net_mlx5 brings kernel network interfaces up during initialization
- because it is affected by their state. Forcing them down prevents packets
- reception.
+Finally per-flow statistics can by queried using ``rte_flow_query`` when attaching a count action for specific flow. The flow counter counts the number of packets received successfully by the port and match the specific flow.
-- **ethtool** operations on related kernel interfaces also affect the PMD.
-Run as non-root
-^^^^^^^^^^^^^^^
+Compilation
+-----------
-In order to run as a non-root user,
-some capabilities must be granted to the application::
+See :ref:`mlx5 common compilation <mlx5_common_compilation>`.
- setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep <dpdk-app>
-Below are the reasons of the need for each capability:
+Configuration
+-------------
-``cap_sys_admin``
- When using physical addresses (PA mode), with Linux >= 4.0,
- for access to ``/proc/self/pagemap``.
+Environment Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~
-``cap_net_admin``
- For device configuration.
+See :ref:`mlx5 common configuration <mlx5_common_env>`.
-``cap_net_raw``
- For raw ethernet queue allocation through kernel driver.
+Firmware configuration
+~~~~~~~~~~~~~~~~~~~~~~
-``cap_ipc_lock``
- For DMA memory pinning.
+See :ref:`mlx5_firmware_config` guide.
Driver options
-^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~
+
+Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>`
+for an additional list of options shared with other mlx5 drivers.
- ``rxq_cqe_comp_en`` parameter [int]
- POWER8 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx,
ConnectX-6 Lx, BlueField and BlueField-2.
+- ``delay_drop`` parameter [int]
+
+ Bitmask value for the Rx queue delay drop attribute. Bit 0 is used for the
+ standard Rx queue and bit 1 is used for the hairpin Rx queue. By default, the
+ delay drop is disabled for all Rx queues. It will be ignored if the port does
+ not support the attribute even if it is enabled explicitly.
+
+ The packets being received will not be dropped immediately when the WQEs are
+ exhausted in a Rx queue with delay drop enabled.
+
+ A timeout value is set in the driver to control the waiting time before
+ dropping a packet. Once the timer is expired, the delay drop will be
+ deactivated for all the Rx queues with this feature enable. To re-activate
+ it, a rearming is needed and it is part of the kernel driver starting from
+ OFED 5.5.
+
+ To enable / disable the delay drop rearming, the private flag ``dropless_rq``
+ can be set and queried via ethtool:
+
+ - ethtool --set-priv-flags <netdev> dropless_rq on (/ off)
+ - ethtool --show-priv-flags <netdev>
+
+ The configuration flag is global per PF and can only be set on the PF, once
+ it is on, all the VFs', SFs' and representors' Rx queues will share the timer
+ and rearming.
+
- ``mprq_en`` parameter [int]
A nonzero value enables configuring Multi-Packet Rx queues. Rx queue is
and each stride receives one packet. MPRQ can improve throughput for
small-packet traffic.
- When MPRQ is enabled, max_rx_pkt_len can be larger than the size of
- user-provided mbuf even if DEV_RX_OFFLOAD_SCATTER isn't enabled. PMD will
- configure large stride size enough to accommodate max_rx_pkt_len as long as
+ When MPRQ is enabled, MTU can be larger than the size of
+ user-provided mbuf even if RTE_ETH_RX_OFFLOAD_SCATTER isn't enabled. PMD will
+ configure large stride size enough to accommodate MTU as long as
device allows. Note that this can waste system memory compared to enabling Rx
scatter and multi-segment packet.
the mbuf by external buffer attachment - ``rte_pktmbuf_attach_extbuf()``.
A mempool for external buffers will be allocated and managed by PMD. If Rx
packet is externally attached, ol_flags field of the mbuf will have
- EXT_ATTACHED_MBUF and this flag must be preserved. ``RTE_MBUF_HAS_EXTBUF()``
+ RTE_MBUF_F_EXTERNAL and this flag must be preserved. ``RTE_MBUF_HAS_EXTBUF()``
checks the flag. The default value is 128, valid only if ``mprq_en`` is set.
- ``rxqs_min_mprq`` parameter [int]
- ``tx_db_nc`` parameter [int]
- The rdma core library can map doorbell register in two ways, depending on the
- environment variable "MLX5_SHUT_UP_BF":
-
- - As regular cached memory (usually with write combining attribute), if the
- variable is either missing or set to zero.
- - As non-cached memory, if the variable is present and set to not "0" value.
-
- The type of mapping may slightly affect the Tx performance, the optimal choice
- is strongly relied on the host architecture and should be deduced practically.
-
- If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular
- memory (with write combining), the PMD will perform the extra write memory barrier
- after writing to doorbell, it might increase the needed CPU clocks per packet
- to send, but latency might be improved.
-
- If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non
- cached memory, the PMD will not perform the extra write memory barrier
- after writing to doorbell, on some architectures it might improve the
- performance.
-
- If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular
- memory, the PMD will use heuristics to decide whether write memory barrier
- should be performed. For bursts with size multiple of recommended one (64 pkts)
- it is supposed the next burst is coming and no need to issue the extra memory
- barrier (it is supposed to be issued in the next coming burst, at least after
- descriptor writing). It might increase latency (on some hosts till next
- packets transmit) and should be used with care.
-
- If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment
- variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF",
- the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others.
+ This parameter name is deprecated and ignored.
+ The new name for this parameter is ``sq_db_nc``.
+ See :ref:`common driver options <mlx5_common_driver_options>`.
- ``tx_pp`` parameter [int]
By default (if the ``tx_pp`` is not specified) send scheduling on timestamps
feature is disabled.
+ Starting with ConnectX-7 the capability to schedule traffic directly
+ on timestamp specified in descriptor is provided,
+ no extra objects are needed anymore and scheduling capability
+ is advertised and handled regardless ``tx_pp`` parameter presence.
+
- ``tx_skew`` parameter [int]
The parameter adjusts the send packet scheduling on timestamps and represents
For the MARK action the last 16 values in the full range are reserved for
internal PMD purposes (to emulate FLAG action). The valid range for the
- MARK action values is 0-0xFFEF for the 16-bit mode and 0-xFFFFEF
+ MARK action values is 0-0xFFEF for the 16-bit mode and 0-0xFFFFEF
for the 24-bit mode, the flows with the MARK action value outside
the specified range will be rejected.
- ``dv_flow_en`` parameter [int]
- A nonzero value enables the DV flow steering assuming it is supported
- by the driver (RDMA Core library version is rdma-core-24.0 or higher).
+ Value 0 means legacy Verbs flow offloading.
- Enabled by default if supported.
+ Value 1 enables the DV flow steering assuming it is supported by the
+ driver (requires rdma-core 24 or higher).
+
+ Value 2 enables the WQE based hardware steering.
+ In this mode, only queue-based flow management is supported.
+
+ It is configured by default to 1 (DV flow steering) if supported.
+ Otherwise, the value is 0 which indicates legacy Verbs flow offloading.
- ``dv_esw_en`` parameter [int]
Disabled by default (set to 0).
-- ``mr_ext_memseg_en`` parameter [int]
-
- A nonzero value enables extending memseg when registering DMA memory. If
- enabled, the number of entries in MR (Memory Region) lookup table on datapath
- is minimized and it benefits performance. On the other hand, it worsens memory
- utilization because registered memory is pinned by kernel driver. Even if a
- page in the extended chunk is freed, that doesn't become reusable until the
- entire memory is freed.
-
- Enabled by default.
-
- ``representor`` parameter [list]
This parameter can be used to instantiate DPDK Ethernet devices from
By default, the PMD will set this value to 0.
-- ``sys_mem_en`` parameter [int]
-
- A non-zero value enables the PMD memory management allocating memory
- from system by default, without explicit rte memory flag.
-
- By default, the PMD will set this value to 0.
-
- ``decap_en`` parameter [int]
Some devices do not support FCS (frame checksum) scattering for
By default, the PMD will set this value to 1.
-.. _mlx5_firmware_config:
-
-Firmware configuration
-~~~~~~~~~~~~~~~~~~~~~~
-
-Firmware features can be configured as key/value pairs.
-
-The command to set a value is::
-
- mlxconfig -d <device> set <key>=<value>
-
-The command to query a value is::
-
- mlxconfig -d <device> query | grep <key>
-
-The device name for the command ``mlxconfig`` can be either the PCI address,
-or the mst device name found with::
-
- mst status
-
-Below are some firmware configurations listed.
-
-- link type::
-
- LINK_TYPE_P1
- LINK_TYPE_P2
- value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense)
-
-- enable SR-IOV::
-
- SRIOV_EN=1
-
-- maximum number of SR-IOV virtual functions::
-
- NUM_OF_VFS=<max>
-
-- enable DevX (required by Direct Rules and other features)::
-
- UCTX_EN=1
-
-- aggressive CQE zipping::
-
- CQE_COMPRESSION=1
-
-- L3 VXLAN and VXLAN-GPE destination UDP port::
-
- IP_OVER_VXLAN_EN=1
- IP_OVER_VXLAN_PORT=<udp dport>
-
-- enable VXLAN-GPE tunnel flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=0
- or
- FLEX_PARSER_PROFILE_ENABLE=2
-
-- enable IP-in-IP tunnel flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=0
-
-- enable MPLS flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=1
-
-- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching::
-
- FLEX_PARSER_PROFILE_ENABLE=2
-
-- enable Geneve flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=0
- or
- FLEX_PARSER_PROFILE_ENABLE=1
-
-- enable Geneve TLV option flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=0
-
-- enable GTP flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=3
-
-- enable eCPRI flow matching::
-
- FLEX_PARSER_PROFILE_ENABLE=4
- PROG_PARSE_GRAPH=1
-
-Linux Prerequisites
--------------------
-
-This driver relies on external libraries and kernel drivers for resources
-allocations and initialization. The following dependencies are not part of
-DPDK and must be installed separately:
-
-- **libibverbs**
-
- User space Verbs framework used by librte_net_mlx5. This library provides
- a generic interface between the kernel and low-level user space drivers
- such as libmlx5.
-
- It allows slow and privileged operations (context initialization, hardware
- resources allocations) to be managed by the kernel and fast operations to
- never leave user space.
-
-- **libmlx5**
-
- Low-level user space driver library for Mellanox
- ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically loaded
- by libibverbs.
-
- This library basically implements send/receive calls to the hardware
- queues.
-
-- **Kernel modules**
-
- They provide the kernel-side Verbs API and low level device drivers that
- manage actual hardware initialization and resources sharing with user
- space processes.
-
- Unlike most other PMDs, these modules must remain loaded and bound to
- their devices:
-
- - mlx5_core: hardware driver managing Mellanox
- ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Ethernet kernel
- network devices.
- - mlx5_ib: InifiniBand device driver.
- - ib_uverbs: user space driver for Verbs (entry point for libibverbs).
-
-- **Firmware update**
-
- Mellanox OFED/EN releases include firmware updates for
- ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters.
-
- Because each release provides new features, these updates must be applied to
- match the kernel modules and libraries they come with.
-
-.. note::
-
- Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
- licensed.
-
-Installation
-~~~~~~~~~~~~
-
-Either RDMA Core library with a recent enough Linux kernel release
-(recommended) or Mellanox OFED/EN, which provides compatibility with older
-releases.
-
-RDMA Core with Linux Kernel
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-- Minimal kernel version : v4.14 or the most recent 4.14-rc (see `Linux installation documentation`_)
-- Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull request #227 from yishaih/tm")
- (see `RDMA Core installation documentation`_)
-- When building for i686 use:
-
- - rdma-core version 18.0 or above built with 32bit support.
- - Kernel version 4.14.41 or above.
-
-- Starting with rdma-core v21, static libraries can be built::
-
- cd build
- CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
- ninja
-
-.. _`Linux installation documentation`: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/Documentation/admin-guide/README.rst
-.. _`RDMA Core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md
-
-
-Mellanox OFED/EN
-^^^^^^^^^^^^^^^^
-
-- Mellanox OFED version: **4.5** and above /
- Mellanox EN version: **4.5** and above
-- firmware version:
-
- - ConnectX-4: **12.21.1000** and above.
- - ConnectX-4 Lx: **14.21.1000** and above.
- - ConnectX-5: **16.21.1000** and above.
- - ConnectX-5 Ex: **16.21.1000** and above.
- - ConnectX-6: **20.27.0090** and above.
- - ConnectX-6 Dx: **22.27.0090** and above.
- - BlueField: **18.25.1010** and above.
-
-While these libraries and kernel modules are available on OpenFabrics
-Alliance's `website <https://www.openfabrics.org/>`__ and provided by package
-managers on most distributions, this PMD requires Ethernet extensions that
-may not be supported at the moment (this is a work in progress).
-
-`Mellanox OFED
-<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and
-`Mellanox EN
-<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__
-include the necessary support and should be used in the meantime. For DPDK,
-only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are
-required from that distribution.
-
-.. note::
-
- Several versions of Mellanox OFED/EN are available. Installing the version
- this DPDK release was developed and tested against is strongly
- recommended. Please check the `linux prerequisites`_.
-
-Windows Prerequisites
----------------------
-
-This driver relies on external libraries and kernel drivers for resources
-allocations and initialization. The dependencies in the following sub-sections
-are not part of DPDK, and must be installed separately.
-
-Compilation Prerequisites
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DevX SDK installation
-^^^^^^^^^^^^^^^^^^^^^
-
-The DevX SDK must be installed on the machine building the Windows PMD.
-Additional information can be found at
-`How to Integrate Windows DevX in Your Development Environment
-<https://docs.mellanox.com/display/winof2v250/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`__.
-
-Runtime Prerequisites
-~~~~~~~~~~~~~~~~~~~~~
-
-WinOF2 version 2.60 or higher must be installed on the machine.
-
-WinOF2 installation
-^^^^^^^^^^^^^^^^^^^
-
-The driver can be downloaded from the following site:
-`WINOF2
-<https://www.mellanox.com/products/adapter-software/ethernet/windows/winof-2>`__
-
-DevX Enablement
-^^^^^^^^^^^^^^^
-
-DevX for Windows must be enabled in the Windows registry.
-The keys ``DevxEnabled`` and ``DevxFsRules`` must be set.
-Additional information can be found in the WinOF2 user manual.
Supported NICs
--------------
* Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G)
* Mellanox\ |reg| ConnectX\ |reg|-6 Lx EN 25G MCX631102AN-ADAT (2x25G)
-Quick Start Guide on OFED/EN
-----------------------------
-
-1. Download latest Mellanox OFED/EN. For more info check the `linux prerequisites`_.
-
-
-2. Install the required libraries and kernel modules either by installing
- only the required set, or by installing the entire Mellanox OFED/EN::
-
- ./mlnxofedinstall --upstream-libs --dpdk
-
-3. Verify the firmware is the correct one::
-
- ibv_devinfo
-
-4. Verify all ports links are set to Ethernet::
-
- mlxconfig -d <mst device> query | grep LINK_TYPE
- LINK_TYPE_P1 ETH(2)
- LINK_TYPE_P2 ETH(2)
-
- Link types may have to be configured to Ethernet::
-
- mlxconfig -d <mst device> set LINK_TYPE_P1/2=1/2/3
-
- * LINK_TYPE_P1=<1|2|3> , 1=Infiniband 2=Ethernet 3=VPI(auto-sense)
-
- For hypervisors, verify SR-IOV is enabled on the NIC::
-
- mlxconfig -d <mst device> query | grep SRIOV_EN
- SRIOV_EN True(1)
-
- If needed, configure SR-IOV::
-
- mlxconfig -d <mst device> set SRIOV_EN=1 NUM_OF_VFS=16
- mlxfwreset -d <mst device> reset
-
-5. Restart the driver::
-
- /etc/init.d/openibd restart
-
- or::
-
- service openibd restart
-
- If link type was changed, firmware must be reset as well::
-
- mlxfwreset -d <mst device> reset
-
- For hypervisors, after reset write the sysfs number of virtual functions
- needed for the PF.
-
- To dynamically instantiate a given number of virtual functions (VFs)::
-
- echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs
-
-6. Install DPDK and you are ready to go.
- See :doc:`compilation instructions <../linux_gsg/build_dpdk>`.
-
-Enable switchdev mode
----------------------
-
-Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF.
-Representor is a port in DPDK that is connected to a VF or SF in such a way
-that assuming there are no offload flows, each packet that is sent from the VF or SF
-will be received by the corresponding representor. While each packet that is or SF
-sent to a representor will be received by the VF or SF.
-This is very useful in case of SRIOV mode, where the first packet that is sent
-by the VF or SF will be received by the DPDK application which will decide if this
-flow should be offloaded to the E-Switch. After offloading the flow packet
-that the VF or SF that are matching the flow will not be received any more by
-the DPDK application.
-1. Enable SRIOV mode::
+Sub-Function
+------------
- mlxconfig -d <mst device> set SRIOV_EN=true
-
-2. Configure the max number of VFs::
-
- mlxconfig -d <mst device> set NUM_OF_VFS=<num of vfs>
-
-3. Reset the FW::
-
- mlxfwreset -d <mst device> reset
-
-3. Configure the actual number of VFs::
-
- echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs
-
-4. Unbind the device (can be rebind after the switchdev mode)::
-
- echo -n "<device pci address" > /sys/bus/pci/drivers/mlx5_core/unbind
-
-5. Enbale switchdev mode::
-
- echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
-
-Sub-Function support
---------------------
-
-Sub-Function is a portion of the PCI device, a SF netdev has its own
-dedicated queues (txq, rxq).
-A SF shares PCI level resources with other SFs and/or with its parent PCI function.
-
-0. Requirement::
-
- OFED version >= 5.4-0.3.3.0
-
-1. Configure SF feature::
-
- # Run mlxconfig on both PFs on host and ECPFs on BlueField.
- mlxconfig -d <mst device> set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12
-
-2. Enable switchdev mode::
-
- mlxdevm dev eswitch set pci/<DBDF> mode switchdev
-
-3. Add SF port::
-
- mlxdevm port add pci/<DBDF> flavour pcisf pfnum 0 sfnum <sfnum>
-
- Get SFID from output: pci/<DBDF>/<SFID>
-
-4. Modify MAC address::
-
- mlxdevm port function set pci/<DBDF>/<SFID> hw_addr <MAC>
-
-5. Activate SF port::
-
- mlxdevm port function set pci/<DBDF>/<ID> state active
-
-6. Devargs to probe SF device::
-
- auxiliary:mlx5_core.sf.<num>,dv_flow_en=1
+See :ref:`mlx5_sub_function`.
Sub-Function representor support
---------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A SF netdev supports E-Switch representation offload
similar to PF and VF representors.
Use <sfnum> to probe SF representor::
- testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1
+ testpmd> port attach <PCI_BDF>,representor=sf<sfnum>,dv_flow_en=1
+
Performance tuning
------------------
| | | rdma-core 26 | | rdma-core 26 |
| | | ConnectX-5 | | ConnectX-5 |
+-----------------------+-----------------+-----------------+
+ | ASO Metering | | DPDK 21.05 | | DPDK 21.05 |
+ | | | OFED 5.3 | | OFED 5.3 |
+ | | | rdma-core 33 | | rdma-core 33 |
+ | | | ConnectX-6 Dx| | ConnectX-6 Dx |
+ +-----------------------+-----------------+-----------------+
+ | Metering Hierarchy | | DPDK 21.08 | | DPDK 21.08 |
+ | | | OFED 5.3 | | OFED 5.3 |
+ | | | N/A | | N/A |
+ | | | ConnectX-6 Dx| | ConnectX-6 Dx |
+ +-----------------------+-----------------+-----------------+
| Sampling | | DPDK 20.11 | | DPDK 20.11 |
| | | OFED 5.1-2 | | OFED 5.1-2 |
| | | rdma-core 32 | | N/A |
add port meter policy 0 2 g_actions meter mtr_id M / end y_actions end r_actions drop / end
create port meter 0 N 2 2 yes 0xffff 1 0
flow create 0 ingress group 1 pattern eth / end actions meter mtr_id N / end
+
+How to configure a VF as trusted
+--------------------------------
+
+This section demonstrates how to configure a virtual function (VF) interface as trusted.
+Trusted VF is needed to offload rules with rte_flow to a group that is bigger than 0.
+The configuration is done in two parts: driver and FW.
+
+The procedure below is an example of using a ConnectX-5 adapter card (pf0) with 2 VFs:
+
+#. Create 2 VFs on the PF pf0 when in Legacy SR-IOV mode::
+
+ $ echo 2 > /sys/class/net/pf0/device/mlx5_num_vfs
+
+#. Verify the VFs are created:
+
+ .. code-block:: console
+
+ $ lspci | grep Mellanox
+ 82:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
+ 82:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
+ 82:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
+ 82:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
+
+#. Unbind all VFs. For each VF PCIe, using the following command to unbind the driver::
+
+ $ echo "0000:82:00.2" >> /sys/bus/pci/drivers/mlx5_core/unbind
+
+#. Set the VFs to be trusted for the kernel by using one of the methods below:
+
+ - Using sysfs file::
+
+ $ echo ON | tee /sys/class/net/pf0/device/sriov/0/trust
+ $ echo ON | tee /sys/class/net/pf0/device/sriov/1/trust
+
+ - Using “ip link” command::
+
+ $ ip link set p0 vf 0 trust on
+ $ ip link set p0 vf 1 trust on
+
+#. Configure all VFs using mlxreg::
+
+ $ mlxreg -d /dev/mst/mt4121_pciconf0 --reg_name VHCA_TRUST_LEVEL --yes --set "all_vhca=0x1,trust_level=0x1"
+
+ .. note::
+
+ Firmware version used must be >= xx.29.1016 and MFT >= 4.18
+
+#. For each VF PCIe, using the following command to bind the driver::
+
+ $ echo "0000:82:00.2" >> /sys/bus/pci/drivers/mlx5_core/bind
git.droids-corp.org / dpdk.git / blobdiff