The MLX5 poll mode driver library (**librte_net_mlx5**) provides support
for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox
-ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx** and
-**Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters
-as well as their virtual functions (VF) in SR-IOV context.
+ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox
+ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families
+of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF)
+in SR-IOV context.
Information and documentation about these adapters can be found on the
`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the
- Multi arch support: x86_64, POWER8, ARMv8, i686.
- Multiple TX and RX queues.
-- Support for scattered TX and RX frames.
+- Support for scattered TX frames.
+- Advanced support for scattered Rx frames with tunable buffer attributes.
- IPv4, IPv6, TCPv4, TCPv6, UDPv4 and UDPv6 RSS on any number of queues.
- RSS using different combinations of fields: L3 only, L4 only or both,
and source only, destination only or both.
- Per packet no-inline hint flag to disable packet data copying into Tx descriptors.
- Hardware LRO.
- Hairpin.
+- Multiple-thread flow insertion.
+- Matching on GTP extension header with raw encap/decap action.
+- Matching on Geneve TLV option header with raw encap/decap action.
+- RSS support in sample action.
+- E-Switch mirroring and jump.
+- E-Switch mirroring and modify.
+- 21844 flow priorities for ingress or egress flow groups greater than 0 and for any transfer
+ flow group.
Limitations
-----------
+- Windows support:
+
+ On Windows, the features are limited:
+
+ - Promiscuous mode is not supported
+ - The following rules are supported:
+
+ - IPv4/UDP with CVLAN filtering
+ - Unicast MAC filtering
+
- For secondary process:
- Forked secondary process not supported.
Will match any ipv4 packet (VLAN included).
-- When using DV flow engine (``dv_flow_en`` = 1), flow pattern without VLAN item
- will match untagged packets only.
+- When using Verbs flow engine (``dv_flow_en`` = 0), multi-tagged(QinQ) match is not supported.
+
+- When using DV flow engine (``dv_flow_en`` = 1), flow pattern with any VLAN specification will match only single-tagged packets unless the ETH item ``type`` field is 0x88A8 or the VLAN item ``has_more_vlan`` field is 1.
The flow rule::
flow create 0 ingress pattern eth / ipv4 / end ...
- Will match untagged packets only.
- The flow rule::
+ Will match any ipv4 packet.
+ The flow rules::
- flow create 0 ingress pattern eth / vlan / ipv4 / end ...
+ flow create 0 ingress pattern eth / vlan / end ...
+ flow create 0 ingress pattern eth has_vlan is 1 / end ...
+ flow create 0 ingress pattern eth type is 0x8100 / end ...
- Will match tagged packets only, with any VLAN ID value.
- The flow rule::
+ Will match single-tagged packets only, with any VLAN ID value.
+ The flow rules::
- flow create 0 ingress pattern eth / vlan vid is 3 / ipv4 / end ...
+ flow create 0 ingress pattern eth type is 0x88A8 / end ...
+ flow create 0 ingress pattern eth / vlan has_more_vlan is 1 / end ...
+
+ Will match multi-tagged packets only, with any VLAN ID value.
- Will only match tagged packets with VLAN ID 3.
+- A flow pattern with 2 sequential VLAN items is not supported.
- VLAN pop offload command:
- OAM
- protocol type
- options length
- Currently, the only supported options length value is 0.
+
+- Match on Geneve TLV option is supported on the following fields:
+
+ - Class
+ - Type
+ - Length
+ - Data
+
+ Only one Class/Type/Length Geneve TLV option is supported per shared device.
+ Class/Type/Length fields must be specified as well as masks.
+ Class/Type/Length specified masks must be full.
+ Matching Geneve TLV option without specifying data is not supported.
+ Matching Geneve TLV option with ``data & mask == 0`` is not supported.
- VF: flow rules created on VF devices can only match traffic targeted at the
configured MAC addresses (see ``rte_eth_dev_mac_addr_add()``).
- msg_type
- teid
+- Match on GTP extension header only for GTP PDU session container (next
+ extension header type = 0x85).
+- Match on GTP extension header is not supported in group 0.
+
- No Tx metadata go to the E-Switch steering domain for the Flow group 0.
The flows within group 0 and set metadata action are rejected by hardware.
the device. In case of ungraceful program termination, some entries may
remain present and should be removed manually by other means.
+- Buffer split offload is supported with regular Rx burst routine only,
+ no MPRQ feature or vectorized code can be engaged.
+
- When Multi-Packet Rx queue is configured (``mprq_en``), a Rx packet can be
externally attached to a user-provided mbuf with having EXT_ATTACHED_MBUF in
ol_flags. As the mempool for the external buffer is managed by PMD, all the
- CRC:
- ``DEV_RX_OFFLOAD_KEEP_CRC`` cannot be supported with decapsulation
- for some NICs (such as ConnectX-6 Dx and BlueField 2).
+ for some NICs (such as ConnectX-6 Dx, ConnectX-6 Lx, and BlueField-2).
The capability bit ``scatter_fcs_w_decap_disable`` shows NIC support.
- Sample flow:
The last extension header item 'next header' field can specify the following
header protocol type.
+- Hairpin:
+
+ - Hairpin between two ports could only manual binding and explicit Tx flow mode. For single port hairpin, all the combinations of auto/manual binding and explicit/implicit Tx flow mode could be supported.
+ - Hairpin in switchdev SR-IOV mode is not supported till now.
+
Statistics
----------
A nonzero value enables the compression of CQE on RX side. This feature
allows to save PCI bandwidth and improve performance. Enabled by default.
+ Different compression formats are supported in order to achieve the best
+ performance for different traffic patterns. Hash RSS format is the default.
- Supported on:
-
- - x86_64 with ConnectX-4, ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx
- and BlueField.
- - POWER9 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx
- and BlueField.
-
-- ``rxq_cqe_pad_en`` parameter [int]
-
- A nonzero value enables 128B padding of CQE on RX side. The size of CQE
- is aligned with the size of a cacheline of the core. If cacheline size is
- 128B, the CQE size is configured to be 128B even though the device writes
- only 64B data on the cacheline. This is to avoid unnecessary cache
- invalidation by device's two consecutive writes on to one cacheline.
- However in some architecture, it is more beneficial to update entire
- cacheline with padding the rest 64B rather than striding because
- read-modify-write could drop performance a lot. On the other hand,
- writing extra data will consume more PCIe bandwidth and could also drop
- the maximum throughput. It is recommended to empirically set this
- parameter. Disabled by default.
+ Specifying 2 as a ``rxq_cqe_comp_en`` value selects Flow Tag format for
+ better compression rate in case of RTE Flow Mark traffic.
+ Specifying 3 as a ``rxq_cqe_comp_en`` value selects Checksum format.
+ Specifying 4 as a ``rxq_cqe_comp_en`` value selects L3/L4 Header format for
+ better compression rate in case of mixed TCP/UDP and IPv4/IPv6 traffic.
Supported on:
- - CPU having 128B cacheline with ConnectX-5 and BlueField.
+ - x86_64 with ConnectX-4, ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx,
+ ConnectX-6 Lx, BlueField and BlueField-2.
+ - POWER9 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx,
+ ConnectX-6 Lx, BlueField and BlueField-2.
- ``rxq_pkt_pad_en`` parameter [int]
Supported on:
- - x86_64 with ConnectX-4, ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx
- and BlueField.
- - POWER8 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx
- and BlueField.
+ - x86_64 with ConnectX-4, ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx,
+ ConnectX-6 Lx, BlueField and BlueField-2.
+ - POWER8 and ARMv8 with ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx,
+ ConnectX-6 Lx, BlueField and BlueField-2.
- ``mprq_en`` parameter [int]
- ``txq_mpw_en`` parameter [int]
A nonzero value enables Enhanced Multi-Packet Write (eMPW) for ConnectX-5,
- ConnectX-6, ConnectX-6 Dx and BlueField. eMPW allows the TX burst function to pack
- up multiple packets in a single descriptor session in order to save PCI bandwidth
- and improve performance at the cost of a slightly higher CPU usage. When
- ``txq_inline_mpw`` is set along with ``txq_mpw_en``, TX burst function copies
- entire packet data on to TX descriptor instead of including pointer of packet.
+ ConnectX-6, ConnectX-6 Dx, ConnectX-6 Lx, BlueField, BlueField-2.
+ eMPW allows the Tx burst function to pack up multiple packets
+ in a single descriptor session in order to save PCI bandwidth
+ and improve performance at the cost of a slightly higher CPU usage.
+ When ``txq_inline_mpw`` is set along with ``txq_mpw_en``,
+ Tx burst function copies entire packet data on to Tx descriptor
+ instead of including pointer of packet.
The Enhanced Multi-Packet Write feature is enabled by default if NIC supports
it, can be disabled by explicit specifying 0 value for ``txq_mpw_en`` option.
- ``tx_vec_en`` parameter [int]
- A nonzero value enables Tx vector on ConnectX-5, ConnectX-6, ConnectX-6 Dx
- and BlueField NICs if the number of global Tx queues on the port is less than
- ``txqs_max_vec``. The parameter is deprecated and ignored.
+ A nonzero value enables Tx vector on ConnectX-5, ConnectX-6, ConnectX-6 Dx,
+ ConnectX-6 Lx, BlueField and BlueField-2 NICs
+ if the number of global Tx queues on the port is less than ``txqs_max_vec``.
+ The parameter is deprecated and ignored.
- ``rx_vec_en`` parameter [int]
or
FLEX_PARSER_PROFILE_ENABLE=1
+- enable Geneve TLV option flow matching::
+
+ FLEX_PARSER_PROFILE_ENABLE=0
+
- enable GTP flow matching::
FLEX_PARSER_PROFILE_ENABLE=3
FLEX_PARSER_PROFILE_ENABLE=4
PROG_PARSE_GRAPH=1
-Prerequisites
--------------
+Linux Prerequisites
+-------------------
This driver relies on external libraries and kernel drivers for resources
allocations and initialization. The following dependencies are not part of
Several versions of Mellanox OFED/EN are available. Installing the version
this DPDK release was developed and tested against is strongly
- recommended. Please check the `prerequisites`_.
+ recommended. Please check the `linux prerequisites`_.
+
+Windows Prerequisites
+---------------------
+
+This driver relies on external libraries and kernel drivers for resources
+allocations and initialization. The dependencies in the following sub-sections
+are not part of DPDK, and must be installed separately.
+
+Compilation Prerequisites
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DevX SDK installation
+^^^^^^^^^^^^^^^^^^^^^
+
+The DevX SDK must be installed on the machine building the Windows PMD.
+Additional information can be found at
+`How to Integrate Windows DevX in Your Development Environment
+<https://docs.mellanox.com/display/winof2v250/RShim+Drivers+and+Usage#RShimDriversandUsage-DevXInterface>`__.
+
+Runtime Prerequisites
+~~~~~~~~~~~~~~~~~~~~~
+
+WinOF2 version 2.60 or higher must be installed on the machine.
+
+WinOF2 installation
+^^^^^^^^^^^^^^^^^^^
+
+The driver can be downloaded from the following site:
+`WINOF2
+<https://www.mellanox.com/products/adapter-software/ethernet/windows/winof-2>`__
+
+DevX Enablement
+^^^^^^^^^^^^^^^
+
+DevX for Windows must be enabled in the Windows registry.
+The keys ``DevxEnabled`` and ``DevxFsRules`` must be set.
+Additional information can be found in the WinOF2 user manual.
Supported NICs
--------------
- ConnectX-5 Ex
- ConnectX-6
- ConnectX-6 Dx
+ - ConnectX-6 Lx
- BlueField
+ - BlueField-2
Below are detailed device names:
* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
* Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 100G MCX623106AN-CDAT (2x100G)
* Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G)
+* Mellanox\ |reg| ConnectX\ |reg|-6 Lx EN 25G MCX631102AN-ADAT (2x25G)
Quick Start Guide on OFED/EN
----------------------------
-1. Download latest Mellanox OFED/EN. For more info check the `prerequisites`_.
+1. Download latest Mellanox OFED/EN. For more info check the `linux prerequisites`_.
2. Install the required libraries and kernel modules either by installing
- Configure per-lcore cache when creating Mempools for packet buffer.
- Refrain from dynamically allocating/freeing memory in run-time.
+Rx burst functions
+------------------
+
+There are multiple Rx burst functions with different advantages and limitations.
+
+.. table:: Rx burst functions
+
+ +-------------------+------------------------+---------+-----------------+------+-------+
+ || Function Name || Enabler || Scatter|| Error Recovery || CQE || Large|
+ | | | | || comp|| MTU |
+ +===================+========================+=========+=================+======+=======+
+ | rx_burst | rx_vec_en=0 | Yes | Yes | Yes | Yes |
+ +-------------------+------------------------+---------+-----------------+------+-------+
+ | rx_burst_vec | rx_vec_en=1 (default) | No | if CQE comp off | Yes | No |
+ +-------------------+------------------------+---------+-----------------+------+-------+
+ | rx_burst_mprq || mprq_en=1 | No | Yes | Yes | Yes |
+ | || RxQs >= rxqs_min_mprq | | | | |
+ +-------------------+------------------------+---------+-----------------+------+-------+
+ | rx_burst_mprq_vec || rx_vec_en=1 (default) | No | if CQE comp off | Yes | Yes |
+ | || mprq_en=1 | | | | |
+ | || RxQs >= rxqs_min_mprq | | | | |
+ +-------------------+------------------------+---------+-----------------+------+-------+
+
.. _mlx5_offloads_support:
Supported hardware offloads
.. table:: Minimal SW/HW versions for queue offloads
- ============== ===== ===== ========= ===== ========== ==========
+ ============== ===== ===== ========= ===== ========== =============
Offload DPDK Linux rdma-core OFED firmware hardware
- ============== ===== ===== ========= ===== ========== ==========
+ ============== ===== ===== ========= ===== ========== =============
common base 17.11 4.14 16 4.2-1 12.21.1000 ConnectX-4
checksums 17.11 4.14 16 4.2-1 12.21.1000 ConnectX-4
Rx timestamp 17.11 4.14 16 4.2-1 12.21.1000 ConnectX-4
TSO 17.11 4.14 16 4.2-1 12.21.1000 ConnectX-4
LRO 19.08 N/A N/A 4.6-4 16.25.6406 ConnectX-5
- ============== ===== ===== ========= ===== ========== ==========
+ Buffer Split 20.11 N/A N/A 5.1-2 22.28.2006 ConnectX-6 Dx
+ ============== ===== ===== ========= ===== ========== =============
.. table:: Minimal SW/HW versions for rte_flow offloads
| | | | | rdma-core 23 |
| | | | | ConnectX-4 |
+-----------------------+-----------------+-----------------+
+ | RSS shared action | | | | DPDK 20.11 |
+ | | | N/A | | OFED 5.2 |
+ | | | | | rdma-core 33 |
+ | | | | | ConnectX-5 |
+ +-----------------------+-----------------+-----------------+
+ | | VLAN | | DPDK 19.11 | | DPDK 19.11 |
+ | | (of_pop_vlan / | | OFED 4.7-1 | | OFED 4.7-1 |
+ | | of_push_vlan / | | ConnectX-5 | | ConnectX-5 |
+ | | of_set_vlan_pcp / | | | | |
+ | | of_set_vlan_vid) | | | | |
+ +-----------------------+-----------------+-----------------+
| Encapsulation | | DPDK 19.05 | | DPDK 19.02 |
| (VXLAN / NVGRE / RAW) | | OFED 4.7-1 | | OFED 4.6 |
| | | rdma-core 24 | | rdma-core 23 |
| | | rdma-core 27 | | rdma-core 27 |
| | | ConnectX-5 | | ConnectX-5 |
+-----------------------+-----------------+-----------------+
+ | Tunnel Offload | | DPDK 20.11 | | DPDK 20.11 |
+ | | | OFED 5.1-2 | | OFED 5.1-2 |
+ | | | rdma-core 32 | | N/A |
+ | | | ConnectX-5 | | ConnectX-5 |
+ +-----------------------+-----------------+-----------------+
| | Header rewrite | | DPDK 19.05 | | DPDK 19.02 |
| | (set_ipv4_src / | | OFED 4.7-1 | | OFED 4.7-1 |
| | set_ipv4_dst / | | rdma-core 24 | | rdma-core 24 |
| | | rdma-core 24 | | rdma-core 23 |
| | | ConnectX-5 | | ConnectX-4 |
+-----------------------+-----------------+-----------------+
+ | Meta data | | DPDK 19.11 | | DPDK 19.11 |
+ | | | OFED 4.7-3 | | OFED 4.7-3 |
+ | | | rdma-core 26 | | rdma-core 26 |
+ | | | ConnectX-5 | | ConnectX-5 |
+ +-----------------------+-----------------+-----------------+
| Port ID | | DPDK 19.05 | | N/A |
| | | OFED 4.7-1 | | N/A |
| | | rdma-core 24 | | N/A |
| | | ConnectX-5 | | N/A |
+-----------------------+-----------------+-----------------+
- | | VLAN | | DPDK 19.11 | | DPDK 19.11 |
- | | (of_pop_vlan / | | OFED 4.7-1 | | OFED 4.7-1 |
- | | of_push_vlan / | | ConnectX-5 | | ConnectX-5 |
- | | of_set_vlan_pcp / | | | | |
- | | of_set_vlan_vid) | | | | |
- +-----------------------+-----------------+-----------------+
| Hairpin | | | | DPDK 19.11 |
| | | N/A | | OFED 4.7-3 |
| | | | | rdma-core 26 |
| | | | | ConnectX-5 |
+-----------------------+-----------------+-----------------+
- | Meta data | | DPDK 19.11 | | DPDK 19.11 |
- | | | OFED 4.7-3 | | OFED 4.7-3 |
- | | | rdma-core 26 | | rdma-core 26 |
- | | | ConnectX-5 | | ConnectX-5 |
+ | 2-port Hairpin | | | | DPDK 20.11 |
+ | | | N/A | | OFED 5.1-2 |
+ | | | | | N/A |
+ | | | | | ConnectX-5 |
+-----------------------+-----------------+-----------------+
| Metering | | DPDK 19.11 | | DPDK 19.11 |
| | | OFED 4.7-3 | | OFED 4.7-3 |
| | | ConnectX-5 | | ConnectX-5 |
+-----------------------+-----------------+-----------------+
| Sampling | | DPDK 20.11 | | DPDK 20.11 |
+ | | | OFED 5.1-2 | | OFED 5.1-2 |
+ | | | rdma-core 32 | | N/A |
+ | | | ConnectX-5 | | ConnectX-5 |
+ +-----------------------+-----------------+-----------------+
+ | Age shared action | | DPDK 20.11 | | DPDK 20.11 |
| | | OFED 5.2 | | OFED 5.2 |
| | | rdma-core 32 | | rdma-core 32 |
- | | | ConnectX-5 | | ConnectX-5 |
+ | | | ConnectX-6 Dx| | ConnectX-6 Dx |
+ +-----------------------+-----------------+-----------------+
+ | Encapsulation | | DPDK 21.02 | | DPDK 21.02 |
+ | GTP PSC | | OFED 5.2 | | OFED 5.2 |
+ | | | rdma-core 35 | | rdma-core 35 |
+ | | | ConnectX-6 Dx| | ConnectX-6 Dx |
+ +-----------------------+-----------------+-----------------+
+ | Encapsulation | | DPDK 21.02 | | DPDK 21.02 |
+ | GENEVE TLV option | | OFED 5.2 | | OFED 5.2 |
+ | | | rdma-core 34 | | rdma-core 34 |
+ | | | ConnectX-6 Dx | | ConnectX-6 Dx |
+-----------------------+-----------------+-----------------+
Notes for metadata
eth32
eth33
-#. Optionally, retrieve their PCI bus addresses for whitelisting::
+#. Optionally, retrieve their PCI bus addresses for to be used with the allow list::
{
for intf in eth2 eth3 eth4 eth5;
(cd "/sys/class/net/${intf}/device/" && pwd -P);
done;
} |
- sed -n 's,.*/\(.*\),-w \1,p'
+ sed -n 's,.*/\(.*\),-a \1,p'
Example output::
- -w 0000:05:00.1
- -w 0000:06:00.0
- -w 0000:06:00.1
- -w 0000:05:00.0
+ -a 0000:05:00.1
+ -a 0000:06:00.0
+ -a 0000:06:00.1
+ -a 0000:05:00.0
#. Request huge pages::
#. Start testpmd with basic parameters::
- testpmd -l 8-15 -n 4 -w 05:00.0 -w 05:00.1 -w 06:00.0 -w 06:00.1 -- --rxq=2 --txq=2 -i
+ testpmd -l 8-15 -n 4 -a 05:00.0 -a 05:00.1 -a 06:00.0 -a 06:00.1 -- --rxq=2 --txq=2 -i
Example output::
git.droids-corp.org / dpdk.git / blobdiff