X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fnics%2Fenic.rst;h=91bdcd065a95a06e424aa5bf7d0e09e1f96a5153;hb=a86144cd9ded2db1b719c9464cff8091edff2474;hp=4505dcf4817d3e01e57c75f39bd0a9e5c7a8d44d;hpb=e53f18e81ccc039e296a3cda01a587b0be0b2e91;p=dpdk.git diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst index 4505dcf481..91bdcd065a 100644 --- a/doc/guides/nics/enic.rst +++ b/doc/guides/nics/enic.rst @@ -1,32 +1,7 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright (c) 2017, Cisco Systems, Inc. All rights reserved. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - 1. Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. - ENIC Poll Mode Driver ===================== @@ -39,19 +14,12 @@ How to obtain ENIC PMD integrated DPDK -------------------------------------- ENIC PMD support is integrated into the DPDK suite. dpdk-.tar.gz -should be downloaded from http://dpdk.org +should be downloaded from https://core.dpdk.org/download/ Configuration information ------------------------- -- **DPDK Configuration Parameters** - - The following configuration options are available for the ENIC PMD: - - - **CONFIG_RTE_LIBRTE_ENIC_PMD** (default y): Enables or disables inclusion - of the ENIC PMD driver in the DPDK compilation. - - **vNIC Configuration Parameters** - **Number of Queues** @@ -132,24 +100,6 @@ Configuration information TCP, IPv4, TCP-IPv4, IPv6, TCP-IPv6, IPv6 Extension, TCP-IPv6 Extension. -.. _enic-flow-director: - -Flow director support ---------------------- - -Advanced filtering support was added to 1300 series VIC firmware starting -with version 2.0.13 for C-series UCS servers and version 3.1.2 for UCSM -managed blade servers. In order to enable advanced filtering the 'Advanced -filter' radio button should be enabled via CIMC or UCSM followed by a reboot -of the server. - -With advanced filters, perfect matching of all fields of IPv4, IPv6 headers -as well as TCP, UDP and SCTP L4 headers is available through flow director. -Masking of these fields for partial match is also supported. - -Without advanced filter support, the flow director is limited to IPv4 -perfect filtering of the 5-tuple with no masking of fields supported. - SR-IOV mode utilization ----------------------- @@ -230,7 +180,7 @@ or ``vfio`` in non-IOMMU mode. In the VM, the kernel enic driver may be automatically bound to the VF during boot. Unbinding it currently hangs due to a known issue with the driver. To -work around the issue, blacklist the enic module as follows. +work around the issue, block the enic module as follows. Please see :ref:`Limitations ` for limitations in the use of SR-IOV. @@ -249,12 +199,18 @@ the use of SR-IOV. passthrough devices do not require libvirt, port profiles, and VM-FEX. -.. _enic-genic-flow-api: +.. _enic-generic-flow-api: Generic Flow API support ------------------------ -Generic Flow API is supported. The baseline support is: +Generic Flow API (also called "rte_flow" API) is supported. More advanced +capabilities are available when "Advanced Filtering" is enabled on the adapter. +Advanced filtering was added to 1300 series VIC firmware starting with version +2.0.13 for C-series UCS servers and version 3.1.2 for UCSM managed blade +servers. Advanced filtering is available on 1400 series adapters and beyond. +To enable advanced filtering, the 'Advanced filter' radio button should be +selected via CIMC or UCSM followed by a reboot of the server. - **1200 series VICs** @@ -272,7 +228,7 @@ Generic Flow API is supported. The baseline support is: in the pattern. - Attributes: ingress - - Items: eth, ipv4, ipv6, udp, tcp, vxlan, inner eth, ipv4, ipv6, udp, tcp + - Items: eth, vlan, ipv4, ipv6, udp, tcp, vxlan, inner eth, vlan, ipv4, ipv6, udp, tcp - Actions: queue and void - Selectors: 'is', 'spec' and 'mask'. 'last' is not supported - In total, up to 64 bytes of mask is allowed across all headers @@ -280,11 +236,26 @@ Generic Flow API is supported. The baseline support is: - **1300 and later series VICS with advanced filters enabled** - Attributes: ingress - - Items: eth, ipv4, ipv6, udp, tcp, vxlan, inner eth, ipv4, ipv6, udp, tcp - - Actions: queue, mark, drop, flag and void + - Items: eth, vlan, ipv4, ipv6, udp, tcp, vxlan, raw, inner eth, vlan, ipv4, ipv6, udp, tcp + - Actions: queue, mark, drop, flag, rss, passthru, and void - Selectors: 'is', 'spec' and 'mask'. 'last' is not supported - In total, up to 64 bytes of mask is allowed across all headers +- **1400 and later series VICs with Flow Manager API enabled** + + - Attributes: ingress, egress + - Items: eth, vlan, ipv4, ipv6, sctp, udp, tcp, vxlan, raw, inner eth, vlan, ipv4, ipv6, sctp, udp, tcp + - Ingress Actions: count, drop, flag, jump, mark, port_id, passthru, queue, rss, vxlan_decap, vxlan_encap, and void + - Egress Actions: count, drop, jump, passthru, vxlan_encap, and void + - Selectors: 'is', 'spec' and 'mask'. 'last' is not supported + - In total, up to 64 bytes of mask is allowed across all headers + +The VIC performs packet matching after applying VLAN strip. If VLAN +stripping is enabled, EtherType in the ETH item corresponds to the +stripped VLAN header's EtherType. Stripping does not affect the VLAN +item. TCI and EtherType in the VLAN item are matched against those in +the (stripped) VLAN header whether stripping is enabled or disabled. + More features may be added in future firmware and new versions of the VIC. Please refer to the release notes. @@ -323,18 +294,120 @@ inner and outer packets can be IPv4 or IPv6. RSS hash calculation, therefore queue selection, is done on inner packets. -In order to enable overlay offload, the 'Enable VXLAN' box should be checked +In order to enable overlay offload, enable VXLAN and/or Geneve on vNIC via CIMC or UCSM followed by a reboot of the server. When PMD successfully -enables overlay offload, it prints the following message on the console. +enables overlay offload, it prints one of the following messages on the console. .. code-block:: console - Overlay offload is enabled + Overlay offload is enabled (VxLAN) + Overlay offload is enabled (Geneve) + Overlay offload is enabled (VxLAN, Geneve) By default, PMD enables overlay offload if hardware supports it. To disable it, set ``devargs`` parameter ``disable-overlay=1``. For example:: - -w 12:00.0,disable-overlay=1 + -a 12:00.0,disable-overlay=1 + +By default, the NIC uses 4789 and 6081 as the VXLAN and Geneve ports, +respectively. The user may change them through +``rte_eth_dev_udp_tunnel_port_{add,delete}``. However, as the current +NIC has a single VXLAN port number and a single Geneve port number, +the user cannot configure multiple port numbers for each tunnel type. + +Geneve offload support has evolved over VIC models. On older models, +Geneve offload and advanced filters are mutually exclusive. This is +enforced by UCSM and CIMC, which only allow one of the two features +to be selected at one time. Newer VIC models do not have this restriction. + +Ingress VLAN Rewrite +-------------------- + +VIC adapters can tag, untag, or modify the VLAN headers of ingress +packets. The ingress VLAN rewrite mode controls this behavior. By +default, it is set to pass-through, where the NIC does not modify the +VLAN header in any way so that the application can see the original +header. This mode is sufficient for many applications, but may not be +suitable for others. Such applications may change the mode by setting +``devargs`` parameter ``ig-vlan-rewrite`` to one of the following. + +- ``pass``: Pass-through mode. The NIC does not modify the VLAN + header. This is the default mode. + +- ``priority``: Priority-tag default VLAN mode. If the ingress packet + is tagged with the default VLAN, the NIC replaces its VLAN header + with the priority tag (VLAN ID 0). + +- ``trunk``: Default trunk mode. The NIC tags untagged ingress packets + with the default VLAN. Tagged ingress packets are not modified. To + the application, every packet appears as tagged. + +- ``untag``: Untag default VLAN mode. If the ingress packet is tagged + with the default VLAN, the NIC removes or untags its VLAN header so + that the application sees an untagged packet. As a result, the + default VLAN becomes `untagged`. This mode can be useful for + applications such as OVS-DPDK performance benchmarks that utilize + only the default VLAN and want to see only untagged packets. + + +Vectorized Rx Handler +--------------------- + +ENIC PMD includes a version of the receive handler that is vectorized using +AVX2 SIMD instructions. It is meant for bulk, throughput oriented workloads +where reducing cycles/packet in PMD is a priority. In order to use the +vectorized handler, take the following steps. + +- Use a recent version of gcc, icc, or clang and build 64-bit DPDK. If + the compiler is known to support AVX2, DPDK build system + automatically compiles the vectorized handler. Otherwise, the + handler is not available. + +- Set ``devargs`` parameter ``enable-avx2-rx=1`` to explicitly request that + PMD consider the vectorized handler when selecting the receive handler. + For example:: + + -a 12:00.0,enable-avx2-rx=1 + + As the current implementation is intended for field trials, by default, the + vectorized handler is not considered (``enable-avx2-rx=0``). + +- Run on a UCS M4 or later server with CPUs that support AVX2. + +PMD selects the vectorized handler when the handler is compiled into +the driver, the user requests its use via ``enable-avx2-rx=1``, CPU +supports AVX2, and scatter Rx is not used. To verify that the +vectorized handler is selected, enable debug logging +(``--log-level=pmd,debug``) and check the following message. + +.. code-block:: console + + enic_use_vector_rx_handler use the non-scatter avx2 Rx handler + +64B Completion Queue Entry +-------------------------- + +Recent VIC adapters support 64B completion queue entries, as well as +16B entries that are available on all adapter models. ENIC PMD enables +and uses 64B entries by default, if available. 64B entries generally +lower CPU cycles per Rx packet, as they avoid partial DMA writes and +reduce cache contention between DMA and polling CPU. The effect is +most pronounced when multiple Rx queues are used on Intel platforms +with Data Direct I/O Technology (DDIO). + +If 64B entries are not available, PMD uses 16B entries. The user may +explicitly disable 64B entries and use 16B entries by setting +``devarg`` parameter ``cq64=0``. For example:: + + -a 12:00.0,cq64=0 + +To verify the selected entry size, enable debug logging +(``--log-level=enic,debug``) and check the following messages. + +.. code-block:: console + + PMD: rte_enic_pmd: Supported CQ entry sizes: 16 32 + PMD: rte_enic_pmd: Using 16B CQ entry size .. _enic_limitations: @@ -351,9 +424,10 @@ Limitations In test setups where an Ethernet port of a Cisco adapter in TRUNK mode is connected point-to-point to another adapter port or connected though a router instead of a switch, all ingress packets will be VLAN tagged. Programs such - as l3fwd which do not account for VLAN tags in packets will misbehave. The - solution is to enable VLAN stripping on ingress. The following code fragment is - an example of how to accomplish this: + as l3fwd may not account for VLAN tags in packets and may misbehave. One + solution is to enable VLAN stripping on ingress so the VLAN tag is removed + from the packet and put into the mbuf->vlan_tci field. Here is an example + of how to accomplish this: .. code-block:: console @@ -361,11 +435,13 @@ Limitations vlan_offload |= ETH_VLAN_STRIP_OFFLOAD; rte_eth_dev_set_vlan_offload(port, vlan_offload); -- Limited flow director support on 1200 series and 1300 series Cisco VIC - adapters with old firmware. Please see :ref:`enic-flow-director`. +Another alternative is modify the adapter's ingress VLAN rewrite mode so that +packets with the default VLAN tag are stripped by the adapter and presented to +DPDK as untagged packets. In this case mbuf->vlan_tci and the PKT_RX_VLAN and +PKT_RX_VLAN_STRIPPED mbuf flags would not be set. This mode is enabled with the +``devargs`` parameter ``ig-vlan-rewrite=untag``. For example:: -- Flow director features are not supported on generation 1 Cisco VIC adapters - (M81KR and P81E) + -a 12:00.0,ig-vlan-rewrite=untag - **SR-IOV** @@ -375,7 +451,7 @@ Limitations - VF devices are not usable directly from the host. They can only be used as assigned devices on VM instances. - Currently, unbind of the ENIC kernel mode driver 'enic.ko' on the VM - instance may hang. As a workaround, enic.ko should be blacklisted or removed + instance may hang. As a workaround, enic.ko should be blocked or removed from the boot process. - pci_generic cannot be used as the uio module in the VM. igb_uio or vfio in non-IOMMU mode can be used. @@ -391,6 +467,16 @@ Limitations 1000 for 1300 series VICs). Filters are checked for matching in the order they were added. Since there currently is no grouping or priority support, 'catch-all' filters should be added last. + - The supported range of IDs for the 'MARK' action is 0 - 0xFFFD. + - RSS and PASSTHRU actions only support "receive normally". They are limited + to supporting MARK + RSS and PASSTHRU + MARK to allow the application to mark + packets and then receive them normally. These require 1400 series VIC adapters + and latest firmware. + - RAW items are limited to matching UDP tunnel headers like VXLAN. + - For 1400 VICs, all flows using the RSS action on a port use same hash + configuration. The RETA is ignored. The queues used in the RSS group must be + sequential. There is a performance hit if the number of queues is not a power of 2. + Only level 0 (outer header) RSS is allowed. - **Statistics** @@ -447,13 +533,12 @@ Supported features - IP checksum offload - Receive side VLAN stripping - Multiple receive and transmit queues -- Flow Director ADD, UPDATE, DELETE, STATS operation support IPv4 and IPv6 - Promiscuous mode - Setting RX VLAN (supported via UCSM/CIMC only) - VLAN filtering (supported via UCSM/CIMC only) - Execution of application by unprivileged system users - IPV4, IPV6 and TCP RSS hashing -- UDP hashing (1400 series and later adapters) +- UDP RSS hashing (1400 series and later adapters) - Scattered Rx - MTU update - SR-IOV on UCS managed servers connected to Fabric Interconnects @@ -512,11 +597,6 @@ The value depends on the memory configuration of the application, DPDK and PMD. Typically, the limit has to be raised to higher than 2GB. e.g., 2621440 -The compilation of any unused drivers can be disabled using the -configuration file in config/ directory (e.g., config/common_linuxapp). -This would help in bringing down the time taken for building the -libraries and the initialization time of the application. - Additional Reference -------------------- @@ -530,4 +610,4 @@ Any questions or bugs should be reported to DPDK community and to the ENIC PMD maintainers: - John Daley -- Nelson Escobar +- Hyong Youb Kim