1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2021 Marvell.
7 The CNXK ETHDEV PMD (**librte_net_cnxk**) provides poll mode ethdev driver
8 support for the inbuilt network device found in **Marvell OCTEON CN9K/CN10K**
9 SoC family as well as for their virtual functions (VF) in SR-IOV context.
11 More information can be found at `Marvell Official Website
12 <https://www.marvell.com/embedded-processors/infrastructure-processors>`_.
17 Features of the CNXK Ethdev PMD are:
19 - Packet type information
24 - Multiple queues for TX and RX
25 - Receiver Side Scaling (RSS)
28 - Inner and Outer Checksum offload
29 - Port hardware statistics
30 - Link state information
33 - Scatter-Gather IO support
34 - Vector Poll mode driver
35 - Debug utilities - Context dump and error interrupt support
36 - Support Rx interrupt
37 - Inline IPsec processing support
38 - Ingress meter support
39 - Queue based priority flow control support
44 See :doc:`../platform/cnxk` for setup information.
47 Driver compilation and testing
48 ------------------------------
50 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
55 Follow instructions available in the document
56 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
61 .. code-block:: console
63 ./<build_dir>/app/dpdk-testpmd -c 0xc -a 0002:02:00.0 -- --portmask=0x1 --nb-cores=1 --port-topology=loop --rxq=1 --txq=1
64 EAL: Detected 4 lcore(s)
65 EAL: Detected 1 NUMA nodes
66 EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
67 EAL: Selected IOVA mode 'VA'
68 EAL: No available hugepages reported in hugepages-16777216kB
69 EAL: No available hugepages reported in hugepages-2048kB
70 EAL: Probing VFIO support...
71 EAL: VFIO support initialized
72 EAL: using IOMMU type 1 (Type 1)
73 [ 2003.202721] vfio-pci 0002:02:00.0: vfio_cap_init: hiding cap 0x14@0x98
74 EAL: Probe PCI driver: net_cn10k (177d:a063) device: 0002:02:00.0 (socket 0)
76 EAL: No legacy callbacks, legacy socket not created
77 testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
78 testpmd: preferred mempool ops selected: cn10k_mempool_ops
79 Configuring Port 0 (socket 0)
80 PMD: Port 0: Link Up - speed 25000 Mbps - full-duplex
82 Port 0: link state change event
83 Port 0: 96:D4:99:72:A5:BF
84 Checking link statuses...
86 No commandline core given, start packet forwarding
87 io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
88 Logical Core 3 (socket 0) forwards packets on 1 streams:
89 RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
91 io packet forwarding packets/burst=32
92 nb forwarding cores=1 - nb forwarding ports=1
93 port 0: RX queue number: 1 Tx queue number: 1
94 Rx offloads=0x0 Tx offloads=0x10000
96 RX desc=4096 - RX free threshold=0
97 RX threshold registers: pthresh=0 hthresh=0 wthresh=0
100 TX desc=512 - TX free threshold=0
101 TX threshold registers: pthresh=0 hthresh=0 wthresh=0
102 TX offloads=0x0 - TX RS bit threshold=0
105 Runtime Config Options
106 ----------------------
108 - ``Rx&Tx scalar mode enable`` (default ``0``)
110 PMD supports both scalar and vector mode, it may be selected at runtime
111 using ``scalar_enable`` ``devargs`` parameter.
113 - ``RSS reta size`` (default ``64``)
115 RSS redirection table size may be configured during runtime using ``reta_size``
116 ``devargs`` parameter.
120 -a 0002:02:00.0,reta_size=256
122 With the above configuration, reta table of size 256 is populated.
124 - ``Flow priority levels`` (default ``3``)
126 RTE Flow priority levels can be configured during runtime using
127 ``flow_max_priority`` ``devargs`` parameter.
131 -a 0002:02:00.0,flow_max_priority=10
133 With the above configuration, priority level was set to 10 (0-9). Max
134 priority level supported is 32.
136 - ``Reserve Flow entries`` (default ``8``)
138 RTE flow entries can be pre allocated and the size of pre allocation can be
139 selected runtime using ``flow_prealloc_size`` ``devargs`` parameter.
143 -a 0002:02:00.0,flow_prealloc_size=4
145 With the above configuration, pre alloc size was set to 4. Max pre alloc
146 size supported is 32.
148 - ``Max SQB buffer count`` (default ``512``)
150 Send queue descriptor buffer count may be limited during runtime using
151 ``max_sqb_count`` ``devargs`` parameter.
155 -a 0002:02:00.0,max_sqb_count=64
157 With the above configuration, each send queue's descriptor buffer count is
158 limited to a maximum of 64 buffers.
160 - ``Switch header enable`` (default ``none``)
162 A port can be configured to a specific switch header type by using
163 ``switch_header`` ``devargs`` parameter.
167 -a 0002:02:00.0,switch_header="higig2"
169 With the above configuration, higig2 will be enabled on that port and the
170 traffic on this port should be higig2 traffic only. Supported switch header
171 types are "chlen24b", "chlen90b", "dsa", "exdsa", "higig2", "vlan_exdsa" and
174 - ``Flow pre_l2 info`` (default ``0x0/0x0/0x0``)
176 pre_l2 headers are custom headers placed before the ethernet header. For
177 parsing custom pre_l2 headers, an offset, mask within the offset and shift
178 direction has to be provided within the custom header that holds the size of
179 the custom header. This is valid only with switch header pre_l2. Maximum
180 supported offset range is 0 to 255 and mask range is 1 to 255 and
181 shift direction, 0: left shift, 1: right shift.
182 Info format will be "offset/mask/shift direction". All parameters has to be
183 in hexadecimal format and mask should be contiguous. Info can be configured
184 using ``flow_pre_l2_info`` ``devargs`` parameter.
188 -a 0002:02:00.0,switch_header="pre_l2",flow_pre_l2_info=0x2/0x7e/0x1
190 With the above configuration, custom pre_l2 header will be enabled on that
191 port and size of the header is placed at byte offset 0x2 in the packet with
192 mask 0x7e and right shift will be used to get the size. That is, size will be
193 (pkt[0x2] & 0x7e) >> shift count. Shift count will be calculated based on
194 mask and shift direction. For example, if mask is 0x7c and shift direction is
195 1 (i.e., right shift) then the shift count will be 2, that is, absolute
196 position of the rightmost set bit. If the mask is 0x7c and shift direction
197 is 0 (i.e., left shift) then the shift count will be 1, that is, (8 - n),
198 where n is the absolute position of leftmost set bit.
200 - ``RSS tag as XOR`` (default ``0``)
202 The HW gives two options to configure the RSS adder i.e
204 * ``rss_adder<7:0> = flow_tag<7:0> ^ flow_tag<15:8> ^ flow_tag<23:16> ^ flow_tag<31:24>``
206 * ``rss_adder<7:0> = flow_tag<7:0>``
208 Latter one aligns with standard NIC behavior vs former one is a legacy
209 RSS adder scheme used in OCTEON 9 products.
211 By default, the driver runs in the latter mode.
212 Setting this flag to 1 to select the legacy mode.
214 For example to select the legacy mode(RSS tag adder as XOR)::
216 -a 0002:02:00.0,tag_as_xor=1
218 - ``Min SPI for inbound inline IPsec`` (default ``0``)
220 Min SPI supported for inbound inline IPsec processing can be specified by
221 ``ipsec_in_min_spi`` ``devargs`` parameter.
225 -a 0002:02:00.0,ipsec_in_min_spi=6
227 With the above configuration, application can enable inline IPsec processing
228 for inbound SA with min SPI of 6.
230 - ``Max SPI for inbound inline IPsec`` (default ``255``)
232 Max SPI supported for inbound inline IPsec processing can be specified by
233 ``ipsec_in_max_spi`` ``devargs`` parameter.
237 -a 0002:02:00.0,ipsec_in_max_spi=128
239 With the above configuration, application can enable inline IPsec processing
242 - ``Max SA's for outbound inline IPsec`` (default ``4096``)
244 Max number of SA's supported for outbound inline IPsec processing can be
245 specified by ``ipsec_out_max_sa`` ``devargs`` parameter.
249 -a 0002:02:00.0,ipsec_out_max_sa=128
251 With the above configuration, application can enable inline IPsec processing
252 for 128 outbound SAs.
254 - ``Outbound CPT LF queue size`` (default ``8200``)
256 Size of Outbound CPT LF queue in number of descriptors can be specified by
257 ``outb_nb_desc`` ``devargs`` parameter.
261 -a 0002:02:00.0,outb_nb_desc=16384
263 With the above configuration, Outbound CPT LF will be created to accommodate
264 at max 16384 descriptors at any given time.
266 - ``Outbound CPT LF count`` (default ``1``)
268 Number of CPT LF's to attach for Outbound processing can be specified by
269 ``outb_nb_crypto_qs`` ``devargs`` parameter.
273 -a 0002:02:00.0,outb_nb_crypto_qs=2
275 With the above configuration, two CPT LF's are setup and distributed among
276 all the Tx queues for outbound processing.
278 - ``Disable using inline ipsec device for inbound`` (default ``0``)
280 In CN10K, in event mode, driver can work in two modes,
282 1. Inbound encrypted traffic received by probed ipsec inline device while
283 plain traffic post decryption is received by ethdev.
285 2. Both Inbound encrypted traffic and plain traffic post decryption are
288 By default event mode works using inline device i.e mode ``1``.
289 This behaviour can be changed to pick mode ``2`` by using
290 ``no_inl_dev`` ``devargs`` parameter.
294 -a 0002:02:00.0,no_inl_dev=1 -a 0002:03:00.0,no_inl_dev=1
296 With the above configuration, inbound encrypted traffic from both the ports
297 is received by ipsec inline device.
299 - ``Inline IPsec device channel and mask`` (default ``none``)
301 Set channel and channel mask configuration for the inline IPSec device. This
302 will be used when creating flow rules with RTE_FLOW_ACTION_TYPE_SECURITY
305 By default, RTE Flow API sets the channel number of the port on which the
306 rule is created in the MCAM entry and matches it exactly. This behaviour can
307 be modified using the ``inl_cpt_channel`` ``devargs`` parameter.
311 -a 0002:1d:00.0,inl_cpt_channel=0x100/0xf00
313 With the above configuration, RTE Flow rules API will set the channel
314 and channel mask as 0x100 and 0xF00 in the MCAM entries of the flow rules
315 created with RTE_FLOW_ACTION_TYPE_SECURITY action. Since channel number is
316 set with this custom mask, inbound encrypted traffic from all ports with
317 matching channel number pattern will be directed to the inline IPSec device.
319 - ``SDP device channel and mask`` (default ``none``)
320 Set channel and channel mask configuration for the SDP device. This
321 will be used when creating flow rules on the SDP device.
323 By default, for rules created on the SDP device, the RTE Flow API sets the
324 channel number and mask to cover the entire SDP channel range in the channel
325 field of the MCAM entry. This behaviour can be modified using the
326 ``sdp_channel_mask`` ``devargs`` parameter.
330 -a 0002:1d:00.0,sdp_channel_mask=0x700/0xf00
332 With the above configuration, RTE Flow rules API will set the channel
333 and channel mask as 0x700 and 0xF00 in the MCAM entries of the flow rules
334 created on the SDP device. This option needs to be used when more than one
335 SDP interface is in use and RTE Flow rules created need to distinguish
336 between traffic from each SDP interface. The channel and mask combination
337 specified should match all the channels(or rings) configured on the SDP
342 Above devarg parameters are configurable per device, user needs to pass the
343 parameters to all the PCIe devices if application requires to configure on
344 all the ethdev ports.
349 ``mempool_cnxk`` external mempool handler dependency
350 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
352 The OCTEON CN9K/CN10K SoC family NIC has inbuilt HW assisted external mempool manager.
353 ``net_cnxk`` PMD only works with ``mempool_cnxk`` mempool handler
354 as it is performance wise most effective way for packet allocation and Tx buffer
355 recycling on OCTEON 9 SoC platform.
360 The OCTEON CN9K/CN10K SoC family NICs strip the CRC for every packet being received by
361 the host interface irrespective of the offload configuration.
366 - ``RTE_FLOW_ITEM_TYPE_GRE_KEY`` works only when checksum and routing
367 bits in the GRE header are equal to 0.
369 RTE flow action port_id support
370 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
372 - ``RTE_FLOW_ACTION_TYPE_PORT_ID`` is only supported between PF and its VFs.
374 Custom protocols supported in RTE Flow
375 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
377 The ``RTE_FLOW_ITEM_TYPE_RAW`` can be used to parse the below custom protocols.
379 * ``vlan_exdsa`` and ``exdsa`` can be parsed at L2 level.
380 * ``NGIO`` can be parsed at L3 level.
382 For ``vlan_exdsa`` and ``exdsa``, the port has to be configured with the
383 respective switch header.
387 -a 0002:02:00.0,switch_header="vlan_exdsa"
389 The below fields of ``struct rte_flow_item_raw`` shall be used to specify the
392 - ``relative`` Selects the layer at which parsing is done.
394 - 0 for ``exdsa`` and ``vlan_exdsa``.
398 - ``offset`` The offset in the header where the pattern should be matched.
399 - ``length`` Length of the pattern.
400 - ``pattern`` Pattern as a byte string.
402 Example usage in testpmd::
404 ./dpdk-testpmd -c 3 -w 0002:02:00.0,switch_header=exdsa -- -i \
405 --rx-offloads=0x00080000 --rxq 8 --txq 8
406 testpmd> flow create 0 ingress pattern eth / raw relative is 0 pattern \
407 spec ab pattern mask ab offset is 4 / end actions queue index 1 / end
409 Inline device support for CN10K
410 -------------------------------
412 CN10K HW provides a misc device Inline device that supports ethernet devices in
413 providing following features.
415 - Aggregate all the inline IPsec inbound traffic from all the CN10K ethernet
416 devices to be processed by the single inline IPSec device. This allows
417 single rte security session to accept traffic from multiple ports.
419 - Support for event generation on outbound inline IPsec processing errors.
421 - Support CN106xx poll mode of operation for inline IPSec inbound processing.
423 Inline IPsec device is identified by PCI PF vendid:devid ``177D:A0F0`` or
426 Runtime Config Options for inline device
427 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
429 - ``Min SPI for inbound inline IPsec`` (default ``0``)
431 Min SPI supported for inbound inline IPsec processing can be specified by
432 ``ipsec_in_min_spi`` ``devargs`` parameter.
436 -a 0002:1d:00.0,ipsec_in_min_spi=6
438 With the above configuration, application can enable inline IPsec processing
439 for inbound SA with min SPI of 6 for traffic aggregated on inline device.
441 - ``Max SPI for inbound inline IPsec`` (default ``255``)
443 Max SPI supported for inbound inline IPsec processing can be specified by
444 ``ipsec_in_max_spi`` ``devargs`` parameter.
448 -a 0002:1d:00.0,ipsec_in_max_spi=128
450 With the above configuration, application can enable inline IPsec processing
451 for inbound SA with max SPI of 128 for traffic aggregated on inline device.
457 .. _table_cnxk_ethdev_debug_options:
459 .. table:: cnxk ethdev debug options
461 +---+------------+-------------------------------------------------------+
462 | # | Component | EAL log command |
463 +===+============+=======================================================+
464 | 1 | NIX | --log-level='pmd\.net.cnxk,8' |
465 +---+------------+-------------------------------------------------------+
466 | 2 | NPC | --log-level='pmd\.net.cnxk\.flow,8' |
467 +---+------------+-------------------------------------------------------+