1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2021 Marvell.
4 Marvell cnxk platform guide
5 ===========================
7 This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON cnxk platform.
10 More information about CN9K and CN10K SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON cnxk SoCs
14 --------------------------
23 Resource Virtualization Unit architecture
24 -----------------------------------------
26 The :numref:`figure_cnxk_resource_virtualization` diagram depicts the
27 RVU architecture and a resource provisioning example.
29 .. _figure_cnxk_resource_virtualization:
31 .. figure:: img/cnxk_resource_virtualization.*
33 cnxk Resource virtualization architecture and provisioning example
36 Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW
37 resources belonging to the network, crypto and other functional blocks onto
38 PCI-compatible physical and virtual functions.
40 Each functional block has multiple local functions (LFs) for
41 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
42 physical functions (PFs) and virtual functions (VFs).
44 The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local
45 functions (LFs) provided by the RVU and its functional mapping to
48 .. _table_cnxk_rvu_dpdk_mapping:
50 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
52 +---+-----+--------------------------------------------------------------+
53 | # | LF | DPDK subsystem mapping |
54 +===+=====+==============================================================+
55 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
56 +---+-----+--------------------------------------------------------------+
57 | 2 | NPA | rte_mempool |
58 +---+-----+--------------------------------------------------------------+
59 | 3 | NPC | rte_flow |
60 +---+-----+--------------------------------------------------------------+
61 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
62 +---+-----+--------------------------------------------------------------+
63 | 5 | SSO | rte_eventdev |
64 +---+-----+--------------------------------------------------------------+
65 | 6 | TIM | rte_event_timer_adapter |
66 +---+-----+--------------------------------------------------------------+
67 | 7 | LBK | rte_ethdev |
68 +---+-----+--------------------------------------------------------------+
69 | 8 | DPI | rte_dmadev |
70 +---+-----+--------------------------------------------------------------+
71 | 9 | SDP | rte_ethdev |
72 +---+-----+--------------------------------------------------------------+
73 | 10| REE | rte_regexdev |
74 +---+-----+--------------------------------------------------------------+
75 | 11| BPHY| rte_rawdev |
76 +---+-----+--------------------------------------------------------------+
77 | 12| GPIO| rte_rawdev |
78 +---+-----+--------------------------------------------------------------+
80 PF0 is called the administrative / admin function (AF) and has exclusive
81 privileges to provision RVU functional block's LFs to each of the PF/VF.
83 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
84 requests from PF/VF, AF does resource provisioning and other HW configuration.
86 AF is always attached to host, but PF/VFs may be used by host kernel itself,
87 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
88 handle provisioning/configuration requests sent by any device from any domain.
90 The AF driver does not receive or process any data.
91 It is only a configuration driver used in control path.
93 The :numref:`figure_cnxk_resource_virtualization` diagram also shows a
94 resource provisioning example where,
96 1. PFx and PFx-VF0 bound to Linux netdev driver.
97 2. PFx-VF1 ethdev driver bound to the first DPDK application.
98 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
103 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
104 The loopback block has N channels and contains data buffering that is shared across
105 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
106 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
107 that is, packets sent on odd VF end up received on even VF and vice versa.
108 This would enable HW accelerated means of communication between two domains
109 where even VF bound to the first domain and odd VF bound to the second domain.
111 Typical application usage models are,
113 #. Communication between the Linux kernel and DPDK application.
114 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
115 #. Communication between two different DPDK applications.
120 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
121 to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when
122 cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive
123 packets to/from remote host machine using input/output queue pairs exposed to it.
124 SDP interface receives input packets from remote host from NIX-RX and sends packets
125 to remote host using NIX-TX. Remote host machine need to use corresponding driver
126 (kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports
127 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
128 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
130 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
132 #. Communication channel between remote host and cnxk SoC over PCIe.
133 #. Transfer packets received from network interface to remote host over PCIe and
137 ----------------------
139 The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts
140 the packet flow on cnxk SoC in conjunction with use of various HW accelerators.
142 .. _figure_cnxk_packet_flow_hw_accelerators:
144 .. figure:: img/cnxk_packet_flow_hw_accelerators.*
146 cnxk packet flow in conjunction with use of HW accelerators
151 This section lists dataplane H/W block(s) available in cnxk SoC.
154 See :doc:`../nics/cnxk` for NIX Ethdev driver information.
156 #. **Mempool Driver**
157 See :doc:`../mempool/cnxk` for NPA mempool driver information.
159 #. **Baseband PHY Driver**
160 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information.
163 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information.
165 #. **Regex Device Driver**
166 See :doc:`../regexdevs/cn9k` for REE Regex device driver information.
168 Procedure to Setup Platform
169 ---------------------------
171 There are three main prerequisites for setting up DPDK on cnxk
174 1. **RVU AF Linux kernel driver**
176 The dependent kernel drivers can be obtained from the
177 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
179 Alternatively, the Marvell SDK also provides the required kernel drivers.
181 Linux kernel should be configured with the following features enabled:
183 .. code-block:: console
185 # 64K pages enabled for better performance
186 CONFIG_ARM64_64K_PAGES=y
187 CONFIG_ARM64_VA_BITS_48=y
188 # huge pages support enabled
190 CONFIG_HUGETLB_PAGE=y
191 # VFIO enabled with TYPE1 IOMMU at minimum
192 CONFIG_VFIO_IOMMU_TYPE1=y
195 CONFIG_VFIO_NOIOMMU=y
197 CONFIG_VFIO_PCI_MMAP=y
200 # ARMv8.1 LSE atomics
201 CONFIG_ARM64_LSE_ATOMICS=y
203 CONFIG_OCTEONTX2_MBOX=y
204 CONFIG_OCTEONTX2_AF=y
205 # Enable if netdev PF driver required
206 CONFIG_OCTEONTX2_PF=y
207 # Enable if netdev VF driver required
208 CONFIG_OCTEONTX2_VF=y
209 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
210 # Enable if OCTEONTX2 DMA PF driver required
211 CONFIG_OCTEONTX2_DPI_PF=n
213 2. **ARM64 Linux Tool Chain**
215 For example, the *aarch64* Linaro Toolchain, which can be obtained from
216 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
218 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
219 optimized for cnxk CPU.
221 3. **Rootfile system**
223 Any *aarch64* supporting filesystem may be used. For example,
224 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
225 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
227 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
228 The SDK includes all the above prerequisites necessary to bring up the cnxk board.
230 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
236 .. _table_cnxk_common_debug_options:
238 .. table:: cnxk common debug options
240 +---+------------+-------------------------------------------------------+
241 | # | Component | EAL log command |
242 +===+============+=======================================================+
243 | 1 | Common | --log-level='pmd\.cnxk\.base,8' |
244 +---+------------+-------------------------------------------------------+
245 | 2 | Mailbox | --log-level='pmd\.cnxk\.mbox,8' |
246 +---+------------+-------------------------------------------------------+
251 The **RVU AF Linux kernel driver** provides support to dump RVU blocks
252 context or stats using debugfs.
254 Enable ``debugfs`` by:
256 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
257 2. Boot OCTEON CN9K/CN10K with debugfs supported kernel.
258 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
260 .. code-block:: console
262 # mount -t debugfs none /sys/kernel/debug
264 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
267 The file structure under ``/sys/kernel/debug`` is as follows
269 .. code-block:: console
287 | |-- cpt_engines_info
288 | |-- cpt_engines_sts
295 | |-- ndc_rx_hits_miss
297 | |-- ndc_tx_hits_miss
310 | '-- rx_miss_act_stats
316 |-- sso_hwgrp_aq_thresh
317 |-- sso_hwgrp_iaq_walk
319 |-- sso_hwgrp_free_list_walk
320 |-- sso_hwgrp_ient_walk
321 '-- sso_hwgrp_taq_walk
323 RVU block LF allocation:
325 .. code-block:: console
327 cat /sys/kernel/debug/cn10k/rsrc_alloc
329 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
336 .. code-block:: console
338 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats
340 =======Link Status======
342 Link is UP 25000 Mbps
344 =======NIX RX_STATS(rpm port level)======
354 =======NIX TX_STATS(rpm port level)======
363 =======rpm RX_STATS======
365 Octets of received packets: 0
366 Octets of received packets with out error: 0
367 Received packets with alignment errors: 0
368 Control/PAUSE packets received: 0
369 Packets received with Frame too long Errors: 0
370 Packets received with a1nrange length Errors: 0
372 Packets received with FrameCheckSequenceErrors: 0
373 Packets received with VLAN header: 0
375 Packets received with unicast DMAC: 0
376 Packets received with multicast DMAC: 0
377 Packets received with broadcast DMAC: 0
379 Total frames received on interface: 0
380 Packets received with an octet count < 64: 0
381 Packets received with an octet count == 64: 0
382 Packets received with an octet count of 65–127: 0
383 Packets received with an octet count of 128-255: 0
384 Packets received with an octet count of 256-511: 0
385 Packets received with an octet count of 512-1023: 0
386 Packets received with an octet count of 1024-1518: 0
387 Packets received with an octet count of > 1518: 0
390 Fragmented Packets: 0
391 CBFC(class based flow control) pause frames received for class 0: 0
392 CBFC pause frames received for class 1: 0
393 CBFC pause frames received for class 2: 0
394 CBFC pause frames received for class 3: 0
395 CBFC pause frames received for class 4: 0
396 CBFC pause frames received for class 5: 0
397 CBFC pause frames received for class 6: 0
398 CBFC pause frames received for class 7: 0
399 CBFC pause frames received for class 8: 0
400 CBFC pause frames received for class 9: 0
401 CBFC pause frames received for class 10: 0
402 CBFC pause frames received for class 11: 0
403 CBFC pause frames received for class 12: 0
404 CBFC pause frames received for class 13: 0
405 CBFC pause frames received for class 14: 0
406 CBFC pause frames received for class 15: 0
407 MAC control packets received: 0
409 =======rpm TX_STATS======
411 Total octets sent on the interface: 0
412 Total octets transmitted OK: 0
413 Control/Pause frames sent: 0
414 Total frames transmitted OK: 0
415 Total frames sent with VLAN header: 0
417 Packets sent to unicast DMAC: 0
418 Packets sent to the multicast DMAC: 0
419 Packets sent to a broadcast DMAC: 0
420 Packets sent with an octet count == 64: 0
421 Packets sent with an octet count of 65–127: 0
422 Packets sent with an octet count of 128-255: 0
423 Packets sent with an octet count of 256-511: 0
424 Packets sent with an octet count of 512-1023: 0
425 Packets sent with an octet count of 1024-1518: 0
426 Packets sent with an octet count of > 1518: 0
427 CBFC(class based flow control) pause frames transmitted for class 0: 0
428 CBFC pause frames transmitted for class 1: 0
429 CBFC pause frames transmitted for class 2: 0
430 CBFC pause frames transmitted for class 3: 0
431 CBFC pause frames transmitted for class 4: 0
432 CBFC pause frames transmitted for class 5: 0
433 CBFC pause frames transmitted for class 6: 0
434 CBFC pause frames transmitted for class 7: 0
435 CBFC pause frames transmitted for class 8: 0
436 CBFC pause frames transmitted for class 9: 0
437 CBFC pause frames transmitted for class 10: 0
438 CBFC pause frames transmitted for class 11: 0
439 CBFC pause frames transmitted for class 12: 0
440 CBFC pause frames transmitted for class 13: 0
441 CBFC pause frames transmitted for class 14: 0
442 CBFC pause frames transmitted for class 15: 0
443 MAC control packets sent: 0
444 Total frames sent on the interface: 0
448 .. code-block:: console
450 cat /sys/kernel/debug/cn10k/cpt/cpt_pc
452 CPT instruction requests 0
453 CPT instruction latency 0
454 CPT NCB read requests 0
455 CPT NCB read latency 0
456 CPT read requests caused by UC fills 0
457 CPT active cycles pc 1395642
458 CPT clock count pc 5579867595493
462 .. code-block:: console
464 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx
465 cat /sys/kernel/debug/cn10k/nix/cq_ctx
466 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx
467 cat /sys/kernel/debug/cn10k/nix/cq_ctx
469 =====cq_ctx for nixlf:0 and qidx:0 is=====
480 W2: update_time 31043
497 .. code-block:: console
499 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx
500 cat /sys/kernel/debug/cn10k/npa/pool_ctx
501 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx
502 cat /sys/kernel/debug/cn10k/npa/pool_ctx
504 ======POOL : 0=======
505 W0: Stack base 1375bff00
512 W2: stack_max_pages 24315
513 W2: stack_pages 24314
523 W4: update_time 62993
525 W6: ptr_start 1593adf00
526 W7: ptr_end 180000000
532 W8: thresh_qint_idx 0
537 .. code-block:: console
539 cat /sys/kernel/debug/cn10k/npc/mcam_info
542 RX keywidth : 224bits
543 TX keywidth : 224bits
555 .. code-block:: console
557 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
558 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
560 ==================================================
561 SSOW HWS[0] Arbitration State 0x0
562 SSOW HWS[0] Guest Machine Control 0x0
563 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
564 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
565 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
566 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
567 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
568 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
569 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
570 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
571 ==================================================
576 DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on
577 an x86 based platform.
582 .. code-block:: console
590 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
594 .. code-block:: console
596 meson build --cross-file config/arm/arm64_cn9k_linux_gcc
601 .. code-block:: console
603 meson build --cross-file config/arm/arm64_cn10k_linux_gcc
608 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
609 if Marvell toolchain is available then it can be used by overriding the
610 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
611 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file.