1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2021 Marvell.
4 Marvell cnxk platform guide
5 ===========================
7 This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON cnxk platform.
10 More information about CN9K and CN10K SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON cnxk SoCs
14 --------------------------
19 Resource Virtualization Unit architecture
20 -----------------------------------------
22 The :numref:`figure_cnxk_resource_virtualization` diagram depicts the
23 RVU architecture and a resource provisioning example.
25 .. _figure_cnxk_resource_virtualization:
27 .. figure:: img/cnxk_resource_virtualization.*
29 cnxk Resource virtualization architecture and provisioning example
32 Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW
33 resources belonging to the network, crypto and other functional blocks onto
34 PCI-compatible physical and virtual functions.
36 Each functional block has multiple local functions (LFs) for
37 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
38 physical functions (PFs) and virtual functions (VFs).
40 The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local
41 functions (LFs) provided by the RVU and its functional mapping to
44 .. _table_cnxk_rvu_dpdk_mapping:
46 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
48 +---+-----+--------------------------------------------------------------+
49 | # | LF | DPDK subsystem mapping |
50 +===+=====+==============================================================+
51 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
52 +---+-----+--------------------------------------------------------------+
53 | 2 | NPA | rte_mempool |
54 +---+-----+--------------------------------------------------------------+
55 | 3 | NPC | rte_flow |
56 +---+-----+--------------------------------------------------------------+
57 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
58 +---+-----+--------------------------------------------------------------+
59 | 5 | SSO | rte_eventdev |
60 +---+-----+--------------------------------------------------------------+
61 | 6 | TIM | rte_event_timer_adapter |
62 +---+-----+--------------------------------------------------------------+
63 | 7 | LBK | rte_ethdev |
64 +---+-----+--------------------------------------------------------------+
65 | 8 | DPI | rte_dmadev |
66 +---+-----+--------------------------------------------------------------+
67 | 9 | SDP | rte_ethdev |
68 +---+-----+--------------------------------------------------------------+
69 | 10| REE | rte_regexdev |
70 +---+-----+--------------------------------------------------------------+
71 | 11| BPHY| rte_rawdev |
72 +---+-----+--------------------------------------------------------------+
74 PF0 is called the administrative / admin function (AF) and has exclusive
75 privileges to provision RVU functional block's LFs to each of the PF/VF.
77 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
78 requests from PF/VF, AF does resource provisioning and other HW configuration.
80 AF is always attached to host, but PF/VFs may be used by host kernel itself,
81 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
82 handle provisioning/configuration requests sent by any device from any domain.
84 The AF driver does not receive or process any data.
85 It is only a configuration driver used in control path.
87 The :numref:`figure_cnxk_resource_virtualization` diagram also shows a
88 resource provisioning example where,
90 1. PFx and PFx-VF0 bound to Linux netdev driver.
91 2. PFx-VF1 ethdev driver bound to the first DPDK application.
92 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
97 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
98 The loopback block has N channels and contains data buffering that is shared across
99 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
100 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
101 that is, packets sent on odd VF end up received on even VF and vice versa.
102 This would enable HW accelerated means of communication between two domains
103 where even VF bound to the first domain and odd VF bound to the second domain.
105 Typical application usage models are,
107 #. Communication between the Linux kernel and DPDK application.
108 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
109 #. Communication between two different DPDK applications.
114 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
115 to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when
116 cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive
117 packets to/from remote host machine using input/output queue pairs exposed to it.
118 SDP interface receives input packets from remote host from NIX-RX and sends packets
119 to remote host using NIX-TX. Remote host machine need to use corresponding driver
120 (kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports
121 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
122 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
124 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
126 #. Communication channel between remote host and cnxk SoC over PCIe.
127 #. Transfer packets received from network interface to remote host over PCIe and
131 ----------------------
133 The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts
134 the packet flow on cnxk SoC in conjunction with use of various HW accelerators.
136 .. _figure_cnxk_packet_flow_hw_accelerators:
138 .. figure:: img/cnxk_packet_flow_hw_accelerators.*
140 cnxk packet flow in conjunction with use of HW accelerators
145 This section lists dataplane H/W block(s) available in cnxk SoC.
148 See :doc:`../nics/cnxk` for NIX Ethdev driver information.
150 #. **Mempool Driver**
151 See :doc:`../mempool/cnxk` for NPA mempool driver information.
153 #. **Baseband PHY Driver**
154 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information.
157 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information.
159 Procedure to Setup Platform
160 ---------------------------
162 There are three main prerequisites for setting up DPDK on cnxk
165 1. **RVU AF Linux kernel driver**
167 The dependent kernel drivers can be obtained from the
168 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
170 Alternatively, the Marvell SDK also provides the required kernel drivers.
172 Linux kernel should be configured with the following features enabled:
174 .. code-block:: console
176 # 64K pages enabled for better performance
177 CONFIG_ARM64_64K_PAGES=y
178 CONFIG_ARM64_VA_BITS_48=y
179 # huge pages support enabled
181 CONFIG_HUGETLB_PAGE=y
182 # VFIO enabled with TYPE1 IOMMU at minimum
183 CONFIG_VFIO_IOMMU_TYPE1=y
186 CONFIG_VFIO_NOIOMMU=y
188 CONFIG_VFIO_PCI_MMAP=y
191 # ARMv8.1 LSE atomics
192 CONFIG_ARM64_LSE_ATOMICS=y
194 CONFIG_OCTEONTX2_MBOX=y
195 CONFIG_OCTEONTX2_AF=y
196 # Enable if netdev PF driver required
197 CONFIG_OCTEONTX2_PF=y
198 # Enable if netdev VF driver required
199 CONFIG_OCTEONTX2_VF=y
200 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
201 # Enable if OCTEONTX2 DMA PF driver required
202 CONFIG_OCTEONTX2_DPI_PF=n
204 2. **ARM64 Linux Tool Chain**
206 For example, the *aarch64* Linaro Toolchain, which can be obtained from
207 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
209 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
210 optimized for cnxk CPU.
212 3. **Rootfile system**
214 Any *aarch64* supporting filesystem may be used. For example,
215 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
216 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
218 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
219 The SDK includes all the above prerequisites necessary to bring up the cnxk board.
221 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
227 .. _table_cnxk_common_debug_options:
229 .. table:: cnxk common debug options
231 +---+------------+-------------------------------------------------------+
232 | # | Component | EAL log command |
233 +===+============+=======================================================+
234 | 1 | Common | --log-level='pmd\.cnxk\.base,8' |
235 +---+------------+-------------------------------------------------------+
236 | 2 | Mailbox | --log-level='pmd\.cnxk\.mbox,8' |
237 +---+------------+-------------------------------------------------------+
242 The **RVU AF Linux kernel driver** provides support to dump RVU blocks
243 context or stats using debugfs.
245 Enable ``debugfs`` by:
247 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
248 2. Boot OCTEON CN9K/CN10K with debugfs supported kernel.
249 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
251 .. code-block:: console
253 # mount -t debugfs none /sys/kernel/debug
255 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
258 The file structure under ``/sys/kernel/debug`` is as follows
260 .. code-block:: console
278 | |-- cpt_engines_info
279 | |-- cpt_engines_sts
286 | |-- ndc_rx_hits_miss
288 | |-- ndc_tx_hits_miss
301 | '-- rx_miss_act_stats
307 |-- sso_hwgrp_aq_thresh
308 |-- sso_hwgrp_iaq_walk
310 |-- sso_hwgrp_free_list_walk
311 |-- sso_hwgrp_ient_walk
312 '-- sso_hwgrp_taq_walk
314 RVU block LF allocation:
316 .. code-block:: console
318 cat /sys/kernel/debug/cn10k/rsrc_alloc
320 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
327 .. code-block:: console
329 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats
331 =======Link Status======
333 Link is UP 25000 Mbps
335 =======NIX RX_STATS(rpm port level)======
345 =======NIX TX_STATS(rpm port level)======
354 =======rpm RX_STATS======
356 Octets of received packets: 0
357 Octets of received packets with out error: 0
358 Received packets with alignment errors: 0
359 Control/PAUSE packets received: 0
360 Packets received with Frame too long Errors: 0
361 Packets received with a1nrange length Errors: 0
363 Packets received with FrameCheckSequenceErrors: 0
364 Packets received with VLAN header: 0
366 Packets received with unicast DMAC: 0
367 Packets received with multicast DMAC: 0
368 Packets received with broadcast DMAC: 0
370 Total frames received on interface: 0
371 Packets received with an octet count < 64: 0
372 Packets received with an octet count == 64: 0
373 Packets received with an octet count of 65–127: 0
374 Packets received with an octet count of 128-255: 0
375 Packets received with an octet count of 256-511: 0
376 Packets received with an octet count of 512-1023: 0
377 Packets received with an octet count of 1024-1518: 0
378 Packets received with an octet count of > 1518: 0
381 Fragmented Packets: 0
382 CBFC(class based flow control) pause frames received for class 0: 0
383 CBFC pause frames received for class 1: 0
384 CBFC pause frames received for class 2: 0
385 CBFC pause frames received for class 3: 0
386 CBFC pause frames received for class 4: 0
387 CBFC pause frames received for class 5: 0
388 CBFC pause frames received for class 6: 0
389 CBFC pause frames received for class 7: 0
390 CBFC pause frames received for class 8: 0
391 CBFC pause frames received for class 9: 0
392 CBFC pause frames received for class 10: 0
393 CBFC pause frames received for class 11: 0
394 CBFC pause frames received for class 12: 0
395 CBFC pause frames received for class 13: 0
396 CBFC pause frames received for class 14: 0
397 CBFC pause frames received for class 15: 0
398 MAC control packets received: 0
400 =======rpm TX_STATS======
402 Total octets sent on the interface: 0
403 Total octets transmitted OK: 0
404 Control/Pause frames sent: 0
405 Total frames transmitted OK: 0
406 Total frames sent with VLAN header: 0
408 Packets sent to unicast DMAC: 0
409 Packets sent to the multicast DMAC: 0
410 Packets sent to a broadcast DMAC: 0
411 Packets sent with an octet count == 64: 0
412 Packets sent with an octet count of 65–127: 0
413 Packets sent with an octet count of 128-255: 0
414 Packets sent with an octet count of 256-511: 0
415 Packets sent with an octet count of 512-1023: 0
416 Packets sent with an octet count of 1024-1518: 0
417 Packets sent with an octet count of > 1518: 0
418 CBFC(class based flow control) pause frames transmitted for class 0: 0
419 CBFC pause frames transmitted for class 1: 0
420 CBFC pause frames transmitted for class 2: 0
421 CBFC pause frames transmitted for class 3: 0
422 CBFC pause frames transmitted for class 4: 0
423 CBFC pause frames transmitted for class 5: 0
424 CBFC pause frames transmitted for class 6: 0
425 CBFC pause frames transmitted for class 7: 0
426 CBFC pause frames transmitted for class 8: 0
427 CBFC pause frames transmitted for class 9: 0
428 CBFC pause frames transmitted for class 10: 0
429 CBFC pause frames transmitted for class 11: 0
430 CBFC pause frames transmitted for class 12: 0
431 CBFC pause frames transmitted for class 13: 0
432 CBFC pause frames transmitted for class 14: 0
433 CBFC pause frames transmitted for class 15: 0
434 MAC control packets sent: 0
435 Total frames sent on the interface: 0
439 .. code-block:: console
441 cat /sys/kernel/debug/cn10k/cpt/cpt_pc
443 CPT instruction requests 0
444 CPT instruction latency 0
445 CPT NCB read requests 0
446 CPT NCB read latency 0
447 CPT read requests caused by UC fills 0
448 CPT active cycles pc 1395642
449 CPT clock count pc 5579867595493
453 .. code-block:: console
455 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx
456 cat /sys/kernel/debug/cn10k/nix/cq_ctx
457 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx
458 cat /sys/kernel/debug/cn10k/nix/cq_ctx
460 =====cq_ctx for nixlf:0 and qidx:0 is=====
471 W2: update_time 31043
488 .. code-block:: console
490 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx
491 cat /sys/kernel/debug/cn10k/npa/pool_ctx
492 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx
493 cat /sys/kernel/debug/cn10k/npa/pool_ctx
495 ======POOL : 0=======
496 W0: Stack base 1375bff00
503 W2: stack_max_pages 24315
504 W2: stack_pages 24314
514 W4: update_time 62993
516 W6: ptr_start 1593adf00
517 W7: ptr_end 180000000
523 W8: thresh_qint_idx 0
528 .. code-block:: console
530 cat /sys/kernel/debug/cn10k/npc/mcam_info
533 RX keywidth : 224bits
534 TX keywidth : 224bits
546 .. code-block:: console
548 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
549 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
551 ==================================================
552 SSOW HWS[0] Arbitration State 0x0
553 SSOW HWS[0] Guest Machine Control 0x0
554 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
555 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
556 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
557 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
558 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
559 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
560 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
561 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
562 ==================================================
567 DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on
568 an x86 based platform.
573 .. code-block:: console
581 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
583 .. code-block:: console
585 meson build --cross-file config/arm/arm64_cn10k_linux_gcc
590 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
591 if Marvell toolchain is available then it can be used by overriding the
592 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
593 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file.