1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2021 Marvell.
4 Marvell cnxk platform guide
5 ===========================
7 This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON cnxk platform.
10 More information about CN9K and CN10K SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON cnxk SoCs
14 --------------------------
22 Resource Virtualization Unit architecture
23 -----------------------------------------
25 The :numref:`figure_cnxk_resource_virtualization` diagram depicts the
26 RVU architecture and a resource provisioning example.
28 .. _figure_cnxk_resource_virtualization:
30 .. figure:: img/cnxk_resource_virtualization.*
32 cnxk Resource virtualization architecture and provisioning example
35 Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW
36 resources belonging to the network, crypto and other functional blocks onto
37 PCI-compatible physical and virtual functions.
39 Each functional block has multiple local functions (LFs) for
40 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
41 physical functions (PFs) and virtual functions (VFs).
43 The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local
44 functions (LFs) provided by the RVU and its functional mapping to
47 .. _table_cnxk_rvu_dpdk_mapping:
49 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
51 +---+-----+--------------------------------------------------------------+
52 | # | LF | DPDK subsystem mapping |
53 +===+=====+==============================================================+
54 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
55 +---+-----+--------------------------------------------------------------+
56 | 2 | NPA | rte_mempool |
57 +---+-----+--------------------------------------------------------------+
58 | 3 | NPC | rte_flow |
59 +---+-----+--------------------------------------------------------------+
60 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
61 +---+-----+--------------------------------------------------------------+
62 | 5 | SSO | rte_eventdev |
63 +---+-----+--------------------------------------------------------------+
64 | 6 | TIM | rte_event_timer_adapter |
65 +---+-----+--------------------------------------------------------------+
66 | 7 | LBK | rte_ethdev |
67 +---+-----+--------------------------------------------------------------+
68 | 8 | DPI | rte_dmadev |
69 +---+-----+--------------------------------------------------------------+
70 | 9 | SDP | rte_ethdev |
71 +---+-----+--------------------------------------------------------------+
72 | 10| REE | rte_regexdev |
73 +---+-----+--------------------------------------------------------------+
74 | 11| BPHY| rte_rawdev |
75 +---+-----+--------------------------------------------------------------+
77 PF0 is called the administrative / admin function (AF) and has exclusive
78 privileges to provision RVU functional block's LFs to each of the PF/VF.
80 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
81 requests from PF/VF, AF does resource provisioning and other HW configuration.
83 AF is always attached to host, but PF/VFs may be used by host kernel itself,
84 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
85 handle provisioning/configuration requests sent by any device from any domain.
87 The AF driver does not receive or process any data.
88 It is only a configuration driver used in control path.
90 The :numref:`figure_cnxk_resource_virtualization` diagram also shows a
91 resource provisioning example where,
93 1. PFx and PFx-VF0 bound to Linux netdev driver.
94 2. PFx-VF1 ethdev driver bound to the first DPDK application.
95 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
100 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
101 The loopback block has N channels and contains data buffering that is shared across
102 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
103 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
104 that is, packets sent on odd VF end up received on even VF and vice versa.
105 This would enable HW accelerated means of communication between two domains
106 where even VF bound to the first domain and odd VF bound to the second domain.
108 Typical application usage models are,
110 #. Communication between the Linux kernel and DPDK application.
111 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
112 #. Communication between two different DPDK applications.
117 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
118 to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when
119 cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive
120 packets to/from remote host machine using input/output queue pairs exposed to it.
121 SDP interface receives input packets from remote host from NIX-RX and sends packets
122 to remote host using NIX-TX. Remote host machine need to use corresponding driver
123 (kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports
124 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
125 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
127 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
129 #. Communication channel between remote host and cnxk SoC over PCIe.
130 #. Transfer packets received from network interface to remote host over PCIe and
134 ----------------------
136 The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts
137 the packet flow on cnxk SoC in conjunction with use of various HW accelerators.
139 .. _figure_cnxk_packet_flow_hw_accelerators:
141 .. figure:: img/cnxk_packet_flow_hw_accelerators.*
143 cnxk packet flow in conjunction with use of HW accelerators
148 This section lists dataplane H/W block(s) available in cnxk SoC.
151 See :doc:`../nics/cnxk` for NIX Ethdev driver information.
153 #. **Mempool Driver**
154 See :doc:`../mempool/cnxk` for NPA mempool driver information.
156 #. **Baseband PHY Driver**
157 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information.
160 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information.
162 #. **Regex Device Driver**
163 See :doc:`../regexdevs/cn9k` for REE Regex device driver information.
165 Procedure to Setup Platform
166 ---------------------------
168 There are three main prerequisites for setting up DPDK on cnxk
171 1. **RVU AF Linux kernel driver**
173 The dependent kernel drivers can be obtained from the
174 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
176 Alternatively, the Marvell SDK also provides the required kernel drivers.
178 Linux kernel should be configured with the following features enabled:
180 .. code-block:: console
182 # 64K pages enabled for better performance
183 CONFIG_ARM64_64K_PAGES=y
184 CONFIG_ARM64_VA_BITS_48=y
185 # huge pages support enabled
187 CONFIG_HUGETLB_PAGE=y
188 # VFIO enabled with TYPE1 IOMMU at minimum
189 CONFIG_VFIO_IOMMU_TYPE1=y
192 CONFIG_VFIO_NOIOMMU=y
194 CONFIG_VFIO_PCI_MMAP=y
197 # ARMv8.1 LSE atomics
198 CONFIG_ARM64_LSE_ATOMICS=y
200 CONFIG_OCTEONTX2_MBOX=y
201 CONFIG_OCTEONTX2_AF=y
202 # Enable if netdev PF driver required
203 CONFIG_OCTEONTX2_PF=y
204 # Enable if netdev VF driver required
205 CONFIG_OCTEONTX2_VF=y
206 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
207 # Enable if OCTEONTX2 DMA PF driver required
208 CONFIG_OCTEONTX2_DPI_PF=n
210 2. **ARM64 Linux Tool Chain**
212 For example, the *aarch64* Linaro Toolchain, which can be obtained from
213 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
215 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
216 optimized for cnxk CPU.
218 3. **Rootfile system**
220 Any *aarch64* supporting filesystem may be used. For example,
221 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
222 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
224 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
225 The SDK includes all the above prerequisites necessary to bring up the cnxk board.
227 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
233 .. _table_cnxk_common_debug_options:
235 .. table:: cnxk common debug options
237 +---+------------+-------------------------------------------------------+
238 | # | Component | EAL log command |
239 +===+============+=======================================================+
240 | 1 | Common | --log-level='pmd\.cnxk\.base,8' |
241 +---+------------+-------------------------------------------------------+
242 | 2 | Mailbox | --log-level='pmd\.cnxk\.mbox,8' |
243 +---+------------+-------------------------------------------------------+
248 The **RVU AF Linux kernel driver** provides support to dump RVU blocks
249 context or stats using debugfs.
251 Enable ``debugfs`` by:
253 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
254 2. Boot OCTEON CN9K/CN10K with debugfs supported kernel.
255 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
257 .. code-block:: console
259 # mount -t debugfs none /sys/kernel/debug
261 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
264 The file structure under ``/sys/kernel/debug`` is as follows
266 .. code-block:: console
284 | |-- cpt_engines_info
285 | |-- cpt_engines_sts
292 | |-- ndc_rx_hits_miss
294 | |-- ndc_tx_hits_miss
307 | '-- rx_miss_act_stats
313 |-- sso_hwgrp_aq_thresh
314 |-- sso_hwgrp_iaq_walk
316 |-- sso_hwgrp_free_list_walk
317 |-- sso_hwgrp_ient_walk
318 '-- sso_hwgrp_taq_walk
320 RVU block LF allocation:
322 .. code-block:: console
324 cat /sys/kernel/debug/cn10k/rsrc_alloc
326 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
333 .. code-block:: console
335 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats
337 =======Link Status======
339 Link is UP 25000 Mbps
341 =======NIX RX_STATS(rpm port level)======
351 =======NIX TX_STATS(rpm port level)======
360 =======rpm RX_STATS======
362 Octets of received packets: 0
363 Octets of received packets with out error: 0
364 Received packets with alignment errors: 0
365 Control/PAUSE packets received: 0
366 Packets received with Frame too long Errors: 0
367 Packets received with a1nrange length Errors: 0
369 Packets received with FrameCheckSequenceErrors: 0
370 Packets received with VLAN header: 0
372 Packets received with unicast DMAC: 0
373 Packets received with multicast DMAC: 0
374 Packets received with broadcast DMAC: 0
376 Total frames received on interface: 0
377 Packets received with an octet count < 64: 0
378 Packets received with an octet count == 64: 0
379 Packets received with an octet count of 65–127: 0
380 Packets received with an octet count of 128-255: 0
381 Packets received with an octet count of 256-511: 0
382 Packets received with an octet count of 512-1023: 0
383 Packets received with an octet count of 1024-1518: 0
384 Packets received with an octet count of > 1518: 0
387 Fragmented Packets: 0
388 CBFC(class based flow control) pause frames received for class 0: 0
389 CBFC pause frames received for class 1: 0
390 CBFC pause frames received for class 2: 0
391 CBFC pause frames received for class 3: 0
392 CBFC pause frames received for class 4: 0
393 CBFC pause frames received for class 5: 0
394 CBFC pause frames received for class 6: 0
395 CBFC pause frames received for class 7: 0
396 CBFC pause frames received for class 8: 0
397 CBFC pause frames received for class 9: 0
398 CBFC pause frames received for class 10: 0
399 CBFC pause frames received for class 11: 0
400 CBFC pause frames received for class 12: 0
401 CBFC pause frames received for class 13: 0
402 CBFC pause frames received for class 14: 0
403 CBFC pause frames received for class 15: 0
404 MAC control packets received: 0
406 =======rpm TX_STATS======
408 Total octets sent on the interface: 0
409 Total octets transmitted OK: 0
410 Control/Pause frames sent: 0
411 Total frames transmitted OK: 0
412 Total frames sent with VLAN header: 0
414 Packets sent to unicast DMAC: 0
415 Packets sent to the multicast DMAC: 0
416 Packets sent to a broadcast DMAC: 0
417 Packets sent with an octet count == 64: 0
418 Packets sent with an octet count of 65–127: 0
419 Packets sent with an octet count of 128-255: 0
420 Packets sent with an octet count of 256-511: 0
421 Packets sent with an octet count of 512-1023: 0
422 Packets sent with an octet count of 1024-1518: 0
423 Packets sent with an octet count of > 1518: 0
424 CBFC(class based flow control) pause frames transmitted for class 0: 0
425 CBFC pause frames transmitted for class 1: 0
426 CBFC pause frames transmitted for class 2: 0
427 CBFC pause frames transmitted for class 3: 0
428 CBFC pause frames transmitted for class 4: 0
429 CBFC pause frames transmitted for class 5: 0
430 CBFC pause frames transmitted for class 6: 0
431 CBFC pause frames transmitted for class 7: 0
432 CBFC pause frames transmitted for class 8: 0
433 CBFC pause frames transmitted for class 9: 0
434 CBFC pause frames transmitted for class 10: 0
435 CBFC pause frames transmitted for class 11: 0
436 CBFC pause frames transmitted for class 12: 0
437 CBFC pause frames transmitted for class 13: 0
438 CBFC pause frames transmitted for class 14: 0
439 CBFC pause frames transmitted for class 15: 0
440 MAC control packets sent: 0
441 Total frames sent on the interface: 0
445 .. code-block:: console
447 cat /sys/kernel/debug/cn10k/cpt/cpt_pc
449 CPT instruction requests 0
450 CPT instruction latency 0
451 CPT NCB read requests 0
452 CPT NCB read latency 0
453 CPT read requests caused by UC fills 0
454 CPT active cycles pc 1395642
455 CPT clock count pc 5579867595493
459 .. code-block:: console
461 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx
462 cat /sys/kernel/debug/cn10k/nix/cq_ctx
463 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx
464 cat /sys/kernel/debug/cn10k/nix/cq_ctx
466 =====cq_ctx for nixlf:0 and qidx:0 is=====
477 W2: update_time 31043
494 .. code-block:: console
496 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx
497 cat /sys/kernel/debug/cn10k/npa/pool_ctx
498 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx
499 cat /sys/kernel/debug/cn10k/npa/pool_ctx
501 ======POOL : 0=======
502 W0: Stack base 1375bff00
509 W2: stack_max_pages 24315
510 W2: stack_pages 24314
520 W4: update_time 62993
522 W6: ptr_start 1593adf00
523 W7: ptr_end 180000000
529 W8: thresh_qint_idx 0
534 .. code-block:: console
536 cat /sys/kernel/debug/cn10k/npc/mcam_info
539 RX keywidth : 224bits
540 TX keywidth : 224bits
552 .. code-block:: console
554 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
555 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
557 ==================================================
558 SSOW HWS[0] Arbitration State 0x0
559 SSOW HWS[0] Guest Machine Control 0x0
560 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
561 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
562 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
563 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
564 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
565 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
566 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
567 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
568 ==================================================
573 DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on
574 an x86 based platform.
579 .. code-block:: console
587 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
591 .. code-block:: console
593 meson build --cross-file config/arm/arm64_cn9k_linux_gcc
598 .. code-block:: console
600 meson build --cross-file config/arm/arm64_cn10k_linux_gcc
605 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
606 if Marvell toolchain is available then it can be used by overriding the
607 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
608 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file.