1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2021 Marvell.
4 Marvell cnxk platform guide
5 ===========================
7 This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON cnxk platform.
10 More information about CN9K and CN10K SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON cnxk SoCs
14 --------------------------
22 Resource Virtualization Unit architecture
23 -----------------------------------------
25 The :numref:`figure_cnxk_resource_virtualization` diagram depicts the
26 RVU architecture and a resource provisioning example.
28 .. _figure_cnxk_resource_virtualization:
30 .. figure:: img/cnxk_resource_virtualization.*
32 cnxk Resource virtualization architecture and provisioning example
35 Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW
36 resources belonging to the network, crypto and other functional blocks onto
37 PCI-compatible physical and virtual functions.
39 Each functional block has multiple local functions (LFs) for
40 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
41 physical functions (PFs) and virtual functions (VFs).
43 The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local
44 functions (LFs) provided by the RVU and its functional mapping to
47 .. _table_cnxk_rvu_dpdk_mapping:
49 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
51 +---+-----+--------------------------------------------------------------+
52 | # | LF | DPDK subsystem mapping |
53 +===+=====+==============================================================+
54 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
55 +---+-----+--------------------------------------------------------------+
56 | 2 | NPA | rte_mempool |
57 +---+-----+--------------------------------------------------------------+
58 | 3 | NPC | rte_flow |
59 +---+-----+--------------------------------------------------------------+
60 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
61 +---+-----+--------------------------------------------------------------+
62 | 5 | SSO | rte_eventdev |
63 +---+-----+--------------------------------------------------------------+
64 | 6 | TIM | rte_event_timer_adapter |
65 +---+-----+--------------------------------------------------------------+
66 | 7 | LBK | rte_ethdev |
67 +---+-----+--------------------------------------------------------------+
68 | 8 | DPI | rte_dmadev |
69 +---+-----+--------------------------------------------------------------+
70 | 9 | SDP | rte_ethdev |
71 +---+-----+--------------------------------------------------------------+
72 | 10| REE | rte_regexdev |
73 +---+-----+--------------------------------------------------------------+
74 | 11| BPHY| rte_rawdev |
75 +---+-----+--------------------------------------------------------------+
76 | 12| GPIO| rte_rawdev |
77 +---+-----+--------------------------------------------------------------+
79 PF0 is called the administrative / admin function (AF) and has exclusive
80 privileges to provision RVU functional block's LFs to each of the PF/VF.
82 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
83 requests from PF/VF, AF does resource provisioning and other HW configuration.
85 AF is always attached to host, but PF/VFs may be used by host kernel itself,
86 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
87 handle provisioning/configuration requests sent by any device from any domain.
89 The AF driver does not receive or process any data.
90 It is only a configuration driver used in control path.
92 The :numref:`figure_cnxk_resource_virtualization` diagram also shows a
93 resource provisioning example where,
95 1. PFx and PFx-VF0 bound to Linux netdev driver.
96 2. PFx-VF1 ethdev driver bound to the first DPDK application.
97 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
102 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
103 The loopback block has N channels and contains data buffering that is shared across
104 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
105 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
106 that is, packets sent on odd VF end up received on even VF and vice versa.
107 This would enable HW accelerated means of communication between two domains
108 where even VF bound to the first domain and odd VF bound to the second domain.
110 Typical application usage models are,
112 #. Communication between the Linux kernel and DPDK application.
113 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
114 #. Communication between two different DPDK applications.
119 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
120 to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when
121 cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive
122 packets to/from remote host machine using input/output queue pairs exposed to it.
123 SDP interface receives input packets from remote host from NIX-RX and sends packets
124 to remote host using NIX-TX. Remote host machine need to use corresponding driver
125 (kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports
126 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
127 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
129 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
131 #. Communication channel between remote host and cnxk SoC over PCIe.
132 #. Transfer packets received from network interface to remote host over PCIe and
136 ----------------------
138 The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts
139 the packet flow on cnxk SoC in conjunction with use of various HW accelerators.
141 .. _figure_cnxk_packet_flow_hw_accelerators:
143 .. figure:: img/cnxk_packet_flow_hw_accelerators.*
145 cnxk packet flow in conjunction with use of HW accelerators
150 This section lists dataplane H/W block(s) available in cnxk SoC.
153 See :doc:`../nics/cnxk` for NIX Ethdev driver information.
155 #. **Mempool Driver**
156 See :doc:`../mempool/cnxk` for NPA mempool driver information.
158 #. **Baseband PHY Driver**
159 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information.
162 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information.
164 #. **Regex Device Driver**
165 See :doc:`../regexdevs/cn9k` for REE Regex device driver information.
167 Procedure to Setup Platform
168 ---------------------------
170 There are three main prerequisites for setting up DPDK on cnxk
173 1. **RVU AF Linux kernel driver**
175 The dependent kernel drivers can be obtained from the
176 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
178 Alternatively, the Marvell SDK also provides the required kernel drivers.
180 Linux kernel should be configured with the following features enabled:
182 .. code-block:: console
184 # 64K pages enabled for better performance
185 CONFIG_ARM64_64K_PAGES=y
186 CONFIG_ARM64_VA_BITS_48=y
187 # huge pages support enabled
189 CONFIG_HUGETLB_PAGE=y
190 # VFIO enabled with TYPE1 IOMMU at minimum
191 CONFIG_VFIO_IOMMU_TYPE1=y
194 CONFIG_VFIO_NOIOMMU=y
196 CONFIG_VFIO_PCI_MMAP=y
199 # ARMv8.1 LSE atomics
200 CONFIG_ARM64_LSE_ATOMICS=y
202 CONFIG_OCTEONTX2_MBOX=y
203 CONFIG_OCTEONTX2_AF=y
204 # Enable if netdev PF driver required
205 CONFIG_OCTEONTX2_PF=y
206 # Enable if netdev VF driver required
207 CONFIG_OCTEONTX2_VF=y
208 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
209 # Enable if OCTEONTX2 DMA PF driver required
210 CONFIG_OCTEONTX2_DPI_PF=n
212 2. **ARM64 Linux Tool Chain**
214 For example, the *aarch64* Linaro Toolchain, which can be obtained from
215 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
217 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
218 optimized for cnxk CPU.
220 3. **Rootfile system**
222 Any *aarch64* supporting filesystem may be used. For example,
223 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
224 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
226 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
227 The SDK includes all the above prerequisites necessary to bring up the cnxk board.
229 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
235 .. _table_cnxk_common_debug_options:
237 .. table:: cnxk common debug options
239 +---+------------+-------------------------------------------------------+
240 | # | Component | EAL log command |
241 +===+============+=======================================================+
242 | 1 | Common | --log-level='pmd\.cnxk\.base,8' |
243 +---+------------+-------------------------------------------------------+
244 | 2 | Mailbox | --log-level='pmd\.cnxk\.mbox,8' |
245 +---+------------+-------------------------------------------------------+
250 The **RVU AF Linux kernel driver** provides support to dump RVU blocks
251 context or stats using debugfs.
253 Enable ``debugfs`` by:
255 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
256 2. Boot OCTEON CN9K/CN10K with debugfs supported kernel.
257 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
259 .. code-block:: console
261 # mount -t debugfs none /sys/kernel/debug
263 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
266 The file structure under ``/sys/kernel/debug`` is as follows
268 .. code-block:: console
286 | |-- cpt_engines_info
287 | |-- cpt_engines_sts
294 | |-- ndc_rx_hits_miss
296 | |-- ndc_tx_hits_miss
309 | '-- rx_miss_act_stats
315 |-- sso_hwgrp_aq_thresh
316 |-- sso_hwgrp_iaq_walk
318 |-- sso_hwgrp_free_list_walk
319 |-- sso_hwgrp_ient_walk
320 '-- sso_hwgrp_taq_walk
322 RVU block LF allocation:
324 .. code-block:: console
326 cat /sys/kernel/debug/cn10k/rsrc_alloc
328 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
335 .. code-block:: console
337 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats
339 =======Link Status======
341 Link is UP 25000 Mbps
343 =======NIX RX_STATS(rpm port level)======
353 =======NIX TX_STATS(rpm port level)======
362 =======rpm RX_STATS======
364 Octets of received packets: 0
365 Octets of received packets with out error: 0
366 Received packets with alignment errors: 0
367 Control/PAUSE packets received: 0
368 Packets received with Frame too long Errors: 0
369 Packets received with a1nrange length Errors: 0
371 Packets received with FrameCheckSequenceErrors: 0
372 Packets received with VLAN header: 0
374 Packets received with unicast DMAC: 0
375 Packets received with multicast DMAC: 0
376 Packets received with broadcast DMAC: 0
378 Total frames received on interface: 0
379 Packets received with an octet count < 64: 0
380 Packets received with an octet count == 64: 0
381 Packets received with an octet count of 65–127: 0
382 Packets received with an octet count of 128-255: 0
383 Packets received with an octet count of 256-511: 0
384 Packets received with an octet count of 512-1023: 0
385 Packets received with an octet count of 1024-1518: 0
386 Packets received with an octet count of > 1518: 0
389 Fragmented Packets: 0
390 CBFC(class based flow control) pause frames received for class 0: 0
391 CBFC pause frames received for class 1: 0
392 CBFC pause frames received for class 2: 0
393 CBFC pause frames received for class 3: 0
394 CBFC pause frames received for class 4: 0
395 CBFC pause frames received for class 5: 0
396 CBFC pause frames received for class 6: 0
397 CBFC pause frames received for class 7: 0
398 CBFC pause frames received for class 8: 0
399 CBFC pause frames received for class 9: 0
400 CBFC pause frames received for class 10: 0
401 CBFC pause frames received for class 11: 0
402 CBFC pause frames received for class 12: 0
403 CBFC pause frames received for class 13: 0
404 CBFC pause frames received for class 14: 0
405 CBFC pause frames received for class 15: 0
406 MAC control packets received: 0
408 =======rpm TX_STATS======
410 Total octets sent on the interface: 0
411 Total octets transmitted OK: 0
412 Control/Pause frames sent: 0
413 Total frames transmitted OK: 0
414 Total frames sent with VLAN header: 0
416 Packets sent to unicast DMAC: 0
417 Packets sent to the multicast DMAC: 0
418 Packets sent to a broadcast DMAC: 0
419 Packets sent with an octet count == 64: 0
420 Packets sent with an octet count of 65–127: 0
421 Packets sent with an octet count of 128-255: 0
422 Packets sent with an octet count of 256-511: 0
423 Packets sent with an octet count of 512-1023: 0
424 Packets sent with an octet count of 1024-1518: 0
425 Packets sent with an octet count of > 1518: 0
426 CBFC(class based flow control) pause frames transmitted for class 0: 0
427 CBFC pause frames transmitted for class 1: 0
428 CBFC pause frames transmitted for class 2: 0
429 CBFC pause frames transmitted for class 3: 0
430 CBFC pause frames transmitted for class 4: 0
431 CBFC pause frames transmitted for class 5: 0
432 CBFC pause frames transmitted for class 6: 0
433 CBFC pause frames transmitted for class 7: 0
434 CBFC pause frames transmitted for class 8: 0
435 CBFC pause frames transmitted for class 9: 0
436 CBFC pause frames transmitted for class 10: 0
437 CBFC pause frames transmitted for class 11: 0
438 CBFC pause frames transmitted for class 12: 0
439 CBFC pause frames transmitted for class 13: 0
440 CBFC pause frames transmitted for class 14: 0
441 CBFC pause frames transmitted for class 15: 0
442 MAC control packets sent: 0
443 Total frames sent on the interface: 0
447 .. code-block:: console
449 cat /sys/kernel/debug/cn10k/cpt/cpt_pc
451 CPT instruction requests 0
452 CPT instruction latency 0
453 CPT NCB read requests 0
454 CPT NCB read latency 0
455 CPT read requests caused by UC fills 0
456 CPT active cycles pc 1395642
457 CPT clock count pc 5579867595493
461 .. code-block:: console
463 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx
464 cat /sys/kernel/debug/cn10k/nix/cq_ctx
465 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx
466 cat /sys/kernel/debug/cn10k/nix/cq_ctx
468 =====cq_ctx for nixlf:0 and qidx:0 is=====
479 W2: update_time 31043
496 .. code-block:: console
498 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx
499 cat /sys/kernel/debug/cn10k/npa/pool_ctx
500 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx
501 cat /sys/kernel/debug/cn10k/npa/pool_ctx
503 ======POOL : 0=======
504 W0: Stack base 1375bff00
511 W2: stack_max_pages 24315
512 W2: stack_pages 24314
522 W4: update_time 62993
524 W6: ptr_start 1593adf00
525 W7: ptr_end 180000000
531 W8: thresh_qint_idx 0
536 .. code-block:: console
538 cat /sys/kernel/debug/cn10k/npc/mcam_info
541 RX keywidth : 224bits
542 TX keywidth : 224bits
554 .. code-block:: console
556 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
557 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info
559 ==================================================
560 SSOW HWS[0] Arbitration State 0x0
561 SSOW HWS[0] Guest Machine Control 0x0
562 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
563 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
564 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
565 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
566 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
567 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
568 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
569 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
570 ==================================================
575 DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on
576 an x86 based platform.
581 .. code-block:: console
589 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
593 .. code-block:: console
595 meson build --cross-file config/arm/arm64_cn9k_linux_gcc
600 .. code-block:: console
602 meson build --cross-file config/arm/arm64_cn10k_linux_gcc
607 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
608 if Marvell toolchain is available then it can be used by overriding the
609 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
610 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file.