1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2019 Marvell International Ltd.
4 Marvell OCTEON TX2 Platform Guide
5 =================================
7 This document gives an overview of **Marvell OCTEON TX2** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON TX2 platform.
10 More information about OCTEON TX2 SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON TX2 SoCs
14 -------------------------
19 OCTEON TX2 Resource Virtualization Unit architecture
20 ----------------------------------------------------
22 The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the
23 RVU architecture and a resource provisioning example.
25 .. _figure_octeontx2_resource_virtualization:
27 .. figure:: img/octeontx2_resource_virtualization.*
29 OCTEON TX2 Resource virtualization architecture and provisioning example
32 Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW
33 resources belonging to the network, crypto and other functional blocks onto
34 PCI-compatible physical and virtual functions.
36 Each functional block has multiple local functions (LFs) for
37 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
38 physical functions (PFs) and virtual functions (VFs).
40 The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local
41 functions (LFs) provided by the RVU and its functional mapping to
44 .. _table_octeontx2_rvu_dpdk_mapping:
46 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
48 +---+-----+--------------------------------------------------------------+
49 | # | LF | DPDK subsystem mapping |
50 +===+=====+==============================================================+
51 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
52 +---+-----+--------------------------------------------------------------+
53 | 2 | NPA | rte_mempool |
54 +---+-----+--------------------------------------------------------------+
55 | 3 | NPC | rte_flow |
56 +---+-----+--------------------------------------------------------------+
57 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
58 +---+-----+--------------------------------------------------------------+
59 | 5 | SSO | rte_eventdev |
60 +---+-----+--------------------------------------------------------------+
61 | 6 | TIM | rte_event_timer_adapter |
62 +---+-----+--------------------------------------------------------------+
64 PF0 is called the administrative / admin function (AF) and has exclusive
65 privileges to provision RVU functional block's LFs to each of the PF/VF.
67 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
68 requests from PF/VF, AF does resource provisioning and other HW configuration.
70 AF is always attached to host, but PF/VFs may be used by host kernel itself,
71 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
72 handle provisioning/configuration requests sent by any device from any domain.
74 The AF driver does not receive or process any data.
75 It is only a configuration driver used in control path.
77 The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a
78 resource provisioning example where,
80 1. PFx and PFx-VF0 bound to Linux netdev driver.
81 2. PFx-VF1 ethdev driver bound to the first DPDK application.
82 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
84 OCTEON TX2 packet flow
85 ----------------------
87 The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts
88 the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators.
90 .. _figure_octeontx2_packet_flow_hw_accelerators:
92 .. figure:: img/octeontx2_packet_flow_hw_accelerators.*
94 OCTEON TX2 packet flow in conjunction with use of HW accelerators
99 This section lists dataplane H/W block(s) available in OCTEON TX2 SoC.
101 #. **Mempool Driver**
102 See :doc:`../mempool/octeontx2` for NPA mempool driver information.
104 #. **Event Device Driver**
105 See :doc:`../eventdevs/octeontx2` for SSO event device driver information.
107 Procedure to Setup Platform
108 ---------------------------
110 There are three main prerequisites for setting up DPDK on OCTEON TX2
113 1. **OCTEON TX2 Linux kernel driver**
115 The dependent kernel drivers can be obtained from the
116 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
118 Alternatively, the Marvell SDK also provides the required kernel drivers.
120 Linux kernel should be configured with the following features enabled:
122 .. code-block:: console
124 # 64K pages enabled for better performance
125 CONFIG_ARM64_64K_PAGES=y
126 CONFIG_ARM64_VA_BITS_48=y
127 # huge pages support enabled
129 CONFIG_HUGETLB_PAGE=y
130 # VFIO enabled with TYPE1 IOMMU at minimum
131 CONFIG_VFIO_IOMMU_TYPE1=y
134 CONFIG_VFIO_NOIOMMU=y
136 CONFIG_VFIO_PCI_MMAP=y
139 # ARMv8.1 LSE atomics
140 CONFIG_ARM64_LSE_ATOMICS=y
142 CONFIG_OCTEONTX2_MBOX=y
143 CONFIG_OCTEONTX2_AF=y
144 # Enable if netdev PF driver required
145 CONFIG_OCTEONTX2_PF=y
146 # Enable if netdev VF driver required
147 CONFIG_OCTEONTX2_VF=y
148 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
150 2. **ARM64 Linux Tool Chain**
152 For example, the *aarch64* Linaro Toolchain, which can be obtained from
153 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
155 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
156 optimized for OCTEON TX2 CPU.
158 3. **Rootfile system**
160 Any *aarch64* supporting filesystem may be used. For example,
161 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
162 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
164 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
165 The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board.
167 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
173 .. _table_octeontx2_common_debug_options:
175 .. table:: OCTEON TX2 common debug options
177 +---+------------+-------------------------------------------------------+
178 | # | Component | EAL log command |
179 +===+============+=======================================================+
180 | 1 | Common | --log-level='pmd\.octeontx2\.base,8' |
181 +---+------------+-------------------------------------------------------+
182 | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' |
183 +---+------------+-------------------------------------------------------+
188 The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks
189 context or stats using debugfs.
191 Enable ``debugfs`` by:
193 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
194 2. Boot OCTEON TX2 with debugfs supported kernel.
195 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
197 .. code-block:: console
199 # mount -t debugfs none /sys/kernel/debug
201 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
204 The file structure under ``/sys/kernel/debug`` is as follows
206 .. code-block:: console
222 │ ├── cpt_engines_info
223 │ ├── cpt_engines_sts
230 │ ├── ndc_rx_hits_miss
232 │ ├── ndc_tx_hits_miss
236 │ └── tx_stall_hwissue
245 │ └── rx_miss_act_stats
251 ├── sso_hwgrp_aq_thresh
252 ├── sso_hwgrp_iaq_walk
254 ├── sso_hwgrp_free_list_walk
255 ├── sso_hwgrp_ient_walk
256 └── sso_hwgrp_taq_walk
258 RVU block LF allocation:
260 .. code-block:: console
262 cat /sys/kernel/debug/octeontx2/rsrc_alloc
264 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
271 .. code-block:: console
273 cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats
275 =======Link Status======
276 Link is UP 40000 Mbps
277 =======RX_STATS======
279 Octets of received packets: 0
280 Received PAUSE packets: 0
281 Received PAUSE and control packets: 0
282 Filtered DMAC0 (NIX-bound) packets: 0
283 Filtered DMAC0 (NIX-bound) octets: 0
284 Packets dropped due to RX FIFO full: 0
285 Octets dropped due to RX FIFO full: 0
287 Filtered DMAC1 (NCSI-bound) packets: 0
288 Filtered DMAC1 (NCSI-bound) octets: 0
289 NCSI-bound packets dropped: 0
290 NCSI-bound octets dropped: 0
291 =======TX_STATS======
292 Packets dropped due to excessive collisions: 0
293 Packets dropped due to excessive deferral: 0
294 Multiple collisions before successful transmission: 0
295 Single collisions before successful transmission: 0
296 Total octets sent on the interface: 0
297 Total frames sent on the interface: 0
298 Packets sent with an octet count < 64: 0
299 Packets sent with an octet count == 64: 0
300 Packets sent with an octet count of 65127: 0
301 Packets sent with an octet count of 128-255: 0
302 Packets sent with an octet count of 256-511: 0
303 Packets sent with an octet count of 512-1023: 0
304 Packets sent with an octet count of 1024-1518: 0
305 Packets sent with an octet count of > 1518: 0
306 Packets sent to a broadcast DMAC: 0
307 Packets sent to the multicast DMAC: 0
308 Transmit underflow and were truncated: 0
309 Control/PAUSE packets sent: 0
313 .. code-block:: console
315 cat /sys/kernel/debug/octeontx2/cpt/cpt_pc
317 CPT instruction requests 0
318 CPT instruction latency 0
319 CPT NCB read requests 0
320 CPT NCB read latency 0
321 CPT read requests caused by UC fills 0
322 CPT active cycles pc 1395642
323 CPT clock count pc 5579867595493
327 .. code-block:: console
329 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx
330 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
331 echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx
332 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
334 =====cq_ctx for nixlf:0 and qidx:0 is=====
345 W2: update_time 31043
362 .. code-block:: console
364 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx
365 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
366 echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx
367 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
369 ======POOL : 0=======
370 W0: Stack base 1375bff00
377 W2: stack_max_pages 24315
378 W2: stack_pages 24314
388 W4: update_time 62993
390 W6: ptr_start 1593adf00
391 W7: ptr_end 180000000
397 W8: thresh_qint_idx 0
402 .. code-block:: console
404 cat /sys/kernel/debug/octeontx2/npc/mcam_info
407 RX keywidth : 224bits
408 TX keywidth : 224bits
420 .. code-block:: console
422 Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
423 echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
425 ==================================================
426 SSOW HWS[0] Arbitration State 0x0
427 SSOW HWS[0] Guest Machine Control 0x0
428 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
429 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
430 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
431 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
432 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
433 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
434 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
435 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
436 ==================================================
441 DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on
442 an x86 based platform.
450 .. code-block:: console
452 make config T=arm64-octeontx2-linux-gcc
455 The example applications can be compiled using the following:
457 .. code-block:: console
461 export RTE_TARGET=build
462 cd examples/<application>
468 .. code-block:: console
476 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
481 .. code-block:: console
483 make config T=arm64-octeontx2-linux-gcc
484 make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n
489 .. code-block:: console
491 meson build --cross-file config/arm/arm64_octeontx2_linux_gcc
496 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
497 if Marvell toolchain is available then it can be used by overriding the
498 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
499 toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.