1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2019 Marvell International Ltd.
4 Marvell OCTEON TX2 Platform Guide
5 =================================
7 This document gives an overview of **Marvell OCTEON TX2** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON TX2 platform.
10 More information about OCTEON TX2 SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON TX2 SoCs
14 -------------------------
19 OCTEON TX2 Resource Virtualization Unit architecture
20 ----------------------------------------------------
22 The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the
23 RVU architecture and a resource provisioning example.
25 .. _figure_octeontx2_resource_virtualization:
27 .. figure:: img/octeontx2_resource_virtualization.*
29 OCTEON TX2 Resource virtualization architecture and provisioning example
32 Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW
33 resources belonging to the network, crypto and other functional blocks onto
34 PCI-compatible physical and virtual functions.
36 Each functional block has multiple local functions (LFs) for
37 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
38 physical functions (PFs) and virtual functions (VFs).
40 The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local
41 functions (LFs) provided by the RVU and its functional mapping to
44 .. _table_octeontx2_rvu_dpdk_mapping:
46 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
48 +---+-----+--------------------------------------------------------------+
49 | # | LF | DPDK subsystem mapping |
50 +===+=====+==============================================================+
51 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
52 +---+-----+--------------------------------------------------------------+
53 | 2 | NPA | rte_mempool |
54 +---+-----+--------------------------------------------------------------+
55 | 3 | NPC | rte_flow |
56 +---+-----+--------------------------------------------------------------+
57 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
58 +---+-----+--------------------------------------------------------------+
59 | 5 | SSO | rte_eventdev |
60 +---+-----+--------------------------------------------------------------+
61 | 6 | TIM | rte_event_timer_adapter |
62 +---+-----+--------------------------------------------------------------+
63 | 7 | LBK | rte_ethdev |
64 +---+-----+--------------------------------------------------------------+
65 | 8 | DPI | rte_rawdev |
66 +---+-----+--------------------------------------------------------------+
68 PF0 is called the administrative / admin function (AF) and has exclusive
69 privileges to provision RVU functional block's LFs to each of the PF/VF.
71 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
72 requests from PF/VF, AF does resource provisioning and other HW configuration.
74 AF is always attached to host, but PF/VFs may be used by host kernel itself,
75 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
76 handle provisioning/configuration requests sent by any device from any domain.
78 The AF driver does not receive or process any data.
79 It is only a configuration driver used in control path.
81 The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a
82 resource provisioning example where,
84 1. PFx and PFx-VF0 bound to Linux netdev driver.
85 2. PFx-VF1 ethdev driver bound to the first DPDK application.
86 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
91 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
92 The loopback block has N channels and contains data buffering that is shared across
93 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
94 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
95 that is, packets sent on odd VF end up received on even VF and vice versa.
96 This would enable HW accelerated means of communication between two domains
97 where even VF bound to the first domain and odd VF bound to the second domain.
99 Typical application usage models are,
101 #. Communication between the Linux kernel and DPDK application.
102 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
103 #. Communication between two different DPDK applications.
105 OCTEON TX2 packet flow
106 ----------------------
108 The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts
109 the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators.
111 .. _figure_octeontx2_packet_flow_hw_accelerators:
113 .. figure:: img/octeontx2_packet_flow_hw_accelerators.*
115 OCTEON TX2 packet flow in conjunction with use of HW accelerators
120 This section lists dataplane H/W block(s) available in OCTEON TX2 SoC.
123 See :doc:`../nics/octeontx2` for NIX Ethdev driver information.
125 #. **Mempool Driver**
126 See :doc:`../mempool/octeontx2` for NPA mempool driver information.
128 #. **Event Device Driver**
129 See :doc:`../eventdevs/octeontx2` for SSO event device driver information.
131 #. **DMA Rawdev Driver**
132 See :doc:`../rawdevs/octeontx2_dma` for DMA driver information.
134 #. **Crypto Device Driver**
135 See :doc:`../cryptodevs/octeontx2` for CPT crypto device driver information.
137 Procedure to Setup Platform
138 ---------------------------
140 There are three main prerequisites for setting up DPDK on OCTEON TX2
143 1. **OCTEON TX2 Linux kernel driver**
145 The dependent kernel drivers can be obtained from the
146 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
148 Alternatively, the Marvell SDK also provides the required kernel drivers.
150 Linux kernel should be configured with the following features enabled:
152 .. code-block:: console
154 # 64K pages enabled for better performance
155 CONFIG_ARM64_64K_PAGES=y
156 CONFIG_ARM64_VA_BITS_48=y
157 # huge pages support enabled
159 CONFIG_HUGETLB_PAGE=y
160 # VFIO enabled with TYPE1 IOMMU at minimum
161 CONFIG_VFIO_IOMMU_TYPE1=y
164 CONFIG_VFIO_NOIOMMU=y
166 CONFIG_VFIO_PCI_MMAP=y
169 # ARMv8.1 LSE atomics
170 CONFIG_ARM64_LSE_ATOMICS=y
172 CONFIG_OCTEONTX2_MBOX=y
173 CONFIG_OCTEONTX2_AF=y
174 # Enable if netdev PF driver required
175 CONFIG_OCTEONTX2_PF=y
176 # Enable if netdev VF driver required
177 CONFIG_OCTEONTX2_VF=y
178 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
179 # Enable if OCTEONTX2 DMA PF driver required
180 CONFIG_OCTEONTX2_DPI_PF=n
182 2. **ARM64 Linux Tool Chain**
184 For example, the *aarch64* Linaro Toolchain, which can be obtained from
185 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
187 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
188 optimized for OCTEON TX2 CPU.
190 3. **Rootfile system**
192 Any *aarch64* supporting filesystem may be used. For example,
193 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
194 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
196 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
197 The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board.
199 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
205 .. _table_octeontx2_common_debug_options:
207 .. table:: OCTEON TX2 common debug options
209 +---+------------+-------------------------------------------------------+
210 | # | Component | EAL log command |
211 +===+============+=======================================================+
212 | 1 | Common | --log-level='pmd\.octeontx2\.base,8' |
213 +---+------------+-------------------------------------------------------+
214 | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' |
215 +---+------------+-------------------------------------------------------+
220 The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks
221 context or stats using debugfs.
223 Enable ``debugfs`` by:
225 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
226 2. Boot OCTEON TX2 with debugfs supported kernel.
227 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
229 .. code-block:: console
231 # mount -t debugfs none /sys/kernel/debug
233 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
236 The file structure under ``/sys/kernel/debug`` is as follows
238 .. code-block:: console
254 | |-- cpt_engines_info
255 | |-- cpt_engines_sts
262 | |-- ndc_rx_hits_miss
264 | |-- ndc_tx_hits_miss
268 | '-- tx_stall_hwissue
277 | '-- rx_miss_act_stats
283 |-- sso_hwgrp_aq_thresh
284 |-- sso_hwgrp_iaq_walk
286 |-- sso_hwgrp_free_list_walk
287 |-- sso_hwgrp_ient_walk
288 '-- sso_hwgrp_taq_walk
290 RVU block LF allocation:
292 .. code-block:: console
294 cat /sys/kernel/debug/octeontx2/rsrc_alloc
296 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
303 .. code-block:: console
305 cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats
307 =======Link Status======
308 Link is UP 40000 Mbps
309 =======RX_STATS======
311 Octets of received packets: 0
312 Received PAUSE packets: 0
313 Received PAUSE and control packets: 0
314 Filtered DMAC0 (NIX-bound) packets: 0
315 Filtered DMAC0 (NIX-bound) octets: 0
316 Packets dropped due to RX FIFO full: 0
317 Octets dropped due to RX FIFO full: 0
319 Filtered DMAC1 (NCSI-bound) packets: 0
320 Filtered DMAC1 (NCSI-bound) octets: 0
321 NCSI-bound packets dropped: 0
322 NCSI-bound octets dropped: 0
323 =======TX_STATS======
324 Packets dropped due to excessive collisions: 0
325 Packets dropped due to excessive deferral: 0
326 Multiple collisions before successful transmission: 0
327 Single collisions before successful transmission: 0
328 Total octets sent on the interface: 0
329 Total frames sent on the interface: 0
330 Packets sent with an octet count < 64: 0
331 Packets sent with an octet count == 64: 0
332 Packets sent with an octet count of 65127: 0
333 Packets sent with an octet count of 128-255: 0
334 Packets sent with an octet count of 256-511: 0
335 Packets sent with an octet count of 512-1023: 0
336 Packets sent with an octet count of 1024-1518: 0
337 Packets sent with an octet count of > 1518: 0
338 Packets sent to a broadcast DMAC: 0
339 Packets sent to the multicast DMAC: 0
340 Transmit underflow and were truncated: 0
341 Control/PAUSE packets sent: 0
345 .. code-block:: console
347 cat /sys/kernel/debug/octeontx2/cpt/cpt_pc
349 CPT instruction requests 0
350 CPT instruction latency 0
351 CPT NCB read requests 0
352 CPT NCB read latency 0
353 CPT read requests caused by UC fills 0
354 CPT active cycles pc 1395642
355 CPT clock count pc 5579867595493
359 .. code-block:: console
361 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx
362 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
363 echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx
364 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
366 =====cq_ctx for nixlf:0 and qidx:0 is=====
377 W2: update_time 31043
394 .. code-block:: console
396 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx
397 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
398 echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx
399 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
401 ======POOL : 0=======
402 W0: Stack base 1375bff00
409 W2: stack_max_pages 24315
410 W2: stack_pages 24314
420 W4: update_time 62993
422 W6: ptr_start 1593adf00
423 W7: ptr_end 180000000
429 W8: thresh_qint_idx 0
434 .. code-block:: console
436 cat /sys/kernel/debug/octeontx2/npc/mcam_info
439 RX keywidth : 224bits
440 TX keywidth : 224bits
452 .. code-block:: console
454 Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
455 echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
457 ==================================================
458 SSOW HWS[0] Arbitration State 0x0
459 SSOW HWS[0] Guest Machine Control 0x0
460 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
461 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
462 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
463 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
464 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
465 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
466 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
467 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
468 ==================================================
473 DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on
474 an x86 based platform.
482 .. code-block:: console
484 make config T=arm64-octeontx2-linux-gcc
487 The example applications can be compiled using the following:
489 .. code-block:: console
493 export RTE_TARGET=build
494 cd examples/<application>
500 .. code-block:: console
508 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
513 .. code-block:: console
515 make config T=arm64-octeontx2-linux-gcc
516 make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n
521 .. code-block:: console
523 meson build --cross-file config/arm/arm64_octeontx2_linux_gcc
528 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
529 if Marvell toolchain is available then it can be used by overriding the
530 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
531 toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.