1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2019 Marvell International Ltd.
4 Marvell OCTEON TX2 Platform Guide
5 =================================
7 This document gives an overview of **Marvell OCTEON TX2** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON TX2 platform.
10 More information about OCTEON TX2 SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON TX2 SoCs
14 -------------------------
20 OCTEON TX2 Resource Virtualization Unit architecture
21 ----------------------------------------------------
23 The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the
24 RVU architecture and a resource provisioning example.
26 .. _figure_octeontx2_resource_virtualization:
28 .. figure:: img/octeontx2_resource_virtualization.*
30 OCTEON TX2 Resource virtualization architecture and provisioning example
33 Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW
34 resources belonging to the network, crypto and other functional blocks onto
35 PCI-compatible physical and virtual functions.
37 Each functional block has multiple local functions (LFs) for
38 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
39 physical functions (PFs) and virtual functions (VFs).
41 The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local
42 functions (LFs) provided by the RVU and its functional mapping to
45 .. _table_octeontx2_rvu_dpdk_mapping:
47 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
49 +---+-----+--------------------------------------------------------------+
50 | # | LF | DPDK subsystem mapping |
51 +===+=====+==============================================================+
52 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
53 +---+-----+--------------------------------------------------------------+
54 | 2 | NPA | rte_mempool |
55 +---+-----+--------------------------------------------------------------+
56 | 3 | NPC | rte_flow |
57 +---+-----+--------------------------------------------------------------+
58 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
59 +---+-----+--------------------------------------------------------------+
60 | 5 | SSO | rte_eventdev |
61 +---+-----+--------------------------------------------------------------+
62 | 6 | TIM | rte_event_timer_adapter |
63 +---+-----+--------------------------------------------------------------+
64 | 7 | LBK | rte_ethdev |
65 +---+-----+--------------------------------------------------------------+
66 | 8 | DPI | rte_rawdev |
67 +---+-----+--------------------------------------------------------------+
68 | 9 | SDP | rte_ethdev |
69 +---+-----+--------------------------------------------------------------+
71 PF0 is called the administrative / admin function (AF) and has exclusive
72 privileges to provision RVU functional block's LFs to each of the PF/VF.
74 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
75 requests from PF/VF, AF does resource provisioning and other HW configuration.
77 AF is always attached to host, but PF/VFs may be used by host kernel itself,
78 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
79 handle provisioning/configuration requests sent by any device from any domain.
81 The AF driver does not receive or process any data.
82 It is only a configuration driver used in control path.
84 The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a
85 resource provisioning example where,
87 1. PFx and PFx-VF0 bound to Linux netdev driver.
88 2. PFx-VF1 ethdev driver bound to the first DPDK application.
89 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
94 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
95 The loopback block has N channels and contains data buffering that is shared across
96 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
97 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
98 that is, packets sent on odd VF end up received on even VF and vice versa.
99 This would enable HW accelerated means of communication between two domains
100 where even VF bound to the first domain and odd VF bound to the second domain.
102 Typical application usage models are,
104 #. Communication between the Linux kernel and DPDK application.
105 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
106 #. Communication between two different DPDK applications.
111 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
112 to DMA packets into and out of OCTEON TX2 SoC. SDP interface comes in to live only when
113 OCTEON TX2 SoC is connected in PCIe endpoint mode. It can be used to send/receive
114 packets to/from remote host machine using input/output queue pairs exposed to it.
115 SDP interface receives input packets from remote host from NIX-RX and sends packets
116 to remote host using NIX-TX. Remote host machine need to use corresponding driver
117 (kernel/user mode) to communicate with SDP interface on OCTEON TX2 SoC. SDP supports
118 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
119 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
121 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
123 #. Communication channel between remote host and OCTEON TX2 SoC over PCIe.
124 #. Transfer packets received from network interface to remote host over PCIe and
127 OCTEON TX2 packet flow
128 ----------------------
130 The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts
131 the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators.
133 .. _figure_octeontx2_packet_flow_hw_accelerators:
135 .. figure:: img/octeontx2_packet_flow_hw_accelerators.*
137 OCTEON TX2 packet flow in conjunction with use of HW accelerators
142 This section lists dataplane H/W block(s) available in OCTEON TX2 SoC.
145 See :doc:`../nics/octeontx2` for NIX Ethdev driver information.
147 #. **Mempool Driver**
148 See :doc:`../mempool/octeontx2` for NPA mempool driver information.
150 #. **Event Device Driver**
151 See :doc:`../eventdevs/octeontx2` for SSO event device driver information.
153 #. **DMA Rawdev Driver**
154 See :doc:`../rawdevs/octeontx2_dma` for DMA driver information.
156 #. **Crypto Device Driver**
157 See :doc:`../cryptodevs/octeontx2` for CPT crypto device driver information.
159 Procedure to Setup Platform
160 ---------------------------
162 There are three main prerequisites for setting up DPDK on OCTEON TX2
165 1. **OCTEON TX2 Linux kernel driver**
167 The dependent kernel drivers can be obtained from the
168 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
170 Alternatively, the Marvell SDK also provides the required kernel drivers.
172 Linux kernel should be configured with the following features enabled:
174 .. code-block:: console
176 # 64K pages enabled for better performance
177 CONFIG_ARM64_64K_PAGES=y
178 CONFIG_ARM64_VA_BITS_48=y
179 # huge pages support enabled
181 CONFIG_HUGETLB_PAGE=y
182 # VFIO enabled with TYPE1 IOMMU at minimum
183 CONFIG_VFIO_IOMMU_TYPE1=y
186 CONFIG_VFIO_NOIOMMU=y
188 CONFIG_VFIO_PCI_MMAP=y
191 # ARMv8.1 LSE atomics
192 CONFIG_ARM64_LSE_ATOMICS=y
194 CONFIG_OCTEONTX2_MBOX=y
195 CONFIG_OCTEONTX2_AF=y
196 # Enable if netdev PF driver required
197 CONFIG_OCTEONTX2_PF=y
198 # Enable if netdev VF driver required
199 CONFIG_OCTEONTX2_VF=y
200 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
201 # Enable if OCTEONTX2 DMA PF driver required
202 CONFIG_OCTEONTX2_DPI_PF=n
204 2. **ARM64 Linux Tool Chain**
206 For example, the *aarch64* Linaro Toolchain, which can be obtained from
207 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
209 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
210 optimized for OCTEON TX2 CPU.
212 3. **Rootfile system**
214 Any *aarch64* supporting filesystem may be used. For example,
215 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
216 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
218 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
219 The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board.
221 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
227 .. _table_octeontx2_common_debug_options:
229 .. table:: OCTEON TX2 common debug options
231 +---+------------+-------------------------------------------------------+
232 | # | Component | EAL log command |
233 +===+============+=======================================================+
234 | 1 | Common | --log-level='pmd\.octeontx2\.base,8' |
235 +---+------------+-------------------------------------------------------+
236 | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' |
237 +---+------------+-------------------------------------------------------+
242 The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks
243 context or stats using debugfs.
245 Enable ``debugfs`` by:
247 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
248 2. Boot OCTEON TX2 with debugfs supported kernel.
249 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
251 .. code-block:: console
253 # mount -t debugfs none /sys/kernel/debug
255 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
258 The file structure under ``/sys/kernel/debug`` is as follows
260 .. code-block:: console
276 | |-- cpt_engines_info
277 | |-- cpt_engines_sts
284 | |-- ndc_rx_hits_miss
286 | |-- ndc_tx_hits_miss
290 | '-- tx_stall_hwissue
299 | '-- rx_miss_act_stats
305 |-- sso_hwgrp_aq_thresh
306 |-- sso_hwgrp_iaq_walk
308 |-- sso_hwgrp_free_list_walk
309 |-- sso_hwgrp_ient_walk
310 '-- sso_hwgrp_taq_walk
312 RVU block LF allocation:
314 .. code-block:: console
316 cat /sys/kernel/debug/octeontx2/rsrc_alloc
318 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
325 .. code-block:: console
327 cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats
329 =======Link Status======
330 Link is UP 40000 Mbps
331 =======RX_STATS======
333 Octets of received packets: 0
334 Received PAUSE packets: 0
335 Received PAUSE and control packets: 0
336 Filtered DMAC0 (NIX-bound) packets: 0
337 Filtered DMAC0 (NIX-bound) octets: 0
338 Packets dropped due to RX FIFO full: 0
339 Octets dropped due to RX FIFO full: 0
341 Filtered DMAC1 (NCSI-bound) packets: 0
342 Filtered DMAC1 (NCSI-bound) octets: 0
343 NCSI-bound packets dropped: 0
344 NCSI-bound octets dropped: 0
345 =======TX_STATS======
346 Packets dropped due to excessive collisions: 0
347 Packets dropped due to excessive deferral: 0
348 Multiple collisions before successful transmission: 0
349 Single collisions before successful transmission: 0
350 Total octets sent on the interface: 0
351 Total frames sent on the interface: 0
352 Packets sent with an octet count < 64: 0
353 Packets sent with an octet count == 64: 0
354 Packets sent with an octet count of 65127: 0
355 Packets sent with an octet count of 128-255: 0
356 Packets sent with an octet count of 256-511: 0
357 Packets sent with an octet count of 512-1023: 0
358 Packets sent with an octet count of 1024-1518: 0
359 Packets sent with an octet count of > 1518: 0
360 Packets sent to a broadcast DMAC: 0
361 Packets sent to the multicast DMAC: 0
362 Transmit underflow and were truncated: 0
363 Control/PAUSE packets sent: 0
367 .. code-block:: console
369 cat /sys/kernel/debug/octeontx2/cpt/cpt_pc
371 CPT instruction requests 0
372 CPT instruction latency 0
373 CPT NCB read requests 0
374 CPT NCB read latency 0
375 CPT read requests caused by UC fills 0
376 CPT active cycles pc 1395642
377 CPT clock count pc 5579867595493
381 .. code-block:: console
383 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx
384 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
385 echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx
386 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
388 =====cq_ctx for nixlf:0 and qidx:0 is=====
399 W2: update_time 31043
416 .. code-block:: console
418 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx
419 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
420 echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx
421 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
423 ======POOL : 0=======
424 W0: Stack base 1375bff00
431 W2: stack_max_pages 24315
432 W2: stack_pages 24314
442 W4: update_time 62993
444 W6: ptr_start 1593adf00
445 W7: ptr_end 180000000
451 W8: thresh_qint_idx 0
456 .. code-block:: console
458 cat /sys/kernel/debug/octeontx2/npc/mcam_info
461 RX keywidth : 224bits
462 TX keywidth : 224bits
474 .. code-block:: console
476 Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
477 echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
479 ==================================================
480 SSOW HWS[0] Arbitration State 0x0
481 SSOW HWS[0] Guest Machine Control 0x0
482 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
483 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
484 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
485 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
486 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
487 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
488 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
489 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
490 ==================================================
495 DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on
496 an x86 based platform.
504 .. code-block:: console
506 make config T=arm64-octeontx2-linux-gcc
509 The example applications can be compiled using the following:
511 .. code-block:: console
515 export RTE_TARGET=build
516 cd examples/<application>
522 .. code-block:: console
530 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
535 .. code-block:: console
537 make config T=arm64-octeontx2-linux-gcc
538 make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n
543 .. code-block:: console
545 meson build --cross-file config/arm/arm64_octeontx2_linux_gcc
550 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
551 if Marvell toolchain is available then it can be used by overriding the
552 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
553 toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.