1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2019 Marvell International Ltd.
4 Marvell OCTEON TX2 Platform Guide
5 =================================
7 This document gives an overview of **Marvell OCTEON TX2** RVU H/W block,
8 packet flow and procedure to build DPDK on OCTEON TX2 platform.
10 More information about OCTEON TX2 SoC can be found at `Marvell Official Website
11 <https://www.marvell.com/embedded-processors/infrastructure-processors/>`_.
13 Supported OCTEON TX2 SoCs
14 -------------------------
19 OCTEON TX2 Resource Virtualization Unit architecture
20 ----------------------------------------------------
22 The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the
23 RVU architecture and a resource provisioning example.
25 .. _figure_octeontx2_resource_virtualization:
27 .. figure:: img/octeontx2_resource_virtualization.*
29 OCTEON TX2 Resource virtualization architecture and provisioning example
32 Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW
33 resources belonging to the network, crypto and other functional blocks onto
34 PCI-compatible physical and virtual functions.
36 Each functional block has multiple local functions (LFs) for
37 provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV
38 physical functions (PFs) and virtual functions (VFs).
40 The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local
41 functions (LFs) provided by the RVU and its functional mapping to
44 .. _table_octeontx2_rvu_dpdk_mapping:
46 .. table:: RVU managed functional blocks and its mapping to DPDK subsystem
48 +---+-----+--------------------------------------------------------------+
49 | # | LF | DPDK subsystem mapping |
50 +===+=====+==============================================================+
51 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security|
52 +---+-----+--------------------------------------------------------------+
53 | 2 | NPA | rte_mempool |
54 +---+-----+--------------------------------------------------------------+
55 | 3 | NPC | rte_flow |
56 +---+-----+--------------------------------------------------------------+
57 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter |
58 +---+-----+--------------------------------------------------------------+
59 | 5 | SSO | rte_eventdev |
60 +---+-----+--------------------------------------------------------------+
61 | 6 | TIM | rte_event_timer_adapter |
62 +---+-----+--------------------------------------------------------------+
63 | 7 | LBK | rte_ethdev |
64 +---+-----+--------------------------------------------------------------+
65 | 8 | DPI | rte_rawdev |
66 +---+-----+--------------------------------------------------------------+
67 | 9 | SDP | rte_ethdev |
68 +---+-----+--------------------------------------------------------------+
70 PF0 is called the administrative / admin function (AF) and has exclusive
71 privileges to provision RVU functional block's LFs to each of the PF/VF.
73 PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving
74 requests from PF/VF, AF does resource provisioning and other HW configuration.
76 AF is always attached to host, but PF/VFs may be used by host kernel itself,
77 or attached to VMs or to userspace applications like DPDK, etc. So, AF has to
78 handle provisioning/configuration requests sent by any device from any domain.
80 The AF driver does not receive or process any data.
81 It is only a configuration driver used in control path.
83 The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a
84 resource provisioning example where,
86 1. PFx and PFx-VF0 bound to Linux netdev driver.
87 2. PFx-VF1 ethdev driver bound to the first DPDK application.
88 3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application.
93 Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX.
94 The loopback block has N channels and contains data buffering that is shared across
95 all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's
96 VFs are exposed as ethdev device and odd-even pairs of VFs are tied together,
97 that is, packets sent on odd VF end up received on even VF and vice versa.
98 This would enable HW accelerated means of communication between two domains
99 where even VF bound to the first domain and odd VF bound to the second domain.
101 Typical application usage models are,
103 #. Communication between the Linux kernel and DPDK application.
104 #. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement.
105 #. Communication between two different DPDK applications.
110 System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host
111 to DMA packets into and out of OCTEON TX2 SoC. SDP interface comes in to live only when
112 OCTEON TX2 SoC is connected in PCIe endpoint mode. It can be used to send/receive
113 packets to/from remote host machine using input/output queue pairs exposed to it.
114 SDP interface receives input packets from remote host from NIX-RX and sends packets
115 to remote host using NIX-TX. Remote host machine need to use corresponding driver
116 (kernel/user mode) to communicate with SDP interface on OCTEON TX2 SoC. SDP supports
117 single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users
118 can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports.
120 The primary use case for SDP is to enable the smart NIC use case. Typical usage models are,
122 #. Communication channel between remote host and OCTEON TX2 SoC over PCIe.
123 #. Transfer packets received from network interface to remote host over PCIe and
126 OCTEON TX2 packet flow
127 ----------------------
129 The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts
130 the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators.
132 .. _figure_octeontx2_packet_flow_hw_accelerators:
134 .. figure:: img/octeontx2_packet_flow_hw_accelerators.*
136 OCTEON TX2 packet flow in conjunction with use of HW accelerators
141 This section lists dataplane H/W block(s) available in OCTEON TX2 SoC.
144 See :doc:`../nics/octeontx2` for NIX Ethdev driver information.
146 #. **Mempool Driver**
147 See :doc:`../mempool/octeontx2` for NPA mempool driver information.
149 #. **Event Device Driver**
150 See :doc:`../eventdevs/octeontx2` for SSO event device driver information.
152 #. **DMA Rawdev Driver**
153 See :doc:`../rawdevs/octeontx2_dma` for DMA driver information.
155 #. **Crypto Device Driver**
156 See :doc:`../cryptodevs/octeontx2` for CPT crypto device driver information.
158 Procedure to Setup Platform
159 ---------------------------
161 There are three main prerequisites for setting up DPDK on OCTEON TX2
164 1. **OCTEON TX2 Linux kernel driver**
166 The dependent kernel drivers can be obtained from the
167 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_.
169 Alternatively, the Marvell SDK also provides the required kernel drivers.
171 Linux kernel should be configured with the following features enabled:
173 .. code-block:: console
175 # 64K pages enabled for better performance
176 CONFIG_ARM64_64K_PAGES=y
177 CONFIG_ARM64_VA_BITS_48=y
178 # huge pages support enabled
180 CONFIG_HUGETLB_PAGE=y
181 # VFIO enabled with TYPE1 IOMMU at minimum
182 CONFIG_VFIO_IOMMU_TYPE1=y
185 CONFIG_VFIO_NOIOMMU=y
187 CONFIG_VFIO_PCI_MMAP=y
190 # ARMv8.1 LSE atomics
191 CONFIG_ARM64_LSE_ATOMICS=y
193 CONFIG_OCTEONTX2_MBOX=y
194 CONFIG_OCTEONTX2_AF=y
195 # Enable if netdev PF driver required
196 CONFIG_OCTEONTX2_PF=y
197 # Enable if netdev VF driver required
198 CONFIG_OCTEONTX2_VF=y
199 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y
200 # Enable if OCTEONTX2 DMA PF driver required
201 CONFIG_OCTEONTX2_DPI_PF=n
203 2. **ARM64 Linux Tool Chain**
205 For example, the *aarch64* Linaro Toolchain, which can be obtained from
206 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_.
208 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is
209 optimized for OCTEON TX2 CPU.
211 3. **Rootfile system**
213 Any *aarch64* supporting filesystem may be used. For example,
214 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained
215 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_.
217 Alternatively, the Marvell SDK provides the buildroot based root filesystem.
218 The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board.
220 - Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment.
226 .. _table_octeontx2_common_debug_options:
228 .. table:: OCTEON TX2 common debug options
230 +---+------------+-------------------------------------------------------+
231 | # | Component | EAL log command |
232 +===+============+=======================================================+
233 | 1 | Common | --log-level='pmd\.octeontx2\.base,8' |
234 +---+------------+-------------------------------------------------------+
235 | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' |
236 +---+------------+-------------------------------------------------------+
241 The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks
242 context or stats using debugfs.
244 Enable ``debugfs`` by:
246 1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``.
247 2. Boot OCTEON TX2 with debugfs supported kernel.
248 3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using.
250 .. code-block:: console
252 # mount -t debugfs none /sys/kernel/debug
254 Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC,
257 The file structure under ``/sys/kernel/debug`` is as follows
259 .. code-block:: console
275 | |-- cpt_engines_info
276 | |-- cpt_engines_sts
283 | |-- ndc_rx_hits_miss
285 | |-- ndc_tx_hits_miss
289 | '-- tx_stall_hwissue
298 | '-- rx_miss_act_stats
304 |-- sso_hwgrp_aq_thresh
305 |-- sso_hwgrp_iaq_walk
307 |-- sso_hwgrp_free_list_walk
308 |-- sso_hwgrp_ient_walk
309 '-- sso_hwgrp_taq_walk
311 RVU block LF allocation:
313 .. code-block:: console
315 cat /sys/kernel/debug/octeontx2/rsrc_alloc
317 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT
324 .. code-block:: console
326 cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats
328 =======Link Status======
329 Link is UP 40000 Mbps
330 =======RX_STATS======
332 Octets of received packets: 0
333 Received PAUSE packets: 0
334 Received PAUSE and control packets: 0
335 Filtered DMAC0 (NIX-bound) packets: 0
336 Filtered DMAC0 (NIX-bound) octets: 0
337 Packets dropped due to RX FIFO full: 0
338 Octets dropped due to RX FIFO full: 0
340 Filtered DMAC1 (NCSI-bound) packets: 0
341 Filtered DMAC1 (NCSI-bound) octets: 0
342 NCSI-bound packets dropped: 0
343 NCSI-bound octets dropped: 0
344 =======TX_STATS======
345 Packets dropped due to excessive collisions: 0
346 Packets dropped due to excessive deferral: 0
347 Multiple collisions before successful transmission: 0
348 Single collisions before successful transmission: 0
349 Total octets sent on the interface: 0
350 Total frames sent on the interface: 0
351 Packets sent with an octet count < 64: 0
352 Packets sent with an octet count == 64: 0
353 Packets sent with an octet count of 65127: 0
354 Packets sent with an octet count of 128-255: 0
355 Packets sent with an octet count of 256-511: 0
356 Packets sent with an octet count of 512-1023: 0
357 Packets sent with an octet count of 1024-1518: 0
358 Packets sent with an octet count of > 1518: 0
359 Packets sent to a broadcast DMAC: 0
360 Packets sent to the multicast DMAC: 0
361 Transmit underflow and were truncated: 0
362 Control/PAUSE packets sent: 0
366 .. code-block:: console
368 cat /sys/kernel/debug/octeontx2/cpt/cpt_pc
370 CPT instruction requests 0
371 CPT instruction latency 0
372 CPT NCB read requests 0
373 CPT NCB read latency 0
374 CPT read requests caused by UC fills 0
375 CPT active cycles pc 1395642
376 CPT clock count pc 5579867595493
380 .. code-block:: console
382 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx
383 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
384 echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx
385 cat /sys/kernel/debug/octeontx2/nix/cq_ctx
387 =====cq_ctx for nixlf:0 and qidx:0 is=====
398 W2: update_time 31043
415 .. code-block:: console
417 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx
418 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
419 echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx
420 cat /sys/kernel/debug/octeontx2/npa/pool_ctx
422 ======POOL : 0=======
423 W0: Stack base 1375bff00
430 W2: stack_max_pages 24315
431 W2: stack_pages 24314
441 W4: update_time 62993
443 W6: ptr_start 1593adf00
444 W7: ptr_end 180000000
450 W8: thresh_qint_idx 0
455 .. code-block:: console
457 cat /sys/kernel/debug/octeontx2/npc/mcam_info
460 RX keywidth : 224bits
461 TX keywidth : 224bits
473 .. code-block:: console
475 Usage: echo [<hws>/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
476 echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info
478 ==================================================
479 SSOW HWS[0] Arbitration State 0x0
480 SSOW HWS[0] Guest Machine Control 0x0
481 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff
482 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff
483 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff
484 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff
485 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff
486 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff
487 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff
488 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff
489 ==================================================
494 DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on
495 an x86 based platform.
503 .. code-block:: console
505 make config T=arm64-octeontx2-linux-gcc
508 The example applications can be compiled using the following:
510 .. code-block:: console
514 export RTE_TARGET=build
515 cd examples/<application>
521 .. code-block:: console
529 Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details.
534 .. code-block:: console
536 make config T=arm64-octeontx2-linux-gcc
537 make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n
542 .. code-block:: console
544 meson build --cross-file config/arm/arm64_octeontx2_linux_gcc
549 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain,
550 if Marvell toolchain is available then it can be used by overriding the
551 c, cpp, ar, strip ``binaries`` attributes to respective Marvell
552 toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.