From: Jerin Jacob Date: Sat, 22 Jun 2019 13:24:05 +0000 (+0530) Subject: doc: add Marvell OCTEON TX2 platform guide X-Git-Url: http://git.droids-corp.org/?a=commitdiff_plain;h=14ad4f018453;p=dpdk.git doc: add Marvell OCTEON TX2 platform guide Platform specific guide for Marvell OCTEON TX2 SoC is added. Signed-off-by: Jerin Jacob Signed-off-by: Pavan Nikhilesh Signed-off-by: Shally Verma Signed-off-by: Vivek Sharma --- diff --git a/MAINTAINERS b/MAINTAINERS index 079089a310..f1eb98c82d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -450,6 +450,8 @@ M: Jerin Jacob M: Nithin Dabilpuram M: Vamsi Attunuru F: drivers/common/octeontx2/ +F: doc/guides/platform/img/octeontx2_* +F: doc/guides/platform/octeontx2.rst Bus Drivers diff --git a/doc/guides/platform/img/octeontx2_packet_flow_hw_accelerators.svg b/doc/guides/platform/img/octeontx2_packet_flow_hw_accelerators.svg new file mode 100644 index 0000000000..ecd575947a --- /dev/null +++ b/doc/guides/platform/img/octeontx2_packet_flow_hw_accelerators.svg @@ -0,0 +1,2804 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + DDDpk + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Tx Rx + HW loop back device + + + + + + + + + + + + + + + + + Ethdev Ports (NIX) + Ingress Classification(NPC) + Egress Classification(NPC) + Rx Queues + Tx Queues + EgressTraffic Manager(NIX) + Scheduler SSO + Supports both poll mode and/or event modeby configuring scheduler + ARMv8Cores + Hardware Libraries + Software Libraries + Mempool(NPA) + Timer(TIM) + Crypto(CPT) + Compress(ZIP) + SharedMemory + SW Ring + HASHLPMACL + Mbuf + De(Frag) + + diff --git a/doc/guides/platform/img/octeontx2_resource_virtualization.svg b/doc/guides/platform/img/octeontx2_resource_virtualization.svg new file mode 100644 index 0000000000..bf976b52af --- /dev/null +++ b/doc/guides/platform/img/octeontx2_resource_virtualization.svg @@ -0,0 +1,2418 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + +   + + + + + + + + + + NIX AF + NPA AF + SSO AF + NPC AF + CPT AF + RVU AF + Linux AF driver(octeontx2_af)PF0 + + + CGX-0 + + + + CGX-1 + + + + + CGX-2 + + CGX-FW Interface + + + + + + + + + AF-PF MBOX + Linux Netdev PFdriver(octeontx2_pf)PFx + + NIX LF + + NPA LF + + + PF-VF MBOX + CGX-x LMAC-y + + + + + + + + Linux Netdev VFdriver(octeontx2_vf)PFx-VF0 + + NIX LF + + NPA LF + DPDK Ethdev VFdriverPFx-VF1 + + NIX LF + + NPA LF + + + DPDK Ethdev PFdriverPFy + + NIX LF + + NPA LF + PF-VF MBOX + + DPDK Eventdev PFdriverPFz + + TIM LF + + SSO LF + Linux Crypto PFdriverPFm + + NIX LF + + NPA LF + DPDK Ethdev VFdriverPFy-VF0 + + CPT LF + DPDK Crypto VFdriverPFm-VF0 + PF-VF MBOX + + DDDpk DPDK-APP1 with one ethdev over Linux PF + + DPDK-APP2 with Two ethdevs(PF,VF) ,eventdev, timer adapter and cryptodev + + + + + CGX-x LMAC-y + + diff --git a/doc/guides/platform/index.rst b/doc/guides/platform/index.rst index a17de2efba..f454ef8773 100644 --- a/doc/guides/platform/index.rst +++ b/doc/guides/platform/index.rst @@ -14,3 +14,4 @@ The following are platform specific guides and setup information. dpaa dpaa2 octeontx + octeontx2 diff --git a/doc/guides/platform/octeontx2.rst b/doc/guides/platform/octeontx2.rst new file mode 100644 index 0000000000..3a5e030508 --- /dev/null +++ b/doc/guides/platform/octeontx2.rst @@ -0,0 +1,494 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Marvell International Ltd. + +Marvell OCTEON TX2 Platform Guide +================================= + +This document gives an overview of **Marvell OCTEON TX2** RVU H/W block, +packet flow and procedure to build DPDK on OCTEON TX2 platform. + +More information about OCTEON TX2 SoC can be found at `Marvell Official Website +`_. + +Supported OCTEON TX2 SoCs +------------------------- + +- CN96xx +- CN93xx + +OCTEON TX2 Resource Virtualization Unit architecture +---------------------------------------------------- + +The :numref:`figure_octeontx2_resource_virtualization` diagram depicts the +RVU architecture and a resource provisioning example. + +.. _figure_octeontx2_resource_virtualization: + +.. figure:: img/octeontx2_resource_virtualization.* + + OCTEON TX2 Resource virtualization architecture and provisioning example + + +Resource Virtualization Unit (RVU) on Marvell's OCTEON TX2 SoC maps HW +resources belonging to the network, crypto and other functional blocks onto +PCI-compatible physical and virtual functions. + +Each functional block has multiple local functions (LFs) for +provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV +physical functions (PFs) and virtual functions (VFs). + +The :numref:`table_octeontx2_rvu_dpdk_mapping` shows the various local +functions (LFs) provided by the RVU and its functional mapping to +DPDK subsystem. + +.. _table_octeontx2_rvu_dpdk_mapping: + +.. table:: RVU managed functional blocks and its mapping to DPDK subsystem + + +---+-----+--------------------------------------------------------------+ + | # | LF | DPDK subsystem mapping | + +===+=====+==============================================================+ + | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security| + +---+-----+--------------------------------------------------------------+ + | 2 | NPA | rte_mempool | + +---+-----+--------------------------------------------------------------+ + | 3 | NPC | rte_flow | + +---+-----+--------------------------------------------------------------+ + | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter | + +---+-----+--------------------------------------------------------------+ + | 5 | SSO | rte_eventdev | + +---+-----+--------------------------------------------------------------+ + | 6 | TIM | rte_event_timer_adapter | + +---+-----+--------------------------------------------------------------+ + +PF0 is called the administrative / admin function (AF) and has exclusive +privileges to provision RVU functional block's LFs to each of the PF/VF. + +PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving +requests from PF/VF, AF does resource provisioning and other HW configuration. + +AF is always attached to host, but PF/VFs may be used by host kernel itself, +or attached to VMs or to userspace applications like DPDK, etc. So, AF has to +handle provisioning/configuration requests sent by any device from any domain. + +The AF driver does not receive or process any data. +It is only a configuration driver used in control path. + +The :numref:`figure_octeontx2_resource_virtualization` diagram also shows a +resource provisioning example where, + +1. PFx and PFx-VF0 bound to Linux netdev driver. +2. PFx-VF1 ethdev driver bound to the first DPDK application. +3. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application. + +OCTEON TX2 packet flow +---------------------- + +The :numref:`figure_octeontx2_packet_flow_hw_accelerators` diagram depicts +the packet flow on OCTEON TX2 SoC in conjunction with use of various HW accelerators. + +.. _figure_octeontx2_packet_flow_hw_accelerators: + +.. figure:: img/octeontx2_packet_flow_hw_accelerators.* + + OCTEON TX2 packet flow in conjunction with use of HW accelerators + +HW Offload Drivers +------------------ + +This section lists dataplane H/W block(s) available in OCTEON TX2 SoC. + + +Procedure to Setup Platform +--------------------------- + +There are three main prerequisites for setting up DPDK on OCTEON TX2 +compatible board: + +1. **OCTEON TX2 Linux kernel driver** + + The dependent kernel drivers can be obtained from the + `kernel.org `_. + + Alternatively, the Marvell SDK also provides the required kernel drivers. + + Linux kernel should be configured with the following features enabled: + +.. code-block:: console + + # 64K pages enabled for better performance + CONFIG_ARM64_64K_PAGES=y + CONFIG_ARM64_VA_BITS_48=y + # huge pages support enabled + CONFIG_HUGETLBFS=y + CONFIG_HUGETLB_PAGE=y + # VFIO enabled with TYPE1 IOMMU at minimum + CONFIG_VFIO_IOMMU_TYPE1=y + CONFIG_VFIO_VIRQFD=y + CONFIG_VFIO=y + CONFIG_VFIO_NOIOMMU=y + CONFIG_VFIO_PCI=y + CONFIG_VFIO_PCI_MMAP=y + # SMMUv3 driver + CONFIG_ARM_SMMU_V3=y + # ARMv8.1 LSE atomics + CONFIG_ARM64_LSE_ATOMICS=y + # OCTEONTX2 drivers + CONFIG_OCTEONTX2_MBOX=y + CONFIG_OCTEONTX2_AF=y + # Enable if netdev PF driver required + CONFIG_OCTEONTX2_PF=y + # Enable if netdev VF driver required + CONFIG_OCTEONTX2_VF=y + CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y + +2. **ARM64 Linux Tool Chain** + + For example, the *aarch64* Linaro Toolchain, which can be obtained from + `here `_. + + Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is + optimized for OCTEON TX2 CPU. + +3. **Rootfile system** + + Any *aarch64* supporting filesystem may be used. For example, + Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained + from ``_. + + Alternatively, the Marvell SDK provides the buildroot based root filesystem. + The SDK includes all the above prerequisites necessary to bring up the OCTEON TX2 board. + +- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment. + + +Debugging Options +----------------- + +.. _table_octeontx2_common_debug_options: + +.. table:: OCTEON TX2 common debug options + + +---+------------+-------------------------------------------------------+ + | # | Component | EAL log command | + +===+============+=======================================================+ + | 1 | Common | --log-level='pmd\.octeontx2\.base,8' | + +---+------------+-------------------------------------------------------+ + | 2 | Mailbox | --log-level='pmd\.octeontx2\.mbox,8' | + +---+------------+-------------------------------------------------------+ + +Debugfs support +~~~~~~~~~~~~~~~ + +The **OCTEON TX2 Linux kernel driver** provides support to dump RVU blocks +context or stats using debugfs. + +Enable ``debugfs`` by: + +1. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUGFS=y``. +2. Boot OCTEON TX2 with debugfs supported kernel. +3. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using. + +.. code-block:: console + + # mount -t debugfs none /sys/kernel/debug + +Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC, +SSO & CGX. + +The file structure under ``/sys/kernel/debug`` is as follows + +.. code-block:: console + + octeontx2/ + ├── cgx + │ ├── cgx0 + │ │ └── lmac0 + │ │ └── stats + │ ├── cgx1 + │ │ ├── lmac0 + │ │ │ └── stats + │ │ └── lmac1 + │ │ └── stats + │ └── cgx2 + │ └── lmac0 + │ └── stats + ├── cpt + │ ├── cpt_engines_info + │ ├── cpt_engines_sts + │ ├── cpt_err_info + │ ├── cpt_lfs_info + │ └── cpt_pc + ├──── nix + │ ├── cq_ctx + │ ├── ndc_rx_cache + │ ├── ndc_rx_hits_miss + │ ├── ndc_tx_cache + │ ├── ndc_tx_hits_miss + │ ├── qsize + │ ├── rq_ctx + │ ├── sq_ctx + │ └── tx_stall_hwissue + ├── npa + │ ├── aura_ctx + │ ├── ndc_cache + │ ├── ndc_hits_miss + │ ├── pool_ctx + │ └── qsize + ├── npc + │ ├── mcam_info + │ └── rx_miss_act_stats + ├── rsrc_alloc + └── sso + ├── hws + │ └── sso_hws_info + └── hwgrp + ├── sso_hwgrp_aq_thresh + ├── sso_hwgrp_iaq_walk + ├── sso_hwgrp_pc + ├── sso_hwgrp_free_list_walk + ├── sso_hwgrp_ient_walk + └── sso_hwgrp_taq_walk + +RVU block LF allocation: + +.. code-block:: console + + cat /sys/kernel/debug/octeontx2/rsrc_alloc + + pcifunc NPA NIX SSO GROUP SSOWS TIM CPT + PF1 0 0 + PF4 1 + PF13 0, 1 0, 1 0 + +CGX example usage: + +.. code-block:: console + + cat /sys/kernel/debug/octeontx2/cgx/cgx2/lmac0/stats + + =======Link Status====== + Link is UP 40000 Mbps + =======RX_STATS====== + Received packets: 0 + Octets of received packets: 0 + Received PAUSE packets: 0 + Received PAUSE and control packets: 0 + Filtered DMAC0 (NIX-bound) packets: 0 + Filtered DMAC0 (NIX-bound) octets: 0 + Packets dropped due to RX FIFO full: 0 + Octets dropped due to RX FIFO full: 0 + Error packets: 0 + Filtered DMAC1 (NCSI-bound) packets: 0 + Filtered DMAC1 (NCSI-bound) octets: 0 + NCSI-bound packets dropped: 0 + NCSI-bound octets dropped: 0 + =======TX_STATS====== + Packets dropped due to excessive collisions: 0 + Packets dropped due to excessive deferral: 0 + Multiple collisions before successful transmission: 0 + Single collisions before successful transmission: 0 + Total octets sent on the interface: 0 + Total frames sent on the interface: 0 + Packets sent with an octet count < 64: 0 + Packets sent with an octet count == 64: 0 + Packets sent with an octet count of 65127: 0 + Packets sent with an octet count of 128-255: 0 + Packets sent with an octet count of 256-511: 0 + Packets sent with an octet count of 512-1023: 0 + Packets sent with an octet count of 1024-1518: 0 + Packets sent with an octet count of > 1518: 0 + Packets sent to a broadcast DMAC: 0 + Packets sent to the multicast DMAC: 0 + Transmit underflow and were truncated: 0 + Control/PAUSE packets sent: 0 + +CPT example usage: + +.. code-block:: console + + cat /sys/kernel/debug/octeontx2/cpt/cpt_pc + + CPT instruction requests 0 + CPT instruction latency 0 + CPT NCB read requests 0 + CPT NCB read latency 0 + CPT read requests caused by UC fills 0 + CPT active cycles pc 1395642 + CPT clock count pc 5579867595493 + +NIX example usage: + +.. code-block:: console + + Usage: echo [cq number/all] > /sys/kernel/debug/octeontx2/nix/cq_ctx + cat /sys/kernel/debug/octeontx2/nix/cq_ctx + echo 0 0 > /sys/kernel/debug/octeontx2/nix/cq_ctx + cat /sys/kernel/debug/octeontx2/nix/cq_ctx + + =====cq_ctx for nixlf:0 and qidx:0 is===== + W0: base 158ef1a00 + + W1: wrptr 0 + W1: avg_con 0 + W1: cint_idx 0 + W1: cq_err 0 + W1: qint_idx 0 + W1: bpid 0 + W1: bp_ena 0 + + W2: update_time 31043 + W2:avg_level 255 + W2: head 0 + W2:tail 0 + + W3: cq_err_int_ena 5 + W3:cq_err_int 0 + W3: qsize 4 + W3:caching 1 + W3: substream 0x000 + W3: ena 1 + W3: drop_ena 1 + W3: drop 64 + W3: bp 0 + +NPA example usage: + +.. code-block:: console + + Usage: echo [pool number/all] > /sys/kernel/debug/octeontx2/npa/pool_ctx + cat /sys/kernel/debug/octeontx2/npa/pool_ctx + echo 0 0 > /sys/kernel/debug/octeontx2/npa/pool_ctx + cat /sys/kernel/debug/octeontx2/npa/pool_ctx + + ======POOL : 0======= + W0: Stack base 1375bff00 + W1: ena 1 + W1: nat_align 1 + W1: stack_caching 1 + W1: stack_way_mask 0 + W1: buf_offset 1 + W1: buf_size 19 + W2: stack_max_pages 24315 + W2: stack_pages 24314 + W3: op_pc 267456 + W4: stack_offset 2 + W4: shift 5 + W4: avg_level 255 + W4: avg_con 0 + W4: fc_ena 0 + W4: fc_stype 0 + W4: fc_hyst_bits 0 + W4: fc_up_crossing 0 + W4: update_time 62993 + W5: fc_addr 0 + W6: ptr_start 1593adf00 + W7: ptr_end 180000000 + W8: err_int 0 + W8: err_int_ena 7 + W8: thresh_int 0 + W8: thresh_int_ena 0 + W8: thresh_up 0 + W8: thresh_qint_idx 0 + W8: err_qint_idx 0 + +NPC example usage: + +.. code-block:: console + + cat /sys/kernel/debug/octeontx2/npc/mcam_info + + NPC MCAM info: + RX keywidth : 224bits + TX keywidth : 224bits + + MCAM entries : 2048 + Reserved : 158 + Available : 1890 + + MCAM counters : 512 + Reserved : 1 + Available : 511 + +SSO example usage: + +.. code-block:: console + + Usage: echo [/all] > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info + echo 0 > /sys/kernel/debug/octeontx2/sso/hws/sso_hws_info + + ================================================== + SSOW HWS[0] Arbitration State 0x0 + SSOW HWS[0] Guest Machine Control 0x0 + SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff + SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff + SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff + SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff + SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff + SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff + SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff + SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff + ================================================== + +Compile DPDK +------------ + +DPDK may be compiled either natively on OCTEON TX2 platform or cross-compiled on +an x86 based platform. + +Native Compilation +~~~~~~~~~~~~~~~~~~ + +make build +^^^^^^^^^^ + +.. code-block:: console + + make config T=arm64-octeontx2-linux-gcc + make -j + +The example applications can be compiled using the following: + +.. code-block:: console + + cd + export RTE_SDK=$PWD + export RTE_TARGET=build + cd examples/ + make -j + +meson build +^^^^^^^^^^^ + +.. code-block:: console + + meson build + ninja -C build + +Cross Compilation +~~~~~~~~~~~~~~~~~ + +Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details. + +make build +^^^^^^^^^^ + +.. code-block:: console + + make config T=arm64-octeontx2-linux-gcc + make -j CROSS=aarch64-marvell-linux-gnu- CONFIG_RTE_KNI_KMOD=n + +meson build +^^^^^^^^^^^ + +.. code-block:: console + + meson build --cross-file config/arm/arm64_octeontx2_linux_gcc + ninja -C build + +.. note:: + + By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain, + if Marvell toolchain is available then it can be used by overriding the + c, cpp, ar, strip ``binaries`` attributes to respective Marvell + toolchain binaries in ``config/arm/arm64_octeontx2_linux_gcc`` file.