1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2016 Cavium, Inc
4 ThunderX NICVF Poll Mode Driver
5 ===============================
7 The ThunderX NICVF PMD (**librte_pmd_thunderx_nicvf**) provides poll mode driver
8 support for the inbuilt NIC found in the **Cavium ThunderX** SoC family
9 as well as their virtual functions (VF) in SR-IOV context.
11 More information can be found at `Cavium, Inc Official Website
12 <http://www.cavium.com/ThunderX_ARM_Processors.html>`_.
17 Features of the ThunderX PMD are:
19 - Multiple queues for TX and RX
20 - Receive Side Scaling (RSS)
21 - Packet type information
25 - Port hardware statistics
27 - Link state information
28 - Scattered and gather for TX and RX
32 - Multi queue set support (up to 96 queues (12 queue sets)) per port
34 Supported ThunderX SoCs
35 -----------------------
42 - Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
44 Pre-Installation Configuration
45 ------------------------------
50 The following options can be modified in the ``config`` file.
51 Please note that enabling debugging options may affect system performance.
53 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD`` (default ``y``)
55 Toggle compilation of the ``librte_pmd_thunderx_nicvf`` driver.
57 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT`` (default ``n``)
59 Toggle display of initialization related messages.
61 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX`` (default ``n``)
63 Toggle display of receive fast path run-time message
65 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX`` (default ``n``)
67 Toggle display of transmit fast path run-time message
69 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER`` (default ``n``)
71 Toggle display of generic debugging messages
73 - ``CONFIG_RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX`` (default ``n``)
75 Toggle display of PF mailbox related run-time check messages
77 Driver compilation and testing
78 ------------------------------
80 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
83 To compile the ThunderX NICVF PMD for Linux arm64 gcc,
84 use arm64-thunderx-linuxapp-gcc as target.
89 SR-IOV: Prerequisites and sample Application Notes
90 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
92 Current ThunderX NIC PF/VF kernel modules maps each physical Ethernet port
93 automatically to virtual function (VF) and presented them as PCIe-like SR-IOV device.
94 This section provides instructions to configure SR-IOV with Linux OS.
96 #. Verify PF devices capabilities using ``lspci``:
98 .. code-block:: console
104 .. code-block:: console
106 0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01)
108 Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
110 Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
112 Kernel driver in use: thunder-nic
117 Unless ``thunder-nic`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_PF`` setting.
119 #. Verify VF devices capabilities and drivers using ``lspci``:
121 .. code-block:: console
127 .. code-block:: console
129 0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01)
131 Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
133 Kernel driver in use: thunder-nicvf
136 0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01)
138 Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
140 Kernel driver in use: thunder-nicvf
145 Unless ``thunder-nicvf`` driver is in use make sure your kernel config includes ``CONFIG_THUNDER_NIC_VF`` setting.
147 #. Pass VF device to VM context (PCIe Passthrough):
149 The VF devices may be passed through to the guest VM using qemu or
150 virt-manager or virsh etc.
152 Example qemu guest launch command:
154 .. code-block:: console
156 sudo qemu-system-aarch64 -name vm1 \
157 -machine virt,gic_version=3,accel=kvm,usb=off \
159 -smp 4,sockets=1,cores=8,threads=1 \
160 -nographic -nodefaults \
161 -kernel <kernel image> \
162 -append "root=/dev/vda console=ttyAMA0 rw hugepagesz=512M hugepages=3" \
163 -device vfio-pci,host=0002:01:00.1 \
164 -drive file=<rootfs.ext3>,if=none,id=disk1,format=raw \
165 -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
166 -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup_thunder \
167 -device virtio-net-device,netdev=net0 \
171 #. Enable **VFIO-NOIOMMU** mode (optional):
173 .. code-block:: console
175 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
179 **VFIO-NOIOMMU** is required only when running in VM context and should not be enabled otherwise.
183 Follow instructions available in the document
184 :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
189 .. code-block:: console
191 ./arm64-thunderx-linuxapp-gcc/app/testpmd -l 0-3 -n 4 -w 0002:01:00.2 \
192 -- -i --disable-hw-vlan-filter --disable-crc-strip --no-flush-rx \
197 PMD: rte_nicvf_pmd_init(): librte_pmd_thunderx nicvf version 1.0
200 EAL: probe driver: 177d:11 rte_nicvf_pmd
201 EAL: using IOMMU type 1 (Type 1)
202 EAL: PCI memory mapped at 0x3ffade50000
203 EAL: Trying to map BAR 4 that contains the MSI-X table.
204 Trying offsets: 0x40000000000:0x0000, 0x10000:0x1f0000
205 EAL: PCI memory mapped at 0x3ffadc60000
206 PMD: nicvf_eth_dev_init(): nicvf: device (177d:11) 2:1:0:2
207 PMD: nicvf_eth_dev_init(): node=0 vf=1 mode=tns-bypass sqs=false
208 loopback_supported=true
209 PMD: nicvf_eth_dev_init(): Port 0 (177d:11) mac=a6:c6:d9:17:78:01
210 Interactive-mode selected
211 Configuring Port 0 (socket 0)
214 PMD: nicvf_dev_configure(): Configured ethdev port0 hwcap=0x0
215 Port 0: A6:C6:D9:17:78:01
216 Checking link statuses...
217 Port 0 Link Up - speed 10000 Mbps - full-duplex
221 Multiple Queue Set per DPDK port configuration
222 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
224 There are two types of VFs:
229 Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
230 When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF.
231 Each secondary VF adds 8 additional queues to the queue set.
233 During PMD driver initialization, the primary VF's are enumerated by checking the
234 specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
235 They are at the beginning of VF list (the remain ones are secondary VF's).
237 The primary VFs are used as master queue sets. Secondary VFs provide
238 additional queue sets for primary ones. If a port is configured for more then
239 8 queues than it will request for additional queues from secondary VFs.
241 Secondary VFs cannot be shared between primary VFs.
243 Primary VFs are present on the beginning of the 'Network devices using kernel
244 driver' list, secondary VFs are on the remaining on the remaining part of the list.
248 The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
249 We need to bind separately each specific queue set device with the ``usertools/dpdk-devbind.py`` utility.
253 Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
254 this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
257 Example device binding
258 ~~~~~~~~~~~~~~~~~~~~~~
260 If a system has three interfaces, a total of 18 VF devices will be created
261 on a non-NUMA machine.
265 NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
267 .. code-block:: console
269 # usertools/dpdk-devbind.py --status
271 Network devices using DPDK-compatible driver
272 ============================================
275 Network devices using kernel driver
276 ===================================
277 0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
278 0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
279 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
280 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
281 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
282 0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
283 0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
284 0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
285 0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
286 0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
287 0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
288 0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
289 0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
290 0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
291 0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
292 0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
293 0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
294 0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
295 0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
296 0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
297 0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
299 Other network devices
300 =====================
301 0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
304 We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
305 and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
306 We will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
309 #. Bind two primary VFs to the ``vfio-pci`` driver:
311 .. code-block:: console
313 usertools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
314 usertools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
316 #. Bind four primary VFs to the ``vfio-pci`` driver:
318 .. code-block:: console
320 usertools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
321 usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
322 usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
323 usertools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
325 The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
333 The ThunderX SoC family NICs strip the CRC for every packets coming into the
334 host interface. So, CRC will be stripped even when the
335 ``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``.
337 Maximum packet length
338 ~~~~~~~~~~~~~~~~~~~~~
340 The ThunderX SoC family NICs support a maximum of a 9K jumbo frame. The value
341 is fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len``
342 member of ``struct rte_eth_conf`` is set to a value lower than 9200, frames
343 up to 9200 bytes can still reach the host interface.
345 Maximum packet segments
346 ~~~~~~~~~~~~~~~~~~~~~~~
348 The ThunderX SoC family NICs support up to 12 segments per packet when working
349 in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
350 frame size does not fit in the maximum number of segments.