X-Git-Url: http://git.droids-corp.org/?a=blobdiff_plain;f=doc%2Fguides%2Fnics%2Fvirtio.rst;h=d1f5fb8986dc5aa94ca9114626488742f47c0b16;hb=7001c8fdb27357c67147c0a13cb3826e48c0f2bf;hp=c90e5175c60c2b033c9ff592f11526b59c5327e2;hpb=c6dab2a873f65c5a4ea9735aa24d9539426adba4;p=dpdk.git diff --git a/doc/guides/nics/virtio.rst b/doc/guides/nics/virtio.rst index c90e5175c6..d1f5fb8986 100644 --- a/doc/guides/nics/virtio.rst +++ b/doc/guides/nics/virtio.rst @@ -1,32 +1,5 @@ -.. BSD LICENSE - Copyright(c) 2010-2015 Intel Corporation. All rights reserved. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Intel Corporation nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2010-2015 Intel Corporation. Poll Mode Driver for Emulated Virtio NIC ======================================== @@ -41,9 +14,6 @@ The DPDK extends kni to support vhost raw socket interface, which enables vhost to directly read/ write packets from/to a physical port. With this enhancement, virtio could achieve quite promising performance. -In future release, we will also make enhancement to vhost backend, -releasing peak performance of virtio PMD driver. - For basic qemu-KVM installation and other Intel EM poll mode driver in guest VM, please refer to Chapter "Driver for VM Emulated Devices". @@ -53,15 +23,19 @@ standard qemu vhost back end and vhost kni back end. Virtio Implementation in DPDK ----------------------------- -For details about the virtio spec, refer to Virtio PCI Card Specification written by Rusty Russell. +For details about the virtio spec, refer to the latest +`VIRTIO (Virtual I/O) Device Specification +`_. -As a PMD, virtio provides packet reception and transmission callbacks virtio_recv_pkts and virtio_xmit_pkts. +As a PMD, virtio provides packet reception and transmission callbacks. -In virtio_recv_pkts, index in range [vq->vq_used_cons_idx , vq->vq_ring.used->idx) in vring is available for virtio to burst out. +In Rx, packets described by the used descriptors in vring are available +for virtio to burst out. -In virtio_xmit_pkts, same index range in vring is available for virtio to clean. -Virtio will enqueue to be transmitted packets into vring, advance the vq->vq_ring.avail->idx, -and then notify the host back end if necessary. +In Tx, packets described by the used descriptors in vring are available +for virtio to clean. Virtio will enqueue to be transmitted packets into +vring, make them available to the device, and then notify the host back +end if necessary. Features and Limitations of virtio PMD -------------------------------------- @@ -73,20 +47,28 @@ In this release, the virtio PMD driver provides the basic functionality of packe * It supports multicast packets and promiscuous mode. -* The descriptor number for the Rx/Tx queue is hard-coded to be 256 by qemu. +* The descriptor number for the Rx/Tx queue is hard-coded to be 256 by qemu 2.7 and below. If given a different descriptor number by the upper application, the virtio PMD generates a warning and fall back to the hard-coded value. + Rx queue size can be configurable and up to 1024 since qemu 2.8 and above. Rx queue size is 256 + by default. Tx queue size is still hard-coded to be 256. * Features of mac/vlan filter are supported, negotiation with vhost/backend are needed to support them. - When backend can't support vlan filter, virtio app on guest should disable vlan filter to make sure - the virtio port is configured correctly. E.g. specify '--disable-hw-vlan' in testpmd command line. + When backend can't support vlan filter, virtio app on guest should not enable vlan filter in order + to make sure the virtio port is configured correctly. E.g. do not specify '--enable-hw-vlan' in testpmd + command line. Note that, mac/vlan filter is best effort: unwanted packets could still arrive. -* RTE_PKTMBUF_HEADROOM should be defined larger than sizeof(struct virtio_net_hdr), which is 10 bytes. +* "RTE_PKTMBUF_HEADROOM" should be defined + no less than "sizeof(struct virtio_net_hdr_mrg_rxbuf)", which is 12 bytes when mergeable or + "VIRTIO_F_VERSION_1" is set. + no less than "sizeof(struct virtio_net_hdr)", which is 10 bytes, when using non-mergeable. * Virtio does not support runtime configuration. * Virtio supports Link State interrupt. +* Virtio supports Rx interrupt (so far, only support 1:1 mapping for queue/interrupt). + * Virtio supports software vlan stripping and inserting. * Virtio supports using port IO to get PCI resource when uio/igb_uio module is not available. @@ -128,7 +110,7 @@ Host2VM communication example .. code-block:: console - examples/kni/build/app/kni -c 0xf -n 4 -- -p 0x1 -P --config="(0,1,3)" + examples/kni/build/app/kni -l 0-3 -n 4 -- -p 0x1 -P --config="(0,1,3)" This command generates one network device vEth0 for physical port. If specify more physical ports, the generated network device will be vEth1, vEth2, and so on. @@ -222,55 +204,325 @@ The packet transmission flow is: Virtio PMD Rx/Tx Callbacks -------------------------- -Virtio driver has 3 Rx callbacks and 2 Tx callbacks. +Virtio driver has 6 Rx callbacks and 3 Tx callbacks. Rx callbacks: #. ``virtio_recv_pkts``: - Regular version without mergeable Rx buffer support. + Regular version without mergeable Rx buffer support for split virtqueue. #. ``virtio_recv_mergeable_pkts``: - Regular version with mergeable Rx buffer support. + Regular version with mergeable Rx buffer support for split virtqueue. #. ``virtio_recv_pkts_vec``: Vector version without mergeable Rx buffer support, also fixes the available - ring indexes and uses vector instructions to optimize performance. + ring indexes and uses vector instructions to optimize performance for split + virtqueue. + +#. ``virtio_recv_pkts_inorder``: + In-order version with mergeable and non-mergeable Rx buffer support + for split virtqueue. + +#. ``virtio_recv_pkts_packed``: + Regular and in-order version without mergeable Rx buffer support for + packed virtqueue. + +#. ``virtio_recv_mergeable_pkts_packed``: + Regular and in-order version with mergeable Rx buffer support for packed + virtqueue. Tx callbacks: #. ``virtio_xmit_pkts``: - Regular version. + Regular version for split virtqueue. -#. ``virtio_xmit_pkts_simple``: - Vector version fixes the available ring indexes to optimize performance. +#. ``virtio_xmit_pkts_inorder``: + In-order version for split virtqueue. +#. ``virtio_xmit_pkts_packed``: + Regular and in-order version for packed virtqueue. By default, the non-vector callbacks are used: -* For Rx: If mergeable Rx buffers is disabled then ``virtio_recv_pkts`` is - used; otherwise ``virtio_recv_mergeable_pkts``. +* For Rx: If mergeable Rx buffers is disabled then ``virtio_recv_pkts`` + or ``virtio_recv_pkts_packed`` will be used, otherwise + ``virtio_recv_mergeable_pkts`` or ``virtio_recv_mergeable_pkts_packed`` + will be used. -* For Tx: ``virtio_xmit_pkts``. +* For Tx: ``virtio_xmit_pkts`` or ``virtio_xmit_pkts_packed`` will be used. Vector callbacks will be used when: -* ``txq_flags`` is set to ``VIRTIO_SIMPLE_FLAGS`` (0xF01), which implies: - - * Single segment is specified. - - * No offload support is needed. - * Mergeable Rx buffers is disabled. The corresponding callbacks are: * For Rx: ``virtio_recv_pkts_vec``. -* For Tx: ``virtio_xmit_pkts_simple``. +There is no vector callbacks for packed virtqueue for now. Example of using the vector version of the virtio poll mode driver in ``testpmd``:: - testpmd -c 0x7 -n 4 -- -i --txqflags=0xF01 --rxq=1 --txq=1 --nb-cores=1 + testpmd -l 0-2 -n 4 -- -i --rxq=1 --txq=1 --nb-cores=1 + +In-order callbacks only work on simulated virtio user vdev. + +For split virtqueue: + +* For Rx: If in-order is enabled then ``virtio_recv_pkts_inorder`` is used. + +* For Tx: If in-order is enabled then ``virtio_xmit_pkts_inorder`` is used. + +For packed virtqueue, the default callbacks already support the +in-order feature. + +Interrupt mode +-------------- + +.. _virtio_interrupt_mode: + +There are three kinds of interrupts from a virtio device over PCI bus: config +interrupt, Rx interrupts, and Tx interrupts. Config interrupt is used for +notification of device configuration changes, especially link status (lsc). +Interrupt mode is translated into Rx interrupts in the context of DPDK. + +.. Note:: + + Virtio PMD already has support for receiving lsc from qemu when the link + status changes, especially when vhost user disconnects. However, it fails + to do that if the VM is created by qemu 2.6.2 or below, since the + capability to detect vhost user disconnection is introduced in qemu 2.7.0. + +Prerequisites for Rx interrupts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To support Rx interrupts, +#. Check if guest kernel supports VFIO-NOIOMMU: + + Linux started to support VFIO-NOIOMMU since 4.8.0. Make sure the guest + kernel is compiled with: + + .. code-block:: console + + CONFIG_VFIO_NOIOMMU=y + +#. Properly set msix vectors when starting VM: + + Enable multi-queue when starting VM, and specify msix vectors in qemu + cmdline. (N+1) is the minimum, and (2N+2) is mostly recommended. + + .. code-block:: console + + $(QEMU) ... -device virtio-net-pci,mq=on,vectors=2N+2 ... + +#. In VM, insert vfio module in NOIOMMU mode: + + .. code-block:: console + + modprobe vfio enable_unsafe_noiommu_mode=1 + modprobe vfio-pci + +#. In VM, bind the virtio device with vfio-pci: + + .. code-block:: console + + python usertools/dpdk-devbind.py -b vfio-pci 00:03.0 + +Example +~~~~~~~ + +Here we use l3fwd-power as an example to show how to get started. + + Example: + + .. code-block:: console + + $ l3fwd-power -l 0-1 -- -p 1 -P --config="(0,0,1)" \ + --no-numa --parse-ptype + + +Virtio PMD arguments +-------------------- + +Below devargs are supported by the PCI virtio driver: + +#. ``vdpa``: + + A virtio device could also be driven by vDPA (vhost data path acceleration) + driver, and works as a HW vhost backend. This argument is used to specify + a virtio device needs to work in vDPA mode. + (Default: 0 (disabled)) + +Below devargs are supported by the virtio-user vdev: + +#. ``path``: + + It is used to specify a path to connect to vhost backend. + +#. ``mac``: + + It is used to specify the MAC address. + +#. ``cq``: + + It is used to enable the control queue. (Default: 0 (disabled)) + +#. ``queue_size``: + + It is used to specify the queue size. (Default: 256) + +#. ``queues``: + + It is used to specify the queue number. (Default: 1) + +#. ``iface``: + + It is used to specify the host interface name for vhost-kernel + backend. + +#. ``server``: + + It is used to enable the server mode when using vhost-user backend. + (Default: 0 (disabled)) + +#. ``mrg_rxbuf``: + + It is used to enable virtio device mergeable Rx buffer feature. + (Default: 1 (enabled)) + +#. ``in_order``: + + It is used to enable virtio device in-order feature. + (Default: 1 (enabled)) + +#. ``packed_vq``: + + It is used to enable virtio device packed virtqueue feature. + (Default: 0 (disabled)) + +Virtio paths Selection and Usage +-------------------------------- + +Logically virtio-PMD has 9 paths based on the combination of virtio features +(Rx mergeable, In-order, Packed virtqueue), below is an introduction of these +features: + +* `Rx mergeable `_: With this feature negotiated, device + can receive large packets by combining individual descriptors. +* `In-order `_: Some devices always use descriptors + in the same order in which they have been made available, these + devices can offer the VIRTIO_F_IN_ORDER feature. With this feature negotiated, + driver will use descriptors in order. +* `Packed virtqueue `_: The structure of packed virtqueue is + different from split virtqueue, split virtqueue is composed of available ring, + used ring and descriptor table, while packed virtqueue is composed of descriptor + ring, driver event suppression and device event suppression. The idea behind + this is to improve performance by avoiding cache misses and make it easier + for hardware to implement. + +Virtio paths Selection +~~~~~~~~~~~~~~~~~~~~~~ + +If packed virtqueue is not negotiated, below split virtqueue paths will be selected +according to below configuration: + +#. Split virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature is + not negotiated, this path will be selected. +#. Split virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, also Rx offload(s) are requested, this path will be selected. +#. Split virtqueue in-order mergeable path: If Rx mergeable and in-order feature are + both negotiated, this path will be selected. +#. Split virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path will be selected. +#. Split virtqueue vectorized Rx path: If Rx mergeable is disabled and no Rx offload + requested, this path will be selected. + +If packed virtqueue is negotiated, below packed virtqueue paths will be selected +according to below configuration: + +#. Packed virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature + is not negotiated, this path will be selected. +#. Packed virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, this path will be selected. +#. Packed virtqueue in-order mergeable path: If in-order and Rx mergeable feature are + both negotiated, this path will be selected. +#. Packed virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path will be selected. + +Rx/Tx callbacks of each Virtio path +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to above description, virtio path and corresponding Rx/Tx callbacks will +be selected automatically. Rx callbacks and Tx callbacks for each virtio path +are shown in below table: + +.. table:: Virtio Paths and Callbacks + + ============================================ ================================= ======================== + Virtio paths Rx callbacks Tx callbacks + ============================================ ================================= ======================== + Split virtqueue mergeable path virtio_recv_mergeable_pkts virtio_xmit_pkts + Split virtqueue non-mergeable path virtio_recv_pkts virtio_xmit_pkts + Split virtqueue in-order mergeable path virtio_recv_pkts_inorder virtio_xmit_pkts_inorder + Split virtqueue in-order non-mergeable path virtio_recv_pkts_inorder virtio_xmit_pkts_inorder + Split virtqueue vectorized Rx path virtio_recv_pkts_vec virtio_xmit_pkts + Packed virtqueue mergeable path virtio_recv_mergeable_pkts_packed virtio_xmit_pkts_packed + Packed virtqueue non-meregable path virtio_recv_pkts_packed virtio_xmit_pkts_packed + Packed virtqueue in-order mergeable path virtio_recv_mergeable_pkts_packed virtio_xmit_pkts_packed + Packed virtqueue in-order non-mergeable path virtio_recv_pkts_packed virtio_xmit_pkts_packed + ============================================ ================================= ======================== + +Virtio paths Support Status from Release to Release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Virtio feature implementation: + +* In-order feature is supported since DPDK 18.08 by adding new Rx/Tx callbacks + ``virtio_recv_pkts_inorder`` and ``virtio_xmit_pkts_inorder``. +* Packed virtqueue is supported since DPDK 19.02 by adding new Rx/Tx callbacks + ``virtio_recv_pkts_packed`` , ``virtio_recv_mergeable_pkts_packed`` and + ``virtio_xmit_pkts_packed``. + +All virtio paths support status are shown in below table: + +.. table:: Virtio Paths and Releases + + ============================================ ============= ============= ============= + Virtio paths 16.11 ~ 18.05 18.08 ~ 18.11 19.02 ~ 19.11 + ============================================ ============= ============= ============= + Split virtqueue mergeable path Y Y Y + Split virtqueue non-mergeable path Y Y Y + Split virtqueue vectorized Rx path Y Y Y + Split virtqueue simple Tx path Y N N + Split virtqueue in-order mergeable path Y Y + Split virtqueue in-order non-mergeable path Y Y + Packed virtqueue mergeable path Y + Packed virtqueue non-mergeable path Y + Packed virtqueue in-order mergeable path Y + Packed virtqueue in-order non-mergeable path Y + ============================================ ============= ============= ============= + +QEMU Support Status +~~~~~~~~~~~~~~~~~~~ + +* Qemu now supports three paths of split virtqueue: Split virtqueue mergeable path, + Split virtqueue non-mergeable path, Split virtqueue vectorized Rx path. +* Since qemu 4.2.0, Packed virtqueue mergeable path and Packed virtqueue non-mergeable + path can be supported. + +How to Debug +~~~~~~~~~~~~ + +If you meet performance drop or some other issues after upgrading the driver +or configuration, below steps can help you identify which path you selected and +root cause faster. + +#. Run vhost/virtio test case; +#. Run "perf top" and check virtio Rx/Tx callback names; +#. Identify which virtio path is selected refer to above table.