+
.. BSD LICENSE
Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
All rights reserved.
Vhost Sample Application
========================
-The vhost sample application demonstrates integration of the Intel® Data Plane Development Kit (Intel® DPDK)
+The vhost sample application demonstrates integration of the Data Plane Development Kit (DPDK)
with the Linux* KVM hypervisor by implementing the vhost-net offload API.
The sample application performs simple packet switching between virtual machines based on Media Access Control
(MAC) address or Virtual Local Area Network (VLAN) tag.
Virtio networking (virtio-net) was developed as the Linux* KVM para-virtualized method for communicating network packets
between host and guest.
It was found that virtio-net performance was poor due to context switching and packet copying between host, guest, and QEMU.
-The following figure shows the system architecture for a virtio- based networking (virtio-net).
+The following figure shows the system architecture for a virtio-based networking (virtio-net).
.. _figure_16:
Sample Code Overview
--------------------
-The Intel® DPDK vhost-net sample code demonstrates KVM (QEMU) offloading the servicing of a Virtual Machine's (VM's)
-virtio-net devices to an Intel® DPDK-based application in place of the kernel's vhost-net module.
+The DPDK vhost-net sample code demonstrates KVM (QEMU) offloading the servicing of a Virtual Machine's (VM's)
+virtio-net devices to a DPDK-based application in place of the kernel's vhost-net module.
+
+The DPDK vhost-net sample code is based on vhost library. Vhost library is developed for user space ethernet switch to
+easily integrate with vhost functionality.
-The Intel® DPDK vhost-net sample code is a simple packet switching application with the following features:
+The vhost library implements the following features:
* Management of virtio-net device creation/destruction events.
-* Mapping of the VM's physical memory into the Intel® DPDK vhost-net sample code's address space.
+* Mapping of the VM's physical memory into the DPDK vhost-net's address space.
* Triggering/receiving notifications to/from VMs via eventfds.
* A virtio-net back-end implementation providing a subset of virtio-net features.
+There are two vhost implementations in vhost library, vhost cuse and vhost user. In vhost cuse, a character device driver is implemented to
+receive and process vhost requests through ioctl messages. In vhost user, a socket server is created to received vhost requests through
+socket messages. Most of the messages share the same handler routine.
+
+.. note::
+ **Any vhost cuse specific requirement in the following sections will be emphasized**.
+
+Two impelmentations are turned on and off statically through configure file. Only one implementation could be turned on. They don't co-exist in current implementation.
+
+The vhost sample code application is a simple packet switching application with the following feature:
+
* Packet switching between virtio-net devices and the network interface card,
including using VMDQs to reduce the switching that needs to be performed in software.
-The following figure shows the architecture of the Vhost sample application.
+The following figure shows the architecture of the Vhost sample application based on vhost-cuse.
.. _figure_18:
* Fedora* 19
+* Fedora* 20
+
Prerequisites
-------------
This section lists prerequisite packages that must be installed.
-Installing Packages on the Host
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Installing Packages on the Host(vhost cuse required)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The vhost sample code uses the following packages; fuse, fuse-devel, and kernel- modules-extra.
+The vhost cuse code uses the following packages; fuse, fuse-devel, and kernel-modules-extra.
+The vhost user code don't rely on those modules as eventfds are already installed into vhost process through
+unix domain socket.
#. Install Fuse Development Libraries and headers:
yum -y install kernel-modules-extra
+QEMU simulator
+~~~~~~~~~~~~~~
+
+For vhost user, qemu 2.2 is required.
+
Setting up the Execution Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Vhost Sample Code**
-In this section, we create a second hugetlbs mount point to allocate hugepages for the Intel® DPDK vhost sample code.
+In this section, we create a second hugetlbs mount point to allocate hugepages for the DPDK vhost sample code.
-#. Allocate sufficient 2 MB pages for the Intel® DPDK vhost sample code:
+#. Allocate sufficient 2 MB pages for the DPDK vhost sample code:
.. code-block:: console
- echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/ nr_hugepages
+ echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
#. Mount hugetlbs at a separate mount point for 2 MB pages:
Setting up the Guest Execution Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-It is recommended for testing purposes that the Intel® DPDK testpmd sample application is used in the guest to forward packets,
+It is recommended for testing purposes that the DPDK testpmd sample application is used in the guest to forward packets,
the reasons for this are discussed in Section 22.7, "Running the Virtual Machine (QEMU)".
The testpmd application forwards packets between pairs of Ethernet devices,
Observe that in the example, "-device" and "-netdev" are repeated for two virtio-net devices.
+For vhost cuse:
+
.. code-block:: console
user@target:~$ qemu-system-x86_64 ... \
-netdev tap,id=hostnet2,vhost=on,vhostfd=<open fd> \
-device virtio-net-pci, netdev=hostnet2,id=net1
+For vhost user:
+
+.. code-block:: console
+
+ user@target:~$ qemu-system-x86_64 ... \
+ -chardev socket,id=char1,path=<sock_path> \
+ -netdev type=vhost-user,id=hostnet1,chardev=char1 \
+ -device virtio-net-pci,netdev=hostnet1,id=net1 \
+ -chardev socket,id=char2,path=<sock_path> \
+ -netdev type=vhost-user,id=hostnet2,chardev=char2 \
+ -device virtio-net-pci,netdev=hostnet2,id=net2
+
+sock_path is the path for the socket file created by vhost.
Compiling the Sample Code
-------------------------
+#. Compile vhost lib:
-#. Go to the examples directory:
+ To enable vhost, turn on vhost library in the configure file config/common_linuxapp.
.. code-block:: console
- export RTE_SDK=/path/to/rte_sdk cd ${RTE_SDK}/examples/vhost-net
+ CONFIG_RTE_LIBRTE_VHOST=n
-#. Set the target (a default target is used if not specified). For example:
+ vhost user is turned on by default in the lib/librte_vhost/Makefile.
+ To enable vhost cuse, uncomment vhost cuse and comment vhost user manually. In future, a configure will be created for switch between two implementations.
.. code-block:: console
- export RTE_TARGET=x86_64-native-linuxapp-gcc
+ SRCS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_cuse/vhost-net-cdev.c vhost_cuse/virtio-net-cdev.c vhost_cuse/eventfd_copy.c
+ #SRCS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_user/vhost-net-user.c vhost_user/virtio-net-user.c vhost_user/fd_man.c
- See the Intel® DPDK Getting Started Guide for possible RTE_TARGET values.
+ After vhost is enabled and the implementation is selected, build the vhost library.
-#. Build the application:
+#. Go to the examples directory:
.. code-block:: console
- make
-
- .. note::
+ export RTE_SDK=/path/to/rte_sdk
+ cd ${RTE_SDK}/examples/vhost
- Note For zero copy, need firstly disable CONFIG_RTE_MBUF_SCATTER_GATHER,
- CONFIG_RTE_LIBRTE_IP_FRAG and CONFIG_RTE_LIBRTE_DISTRIBUTOR
- in the config file and then re-configure and compile the core lib, and then build the application:
+#. Set the target (a default target is used if not specified). For example:
.. code-block:: console
- vi ${RTE_SDK}/config/common_linuxapp
-
- change it as follows:
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
- ::
+ See the DPDK Getting Started Guide for possible RTE_TARGET values.
- CONFIG_RTE_MBUF_SCATTER_GATHER=n
- CONFIG_RTE_LIBRTE_IP_FRAG=n
- CONFIG_RTE_LIBRTE_DISTRIBUTOR=n
+#. Build the application:
.. code-block:: console
cd ${RTE_SDK}/examples/vhost
make
-#. Go to the eventfd_link directory:
+#. Go to the eventfd_link directory(vhost cuse required):
.. code-block:: console
- cd ${RTE_SDK}/examples/vhost-net/eventfd_link
+ cd ${RTE_SDK}/lib/librte_vhost/eventfd_link
-#. Build the eventfd_link kernel module:
+#. Build the eventfd_link kernel module(vhost cuse required):
.. code-block:: console
Running the Sample Code
-----------------------
-#. Install the cuse kernel module:
+#. Install the cuse kernel module(vhost cuse required):
.. code-block:: console
modprobe cuse
-#. Go to the eventfd_link directory:
+#. Go to the eventfd_link directory(vhost cuse required):
.. code-block:: console
export RTE_SDK=/path/to/rte_sdk
- cd ${RTE_SDK}/examples/vhost-net/eventfd_link
+ cd ${RTE_SDK}/lib/librte_vhost/eventfd_link
-#. Install the eventfd_link module:
+#. Install the eventfd_link module(vhost cuse required):
.. code-block:: console
.. code-block:: console
export RTE_SDK=/path/to/rte_sdk
- cd ${RTE_SDK}/examples/vhost-net
+ cd ${RTE_SDK}/examples/vhost
#. Run the vhost-switch sample code:
+ vhost cuse:
+
.. code-block:: console
user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- -p 0x1 --dev-basename usvhost --dev-index 1
+ vhost user: a socket file named usvhost will be created under current directory. Use its path as the socket path in guest's qemu commandline.
+
+ .. code-block:: console
+
+ user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- -p 0x1 --dev-basename usvhost
+
.. note::
- Please note the huge-dir parameter instructs the Intel® DPDK to allocate its memory from the 2 MB page hugetlbfs.
+ Please note the huge-dir parameter instructs the DPDK to allocate its memory from the 2 MB page hugetlbfs.
Parameters
~~~~~~~~~~
**Basename and Index.**
-The Intel® DPDK vhost-net sample code uses a Linux* character device to communicate with QEMU.
+vhost cuse uses a Linux* character device to communicate with QEMU.
The basename and the index are used to generate the character devices name.
/dev/<basename>-<index>
user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --zero-copy 1 --tx-desc-num [0, n]
+**VLAN strip.**
+The VLAN strip option enable/disable the VLAN strip on host, if disabled, the guest will receive the packets with VLAN tag.
+It is enabled by default.
+
+.. code-block:: console
+
+ user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --vlan-strip [0, 1]
+
Running the Virtual Machine (QEMU)
----------------------------------
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci, netdev=hostnet1,id=net1 ...
+ user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1,id=net1 ...
* Ensure the guest's virtio-net network adapter is configured with offloads disabled.
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci, netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_ tso6=off,guest_ecn=off
+ user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
-* Redirect QEMU to communicate with the Intel® DPDK vhost-net sample code in place of the vhost-net kernel module.
+* Redirect QEMU to communicate with the DPDK vhost-net sample code in place of the vhost-net kernel module(vhost cuse).
.. code-block:: console
The QEMU wrapper (qemu-wrap.py) is a Python script designed to automate the QEMU configuration described above.
It also facilitates integration with libvirt, although the script may also be used standalone without libvirt.
-Redirecting QEMU to vhost-net Sample Code
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Redirecting QEMU to vhost-net Sample Code(vhost cuse)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To redirect QEMU to the vhost-net sample code implementation of the vhost-net API,
an open file descriptor must be passed to QEMU running as a child process.
.. note::
- This process is automated in the QEMU wrapper script discussed in Section 22.7.3.
+ This process is automated in the QEMU wrapper script discussed in Section 24.7.3.
Mapping the Virtual Machine's Memory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-For the Intel® DPDK vhost-net sample code to be run correctly, QEMU must allocate the VM's memory on hugetlbfs.
+For the DPDK vhost-net sample code to be run correctly, QEMU must allocate the VM's memory on hugetlbfs.
This is done by specifying mem-prealloc and mem-path when executing QEMU.
The vhost-net sample code accesses the virtio-net device's virtual rings and packet buffers
by finding and mapping the VM's physical memory on hugetlbfs.
.. note::
- This process is automated in the QEMU wrapper script discussed in Section 22.7.3.
+ This process is automated in the QEMU wrapper script discussed in Section 24.7.3.
+ The following two sections only applies to vhost cuse. For vhost-user, please make corresponding changes to qemu-wrapper script and guest XML file.
QEMU Wrapper Script
~~~~~~~~~~~~~~~~~~~
.. code-block:: console
/usr/local/bin/qemu-system-x86_64 -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1
- -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> -device virtio-net- pci,netdev=hostnet1,id=net1,
- csum=off,gso=off,guest_tso4=off,gu est_tso6=off,guest_ecn=off -hda <disk img> -m 4096 -mem-path /dev/hugepages -mem-prealloc
+ -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> -device virtio-net-pci,netdev=hostnet1,id=net1,
+ csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -hda <disk img> -m 4096 -mem-path /dev/hugepages -mem-prealloc
Libvirt Integration
~~~~~~~~~~~~~~~~~~~
<driver name="vhost"/>
<interface/>
- * Enable libvirt to access the Intel® DPDK Vhost sample code's character device file by adding it
+ * Enable libvirt to access the DPDK Vhost sample code's character device file by adding it
to controllers cgroup for libvirtd using the following steps:
.. code-block:: xml
emul_path = "/usr/local/bin/qemu-system-x86_64"
- * Configure the "us_vhost_path" variable to point to the Intel® DPDK vhost- net sample code's character devices name.
- Intel® DPDK vhost-net sample code's character device will be in the format "/dev/<basename>-<index>".
+ * Configure the "us_vhost_path" variable to point to the DPDK vhost-net sample code's character devices name.
+ DPDK vhost-net sample code's character device will be in the format "/dev/<basename>-<index>".
.. code-block:: xml
Common Issues
~~~~~~~~~~~~~
-**QEMU failing to allocate memory on hugetlbfs.**
+* QEMU failing to allocate memory on hugetlbfs, with an error like the following::
-file_ram_alloc: can't mmap RAM pages: Cannot allocate memory
+ file_ram_alloc: can't mmap RAM pages: Cannot allocate memory
-When running QEMU the above error implies that it has failed to allocate memory for the Virtual Machine on the hugetlbfs.
-This is typically due to insufficient hugepages being free to support the allocation request.
-The number of free hugepages can be checked as follows:
+ When running QEMU the above error indicates that it has failed to allocate memory for the Virtual Machine on
+ the hugetlbfs. This is typically due to insufficient hugepages being free to support the allocation request.
+ The number of free hugepages can be checked as follows:
-.. code-block:: console
+ .. code-block:: console
+
+ cat /sys/kernel/mm/hugepages/hugepages-<pagesize>/nr_hugepages
+
+ The command above indicates how many hugepages are free to support QEMU's allocation request.
+
+* User space VHOST when the guest has 2MB sized huge pages:
+
+ The guest may have 2MB or 1GB sized huge pages. The user space VHOST should work properly in both cases.
+
+* User space VHOST will not work with QEMU without the ``-mem-prealloc`` option:
+
+ The current implementation works properly only when the guest memory is pre-allocated, so it is required to
+ use a QEMU version (e.g. 1.6) which supports ``-mem-prealloc``. The ``-mem-prealloc`` option must be
+ specified explicitly in the QEMU command line.
+
+* User space VHOST will not work with a QEMU version without shared memory mapping:
+
+ As shared memory mapping is mandatory for user space VHOST to work properly with the guest, user space VHOST
+ needs access to the shared memory from the guest to receive and transmit packets. It is important to make sure
+ the QEMU version supports shared memory mapping.
+
+* Issues with ``virsh destroy`` not destroying the VM:
+
+ Using libvirt ``virsh create`` the ``qemu-wrap.py`` spawns a new process to run ``qemu-kvm``. This impacts the behavior
+ of ``virsh destroy`` which kills the process running ``qemu-wrap.py`` without actually destroying the VM (it leaves
+ the ``qemu-kvm`` process running):
+
+ This following patch should fix this issue:
+ http://dpdk.org/ml/archives/dev/2014-June/003607.html
+
+* In an Ubuntu environment, QEMU fails to start a new guest normally with user space VHOST due to not being able
+ to allocate huge pages for the new guest:
+
+ The solution for this issue is to add ``-boot c`` into the QEMU command line to make sure the huge pages are
+ allocated properly and then the guest should start normally.
+
+ Use ``cat /proc/meminfo`` to check if there is any changes in the value of ``HugePages_Total`` and ``HugePages_Free``
+ after the guest startup.
+
+* Log message: ``eventfd_link: module verification failed: signature and/or required key missing - tainting kernel``:
- user@target:cat /sys/kernel/mm/hugepages/hugepages-<pagesize> / nr_hugepages
+ This log message may be ignored. The message occurs due to the kernel module ``eventfd_link``, which is not a standard
+ Linux module but which is necessary for the user space VHOST current implementation (CUSE-based) to communicate with
+ the guest.
-The command above indicates how many hugepages are free to support QEMU's allocation request.
-Running Intel® DPDK in the Virtual Machine
-------------------------------------------
+Running DPDK in the Virtual Machine
+-----------------------------------
-For the Intel® DPDK vhost-net sample code to switch packets into the VM,
+For the DPDK vhost-net sample code to switch packets into the VM,
the sample code must first learn the MAC address of the VM's virtio-net device.
The sample code detects the address from packets being transmitted from the VM, similar to a learning switch.
This behavior requires no special action or configuration with the Linux* virtio-net driver in the VM
as the Linux* Kernel will automatically transmit packets during device initialization.
-However, Intel® DPDK-based applications must be modified to automatically transmit packets during initialization
-to facilitate the Intel® DPDK vhost- net sample code's MAC learning.
+However, DPDK-based applications must be modified to automatically transmit packets during initialization
+to facilitate the DPDK vhost- net sample code's MAC learning.
-The Intel® DPDK testpmd application can be configured to automatically transmit packets during initialization
+The DPDK testpmd application can be configured to automatically transmit packets during initialization
and to act as an L2 forwarding switch.
Testpmd MAC Forwarding
To resolve this issue, a "wait and retry" mode is implemented in the testpmd and vhost sample code.
In the "wait and retry" mode if the virtqueue is found to be full, then testpmd waits for a period of time before retrying to enqueue packets.
-The "wait and retry" algorithm is implemented in Intel® DPDK testpmd as a forwarding method call "mac_retry".
+The "wait and retry" algorithm is implemented in DPDK testpmd as a forwarding method call "mac_retry".
The following sequence diagram describes the algorithm in detail.
.. _figure_20:
Running Testpmd
~~~~~~~~~~~~~~~
-The testpmd application is automatically built when Intel® DPDK is installed.
+The testpmd application is automatically built when DPDK is installed.
Run the testpmd application as follows:
.. code-block:: console
testpmd> set fwd mac_retry
The testpmd can then be configured to start processing packets,
-transmitting packets first so the Intel® DPDK vhost sample code on the host can learn the MAC address:
+transmitting packets first so the DPDK vhost sample code on the host can learn the MAC address:
.. code-block:: console
For a virtio-net device to receive traffic,
the traffic's Layer 2 header must include both the virtio-net device's MAC address and VLAN tag.
-The Intel® DPDK sample code behaves in a similar manner to a learning switch in that
+The DPDK sample code behaves in a similar manner to a learning switch in that
it learns the MAC address of the virtio-net devices from the first transmitted packet.
On learning the MAC address,
-the Intel® DPDK vhost sample code prints a message with the MAC address and VLAN tag virtio-net device.
+the DPDK vhost sample code prints a message with the MAC address and VLAN tag virtio-net device.
For example:
.. code-block:: console
The above message indicates that device 0 has been registered with MAC address cc:bb:bb:bb:bb:bb and VLAN tag 1000.
Any packets received on the NIC with these values is placed on the devices receive queue.
-When a virtio-net device transmits packets, the VLAN tag is added to the packet by the Intel® DPDK vhost sample code.
+When a virtio-net device transmits packets, the VLAN tag is added to the packet by the DPDK vhost sample code.
-.. |vhost_net_arch| image:: img/vhost_net_arch.png
+.. |vhost_net_arch| image:: img/vhost_net_arch.*
-.. |qemu_virtio_net| image:: img/qemu_virtio_net.png
+.. |qemu_virtio_net| image:: img/qemu_virtio_net.*
-.. |tx_dpdk_testpmd| image:: img/tx_dpdk_testpmd.png
+.. |tx_dpdk_testpmd| image:: img/tx_dpdk_testpmd.*
-.. |vhost_net_sample_app| image:: img/vhost_net_sample_app.png
+.. |vhost_net_sample_app| image:: img/vhost_net_sample_app.*
-.. |virtio_linux_vhost| image:: img/virtio_linux_vhost.png
+.. |virtio_linux_vhost| image:: img/virtio_linux_vhost.*
git.droids-corp.org / dpdk.git / blobdiff