3 Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
6 Redistribution and use in source and binary forms, with or without
7 modification, are permitted provided that the following conditions
10 * Redistributions of source code must retain the above copyright
11 notice, this list of conditions and the following disclaimer.
12 * Redistributions in binary form must reproduce the above copyright
13 notice, this list of conditions and the following disclaimer in
14 the documentation and/or other materials provided with the
16 * Neither the name of Intel Corporation nor the names of its
17 contributors may be used to endorse or promote products derived
18 from this software without specific prior written permission.
20 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
21 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
22 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
23 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
25 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
26 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
27 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
28 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
29 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
30 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
33 Vhost Sample Application
34 ========================
36 The vhost sample application demonstrates integration of the Data Plane Development Kit (DPDK)
37 with the Linux* KVM hypervisor by implementing the vhost-net offload API.
38 The sample application performs simple packet switching between virtual machines based on Media Access Control
39 (MAC) address or Virtual Local Area Network (VLAN) tag.
40 The splitting of Ethernet traffic from an external switch is performed in hardware by the Virtual Machine Device Queues
41 (VMDQ) and Data Center Bridging (DCB) features of the IntelĀ® 82599 10 Gigabit Ethernet Controller.
46 Virtio networking (virtio-net) was developed as the Linux* KVM para-virtualized method for communicating network packets
47 between host and guest.
48 It was found that virtio-net performance was poor due to context switching and packet copying between host, guest, and QEMU.
49 The following figure shows the system architecture for a virtio-based networking (virtio-net).
51 .. _figure_qemu_virtio_net:
53 .. figure:: img/qemu_virtio_net.*
55 System Architecture for Virtio-based Networking (virtio-net).
58 The Linux* Kernel vhost-net module was developed as an offload mechanism for virtio-net.
59 The vhost-net module enables KVM (QEMU) to offload the servicing of virtio-net devices to the vhost-net kernel module,
60 reducing the context switching and packet copies in the virtual dataplane.
62 This is achieved by QEMU sharing the following information with the vhost-net module through the vhost-net API:
64 * The layout of the guest memory space, to enable the vhost-net module to translate addresses.
66 * The locations of virtual queues in QEMU virtual address space,
67 to enable the vhost module to read/write directly to and from the virtqueues.
69 * An event file descriptor (eventfd) configured in KVM to send interrupts to the virtio- net device driver in the guest.
70 This enables the vhost-net module to notify (call) the guest.
72 * An eventfd configured in KVM to be triggered on writes to the virtio-net device's
73 Peripheral Component Interconnect (PCI) config space.
74 This enables the vhost-net module to receive notifications (kicks) from the guest.
76 The following figure shows the system architecture for virtio-net networking with vhost-net offload.
78 .. _figure_virtio_linux_vhost:
80 .. figure:: img/virtio_linux_vhost.*
88 The DPDK vhost-net sample code demonstrates KVM (QEMU) offloading the servicing of a Virtual Machine's (VM's)
89 virtio-net devices to a DPDK-based application in place of the kernel's vhost-net module.
91 The DPDK vhost-net sample code is based on vhost library. Vhost library is developed for user space Ethernet switch to
92 easily integrate with vhost functionality.
94 The vhost library implements the following features:
96 * Management of virtio-net device creation/destruction events.
98 * Mapping of the VM's physical memory into the DPDK vhost-net's address space.
100 * Triggering/receiving notifications to/from VMs via eventfds.
102 * A virtio-net back-end implementation providing a subset of virtio-net features.
104 There are two vhost implementations in vhost library, vhost cuse and vhost user. In vhost cuse, a character device driver is implemented to
105 receive and process vhost requests through ioctl messages. In vhost user, a socket server is created to received vhost requests through
106 socket messages. Most of the messages share the same handler routine.
109 **Any vhost cuse specific requirement in the following sections will be emphasized**.
111 Two implementations are turned on and off statically through configure file. Only one implementation could be turned on. They don't co-exist in current implementation.
113 The vhost sample code application is a simple packet switching application with the following feature:
115 * Packet switching between virtio-net devices and the network interface card,
116 including using VMDQs to reduce the switching that needs to be performed in software.
118 The following figure shows the architecture of the Vhost sample application based on vhost-cuse.
120 .. _figure_vhost_net_arch:
122 .. figure:: img/vhost_net_arch.*
124 Vhost-net Architectural Overview
127 The following figure shows the flow of packets through the vhost-net sample application.
129 .. _figure_vhost_net_sample_app:
131 .. figure:: img/vhost_net_sample_app.*
133 Packet Flow Through the vhost-net Sample Application
136 Supported Distributions
137 -----------------------
139 The example in this section have been validated with the following distributions:
147 .. _vhost_app_prerequisites:
152 This section lists prerequisite packages that must be installed.
154 Installing Packages on the Host(vhost cuse required)
155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
157 The vhost cuse code uses the following packages; fuse, fuse-devel, and kernel-modules-extra.
158 The vhost user code don't rely on those modules as eventfds are already installed into vhost process through
161 #. Install Fuse Development Libraries and headers:
163 .. code-block:: console
165 yum -y install fuse fuse-devel
167 #. Install the Cuse Kernel Module:
169 .. code-block:: console
171 yum -y install kernel-modules-extra
176 For vhost user, qemu 2.2 is required.
178 Setting up the Execution Environment
179 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
181 The vhost sample code requires that QEMU allocates a VM's memory on the hugetlbfs file system.
182 As the vhost sample code requires hugepages,
183 the best practice is to partition the system into separate hugepage mount points for the VMs and the vhost sample code.
187 This is best-practice only and is not mandatory.
188 For systems that only support 2 MB page sizes,
189 both QEMU and vhost sample code can use the same hugetlbfs mount point without issue.
193 VMs with gigabytes of memory can benefit from having QEMU allocate their memory from 1 GB huge pages.
194 1 GB huge pages must be allocated at boot time by passing kernel parameters through the grub boot loader.
196 #. Calculate the maximum memory usage of all VMs to be run on the system.
197 Then, round this value up to the nearest Gigabyte the execution environment will require.
199 #. Edit the /etc/default/grub file, and add the following to the GRUB_CMDLINE_LINUX entry:
201 .. code-block:: console
203 GRUB_CMDLINE_LINUX="... hugepagesz=1G hugepages=<Number of hugepages required> default_hugepagesz=1G"
205 #. Update the grub boot loader:
207 .. code-block:: console
209 grub2-mkconfig -o /boot/grub2/grub.cfg
211 #. Reboot the system.
213 #. The hugetlbfs mount point (/dev/hugepages) should now default to allocating gigabyte pages.
217 Making the above modification will change the system default hugepage size to 1 GB for all applications.
219 **Vhost Sample Code**
221 In this section, we create a second hugetlbs mount point to allocate hugepages for the DPDK vhost sample code.
223 #. Allocate sufficient 2 MB pages for the DPDK vhost sample code:
225 .. code-block:: console
227 echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
229 #. Mount hugetlbs at a separate mount point for 2 MB pages:
231 .. code-block:: console
233 mount -t hugetlbfs nodev /mnt/huge -o pagesize=2M
235 The above steps can be automated by doing the following:
237 #. Edit /etc/fstab to add an entry to automatically mount the second hugetlbfs mount point:
241 hugetlbfs <tab> /mnt/huge <tab> hugetlbfs defaults,pagesize=1G 0 0
243 #. Edit the /etc/default/grub file, and add the following to the GRUB_CMDLINE_LINUX entry:
247 GRUB_CMDLINE_LINUX="... hugepagesz=2M hugepages=256 ... default_hugepagesz=1G"
249 #. Update the grub bootloader:
251 .. code-block:: console
253 grub2-mkconfig -o /boot/grub2/grub.cfg
255 #. Reboot the system.
259 Ensure that the default hugepage size after this setup is 1 GB.
261 Setting up the Guest Execution Environment
262 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
264 It is recommended for testing purposes that the DPDK testpmd sample application is used in the guest to forward packets,
265 the reasons for this are discussed in `Running the Virtual Machine (QEMU)`_.
267 The testpmd application forwards packets between pairs of Ethernet devices,
268 it requires an even number of Ethernet devices (virtio or otherwise) to execute.
269 It is therefore recommended to create multiples of two virtio-net devices for each Virtual Machine either through libvirt or
270 at the command line as follows.
274 Observe that in the example, "-device" and "-netdev" are repeated for two virtio-net devices.
278 .. code-block:: console
280 qemu-system-x86_64 ... \
281 -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> \
282 -device virtio-net-pci, netdev=hostnet1,id=net1 \
283 -netdev tap,id=hostnet2,vhost=on,vhostfd=<open fd> \
284 -device virtio-net-pci, netdev=hostnet2,id=net1
288 .. code-block:: console
290 qemu-system-x86_64 ... \
291 -chardev socket,id=char1,path=<sock_path> \
292 -netdev type=vhost-user,id=hostnet1,chardev=char1 \
293 -device virtio-net-pci,netdev=hostnet1,id=net1 \
294 -chardev socket,id=char2,path=<sock_path> \
295 -netdev type=vhost-user,id=hostnet2,chardev=char2 \
296 -device virtio-net-pci,netdev=hostnet2,id=net2
298 sock_path is the path for the socket file created by vhost.
300 Compiling the Sample Code
301 -------------------------
302 #. Compile vhost lib:
304 To enable vhost, turn on vhost library in the configure file config/common_linuxapp.
306 .. code-block:: console
308 CONFIG_RTE_LIBRTE_VHOST=n
310 vhost user is turned on by default in the configure file config/common_linuxapp.
311 To enable vhost cuse, disable vhost user.
313 .. code-block:: console
315 CONFIG_RTE_LIBRTE_VHOST_USER=y
317 After vhost is enabled and the implementation is selected, build the vhost library.
319 #. Go to the examples directory:
321 .. code-block:: console
323 export RTE_SDK=/path/to/rte_sdk
324 cd ${RTE_SDK}/examples/vhost
326 #. Set the target (a default target is used if not specified). For example:
328 .. code-block:: console
330 export RTE_TARGET=x86_64-native-linuxapp-gcc
332 See the DPDK Getting Started Guide for possible RTE_TARGET values.
334 #. Build the application:
336 .. code-block:: console
339 make config ${RTE_TARGET}
340 make install ${RTE_TARGET}
341 cd ${RTE_SDK}/examples/vhost
344 #. Go to the eventfd_link directory(vhost cuse required):
346 .. code-block:: console
348 cd ${RTE_SDK}/lib/librte_vhost/eventfd_link
350 #. Build the eventfd_link kernel module(vhost cuse required):
352 .. code-block:: console
356 Running the Sample Code
357 -----------------------
359 #. Install the cuse kernel module(vhost cuse required):
361 .. code-block:: console
365 #. Go to the eventfd_link directory(vhost cuse required):
367 .. code-block:: console
369 export RTE_SDK=/path/to/rte_sdk
370 cd ${RTE_SDK}/lib/librte_vhost/eventfd_link
372 #. Install the eventfd_link module(vhost cuse required):
374 .. code-block:: console
376 insmod ./eventfd_link.ko
378 #. Go to the examples directory:
380 .. code-block:: console
382 export RTE_SDK=/path/to/rte_sdk
383 cd ${RTE_SDK}/examples/vhost/build/app
385 #. Run the vhost-switch sample code:
389 .. code-block:: console
391 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
392 -- -p 0x1 --dev-basename usvhost
394 vhost user: a socket file named usvhost will be created under current directory. Use its path as the socket path in guest's qemu commandline.
396 .. code-block:: console
398 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
399 -- -p 0x1 --dev-basename usvhost
403 Please note the huge-dir parameter instructs the DPDK to allocate its memory from the 2 MB page hugetlbfs.
407 The number used with the --socket-mem parameter may need to be more than 1024.
408 The number required depends on the number of mbufs allocated by vhost-switch.
410 .. _vhost_app_parameters:
416 vhost cuse uses a Linux* character device to communicate with QEMU.
417 The basename is used to generate the character devices name.
421 For compatibility with the QEMU wrapper script, a base name of "usvhost" should be used:
423 .. code-block:: console
425 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
426 -- -p 0x1 --dev-basename usvhost
429 The vm2vm parameter disable/set mode of packet switching between guests in the host.
430 Value of "0" means disabling vm2vm implies that on virtual machine packet transmission will always go to the Ethernet port;
431 Value of "1" means software mode packet forwarding between guests, it needs packets copy in vHOST,
432 so valid only in one-copy implementation, and invalid for zero copy implementation;
433 value of "2" means hardware mode packet forwarding between guests, it allows packets go to the Ethernet port,
434 hardware L2 switch will determine which guest the packet should forward to or need send to external,
435 which bases on the packet destination MAC address and VLAN tag.
437 .. code-block:: console
439 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
442 **Mergeable Buffers.**
443 The mergeable buffers parameter controls how virtio-net descriptors are used for virtio-net headers.
444 In a disabled state, one virtio-net header is used per packet buffer;
445 in an enabled state one virtio-net header is used for multiple packets.
446 The default value is 0 or disabled since recent kernels virtio-net drivers show performance degradation with this feature is enabled.
448 .. code-block:: console
450 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
454 The stats parameter controls the printing of virtio-net device statistics.
455 The parameter specifies an interval second to print statistics, with an interval of 0 seconds disabling statistics.
457 .. code-block:: console
459 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
463 The rx-retry option enables/disables enqueue retries when the guests RX queue is full.
464 This feature resolves a packet loss that is observed at high data-rates,
465 by allowing it to delay and retry in the receive path.
466 This option is enabled by default.
468 .. code-block:: console
470 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
474 The rx-retry-num option specifies the number of retries on an RX burst,
475 it takes effect only when rx retry is enabled.
476 The default value is 4.
478 .. code-block:: console
480 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
481 -- --rx-retry 1 --rx-retry-num 5
483 **RX Retry Delay Time.**
484 The rx-retry-delay option specifies the timeout (in micro seconds) between retries on an RX burst,
485 it takes effect only when rx retry is enabled.
486 The default value is 15.
488 .. code-block:: console
490 ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
491 -- --rx-retry 1 --rx-retry-delay 20
494 Zero copy mode is removed, due to it has not been working for a while. And
495 due to the large and complex code, it's better to redesign it than fixing
496 it to make it work again. Hence, zero copy may be added back later.
499 VLAN strip option is removed, because different NICs have different behaviors
500 when disabling VLAN strip. Such feature, which heavily depends on hardware,
501 should be removed from this example to reduce confusion. Now, VLAN strip is
502 enabled and cannot be disabled.
504 .. _vhost_app_running:
506 Running the Virtual Machine (QEMU)
507 ----------------------------------
509 QEMU must be executed with specific parameters to:
511 * Ensure the guest is configured to use virtio-net network adapters.
513 .. code-block:: console
515 qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1, \
518 * Ensure the guest's virtio-net network adapter is configured with offloads disabled.
520 .. code-block:: console
522 qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1, \
523 id=net1, csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
525 * Redirect QEMU to communicate with the DPDK vhost-net sample code in place of the vhost-net kernel module(vhost cuse).
527 .. code-block:: console
529 qemu-system-x86_64 ... -netdev tap,id=hostnet1,vhost=on, \
530 vhostfd=<open fd> ...
532 * Enable the vhost-net sample code to map the VM's memory into its own process address space.
534 .. code-block:: console
536 qemu-system-x86_64 ... -mem-prealloc -mem-path /dev/hugepages ...
540 The QEMU wrapper (qemu-wrap.py) is a Python script designed to automate the QEMU configuration described above.
541 It also facilitates integration with libvirt, although the script may also be used standalone without libvirt.
543 Redirecting QEMU to vhost-net Sample Code(vhost cuse)
544 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
546 To redirect QEMU to the vhost-net sample code implementation of the vhost-net API,
547 an open file descriptor must be passed to QEMU running as a child process.
549 .. code-block:: python
552 fd = os.open("/dev/usvhost-1", os.O_RDWR)
554 ("qemu-system-x86_64 ... -netdev tap,id=vhostnet0,vhost=on,vhostfd="
555 + fd +"...", shell=True)
559 This process is automated in the `QEMU Wrapper Script`_.
561 Mapping the Virtual Machine's Memory
562 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
564 For the DPDK vhost-net sample code to be run correctly, QEMU must allocate the VM's memory on hugetlbfs.
565 This is done by specifying mem-prealloc and mem-path when executing QEMU.
566 The vhost-net sample code accesses the virtio-net device's virtual rings and packet buffers
567 by finding and mapping the VM's physical memory on hugetlbfs.
568 In this case, the path passed to the guest should be that of the 1 GB page hugetlbfs:
570 .. code-block:: console
572 qemu-system-x86_64 ... -mem-prealloc -mem-path /dev/hugepages ...
576 This process is automated in the `QEMU Wrapper Script`_.
577 The following two sections only applies to vhost cuse.
578 For vhost-user, please make corresponding changes to qemu-wrapper script and guest XML file.
583 The QEMU wrapper script automatically detects and calls QEMU with the necessary parameters required
584 to integrate with the vhost sample code.
585 It performs the following actions:
587 * Automatically detects the location of the hugetlbfs and inserts this into the command line parameters.
589 * Automatically open file descriptors for each virtio-net device and inserts this into the command line parameters.
591 * Disables offloads on each virtio-net device.
593 * Calls Qemu passing both the command line parameters passed to the script itself and those it has auto-detected.
595 The QEMU wrapper script will automatically configure calls to QEMU:
597 .. code-block:: console
599 qemu-wrap.py -machine pc-i440fx-1.4,accel=kvm,usb=off \
600 -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1 \
601 -netdev tap,id=hostnet1,vhost=on \
602 -device virtio-net-pci,netdev=hostnet1,id=net1 \
603 -hda <disk img> -m 4096
605 which will become the following call to QEMU:
607 .. code-block:: console
609 qemu-system-x86_64 -machine pc-i440fx-1.4,accel=kvm,usb=off \
610 -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1 \
611 -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> \
612 -device virtio-net-pci,netdev=hostnet1,id=net1, \
613 csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off \
614 -hda <disk img> -m 4096 -mem-path /dev/hugepages -mem-prealloc
619 The QEMU wrapper script (qemu-wrap.py) "wraps" libvirt calls to QEMU,
620 such that QEMU is called with the correct parameters described above.
621 To call the QEMU wrapper automatically from libvirt, the following configuration changes must be made:
623 * Place the QEMU wrapper script in libvirt's binary search PATH ($PATH).
624 A good location is in the directory that contains the QEMU binary.
626 * Ensure that the script has the same owner/group and file permissions as the QEMU binary.
628 * Update the VM xml file using virsh edit <vm name>:
630 * Set the VM to use the launch script
632 * Set the emulator path contained in the #<emulator><emulator/> tags For example,
633 replace <emulator>/usr/bin/qemu-kvm<emulator/> with <emulator>/usr/bin/qemu-wrap.py<emulator/>
635 * Set the VM's virtio-net device's to use vhost-net offload:
639 <interface type="network">
640 <model type="virtio"/>
641 <driver name="vhost"/>
644 * Enable libvirt to access the DPDK Vhost sample code's character device file by adding it
645 to controllers cgroup for libvirtd using the following steps:
649 cgroup_controllers = [ ... "devices", ... ] clear_emulator_capabilities = 0
650 user = "root" group = "root"
651 cgroup_device_acl = [
652 "/dev/null", "/dev/full", "/dev/zero",
653 "/dev/random", "/dev/urandom",
654 "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
655 "/dev/rtc", "/dev/hpet", "/dev/net/tun",
656 "/dev/<devbase-name>-<index>",
659 * Disable SELinux or set to permissive mode.
662 * Mount cgroup device controller:
664 .. code-block:: console
667 mount -t cgroup none /dev/cgroup -o devices
669 * Restart the libvirtd system process
671 For example, on Fedora* "systemctl restart libvirtd.service"
673 * Edit the configuration parameters section of the script:
675 * Configure the "emul_path" variable to point to the QEMU emulator.
679 emul_path = "/usr/local/bin/qemu-system-x86_64"
681 * Configure the "us_vhost_path" variable to point to the DPDK vhost-net sample code's character devices name.
682 DPDK vhost-net sample code's character device will be in the format "/dev/<basename>".
686 us_vhost_path = "/dev/usvhost"
691 * QEMU failing to allocate memory on hugetlbfs, with an error like the following::
693 file_ram_alloc: can't mmap RAM pages: Cannot allocate memory
695 When running QEMU the above error indicates that it has failed to allocate memory for the Virtual Machine on
696 the hugetlbfs. This is typically due to insufficient hugepages being free to support the allocation request.
697 The number of free hugepages can be checked as follows:
699 .. code-block:: console
701 cat /sys/kernel/mm/hugepages/hugepages-<pagesize>/nr_hugepages
703 The command above indicates how many hugepages are free to support QEMU's allocation request.
705 * User space VHOST when the guest has 2MB sized huge pages:
707 The guest may have 2MB or 1GB sized huge pages. The user space VHOST should work properly in both cases.
709 * User space VHOST will not work with QEMU without the ``-mem-prealloc`` option:
711 The current implementation works properly only when the guest memory is pre-allocated, so it is required to
712 use a QEMU version (e.g. 1.6) which supports ``-mem-prealloc``. The ``-mem-prealloc`` option must be
713 specified explicitly in the QEMU command line.
715 * User space VHOST will not work with a QEMU version without shared memory mapping:
717 As shared memory mapping is mandatory for user space VHOST to work properly with the guest, user space VHOST
718 needs access to the shared memory from the guest to receive and transmit packets. It is important to make sure
719 the QEMU version supports shared memory mapping.
721 * In an Ubuntu environment, QEMU fails to start a new guest normally with user space VHOST due to not being able
722 to allocate huge pages for the new guest:
724 The solution for this issue is to add ``-boot c`` into the QEMU command line to make sure the huge pages are
725 allocated properly and then the guest should start normally.
727 Use ``cat /proc/meminfo`` to check if there is any changes in the value of ``HugePages_Total`` and ``HugePages_Free``
728 after the guest startup.
730 * Log message: ``eventfd_link: module verification failed: signature and/or required key missing - tainting kernel``:
732 This log message may be ignored. The message occurs due to the kernel module ``eventfd_link``, which is not a standard
733 Linux module but which is necessary for the user space VHOST current implementation (CUSE-based) to communicate with
736 .. _vhost_app_running_dpdk:
738 Running DPDK in the Virtual Machine
739 -----------------------------------
741 For the DPDK vhost-net sample code to switch packets into the VM,
742 the sample code must first learn the MAC address of the VM's virtio-net device.
743 The sample code detects the address from packets being transmitted from the VM, similar to a learning switch.
745 This behavior requires no special action or configuration with the Linux* virtio-net driver in the VM
746 as the Linux* Kernel will automatically transmit packets during device initialization.
747 However, DPDK-based applications must be modified to automatically transmit packets during initialization
748 to facilitate the DPDK vhost- net sample code's MAC learning.
750 The DPDK testpmd application can be configured to automatically transmit packets during initialization
751 and to act as an L2 forwarding switch.
753 Testpmd MAC Forwarding
754 ~~~~~~~~~~~~~~~~~~~~~~
756 At high packet rates, a minor packet loss may be observed.
757 To resolve this issue, a "wait and retry" mode is implemented in the testpmd and vhost sample code.
758 In the "wait and retry" mode if the virtqueue is found to be full, then testpmd waits for a period of time before retrying to enqueue packets.
760 The "wait and retry" algorithm is implemented in DPDK testpmd as a forwarding method call "mac_retry".
761 The following sequence diagram describes the algorithm in detail.
763 .. _figure_tx_dpdk_testpmd:
765 .. figure:: img/tx_dpdk_testpmd.*
767 Packet Flow on TX in DPDK-testpmd
773 The testpmd application is automatically built when DPDK is installed.
774 Run the testpmd application as follows:
776 .. code-block:: console
778 cd ${RTE_SDK}/x86_64-native-linuxapp-gcc/app
779 ./testpmd -c 0x3 -n 4 --socket-mem 512 \
780 -- --burst=64 --i --disable-hw-vlan-filter
782 The destination MAC address for packets transmitted on each port can be set at the command line:
784 .. code-block:: console
786 ./testpmd -c 0x3 -n 4 --socket-mem 512 \
787 -- --burst=64 --i --disable-hw-vlan-filter \
788 --eth-peer=0,aa:bb:cc:dd:ee:ff --eth-peer=1,ff:ee:dd:cc:bb:aa
790 * Packets received on port 1 will be forwarded on port 0 to MAC address
794 * Packets received on port 0 will be forwarded on port 1 to MAC address
798 The testpmd application can then be configured to act as an L2 forwarding application:
800 .. code-block:: console
802 testpmd> set fwd mac_retry
804 The testpmd can then be configured to start processing packets,
805 transmitting packets first so the DPDK vhost sample code on the host can learn the MAC address:
807 .. code-block:: console
809 testpmd> start tx_first
813 Please note "set fwd mac_retry" is used in place of "set fwd mac_fwd" to ensure the retry feature is activated.
815 Passing Traffic to the Virtual Machine Device
816 ---------------------------------------------
818 For a virtio-net device to receive traffic,
819 the traffic's Layer 2 header must include both the virtio-net device's MAC address and VLAN tag.
820 The DPDK sample code behaves in a similar manner to a learning switch in that
821 it learns the MAC address of the virtio-net devices from the first transmitted packet.
822 On learning the MAC address,
823 the DPDK vhost sample code prints a message with the MAC address and VLAN tag virtio-net device.
826 .. code-block:: console
828 DATA: (0) MAC_ADDRESS cc:bb:bb:bb:bb:bb and VLAN_TAG 1000 registered
830 The above message indicates that device 0 has been registered with MAC address cc:bb:bb:bb:bb:bb and VLAN tag 1000.
831 Any packets received on the NIC with these values is placed on the devices receive queue.
832 When a virtio-net device transmits packets, the VLAN tag is added to the packet by the DPDK vhost sample code.
834 Running virtio_user with vhost-switch
835 -------------------------------------
837 We can also use virtio_user with vhost-switch now.
838 Virtio_user is a virtual device that can be run in a application (container) parallelly with vhost in the same OS,
839 aka, there is no need to start a VM. We just run it with a different --file-prefix to avoid startup failure.
841 .. code-block:: console
843 cd ${RTE_SDK}/x86_64-native-linuxapp-gcc/app
844 ./testpmd -c 0x3 -n 4 --socket-mem 1024 --no-pci --file-prefix=virtio_user-testpmd \
845 --vdev=virtio_user0,mac=00:01:02:03:04:05,path=$path_vhost \
846 -- -i --txqflags=0xf01 --disable-hw-vlan
848 There is no difference on the vhost side.
849 Pleae note that there are some limitations (see release note for more information) in the usage of virtio_user.