.. BSD LICENSE
- Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+ Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
All rights reserved.
Redistribution and use in source and binary forms, with or without
* Fedora* 20
+.. _vhost_app_prerequisites:
+
Prerequisites
-------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is recommended for testing purposes that the DPDK testpmd sample application is used in the guest to forward packets,
-the reasons for this are discussed in Section 22.7, "Running the Virtual Machine (QEMU)".
+the reasons for this are discussed in `Running the Virtual Machine (QEMU)`_.
The testpmd application forwards packets between pairs of Ethernet devices,
it requires an even number of Ethernet devices (virtio or otherwise) to execute.
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... \
+ qemu-system-x86_64 ... \
-netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> \
-device virtio-net-pci, netdev=hostnet1,id=net1 \
-netdev tap,id=hostnet2,vhost=on,vhostfd=<open fd> \
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... \
+ qemu-system-x86_64 ... \
-chardev socket,id=char1,path=<sock_path> \
-netdev type=vhost-user,id=hostnet1,chardev=char1 \
-device virtio-net-pci,netdev=hostnet1,id=net1 \
.. code-block:: console
export RTE_SDK=/path/to/rte_sdk
- cd ${RTE_SDK}/examples/vhost
+ cd ${RTE_SDK}/examples/vhost/build/app
#. Run the vhost-switch sample code:
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- -p 0x1 --dev-basename usvhost
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- -p 0x1 --dev-basename usvhost
vhost user: a socket file named usvhost will be created under current directory. Use its path as the socket path in guest's qemu commandline.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- -p 0x1 --dev-basename usvhost
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- -p 0x1 --dev-basename usvhost
.. note::
Please note the huge-dir parameter instructs the DPDK to allocate its memory from the 2 MB page hugetlbfs.
+.. note::
+
+ The number used with the --socket-mem parameter may need to be more than 1024.
+ The number required depends on the number of mbufs allocated by vhost-switch.
+
+.. _vhost_app_parameters:
+
Parameters
~~~~~~~~~~
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- -p 0x1 --dev-basename usvhost
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- -p 0x1 --dev-basename usvhost
**vm2vm.**
The vm2vm parameter disable/set mode of packet switching between guests in the host.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --vm2vm [0,1,2]
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --vm2vm [0,1,2]
**Mergeable Buffers.**
The mergeable buffers parameter controls how virtio-net descriptors are used for virtio-net headers.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- --mergeable [0,1]
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --mergeable [0,1]
**Stats.**
The stats parameter controls the printing of virtio-net device statistics.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- --stats [0,n]
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --stats [0,n]
**RX Retry.**
The rx-retry option enables/disables enqueue retries when the guests RX queue is full.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- --rx-retry [0,1]
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --rx-retry [0,1]
**RX Retry Number.**
The rx-retry-num option specifies the number of retries on an RX burst,
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- --rx-retry 1 --rx-retry-num 5
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --rx-retry 1 --rx-retry-num 5
**RX Retry Delay Time.**
The rx-retry-delay option specifies the timeout (in micro seconds) between retries on an RX burst,
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir / mnt/huge -- --rx-retry 1 --rx-retry-delay 20
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --rx-retry 1 --rx-retry-delay 20
**Zero copy.**
-The zero copy option enables/disables the zero copy mode for RX/TX packet,
-in the zero copy mode the packet buffer address from guest translate into host physical address
-and then set directly as DMA address.
-If the zero copy mode is disabled, then one copy mode is utilized in the sample.
-This option is disabled by default.
-
-.. code-block:: console
-
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --zero-copy [0,1]
-
-**RX descriptor number.**
-The RX descriptor number option specify the Ethernet RX descriptor number,
-Linux legacy virtio-net has different behavior in how to use the vring descriptor from DPDK based virtio-net PMD,
-the former likely allocate half for virtio header, another half for frame buffer,
-while the latter allocate all for frame buffer,
-this lead to different number for available frame buffer in vring,
-and then lead to different Ethernet RX descriptor number could be used in zero copy mode.
-So it is valid only in zero copy mode is enabled. The value is 32 by default.
-
-.. code-block:: console
-
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --zero-copy 1 --rx-desc-num [0, n]
-
-**TX descriptor number.**
-The TX descriptor number option specify the Ethernet TX descriptor number, it is valid only in zero copy mode is enabled.
-The value is 64 by default.
-
-.. code-block:: console
-
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --zero-copy 1 --tx-desc-num [0, n]
+Zero copy mode is removed, due to it has not been working for a while. And
+due to the large and complex code, it's better to redesign it than fixing
+it to make it work again. Hence, zero copy may be added back later.
**VLAN strip.**
The VLAN strip option enable/disable the VLAN strip on host, if disabled, the guest will receive the packets with VLAN tag.
.. code-block:: console
- user@target:~$ ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge -- --vlan-strip [0, 1]
+ ./vhost-switch -c f -n 4 --socket-mem 1024 --huge-dir /mnt/huge \
+ -- --vlan-strip [0, 1]
+
+.. _vhost_app_running:
Running the Virtual Machine (QEMU)
----------------------------------
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1,id=net1 ...
+ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1, \
+ id=net1 ...
* Ensure the guest's virtio-net network adapter is configured with offloads disabled.
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
+ qemu-system-x86_64 ... -device virtio-net-pci,netdev=hostnet1, \
+ id=net1, csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
* Redirect QEMU to communicate with the DPDK vhost-net sample code in place of the vhost-net kernel module(vhost cuse).
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> ...
+ qemu-system-x86_64 ... -netdev tap,id=hostnet1,vhost=on, \
+ vhostfd=<open fd> ...
* Enable the vhost-net sample code to map the VM's memory into its own process address space.
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -mem-prealloc -mem-path / dev/hugepages ...
+ qemu-system-x86_64 ... -mem-prealloc -mem-path /dev/hugepages ...
.. note::
#!/usr/bin/python
fd = os.open("/dev/usvhost-1", os.O_RDWR)
- subprocess.call("qemu-system-x86_64 ... . -netdev tap,id=vhostnet0,vhost=on,vhostfd=" + fd +"...", shell=True)
+ subprocess.call
+ ("qemu-system-x86_64 ... -netdev tap,id=vhostnet0,vhost=on,vhostfd="
+ + fd +"...", shell=True)
.. note::
- This process is automated in the QEMU wrapper script discussed in Section 24.7.3.
+ This process is automated in the `QEMU Wrapper Script`_.
Mapping the Virtual Machine's Memory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: console
- user@target:~$ qemu-system-x86_64 ... -mem-prealloc -mem-path / dev/hugepages ...
+ qemu-system-x86_64 ... -mem-prealloc -mem-path /dev/hugepages ...
.. note::
- This process is automated in the QEMU wrapper script discussed in Section 24.7.3.
- The following two sections only applies to vhost cuse. For vhost-user, please make corresponding changes to qemu-wrapper script and guest XML file.
+ This process is automated in the `QEMU Wrapper Script`_.
+ The following two sections only applies to vhost cuse.
+ For vhost-user, please make corresponding changes to qemu-wrapper script and guest XML file.
QEMU Wrapper Script
~~~~~~~~~~~~~~~~~~~
.. code-block:: console
- user@target:~$ qemu-wrap.py -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1
- -netdev tap,id=hostnet1,vhost=on -device virtio-net-pci,netdev=hostnet1,id=net1 -hda <disk img> -m 4096
+ qemu-wrap.py -machine pc-i440fx-1.4,accel=kvm,usb=off \
+ -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1 \
+ -netdev tap,id=hostnet1,vhost=on \
+ -device virtio-net-pci,netdev=hostnet1,id=net1 \
+ -hda <disk img> -m 4096
which will become the following call to QEMU:
.. code-block:: console
- /usr/local/bin/qemu-system-x86_64 -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1
- -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> -device virtio-net-pci,netdev=hostnet1,id=net1,
- csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -hda <disk img> -m 4096 -mem-path /dev/hugepages -mem-prealloc
+ qemu-system-x86_64 -machine pc-i440fx-1.4,accel=kvm,usb=off \
+ -cpu SandyBridge -smp 4,sockets=4,cores=1,threads=1 \
+ -netdev tap,id=hostnet1,vhost=on,vhostfd=<open fd> \
+ -device virtio-net-pci,netdev=hostnet1,id=net1, \
+ csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off \
+ -hda <disk img> -m 4096 -mem-path /dev/hugepages -mem-prealloc
Libvirt Integration
~~~~~~~~~~~~~~~~~~~
.. code-block:: console
- user@target:~$ mkdir /dev/cgroup
- user@target:~$ mount -t cgroup none /dev/cgroup -o devices
+ mkdir /dev/cgroup
+ mount -t cgroup none /dev/cgroup -o devices
* Restart the libvirtd system process
needs access to the shared memory from the guest to receive and transmit packets. It is important to make sure
the QEMU version supports shared memory mapping.
-* Issues with ``virsh destroy`` not destroying the VM:
-
- Using libvirt ``virsh create`` the ``qemu-wrap.py`` spawns a new process to run ``qemu-kvm``. This impacts the behavior
- of ``virsh destroy`` which kills the process running ``qemu-wrap.py`` without actually destroying the VM (it leaves
- the ``qemu-kvm`` process running):
-
- This following patch should fix this issue:
- http://dpdk.org/ml/archives/dev/2014-June/003607.html
-
* In an Ubuntu environment, QEMU fails to start a new guest normally with user space VHOST due to not being able
to allocate huge pages for the new guest:
Linux module but which is necessary for the user space VHOST current implementation (CUSE-based) to communicate with
the guest.
+.. _vhost_app_running_dpdk:
Running DPDK in the Virtual Machine
-----------------------------------
.. code-block:: console
- user@target:~$ x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -- n 4 -socket-mem 128 -- --burst=64 -i
+ cd ${RTE_SDK}/x86_64-native-linuxapp-gcc/app
+ ./testpmd -c 0x3 -n 4 --socket-mem 512 \
+ -- --burst=64 --i --disable-hw-vlan-filter
The destination MAC address for packets transmitted on each port can be set at the command line:
.. code-block:: console
- user@target:~$ x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -- n 4 -socket-mem 128 -- --burst=64 -i --eth- peer=0,aa:bb:cc:dd:ee:ff --eth-peer=1,ff,ee,dd,cc,bb,aa
+ ./testpmd -c 0x3 -n 4 --socket-mem 512 \
+ -- --burst=64 --i --disable-hw-vlan-filter \
+ --eth-peer=0,aa:bb:cc:dd:ee:ff --eth-peer=1,ff:ee:dd:cc:bb:aa
* Packets received on port 1 will be forwarded on port 0 to MAC address
- aa:bb:cc:dd:ee:ff.
+ aa:bb:cc:dd:ee:ff
* Packets received on port 0 will be forwarded on port 1 to MAC address
- ff,ee,dd,cc,bb,aa.
+ ff:ee:dd:cc:bb:aa
The testpmd application can then be configured to act as an L2 forwarding application:
The above message indicates that device 0 has been registered with MAC address cc:bb:bb:bb:bb:bb and VLAN tag 1000.
Any packets received on the NIC with these values is placed on the devices receive queue.
When a virtio-net device transmits packets, the VLAN tag is added to the packet by the DPDK vhost sample code.
+
+Running virtio-user with vhost-switch
+-------------------------------------
+
+We can also use virtio-user with vhost-switch now.
+Virtio-user is a virtual device that can be run in a application (container) parallelly with vhost in the same OS,
+aka, there is no need to start a VM. We just run it with a different --file-prefix to avoid startup failure.
+
+.. code-block:: console
+
+ cd ${RTE_SDK}/x86_64-native-linuxapp-gcc/app
+ ./testpmd -c 0x3 -n 4 --socket-mem 1024 --no-pci --file-prefix=virtio-user-testpmd \
+ --vdev=virtio-user0,mac=00:01:02:03:04:05,path=$path_vhost \
+ -- -i --txqflags=0xf01 --disable-hw-vlan
+
+There is no difference on the vhost side.
+Pleae note that there are some limitations (see release note for more information) in the usage of virtio-user.
git.droids-corp.org / dpdk.git / blobdiff