Introduction
------------
Virtio-vhost-user[1] is a promising virtualisation technology that allows
virtual devices that are exposed to VMs to themselves be implemented
in VMs.
Let's break down its name a bit to understand how it works:
* Virtio[2] is a standard driver interface for virtualisation.
Interfaces are available for all sorts of types of virtual devices,
e.g. virtio-net, virtio-blk, and virtio-scsi. Typically, virtio
devices are implemented by a virtual machine monitor (VMM).
* Vhost[3] is kernelspace implementation of virtio virtual devices,
created for their performance benefit. Instead of implementing the
virtual devices itself, the VMM talks to the kernel implementation
of them using a special ioctl protocol.
* Vhost-user[4] allows another process to implement the vhost
protocol, instead of the kernel, by using a UNIX socket instead of
ioctls on a special character device. This doesn't provide the raw
performance of vhost, but it serves a different purpose -- it
allows virtual devices to be implemented by external programs, in a
standardised way so they're portable between VMMs.
Virtio-vhost-user allows the program implementing the virtual device
to run in a VM of its own, by having the VMM for that VM create the
vhost-user socket, and transferring messages over it to its guest
using virtio. This is exciting for Spectrum, because it would mean
that the host system doesn't have to interact with physical hardware
directly beyond the PCI level, and can instead pass it through to a
VM, which is responsible for implementing the virtual device backed by
that physical hardware, which can be exposed to other VMs.
Last year I spent a while looking into virtio-vhost-user[5][6][7].
It's a long way from being ready to use, and it seems to be maturing
very slowly. It might be useful to us eventually for driver
isolation, or something else might come along. My conclusion from my
research was that we should decide later, once the ecosystem has had a
chance to develop. But I wanted something to come out of the research
I did anyway, and so I've prepared a demonstration.
[1]: https://wiki.qemu.org/Features/VirtioVhostUser
[2]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
[3]: https://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
[4]: https://qemu.readthedocs.io/en/latest/interop/vhost-user.html
[5]: https://spectrum-os.org/lists/archives/spectrum-devel/87pn8rezqn.fsf@alyssa....
[6]: https://spectrum-os.org/lists/archives/spectrum-devel/87blk1pwph.fsf@alyssa....
[7]: https://spectrum-os.org/lists/archives/spectrum-devel/87wo2glkg0.fsf@alyssa....
What the demo does
------------------
Using any sort of non-host-based virtual device implementation is
going to have to start with taking the virtual device implementation
out of the VMM running an application VM, and vhost-user the clear
solution to this. Vhost-user isn't supported by crosvm -- its focus
is on doing all the virtualisation required by Chromium OS, so there's
no need for it to allow other programs to provide virtual device
implementations. So another part of the research I did was to try to
port the vhost-user implementation from cloud-hypervisor to crosvm,
which I was able to do successfully[8]. This has implications beyond
vhost-user, and even beyond crosvm, because it demonstrates that it's
practical to port features between rust-vmm[9] VMMs, which means we
don't have to worry about finding one that provides every feature we
need (which is just as well, because there isn't one).
So here I demonstrate not just a "standard" virtio-vhost-user setup
(to the extent that such a thing can exist at this early stage), but
also that my patched crosvm with vhost-user support is capable of
interoperating with the experimental virtio-vhost-user implementation
for QEMU, because both are speaking the standardised vhost-user
protocol.
The demo sets up two VMs. One is run with my patched crosvm, and
expects a virtual ethernet device to be provided by a vhost-user
socket. When it boots, it brings up its network interface, tries to
run a DHCP client, and then exits. The other VM is run with Nikos
Dragazis and Stefan Hajnoczi's experimental virtio-vhost-user
implementation in QEMU[10]. It gets a standard virtual ethernet
device (backed by a TAP device on the host), and the virtio-vhost-user
device hooked up to the socket that crosvm will be connecting to.
Inside the VM, a userspace networking stack (DPDK, again modified to
support virtio-vhost-user by Nikos Dragazis[11]) implements the device
side of virtio-vhost-user, and forwards packets sent by crosvm's guest
to the virtual ethernet device backed by the host TAP.
+-----------------------------------------------------------------------------+
| |
| +------------------------+ +------------------------+ |
| | | | | |
| | +----------------+ | | +----------------+ | |
| | | | | | | | | |
| +-----+ | | +--------+ | | | | | | |
| | TAP +------+---+---+ DPDK +---+---+------+---+ | | |
| +-----+ | | +--------+ | | | | | | |
| | | | | | | | | |
| | | Linux | | | | Linux | | |
| | +----------------+ | | +----------------+ | |
| | | | | |
| | QEMU | | crosvm | |
| +------------------------+ +------------------------+ |
| |
| Linux |
+-----------------------------------------------------------------------------+
A complicating factor is that the virtio-vhost-user implementation for
DPDK only supports outgoing traffic[12]. So packets coming from
crosvm will be relayed to the TAP, but not the other way around. This
means that we can't just use ping inside the crosvm VM to verify that
the connection is working. Instead, we have to tcpdump on the host
and verify that the packets the DHCP client inside the crosvm VM is
sending are arriving on the TAP.
For this to be useful for our intended purpose of isolating drivers
for physical devices, we'd pass through the device here rather than
using a TAP. It would otherwise work exactly the same, but it's more
difficult to test it's working correctly. (I have tested it though --
for the first version of this I got working last year, I verified it
worked by checking the logs of my local network's DHCP server.)
[8]: https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540-...
[9]: https://github.com/rust-vmm
[10]: https://github.com/ndragazis/qemu/tree/virtio-vhost-user
[11]: https://github.com/ndragazis/dpdk-next-virtio/tree/virtio-vhost-user
[12]: https://github.com/ndragazis/dpdk-next-virtio/blob/2d60e63/drivers/virtio_vh...
Running the demo
----------------
First, create a TAP device for QEMU to use:
# ip tuntap add qemutap mode tap
# ip link set qemutap up
Start tcpdump, so we can see if packets arrive on the TAP:
# tcpdump -i qemutap
Start the QEMU VM:
$ $(nix-build -A qemuVm /path/to/demo.nix)
When you see "Press enter to exit", DPDK is ready to receive a
virtio-vhost-user connection.
Start the crosvm VM:
$ $(nix-build -A crosvmVm /path/to/demo.nix)
Once that VM boots, you should see some "BOOTP/DHCP" lines in the
tcpdump output. This demonstrates that traffic from the crosvm guest
has been relayed over virtio-vhost-user to DPDK, and then to the TAP
on the host over virtio-net.
You'll want to press enter to shut down the QEMU VM now, because DPDK
pegs a CPU core (for reasons[*] unrelated to virtio-vhost-user that
are out of scope here).
Then you can remove the TAP device:
# ip link delete qemutap
Nix expression for the demo
---------------------------
# SPDX-License-Identifier: MIT OR Apache-2.0
# SPDX-FileCopyrightText: 2021 Alyssa Ross