Hi everyone, especially people who're new to the project after seeing it
on Hacker News recently.
Those of you who've been around for a while will remember that the end
of last year was a busy time for Spectrum because of funding cycles, and
that's going to be the case again this year.
As a result of this, I'm going to have my head down until the end of the
year, and am not planning on doing more This Week in Spectrum this year.
I love TWiS, but it takes hours to write every week I manage to get it
done, and I need to spend those hours elsewhere at the moment. It'll be
back next year.
If you do still want to keep up with development (and I hope you do!),
I'm trying to talk in more detail about what I'm working on in the
#spectrum IRC channel. There's been a good reaction to this so far, and
the conversation has been extremely lively for the last few days. There
are consistently over 120 people in the channel now, which is amazing!
You can find information about how to join the channel through Matrix or
IRC on the website[1].
Hope to see you there!
[1]: https://spectrum-os.org/participating.html#irc
What a week. Progress has felt a bit slow, but the work has been
consistently interesting, and there have been some exciting developments
in the ecosystem.
Nixpkgs
-------
Last week I'd posted the patches required to get all of Spectrum working
with a more up-to-date Nixpkgs. After last week's email, Cole reviewed
the patch set, so I applied it, and merged the nixpkgs-update branch
into master, so now Spectrum is using a reasonably up-to-date Nixpkgs
for the first time in a long time.
I also said
> Once that's done, it's time for another chromiumOSPackages upgrade, but
> that should be pretty easy this time because we're only one version behind.
Alas, it's never that simple.
Firstly, the chromiumOSPackages update script broke, because the
information about the currently released Chromium OS build seems to have
gone missing. Google is apparently[1] serving build number 13982, but
their published build metadata[2] includes builds 13981 and 13983, but
not 13982. This means it's not possible for me to know what Git
revisions are used in the currently released Chromium OS. Assuming this
is just a one time thing, I hacked the update script to just look at the
previous build, but we should keep an eye on this. If it ever happens
again we should probably implement some sort of mitigation in the update
script.
Once I had new versions of the Chromium OS packages it was time to get
them to build, which was straightforward enough for everything except
crosvm. For Spectrum, we have a patch[3] for crosvm to make it support
VIRTIO_NET_F_MAC[4], which is a mechanism by which the host system can
indicate to the guest kernel what it should set as the MAC address of a
virtual network device. After the update, this patch no longer applied,
because all of a sudden crosvm has two different virtual network device
implementations.
This turns out to be because crosvm has implemented vhost-user[5], a
protocol to allow virtual devices to be implemented outside the VMM
program! This is great news, but it's also surprising. virtiofs[6] was
designed to be implemented with vhost-user, but when crosvm implemented
it a while ago, they became the only implementer to do so in-VMM. It's
great to see them moving in the vhost-user direction, because it makes
it much easier to mix and match virtual device implementation and VMMs.
Most excitingly, I saw a reference to vhost-user-wl, meaning a
standalone implementation of Virtio Wayland. This would allow us to use
Virtio Wayland with other, non-crosvm VMMs, which is great because I
think cloud-hypervisor[7] is probably going to end up being a better fit
for Spectrum, but Virtio Wayland was crosvm's killer app. I'd even
thought about trying to port crosvm's virtio wayland implementation to
vhost-user myself, so it's great to know that when the time comes,
that'll already have been done for me.
While I could have fixed my crosvm patch for all this new virtual
network device code, the introduction of vhost-user support means we
should be able to drop the patch altogether. cloud-hypervisor provides
a vhost-user-net implementation[8] that already supports
VIRTIO_NET_F_MAC, so if we can just get crosvm to talk to
cloud-hypervisor's vhost-user-net implementation we shouldn't have to
carry any patch for this any more.
[1]: https://cros-updates-serving.appspot.com/
[2]: https://chromium.googlesource.com/chromiumos/manifest-versions
[3]: https://spectrum-os.org/git/nixpkgs/tree/pkgs/os-specific/linux/chromium-os…
[4]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
[5]: https://qemu.readthedocs.io/en/latest/interop/vhost-user.html
[6]: https://virtio-fs.gitlab.io/index.html#overview
[7]: https://github.com/cloud-hypervisor/cloud-hypervisor
[8]: https://github.com/cloud-hypervisor/cloud-hypervisor/tree/db2159b5638eb4ccf…
cloud-hypervisor / rust-vmm
---------------------------
So I started looking at cloud-hypervisor to try to hook this all up, but
it looks like cloud-hypervisor still has some issues where it doesn't
quite follow the vhost-user specification. As I was trying to debug
this, I noticed some UB in rust-vmm[9] (the shared utility code project
for crosvm and its derivatives like cloud-hypervisor and
Firecracker[10]).
Being a good citizen, I started working on a fix for this, and then I
encountered some more issues with functions that should have been marked
as unsafe but weren't. So that's going to need to be fixed too.
But the rust-vmm code is otherwise very high quality and easy to work
with, and they're very responsive, so it shouldn't take long to get
these issues fixed. The affected code is fortunately all to do with
communication between vhost-user backends and the VMM, so it's very very
unlikely that it's anything that could be exploited by a guest.
These sorts of problems are exactly the sort of thing that Rust is
supposed to prevent, so it was disappointing to discover these issues.
But on the other hand, Rust made them stand out like sore thumbs to me,
so even though these issues managed to sneak in, there's definitely
still a huge benefit to using Rust for these sorts of programs.
[9]: https://github.com/rust-vmm
[10]: https://firecracker-microvm.github.io/
In the next week, I'm hoping to get all the rust-vmm issues I've
discovered fixed, and maybe get cloud-hypervisor's backends to speak
proper vhost-user. I'm getting my second vaccination next week as well
though, so we'll see. I might just end up being sick.
Hi! I've had a busy day today and am pretty tired, so I'm not sure how
coherent my writing is at the moment. But I'd rather get this out on
time, especially since tomorrow is also busy today.
As I said last week[1], I took some time off this week as a preventative
measure against burnout.
[1]: https://spectrum-os.org/lists/archives/spectrum-devel/87lf51ybwk.fsf@alyssa…
rust-vmm
--------
One of my patches[2] was accepted, but another[3] is still waiting. As
I said last week, I have more fixes for rust-vmm planned, but want to
let them catch up with the changes I've already sent upstream first so I
know what base I'm using for the next stuff. I expect the one that's
still waiting to be accepted next week.
[2]: https://github.com/rust-vmm/vmm-sys-util/pull/135
[3]: https://github.com/rust-vmm/vhost/pull/69
spectrum-live
-------------
Last week, I'd just integrated dm-verity into the Spectrum live image
I've been working on. When it came time to work on the actual root
filesystem, instead of the initramfs, I hit a bit of a brick wall. I
realised that trying to generate a whole operating system image using
Nix was giving me real writers' block. There was too much inbetween me
and how the files ended up on disk, and that meant there was too much
overhead to keep in mind when I was thinking about how things should be
designed and laid out. It might feel like making a Linux root
filesystem should be a solved problem, but Spectrum has a bunch of
special requirements. You might want to do something like start a VM
for each hardware device of a specific type, and that's something that
isn't really addressed by most standard stuff. All this stuff is
definitely solveable, but it requires some experimentation to get right,
and Nix was getting a bit in the way of that.
So I created a new directory, and I wrote a Makefile that builds an
ext4 image, and I just started putting files in an etc/ directory. This
made reasoning about the system way easier, and I was immediately making
progress.
Nix is great for building known targets, and for making customisable
systems (there's no way my Makefile-based system would allow the amount
of customisation I'd easily be able to provide with Nix), but for
experimentation, it's a lot nicer to be closer to the end product. So
once I know how all this should look I'll make it Nix-aware.
Currently, I have a root filesystem with a service manager that can
respond to hardware appearing and disappearing. The next (more exciting)
step will be to have it start some VMs, and assign hardware to them
appropriately. I'm looking forward to getting to that next week. An
interesting challenge I'll have to solve will be figuring out simple
categories (e.g. "ethernet device") from the huge amount of very
specific information the kernel provides. I think I might be able
abuse the modules.alias file from the kernel, that defines the mappings
from PCI etc. information to default drivers. Then all I'll have to do
will be to write mappings from default drivers to whatever categories I
come up with, or what VM I want to assign them to, or whatever.
One neat thing I'm using for the first time here is tar2ext4[4], a
utility program that's part of a larger Microsoft open source project I
don't entirely understand the purpose of. It's really useful for me
it builds ext4 images entirely in userspace, which will be great for
using in Nix derivations where it's not possible to just mount an ext4
image and write directly to it. For previous Spectrum experiments, I'd
always used SquashFS[5], entirely because I already knew of a tar2sqfs
program[6] that made creating filesystem images really easy. I've added
tar2ext4 to Nixpkgs[7], which will hopefully help other people who have
similar problems discover it.
[4]: https://github.com/microsoft/hcsshim/blob/master/cmd/tar2ext4/tar2ext4.go
[5]: https://en.wikipedia.org/wiki/SquashFS
[6]: https://github.com/AgentD/squashfs-tools-ng
[7]: https://github.com/NixOS/nixpkgs/pull/134650
musl
----
While I was working on the root filesystem, I noticed that mount -a,
which mounts all filesystems described in fstab(5), wasn't working.
This turned out to be because of a bug in Musl's implementation of
getmntent(3), a libc function for parsing fstab files.
So I wrote some tests and a fix and sent them to Musl[8].
[8]: https://inbox.vuxu.org/musl/20210821085420.474615-1-hi@alyssa.is/
Hope that all made sense.
Next week, I'll continue working on Spectrum live, and maybe fix the
next rust-vmm issue I have on my todo list if my final outstanding PR
is merged.
It's a short update this week, because most of what I did was a
continuation of stuff from last week.
rust-vmm
--------
Last week, I mentioned I'd identified some Rust safety issues in
rust-vmm. Most of the patches for these are now up[1][2][3]. The first
has been accepted already, and I expect another to be accepted later
today. There's still a UB issue I'm aware of and haven't sent a fix for
yet, because there are a number of ways to fix it and I wanted to get my
other patches in first before I decided how to fix that one.
I deliberately haven't made any progress on using cloud-hypervisor's
vhost-user-net backend with crosvm, which is what got me looking at this
code in the first place. I want to make sure I can work on
rust-vmm-adjacent things at a pace where I don't get overwhelmed with
having to keep track of loads of patches and whether I've got them
upstream yet. So I'll be putting that work on hold until the current
round of patches are upstreamed.
[1]: https://github.com/rust-vmm/vhost/pull/68
[2]: https://github.com/rust-vmm/vmm-sys-util/pull/135
[3]: https://github.com/rust-vmm/vhost/pull/69
spectrum-live
-------------
For the past little while, in the time when I wasn't writing regular
updates, I've been working on a live system for testing Spectrum. This
will be especially useful for testing things like GPU support, because I
can just build a live image with everything I might need, plug it into
all the computers I want to test, and have everything be automatic from
there. It will also probably evolve directly into what becomes the
Spectrum base system that we'll hopefully all be running as the host
system on our machines at some point.
I shifted my focus back to this this week because of wanting to not get
ahead of myself with rust-vmm. (I have a funding milestone for GPU
support, so getting that checked off soon would be good.) The main
thing I did this week was integrate dm-verity[4], which I did mostly for
fun and to satisfy my curiosity.
dm-verity is a Linux mechanism to efficiently ensure that a read-only
filesystem hasn't been tampered with, by constructing a Merkle tree out
of filesystem block hashes, and providing the root hash to the kernel
when the filesystem is mounted. dm-verity is a _great_ fit for Nix,
because we can generate the hashes at the same time as creating the
filesystem image, and then embed the hash into the initramfs we're also
building. Getting this all working took less than a day. The idea is
that (long) in the future, we'll also implement Secure Boot, which will
make sure the kernel and initramfs haven't been tampered with, and
dm-verity will extend that integrity guarantee to the host system's root
filesystem. I recommend reading "Producing a trustworthy x86-based
Linux appliance"[5] by Matthew Garrett for an overview of how this all
comes together.
dm-verity is something that's particularly exciting to me, because it's
very useful to us, but it's something that's generally used to frustrate
end user attempts to control computers they own. In Spectrum, it's
instead a tool that protects the end user against malicious filesystem
changes, while being almost completely transparent to the user if they
do want to modify their own system.
Protecting against root filesystem tampering (which would require a VM
escape or physical device access) is hardly the biggest priority for
Spectrum, but integrating dm-verity was fun, interesting, and provided
good motivation for working on the live image, which is one of the
highest priority bits of the system. (Because I'm tired of having to
say "you can't" when people ask me how they can try out Spectrum.)
[4]: https://lwn.net/Articles/459420/
[5]: https://mjg59.dreamwidth.org/57199.html
This week, I'm going to take a bit of time off as an anti-burnout
defense, but probably not the whole week. I'll still keep an eye on the
rust-vmm patches throughout this time as well, to make sure they're not
delayed in getting accepted upstream.
Hi, it's been a long time since I've done one of these, but so much
interesting stuff happened this week, and there's been a bit of renewed
interest in the project, that it _really_ needed an update. So here it
is. As always when I'm not keeping up with TWiS, this update is limited
to only things that happened in the last week, because otherwise it
would just be far too long.
Wayland
-------
A problem I've been thinking about for a long time is Wayland access
control. The Linux desktop ecosystem is moving towards access controls
for most functionality through xdg-desktop-portal[1] (which sadly
doesn't seem to have a website). But it doesn't cover stuff that's part
of the core Wayland protocol, like access to the clipboard. And some
compositors (wlroots-based ones) provide extra Wayland protocols for
things like screenshots. And up to now, Wayland compositors haven't
really done any authorization for these potentially dangerous APIs.
(All of this is only really meaningful for sandboxed applications --
running in Flatpak or a Spectrum VM or something. Linux isn't
really set up to do separation between processes running as the same
user without namespaces.)
So earlier this week, I posted[2] to the Wayland mailing list with an
idea I'd been thinking about for a while[3], which was to place a proxy
program in front of the Wayland compositor, that would intercept
client->compositor requests and handle access control. I was quickly
convinced that a proxy wasn't a good idea, but there was a lot of
discussion, and it was really helpful to me figuring out what the right
way to do it might be.
There are really two problems to be solved here, one of which I hadn't
even thought much about. The first is securely identifying a Wayland
client. A compositor needs to be able to form the question "Should
client X be able to do this?", and to do that it needs to be able to
identify a client as client X, and know if it tries to interact with
client X later, that it'll be talking to the same client. In a
non-virtualized system, the obvious way to do this would be getting the
pid of the client from the connection socket and then looking it up in
proc(5) to find out executable path, but this approach is fundamentally
insecure[4].
The way forward here (and one that would work for Spectrum) appears to
be the proposed Security Contexts protocol[5], which would allow a
sandbox implementation to provide a security context identifier for a
client before handing off the Wayland connection to the client. Once
the security context had been set up, it wouldn't be allowed to be
modified, so once the sandboxed client was given the connection, it
wouldn't be able to change the security context identifier.
In Spectrum, the security context identifier here would likely be a
unique, user-provided name for the VM, and the security context setup
would be done by the virtio wayland implementation in the VMM.
The second part of this puzzle is how the compositor should decide
whether a client should be allowed to perform a particular operation,
like a paste or going fullscreen (which is risky because it might spoof
a desktop).
There was actually a previous attempt at this a long time ago.
libwsm[7] (short for Wayland Security Module) was a library that
compositors would have integrated to make authorization decisions. But
it wasn't adopted by compositors, and it some things we now know[8] to
be bad ideas, like setting policy based on the executable path. It also
made compositors responsible for any sort of authorization UI. In my
opinion, it's better to have that done by the external piece, so that
compositors have as little work to do as possible and therefore
authorization is implemented as widely as possible.
The compositor could implement an authorization system entirely on its
own, but this would be a lot of code for each compositor to write, and
it would limit the user to whatever permissions system the compositor
came up with, which might not be able to accommodate their needs. (An
example of this would be a Spectrum user that wanted to allow pasting
between two applications, but not allow pasting between either of those
and any other application.)
It could also be implemented by having the compositors integrate with
something like polkit[6], but compositor authors are reluctant to
integrate directly with a single system like that, and even polkit might
not support everything that would be desirable in an authorization
system. (For example, it might be nice to implement libwsm's concept of
a soft allow, where an action is permitted, but a notification is shown
so the user is aware it's happened.)
So a third solution, suggested by ifreund, a Wayland compositor author,
in the Spectrum IRC channel, would be to have a privileged Wayland
client that the compositor could ask authorization questions to, with a
new protocol. The compositor would know that this particular client
should be privileged because it either wouldn't be in a security
context, or would be given a special security context identifier known
to the compositor ahead of time. Then, implementations of this protocol
could do authorization however they wanted, with the only limitation
being the questions the compositor was programmed to ask them. I think
this is a good way forward, but it'd be important to discuss with more
compositor authors before getting to excited about it.
Next steps from here with Wayland are:
* Figure out what needs to happen to move the security contexts
proposal forward. If it needs an experimental implementation, maybe
we could help with that?
* Inquire about the authorization protocol idea, and see how other
compositor offers would feel about it. If there's a generally
positive reaction, figure out how to move forward with it.
[1]: https://github.com/flatpak/xdg-desktop-portal/
[2]: https://lists.freedesktop.org/archives/wayland-devel/2021-July/041896.html
[3]: https://spectrum-os.org/lists/archives/spectrum-discuss/8735ueudel.fsf@alys…
[4]: https://gitlab.freedesktop.org/wayland/weston/-/issues/206#note_176699
[5]: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/68
[6]: https://www.freedesktop.org/software/polkit/docs/latest/polkit.8.html
[7]: https://github.com/mupuf/libwsm
[8]: https://lore.kernel.org/lkml/87r1xplsku.fsf@x220.int.ebiederm.org/
Nixpkgs
-------
I created a branch in Spectrum's nixpkgs repository[9] for the
long-overdue merge with upstream, did the merge, and posted some
patches[10] to spectrum-devel that fix all the builds that broke as a
result. I've been letting them sit for a few days hoping for a review.
Once that's done, it's time for another chromiumOSPackages upgrade, but
that should be pretty easy this time because we're only one version behind.
[9]: https://spectrum-os.org/git/nixpkgs
[10]: https://spectrum-os.org/lists/archives/spectrum-devel/20210729100928.196534…
Thanks for keeping up with Spectrum. :)
The Spectrum IRC channel is now #spectrum on irc.libera.chat.
It can also be joined through the Matrix bridge as
#spectrum:libera.chat. I understand there are some teething problems
with the Matrix bridge, but hopefully they should be resolved soon.
Apologies for the inconvenience.
Tommorow (2021-05-19) at 7:30 UTC, I'm going to be part of a panel
discussion at the NGI Forum. NGI is the EU initiative that funds my
work on Spectrum.
The topic of the panel is "Internet of Trust", and it'll be streamed
here:
https://2021.ngiforum.eu/
There's a bit more information about the panel in the agenda here:
https://prod5.assets-cdn.io/event/6431/assets/8379007230-30f8d8b6fb.pdf
I've been thinking a lot about this for a while, thanks to conversations
with Thomas and Puck, CCed here. I think it's time to put it into words
properly and start working towards making it happen.
One of the benefits that Wayland is supposed to have over X11 is
security. A Wayland application isn't supposed to be able to record the
screen without user permission, for example. But in most compositors,
it can, with no restrictions. Existing Wayland compositors are
monolithic, and each one would have to implement its own access
controls. (Mutter already does this to some extent, at least for screen
sharing, I believe.) The popular Wayland compositors are largely
focused on being feature-complete reimplementations of their X11
equivalents, and so taking advantage of the security features and access
controls the Wayland protocol makes possible hasn't been a priority for
them. Additionally, every popular Wayland compositor is written in a
memory-unsafe language, and this combined with the complexity of the
Wayland protocol, with all the extensions involved, presents a serious
concern to applications of Wayland that involve untrusted clients.
To solve these problems, I propose a proxy program that sits between
Wayland clients and the compositor, in the same privelege domain as the
compositor. The proxy would decode and re-encode every Wayland request
(client->compositor message), and would discard any request it didn't
understand. This would mitigate the problem of a large, privileged
program written in a memory-unsafe language being exposed to untrusted
inputs. Additionally, the proxy would support a plugin interface,
through which the user of the proxy (or their distributor) could
configure custom behaviour. This could be used to prompt the user for
confirmation before allowing a screen capture request, or even to
implement a similar thing for e.g. clipboard access, for which there is
no support in the Wayland protocol. It could even be used to modify
surfaces, to implement things like Qubes-style unspoofable coloured
window borders.
This approach would allow permissions systems and other custom Wayland
behaviour to be implemented in a compositor-independent manner.
Distributions which suppor tseveral compositors could implement
customisations in a single place, and users of compositors which lack
security features and the assurances memory-safety can provide against
untrusted input would gain access to those things.
I'd like to hear feedback here, but I think early in the life of this
idea we should also reach out to the broader Wayland community. I think
there's a lot of potential for this idea beyond Spectrum, and it would
be great if it could be something developed with input from a big
breadth of Wayland users. If we can do that, it might be sensible for
it to live at freedesktop.org? I'm not sure how that works.
Let me know what you all think. :)
On May 22, 2021, at 8:05 AM, Alyssa Ross <hi(a)alyssa.is> wrote:
>
> One of the benefits that Wayland is supposed to have over X11 is
> security. A Wayland application isn't supposed to be able to record the
> screen without user permission, for example. But in most compositors,
> it can, with no restrictions.
<snip>
>
> To solve these problems, I propose a proxy program that sits between
> Wayland clients and the compositor, in the same privelege domain as the
> compositor.
<snip>
> If we can do that, it might be sensible for
> it to live at freedesktop.org? I'm not sure how that works.
I am curious, if you have time, to hear more on why the approach of a proxy vs picking a compositor and implementing security there.
If the problem is that the Wayland community so far has not considered security a priority, it seems that a security proxy may suffer from those same forces. Basically, will it be easier to attract developers or gain widespread adoption of a proxy as opposed to getting buy-in to do security directly in a compositor? You mention writing in a memory safe language and having a compositor neutral solution as technical advantages.
Do you think a proxy is a good choice primarily because it can achieve a better technical result, or is the choice of a new component more a matter of difficulty getting community buy-in from a popular compositor and doing security there? How would you weigh the upsides of a new project against the difficulties of getting a new thing off the ground and adopted?
(This is really just curiosity on my part and my $0.02 from the outside. You may have already had a lot of discussions about that, or even already tried talking to compositor folk and not gotten traction. Seems worth some explicit consideration.)
Introduction
------------
Virtio-vhost-user[1] is a promising virtualisation technology that allows
virtual devices that are exposed to VMs to themselves be implemented
in VMs.
Let's break down its name a bit to understand how it works:
* Virtio[2] is a standard driver interface for virtualisation.
Interfaces are available for all sorts of types of virtual devices,
e.g. virtio-net, virtio-blk, and virtio-scsi. Typically, virtio
devices are implemented by a virtual machine monitor (VMM).
* Vhost[3] is kernelspace implementation of virtio virtual devices,
created for their performance benefit. Instead of implementing the
virtual devices itself, the VMM talks to the kernel implementation
of them using a special ioctl protocol.
* Vhost-user[4] allows another process to implement the vhost
protocol, instead of the kernel, by using a UNIX socket instead of
ioctls on a special character device. This doesn't provide the raw
performance of vhost, but it serves a different purpose -- it
allows virtual devices to be implemented by external programs, in a
standardised way so they're portable between VMMs.
Virtio-vhost-user allows the program implementing the virtual device
to run in a VM of its own, by having the VMM for that VM create the
vhost-user socket, and transferring messages over it to its guest
using virtio. This is exciting for Spectrum, because it would mean
that the host system doesn't have to interact with physical hardware
directly beyond the PCI level, and can instead pass it through to a
VM, which is responsible for implementing the virtual device backed by
that physical hardware, which can be exposed to other VMs.
Last year I spent a while looking into virtio-vhost-user[5][6][7].
It's a long way from being ready to use, and it seems to be maturing
very slowly. It might be useful to us eventually for driver
isolation, or something else might come along. My conclusion from my
research was that we should decide later, once the ecosystem has had a
chance to develop. But I wanted something to come out of the research
I did anyway, and so I've prepared a demonstration.
[1]: https://wiki.qemu.org/Features/VirtioVhostUser
[2]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
[3]: https://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
[4]: https://qemu.readthedocs.io/en/latest/interop/vhost-user.html
[5]: https://spectrum-os.org/lists/archives/spectrum-devel/87pn8rezqn.fsf@alyssa…
[6]: https://spectrum-os.org/lists/archives/spectrum-devel/87blk1pwph.fsf@alyssa…
[7]: https://spectrum-os.org/lists/archives/spectrum-devel/87wo2glkg0.fsf@alyssa…
What the demo does
------------------
Using any sort of non-host-based virtual device implementation is
going to have to start with taking the virtual device implementation
out of the VMM running an application VM, and vhost-user the clear
solution to this. Vhost-user isn't supported by crosvm -- its focus
is on doing all the virtualisation required by Chromium OS, so there's
no need for it to allow other programs to provide virtual device
implementations. So another part of the research I did was to try to
port the vhost-user implementation from cloud-hypervisor to crosvm,
which I was able to do successfully[8]. This has implications beyond
vhost-user, and even beyond crosvm, because it demonstrates that it's
practical to port features between rust-vmm[9] VMMs, which means we
don't have to worry about finding one that provides every feature we
need (which is just as well, because there isn't one).
So here I demonstrate not just a "standard" virtio-vhost-user setup
(to the extent that such a thing can exist at this early stage), but
also that my patched crosvm with vhost-user support is capable of
interoperating with the experimental virtio-vhost-user implementation
for QEMU, because both are speaking the standardised vhost-user
protocol.
The demo sets up two VMs. One is run with my patched crosvm, and
expects a virtual ethernet device to be provided by a vhost-user
socket. When it boots, it brings up its network interface, tries to
run a DHCP client, and then exits. The other VM is run with Nikos
Dragazis and Stefan Hajnoczi's experimental virtio-vhost-user
implementation in QEMU[10]. It gets a standard virtual ethernet
device (backed by a TAP device on the host), and the virtio-vhost-user
device hooked up to the socket that crosvm will be connecting to.
Inside the VM, a userspace networking stack (DPDK, again modified to
support virtio-vhost-user by Nikos Dragazis[11]) implements the device
side of virtio-vhost-user, and forwards packets sent by crosvm's guest
to the virtual ethernet device backed by the host TAP.
+-----------------------------------------------------------------------------+
| |
| +------------------------+ +------------------------+ |
| | | | | |
| | +----------------+ | | +----------------+ | |
| | | | | | | | | |
| +-----+ | | +--------+ | | | | | | |
| | TAP +------+---+---+ DPDK +---+---+------+---+ | | |
| +-----+ | | +--------+ | | | | | | |
| | | | | | | | | |
| | | Linux | | | | Linux | | |
| | +----------------+ | | +----------------+ | |
| | | | | |
| | QEMU | | crosvm | |
| +------------------------+ +------------------------+ |
| |
| Linux |
+-----------------------------------------------------------------------------+
A complicating factor is that the virtio-vhost-user implementation for
DPDK only supports outgoing traffic[12]. So packets coming from
crosvm will be relayed to the TAP, but not the other way around. This
means that we can't just use ping inside the crosvm VM to verify that
the connection is working. Instead, we have to tcpdump on the host
and verify that the packets the DHCP client inside the crosvm VM is
sending are arriving on the TAP.
For this to be useful for our intended purpose of isolating drivers
for physical devices, we'd pass through the device here rather than
using a TAP. It would otherwise work exactly the same, but it's more
difficult to test it's working correctly. (I have tested it though --
for the first version of this I got working last year, I verified it
worked by checking the logs of my local network's DHCP server.)
[8]: https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…
[9]: https://github.com/rust-vmm
[10]: https://github.com/ndragazis/qemu/tree/virtio-vhost-user
[11]: https://github.com/ndragazis/dpdk-next-virtio/tree/virtio-vhost-user
[12]: https://github.com/ndragazis/dpdk-next-virtio/blob/2d60e63/drivers/virtio_v…
Running the demo
----------------
First, create a TAP device for QEMU to use:
# ip tuntap add qemutap mode tap
# ip link set qemutap up
Start tcpdump, so we can see if packets arrive on the TAP:
# tcpdump -i qemutap
Start the QEMU VM:
$ $(nix-build -A qemuVm /path/to/demo.nix)
When you see "Press enter to exit", DPDK is ready to receive a
virtio-vhost-user connection.
Start the crosvm VM:
$ $(nix-build -A crosvmVm /path/to/demo.nix)
Once that VM boots, you should see some "BOOTP/DHCP" lines in the
tcpdump output. This demonstrates that traffic from the crosvm guest
has been relayed over virtio-vhost-user to DPDK, and then to the TAP
on the host over virtio-net.
You'll want to press enter to shut down the QEMU VM now, because DPDK
pegs a CPU core (for reasons[*] unrelated to virtio-vhost-user that
are out of scope here).
Then you can remove the TAP device:
# ip link delete qemutap
Nix expression for the demo
---------------------------
# SPDX-License-Identifier: MIT OR Apache-2.0
# SPDX-FileCopyrightText: 2021 Alyssa Ross <hi(a)alyssa.is>
let
pinned = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/tarball/b14062b75c4e8ef4dd4110282f7105be87…";
sha256 = "1hzs0w6pcwwbzl2gkqyk46yrzizzm03mph4kggws02a6vlwphsib";
};
in
{ pkgs ? import pinned {} }: with pkgs;
rec {
linux = pkgs.linux.override {
structuredExtraConfig = with lib.kernel; {
"9P_FS" = yes;
NET_9P = yes;
NET_9P_VIRTIO = yes;
PACKET = yes;
VFIO = yes;
VFIO_NOIOMMU = yes;
VFIO_PCI = yes;
VIRTIO_NET = yes;
VIRTIO_PCI = yes;
};
};
dpdk = stdenv.mkDerivation {
name = "dpdk-virtio-vhost-user";
src = fetchFromGitHub {
owner = "ndragazis";
repo = "dpdk-next-virtio";
rev = "0a46582dc1d02c0dc5069347ffff1a64239385f2";
sha256 = "169cxdps9k764jj420q44262x3291h2jcqsbrh7038hqjczjkgif";
};
buildInputs = [ numactl ];
configurePhase = ''
runHook preConfigure
make $makeFlags defconfig
runHook postConfigure
'';
enableParallelBuilding = true;
RTE_KERNELDIR = "${linux.dev}/lib/modules/${linux.modDirVersion}/build";
NIX_CFLAGS_COMPILE = [
"-Wno-error=implicit-fallthrough"
"-Wno-error=incompatible-pointer-types"
];
makeFlags = [
"RTE_OUTPUT=$(out)/lib"
"kerneldir=$(out)/lib/modules/${linux.modDirVersion}/build"
"prefix=$(out)"
];
inherit (pkgs.dpdk) meta;
};
# DPDK is huge! We just need one program from it.
testpmd = runCommandNoCC "testpmd" {} ''
mkdir -p $out/bin
cp ${dpdk}/bin/testpmd $out/bin
'';
# qemu has changed build system since the virtio-vhost-user branch
# was last updated, so it's simpler to just make a new derivation
# and inherit the bits that are the same than to override the
# existing one.
qemu = stdenv.mkDerivation {
name = "qemu-virtio-vhost-user";
src = fetchFromGitHub {
owner = "ndragazis";
repo = "qemu";
rev = "f9ab08c0c8cfc58036ed95b895f9780397448071";
sha256 = "0p6v4i7gj70d6x7s28x3i3x9z8vlswcbbqdwfbhlx87bbnxjrn3b";
fetchSubmodules = true;
};
enableParallelBuilding = true;
nativeBuildInputs =
lib.subtractLists [ ninja meson ] qemu_kvm.nativeBuildInputs;
postPatch = ''
sed -i '/$(INSTALL_DIR) "$(DESTDIR)$(qemu_localstatedir)/d' Makefile
# The virtio-vhost-user implementation tries to allocate a huge
# PCI bar, that's bigger than some CPUs can support! If you see
# a kernel panic in vp_reset(), lower this further.
substituteInPlace hw/virtio/virtio-vhost-user-pci.c \
--replace '1ULL << 36' '1ULL << 34'
'';
inherit (qemu_kvm) buildInputs configureFlags meta;
};
qemuInitramfs = makeInitrd {
contents = [
{
symlink = "/init";
object = writeScript "init" ''
#!${busybox}/bin/sh -eux
export PATH=${busybox}/bin
mkdir -p /nix/store /run /var
mount -t sysfs none /sys
mount -t proc none /proc
mount -t tmpfs none /run
mount -t devtmpfs none /dev
mkdir /dev/hugepages
mount -t hugetlbfs none /dev/hugepages
ln -s /run /var
# Unbind the virtio-net (host TAP) and virtio-vhost-user devices
# from their default drivers, since we'll be passing them
# through to DPDK.
echo 0000:00:04.0 > /sys/bus/pci/devices/0000:00:04.0/driver/unbind
echo 0000:00:05.0 > /sys/bus/pci/devices/0000:00:05.0/driver/unbind
# Tell the vfio-pci driver it can support virtio-net and
# virtio-vhost-user devices. Since our devices are not
# bound to any driver at the moment, doing this will bind
# them to vfio-pci automatically.
echo 1af4 1000 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 1af4 1017 > /sys/bus/pci/drivers/vfio-pci/new_id
echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
${testpmd}/bin/testpmd \
-l 0-1 \
-w 0000:00:05.0 \
--vdev net_vhost0,iface=0000:00:05.0,virtio-transport=1 \
-w 0000:00:04.0
poweroff -f
'';
}
];
};
qemuVm = writeShellScript "qemu-vm" ''
exec ${qemu}/bin/qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 1G \
-M q35,kernel-irqchip=split \
-initrd ${qemuInitramfs}/initrd \
-netdev tap,id=net0,ifname=qemutap,script=no,downscript=no \
-device virtio-net-pci,netdev=net0,addr=04 \
-chardev socket,id=chardev0,path="$XDG_RUNTIME_DIR/vhost-user0.sock",server,nowait \
-device virtio-vhost-user-pci,addr=05,chardev=chardev0 \
-kernel ${linux}/${stdenv.hostPlatform.linux-kernel.target} \
-append "console=ttyS0 vfio.enable_unsafe_noiommu_mode=1" \
-nographic
'';
# Can't use overrideAttrs because of cargoSha256.
crosvm = rustPlatform.buildRustPackage rec {
name = "crosvm-virtio-vhost-user";
src = fetchFromGitiles {
url = "https://chromium.googlesource.com/chromiumos/platform/crosvm";
rev = "8a7e4e902a4950b060ea23b40c0dfce7bfa1b2cb";
sha256 = "1lm6psp0xakb66nhgmmh94valc4wzbb967chk80msk8bcvsfpdn4";
};
unpackPhase =
let origSrc = pkgs.crosvm.passthru.src; in
builtins.replaceStrings [ "${origSrc}" origSrc.name ] [ "$src" src.name ]
pkgs.crosvm.unpackPhase;
cargoPatches = [
(fetchpatch {
url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…";
sha256 = "0yzqrpgq35s9wxvbf9s3dgs5cpyxgdc5hr14hsdjr0gd18a6camg";
})
];
patches = pkgs.crosvm.patches ++ [
(fetchpatch {
url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…";
sha256 = "0g2rvqqa4lvq7bjq0s1ynsjx7lmrxql7lsdv8wyzb7d2z9j6mj13";
})
(fetchpatch {
url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…";
sha256 = "051sz87i8kzc5sbygk2bpiqp4g32y9fxswg2yax1nd3lg4rxh43r";
})
(fetchpatch {
url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…";
sha256 = "1jpas65masn2xg9jxha16vi0y7scarzhl221y9wxh4chi4aa4m3f";
})
];
cargoSha256 = "07yizbhs64jrb05fq5g7sx812xbz2989bsficacq5l19ziax5164";
passthru = pkgs.crosvm.passthru // { inherit src; };
inherit (pkgs.crosvm) sourceRoot postPatch nativeBuildInputs buildInputs
preBuild postInstall CROSVM_CARGO_TEST_KERNEL_BINARY meta;
};
crosvmInitramfs = makeInitrd {
contents = [
{
symlink = "/init";
object = writeScript "init" ''
#!${busybox}/bin/sh -eux
export PATH=${busybox}/bin
mount -t sysfs none /sys
mount -t proc none /proc
ip link set eth0 up
udhcpc -n || :
reboot -f
'';
}
];
};
crosvmVm = writeShellScript "crosvm-vm" ''
# In our patched crosvm, suppling --mac without --host_ip or
# --netmask will put it into vhost-user mode.
exec ${crosvm}/bin/crosvm run \
--mac 0A:B3:EC:FF:FF:FF \
-i ${crosvmInitramfs}/initrd \
${linux}/${stdenv.hostPlatform.linux-kernel.target}
'';
}
