This week, I want to give a special shoutout to Cole Helbling, who sent
three patches to improve our documentation. Non-Alyssa Spectrum patches
are quite rare at the moment, so it was very very nice to get these all
at once. Thanks Cole!
This week has mostly been tidying up loose ends and working on
documentation. But there's some good, concrete progress as well.
<https://spectrum-os.org/> now links directly to my blog and This Week
in Spectrum, so that people who come to the website can more easily see
that development is ongoing and progress is being made. I've also
made it a little more obvious how to find the IRC channel and mailing
lists from the homepage, because a conversation on the fediverse made
me realise that wasn't clear at all before.
Cole fixed the advice in the website's CONTRIBUTING file on sending
patches for the website. I had written to set Git's
"sendemail.subjectPrefix" configuration option, but the option is
actually called "format.subjectPrefix".
Cole fixed a broken link, and I made the incorrect URL redirect to
the right one as well, just in case.
Did you know there's a developer manual for Spectrum? There's not a
lot in there yet, so for now it might not be all that useful, but it
will be one day! I added a section about how to hack on Spectrum's
crosvm, because it can be a bit non-obvious how to get up and running
with it. I got some great feedback on it on IRC, and a diff from Cole
with some fixes. Thanks to everybody who gave me feedback and helped
get that documentation into shape. Setting up a crosvm build still
isn't all that easy, but at least it's documented.
Two(?) people trying to clone all of Nixpkgs from the tiny VPS running
spectrum-os.org caused the server to run out of memory and return 504 to
those clones. It's quite important that people are able to get
Spectrum's code, so I've arranged for the VPS to be upgraded with more
memory, and so now I just need to reboot the machine so it picks it up.
I plan on doing this tomorrow, since there's less likely to be mailing
list traffic then than there is immediately after I send this.
In the medium term I'll host all this stuff on a more sensible machine,
because it's quite clear the workload on this server (which also runs
some other stuff) is starting to get beyond the "tiny VPS" territory.
Activity in #spectrum has really picked up all of a sudden. Hundreds of
messages were sent in the second half of this week, mostly talking
either about Spectrum or about the future of secure computing in
general, and I'm extremely here for all of it. The best thing is that I
wasn't even around for most of these conversations!
It's hugely motivating to me to feel that people are so interested in
Spectrum, and it's great to have people with all sorts of useful
knowledge and questions around.
If you weren't there, I recommend actually checking out the logs from
this period, because the conversations being had were just a great
read full of interesting stuff.
#spectrum feels like it's now at the point where you can start a
discussion about secure computing, and be confident that you'll get
at least a couple of extremely knowledgeable people involved in the
conversation, and that's just fantastic.
I replaced Sway in the test VM derivation with our newly packaged
Wayfire. I was worried at first because wf-shell wasn't starting
properly, but strace helped me realise that wf-shell likes HOME to be
set, and I hadn't actually done that in the VMs yet. So now that works.
There hasn't been much movement on my Wayfire PRs, or my PR to add
Wayfire to upstream Nixpkgs, and that's down to me. I felt like after
working on it all week last week, I needed a bit of a break. I'll try
to get back to it next week so all those PRs can be gotten over the
I merged the latest crosvm changes from Google into Spectrum's crosvm
tree. Doing this at the moment takes quite a while, because my
work-in-progress "interguest" branch has about 50 commits that have to
be rebased onto the new master, and quite a lot of those usually result
in conflicts. This won't be as much of a problem once that code is
ready to be included in master, because updating then will just be a
single merge commit, but for now it's a bit of a pain, especially since
I try to make sure every commit works.
I haven't had much of a look over what's changed yet. There's usually
at least one cool new toy to be excited about when I merge Google's
crosvm, though, so I look forward to finding out what it is this time.
I think that's it for this week, although honestly there have been so
many different things going on I've probably missed something.
Hopefully next week will be a bit more focused and I'll make more
progress with interguest communication. This week, though, we've made
some great improvements in several important areas. I'm optimistic that
the increased IRC and patch volume are a sign of things to come. I'm
looking forward to seeing where things will go from here. :)
A belated Happy International Workers' Day. :)
I had some other stuff going on this week but I'm still happy with what
I got done.
I've been working on packaging Wayfire in Nixpkgs so we can use it in
Spectrum. (See last week's update for why I've chosen Wayfire for
this.) This has required quite a bit of work on Wayfire itself, because
up until now Wayfire has assumed that every component, including
plugins, will be installed into Wayfire's prefix. With Nix, a package's
prefix is only writable by that package, so we need to make some
changes to Wayfire to support each component having its own prefix.
This is necessary even for our simple case because even a basic Wayfire
setup has at least one plugin to provide the desktop shell. Everything
is supposed to be a plugin, after all. And we'll want to add our own
plugins at some point.
To implement plugins in other search paths, I added two environment
variables WAYFIRE_PLUGIN_PATH (for libraries) and
WAYFIRE_PLUGIN_XML_PATH (for the XML files that define Wayfire plugin
options). These are then searched in addition to Wayfire's prefix.
Support for plugins coming from multiple prefixes required some small
changes in several parts of the Wayfire ecosystem.
Wayfire has so far proven to be a very friendly upstream. They
understood the problem I was having, even though it's less of an issue
on traditional distributions, and I always got quick answers on IRC when
I needed to ask for advice on implementing the search paths.
Here's the issue I filed describing the need for search paths:
And here are all the pull requests I made to implement them across the
With these changes, I can have Wayfire all nicely packaged up. We can
even provide a nice Nix interface for configuring a Wayfire with some
(wayfireApplications.withPlugins (plugins: with plugins; [ wf-shell ])).wayfire
I made a pull request to add Wayfire, with the interface demonstrated
above, to Nixpkgs:
Unbeknownst to me, while I was working on this, another PR was also
opened to add the latest version of Wayfire to Nixpkgs:
This PR does a good job of making the basic Wayfire installation work,
but it doesn't support extra plugins. Still, it's likely that if this
had been available before I started my work, it would have been good
enough for us for now. This sort of thing is always a risk, and even if
I hadn't done the Wayfire work now I'd have had to come back later
anyway to do plugins, so it's not too bad. The author of that PR and I
have agreed to work together from here to get Wayfire into Nixpkgs,
using my PR as a base because it has the plugins support.
So, hopefully by the time I write my next status update all my Wayfire
pull requests will have been merged, and Wayfire will be in Nixpkgs.
Reviews on the Nixpkgs PR in particular are very much appreciated. :)
From here, I'll start trying to modify Wayfire/wlroots to make it
request host memory rather than allocating its own, as described in last
week's status update. I had a quick look, and it appears that there
aren't many places these allocations are happening, so hopefully that
shouldn't require too much modification. Then, we should be able to run
an application in one VM, displayed on a compositor in another. :)
It's been a bit of a slow week, I'm afraid.
As mentioned last week, it was becoming extremely inconvenient to test
out custom kernel/crosvm/rootfs builds by modifying a Nix expression
every time. So I wrote spectrum-vm, a small program that takes command
line arguments for kernel, crosvm, and rootfs paths to use, and falls
back to pre-built defaults if any of those aren't specified. This makes
turnaround time on testing different kernel builds much quicker, because
they can be built incrementally outside of nix sandboxes.
To make things more concrete, suppose I have built a kernel in
/home/src/linux/arch/x86_64/boot/bzImage, and I want to test it with the
Spectrum crosvm demo setup. I can run
spectrum-vm -k /home/src/linux/arch/x86_64/boot/bzImage
and I get a VM booted from that kernel, with no need to nix-build
By default, the launched VM will run Weston Terminal inside a Wayland
compositor. But you can even replace this with a custom command:
spectrum-vm -c 'echo "Hello, world!"'
The implementation of that last bit is quite the hack. I'll leave
figuring out how it works as an exercise to the reader. ;)
spectrum-vm is available now in Spectrum's Nixpkgs as
spectrumPackages.spectrum-vm, as of commit
817a1a9b0de30d8bbc2ad2e80a802d87c56e96ff. Implementing this has meant
that the huge start-vm.nix file that used to exist on the crosvm-demo
branch had to be split up into a bunch of discreet units. This is
really great, because start-vm.nix was starting to grow enormous, and
experimenting with changes was becoming A Problem. Now, that's all nice
and modular under the new spectrumPackages namespace. And _that_ means
that the crosvm-demo branch can finally go away, and Spectrum's Nixpkgs
is no longer split across two branches. (The purpose of the separation
was to avoid there being a horrible start-vm.nix file in the root on
master.) So in addition to a lovely helpful tool, this has also
resulted in much cleaner code. Yay!
One feature I would still like to add is an option that maps to crosvm
run's --shared-dir, to mount a directory into a VM over 9p. That would
make getting code into a VM to test it extremely convenient. I haven't
gotten around to it yet though.
Guest memory sharing
Last week I shared a bunch of research I had done on sharing
guest-allocated buffers with other VMs. I was hoping to avoid having to
patch a Wayland compositor to request buffers from the host instead of
allocating them itself, but I've since come to the conclusion that
avoiding that won't be feasible, for now at least.
Basically, the kernel primitives aren't there yet, and it would be silly
for me, with almost zero kernel development experience, to attempt to
write them. There are all sorts of horrible edge cases to think about,
like what happens if the guest-allocated memory moves around. Compared
to this, patching a Wayland compositor doesn't seem too bad. Having to
go through a request/response over a vsock to the host will be a bit of
a performance overhead compared to a smarter solution, but I think this
kind of allocation happens rarely enough that it probably doesn't matter
A nice middle ground would have been to use virtio-gpu resources rather
than a custom message over a vsock. It would have been a much cleaner
way to do it, and it might not even involve writing any kernel code, but
the practical benefits of it would be extremely limited when contrasted
with the time it would take me to implement that.
I don't expect the approach I'm taking now of patching the Wayland
compositor to necessarily be permanent. As I linked to last week,
there are kernel efforts happening to add the necessary primitives to
share guest allocations. At some point in the future we might be able
to use them. And at basically any time we could switch to using
virtio-gpu resources if somebody wanted to put in the work.
Patching a compositor really isn't a big deal, though. It's unlikely
Spectrum will support more than one compositor anyway, because we'll
need to customise it to do things like Qubes-style window borders that
are coloured based on security domain. And, even then, if we're using a
wlroots-based compositor, it's probably only wlroots that needs
patching, so in that case any other wlroots-based compositor would
probably work. Speaking of compositors...
If you know what Wayfire is, you're probably surprised to see it as an
item in a Spectrum status update it. If you don't, Wayfire is a Wayland
compositor notable for having a bunch of over-the-top Compiz-style
graphical effects, like wobbly windows and the titular fire. I
recommend checking out the demo videos on https://wayfire.org/ if you
haven't seen them -- it's quite something.
But I bring this up not because I think support for wobbly windows is
imperative to Spectrum's chances of success, but because fundamentally
Wayfire is just an extremely modular Wayland compositor. Virtually
everything in Wayfire is implemented as a plugin. Switching between
windows is a plugin, screen zoom is a plugin, and so on.
This is really cool because in Spectrum we'll want to have, for example,
coloured window decorations like in Qubes. With any other compositor,
we'd have to hack that into the compositor internals somehow. With
Wayfire, it can just be a plugin.
Wayfire is also a pleasantly simple but featureful windowing
experience. You can drag windows around, put two side by side by
dragging them to screen edges, have multiple workspaces, etc. I think
it will be straightforward enough to not be a problem for people coming
to Spectrum regardless of their computing background, and powerful
enough to not get in the way of Serious Computer Business. And anything
missing could of course be implemented as a plugin. ;)
So, I've been doing some work to package Wayfire in Nixpkgs. This is
not all that easy, because Wayfire seems to assume that plugins will be
installed into Wayfire's prefix. This isn't really possible with Nix
unless you compile all the plugins along with Wayfire, so I'm working
some patches for Wayfire to look up plugins in a search path specified
as an environment variable, which on the Nix side we can then wrap in
the idiomatic wayfireWithPackages style. My initial impressions of the
Wayfire codebase are that it is relatively clean, and straightforward
enough to understand. The core compositor is about 13000 SLoC, which is
That, unfortunately, is it for this week. I didn't get as much done as
I'd hoped, but I did at least have loads of good thoughts about things
to do later in Spectrum's development, that aren't quite concrete enough
to put into words yet.
Wow, a week isn't very long at all. But still, there's definitely
enough to write about, even if I spent most of the week just reading
mailing list threads. More on that in a bit.
Spectrum Live + nixpkgs
I did another stream. This time, we looked at updating the Chromium OS
packages in Spectrum. The script I used to update them had broken -- it
was stuck on an old version, so first we had to fix the script, and then
fix the package builds. A lot of the Chromium OS packages aren't
actually necessary at the moment, but I made them a while ago when it
looked like they would be. I couldn't get one particular such package,
libchrome, to build any more at all after spending quite a few hours on
it, so I ended up just dropping it, along with libbrillo, which depended
on it. Some other package updates had some complexity, but we managed
to work through them all on the stream.
This stream was much more focused on what I was working on than the last
one. The last one was more about people asking general questions and me
answering them and demonstrating things. This time I stayed on the
package updates for the whole stream. I figure people asked all their
questions last time. But there were lots of questions about specific
issues in the package updates, etc. It was nice to know that I had
explained things sufficiently well that people could follow along with
what I was doing. Viewer numbers were up too, which was nice. We were
on 15 for most of the stream.
The packaging work I mostly did on the stream, and then finished up
afterwards is here:
And a couple of small upstream Nixpkgs PRs:
crosvm still needs to be updated in upstream Nixpkgs. If you want to
help out, this might be a good thing to pick up! You'd need to copy the
update.py change I made in Spectrum's Nixpkgs to the crosvm update.py in
upstream Nixpkgs, and then copy the changes in Spectrum nixpkgs' crosvm
to upstream Nixpkgs's crosvm.
If not, I'll probably get to it in a few days.
Last week, I said that we might me able to use Virtio Wayland as a
transport for crosvm device <-> crosvm communication across a VM
boundary, but that I'd just had the idea and I might decide it couldn't
Having researched it extensively this week, reading over the code for
the virtio_wl kernel driver with puck and consuming a huge number of
messages on assorted kernel-adjacent mailing lists, I think we can make
it work, and it's our best path forward, but I'm still not sure how
exactly. The main problem is that Virtio Wayland can only send shared
memory from host to guest, not the other way around. So when a Wayland
compositor in a guest hands the Wayland crosvm device, it's not going to
be able to pass it over the VM boundary to the crosvm instance running a
Sharing a guest memory buffer with the host, or another guest, turns out
not to be a solved problem in general. As an example, here is a very
long mailing list thread about how a generic way to do this could work,
from November/December, which reached no firm conclusions:
I expect that for most devices, e.g. the network driver, this won't be a
problem. But we're definitely going to need this for Wayland. One
option would be to modify a Wayland compositor to make it ask the host
for some shared memory, rather than allocating its own shared memory
buffer. That would work, but it's the last resort, because it would
have to be a modification made to any Wayland compositor we might want
At a high level, what we want to do is have a crosvm device running
inside a device VM tell crosvm that a certain address in guest memory
is some shared memory, and have it handle that as it would shared memory
it had allocated itself.
crosvm does support sending dmabufs from guest to host, because of
virtio-gpu. There's a relatively new kernel API called udmabuf that
converts a memfd into a dmabuf, which sounds like it should solve
exactly this problem. It has no documentation in the kernel, but I did
manage to find a small amount of documentation in a patch that was for
some reason never applied. I then managed to write a small test
program that at least called all the ioctls properly and ended up with a
dmabuf. Actually testing this any further will involve a VM, and
testing this sort of thing in a VM at the moment is a real pain, and I
want to make a small tooling improvement around that before I proceed
any further with testing udmabuf. There are also good reasons to
suspect that the udmabuf idea won't work anyway. puck thinks that the
only dmabufs Virtio Wayland will be able to transfer are those allocated
by virtio-gpu. So, not dmabufs that we just made out of a memfd given
to us by a Wayland compositor.
If udmabuf ends up not working, I'll want to see whether we can use
virtio-gpu to allocate dmabufs. The problem with this is that I don't
know how we'll bridge compositor shared memory to virtio-gpu dmabufs.
It'll presumably involve writing kernel code, which I've been pleased to
avoid so far. I can at least probably use the udmabuf implementation as
a starting point.
The third least bad option would be to modify the virtio_wl kernel
driver so it supports shared memory originating from the guest. This
would be very nice to use from userspace, but I just don't think there's
any way I'll be able to write the required kernel code.
And the fourth least bad option (the worst one) would be to modify the
compositor as mentioned above.
The current way of running a Spectrum test VM is to build and run a file
called start-vm.nix that lives in the root of a non-default branch of
Spectrum's nixpkgs called crosvm-demo. Any time you want to try a
custom kernel, rootfs, or crosvm, you need to modify this file. This is
extremely annoying. Also, as time goes on, start-vm.nix is getting
longer and longer, and more and more difficult to understand. So I
think it's finally time to do something better. The something better is
that we introduce derivations for a Spectrum rootfs, guest kernel, and
crosvm, and then provide a program, spectrum-vm, that by default runs a
VM with those three components. It would also have options for
overriding those components, so if you built a custom kernel you could
just drop that one in instead of the default one, for example. And
finally, it would allow changing the flags passed to crosvm.
I started working on this this morning, and I'm quite excited about it.
It's not going to take very long at all to do, but it's going to be such
a big improvement. It will also mean I no longer have a separate
crosvm-demo branch, to merge Spectrum nixpkgs master into every time I
update it. So I'm going to spend the next couple of days on this, and
then get back into investigating udmabuf with a nice new tool.
So, that's what happened this week. I feel like I just spent most of
the week reading mailing lists and not making much progress, but I've
managed to write quite a lot (again) so I suppose I've accomplished
Now, I've been extremely immersed in this stuff all week, and I suspect
that some of this update, especially the crosvm stuff, will not be clear
to somebody who has not been similarly immersed. And, it's getting
dangerously close to midnight UTC, my self-imposed This Week in Spectrum
deadline. So I'm going to send this as is, even though it's probably
confusing. But! I do want you to understand what's going on. So
please feel free to reply and point out anything I didn't explain very
well and ask me for a better explanation. Just be prepared for me not
to know the answer, since I'm still trying to figure out a lot of this
stuff! As a nice side effect, maybe we'll get a mailing list discussion