I began this week feeling very burnt out from some Nix stuff. Later in
the week, I got sick as well (thankfully not with any COVID-19 symptoms)
and ended up spending several days not being able to do much beyond
sleep. So, not a lot from me to report this week I'm afraid. This'll
be a short one.
Documentation
-------------
I documented the memfd server I recently added to crosvm[1]. I sent the
documentation patch to the list, and was very pleased to receive
reviews[2] from impaqt and Cole Helbling. This is really the first time
a patch from me has gone through an actual review cycle with other
people involved, so that's very exciting. Here's to more of that! :)
With that documentation, I was then able to add a memfd server quota to
the Contribution Ideas page[3]. This is an important feature, so if
nobody is interested in working on it I'll do it myself, but it's also a
nice opportunity to do a pretty self-contained bit of work on crosvm, so
I'm going to leave it up for a bit in case somebody else wants to try
it.
[1]: https://spectrum-os.org/doc/developer-manual.html#_the_memfd_server
[2]: https://spectrum-os.org/lists/archives/spectrum-devel/C312TALEYVQS.RICE3VLE…
[3]: https://spectrum-os.org/todo.html
Infrastructure
--------------
Increased the maximum message size on devel@ to 80K from 40K, because a
patch I sent was too big and got held for moderation.
Nixpkgs
-------
The expected Chromium OS update was released this week (instead of last
as I'd expected). crosvm now wants the minijail source code, which
isn't hosted on chromium.googlesource.com. This required making the
update script a bit smarter. I posted the patch series improving the
update script and then updating chromiumOSPackages to the list[1]. More
details there.
Another shout out to Cole for having a go at doing this update, even if
he didn't get all the way because he needed the cores on his computer
for other things. Thanks Cole!
[1]: https://spectrum-os.org/lists/archives/spectrum-devel/20200530190028.6388-1…
I'm mostly feeling better now (although still getting tired very
easily), so hopefully things will go faster again next week. Too early
to be certain, though. I'm happy to have at least got the
chromiumOSPackages update out the way, and delighted to have received
the reviews on the documentation patch.
A week of results!
Infrastructure
--------------
Fixed a misconfigured spam filter that allowed an obvious spam message
through to devel@. Oops.
crosvm
------
Integrated the memfd server[1] on the interguest branch. It's now all
sandboxed, and optionally enabled with a command line argument to crosvm
run. Not all that much to say here, but it's what took me most of the
week!
Getting the sandbox working was a bit weird. When I tried to get it to
log seccomp failures, it seemed to just disable the sandbox. I had to
track them down with strace instead. Annoying. But the sandbox does
work in normal operation.
I still haven't limited how much memory can be requested this way. I
think implementing that would be relatively straightforward for another
contributor, so I think I'll add it to the ideas list[2] and see if a
patch is forthcoming. Otherwise I'll do it myself.
[1]: https://spectrum-os.org/git/crosvm/commit/?h=interguest
[2]: https://spectrum-os.org/todo.html
wlroots
-------
I took my standalone virtio_wl test program, and integrated it into
wlroots' allocate_shm_file function. This has the result that, when
running under Sommelier, this patched wlroots will request shared memory
from the host, rather than allocating it itself. Porting from the
standalone test program was nice, because it meant that this all just
worked, first try! (Once I got it to compile under Nixpkgs' or wlroots'
strict compiler errors, at least.) This will allow that memory to be
sent between VMs!
I haven't pushed the patch yet because I haven't integrated it into
Spectrum's Nixpkgs yet. I plan to do that next week. I'm starting to
think about moving the stuff specific to Spectrum VMs into an overlay,
but I need to think a bit about how to structure that.
Nixpkgs
-------
There's no sign of the expected Chromium OS release so far, so I
backported[3] support for multiple virtio_wl sockets from a more recent
Chromium OS kernel to the one in Spectrum's Nixpkgs. We need this to be
able to dedicate a named socket to the memfd server.
[3]: https://spectrum-os.org/git/nixpkgs/commit/?id=f24d310275909265de32cbc831d5…
It's been another week where I've been very focused on one task. I'm
quite excited about the direction this is all going. It's looking like
we'll be able to do almost everything inside VMs, which means it might
be possible to have a host kernel that does almost nothing apart from
KVM and PCI passthrough?? This would mean we'd end up with a tiny Linux
a little bit (but not all that much) like a microkernel, with most
hardware interaction and all user programs running in VMs. Cool stuff!
It's not clear to me yet the exact extent to which this is achievable,
but it's a nice vision to keep in mind. It might also make it easier
for us to transition to a true microkernel at some point in the future.
I'm hoping that I'll hit an NLnet milestone related to this stuff fairly
soon. Until I do, I'm now living on the money I've received in the past
six months through GitHub Sponsors. Thank you so much to everyone who
is helping to make it possible for me to spend this time on the
fundamentals so we have a good foundation to build Spectrum on. <3
It feels like things are finally starting to come together.
Infrastructure
--------------
Server memory upgraded. We should now be able to handle more than one
person cloning Spectrum's nixpkgs repo at once!
Documentation
-------------
I added a much requested page[1] to the website with a few tasks a
potential contributor might want to look into. More to come here, too!
[1]: https://spectrum-os.org/todo.html
virtio_wl
---------
I wrote a simple guest C program that connected to a host socket over
virtio_wl, and sent some messages back and forth. In doing this, I've
ended up with a nice set of generic functions for interacting with
virtio_wl -- virtio_wl_connect(), virtio_wl_send(), etc. This is nice,
because it allows me to treat virtio_wl inside a guest much like a
regular socket (albeit one that can only be interacted with using a
custom set of functions). Otherwise, I'd have to do ioctl operations
every time (which seems to be what the various virtio_wl-using Chromium
OS components do).
Having a simple test program (rather than integrating directly into
wlroots) gives me a more simple environment with which to test the other
end of this connection.
crosvm
------
I implemented a little Rust server that listens on a Unix stream socket.
When it receives a message (consisting of an identifier {for debug
purposes} and a size), it will allocate a memfd of the requested size,
and then send that back to the socket, along with a status byte
indicating success or failure.
I then integrated this program into crosvm, so it is started as part of
the VM initialization, and exposed over virtio_wl as an additional
socket. Integrating this into crosvm required some careful thought,
because there's no precedent in crosvm for a virtio_wl server of this
sort. It has almost nothing with the existing concept in crosvm of a
"device", which is something that communicates with a guest kernel
driver over PCI. So Spectrum's crosvm now has the concept of
"servers". I can see us wanting to provide other VM services over
virtio_wl in the future. It allows us to exchange various resources
between host and guest (or between guests) without having to write a
kernel driver specific to each task.
This all now works well enough that my program inside the VM can request
a chunk of memory, and receive back something that looks a bit like a
memfd, that can then be sent back to the host over a virtio_wl socket.
It was great to see this working the first time!
One thing crosvm devices do that I would also like to apply to this type
of server is sandboxing. The sandboxing mechanism used in crosvm is
pretty generic (it's really just a thin wrapper around Minijail[2]), so
it should be straightforward to reuse. I just have to understand a bit
better how to use it.
The other crosvm change I still need to make is to put this
functionality behind a command line flag. Most VMs should not be able
to give themselves extra memory this way -- really only the Wayland
compositor should be using it. I should also impose some upper bound on
how much memory is allowed to be allocated this way. Such a limit will
probably get harder to add (from a testing point of view) the longer I
wait.
[2]: https://lwn.net/Articles/700557/
So, at lot of very focused progress this week. From here I plan to
finish up the crosvm stuff, and then integrate this protocol into
wlroots. Then, not too long from now, we'll finally be able to move the
Wayland compositor into its own VM, separate from applications.
Exciting!
There will also probably be a Chromium OS release next week, which will
require some updates in Spectrum. If all goes well, I won't actually be
the one doing that this time. ;)
This week, I want to give a special shoutout to Cole Helbling, who sent
three patches to improve our documentation. Non-Alyssa Spectrum patches
are quite rare at the moment, so it was very very nice to get these all
at once. Thanks Cole!
This week has mostly been tidying up loose ends and working on
documentation. But there's some good, concrete progress as well.
Website
-------
<https://spectrum-os.org/> now links directly to my blog and This Week
in Spectrum, so that people who come to the website can more easily see
that development is ongoing and progress is being made[1]. I've also
made it a little more obvious how to find the IRC channel and mailing
lists from the homepage[2], because a conversation on the fediverse made
me realise that wasn't clear at all before.
Cole fixed the advice in the website's CONTRIBUTING file on sending
patches for the website[3]. I had written to set Git's
"sendemail.subjectPrefix" configuration option, but the option is
actually called "format.subjectPrefix".
Cole fixed a broken link[4], and I made the incorrect URL redirect to
the right one as well, just in case.
[1]: https://spectrum-os.org/git/www/commit/?id=8f37213cfd4d5dbd4a982bfc1899a3b3…
[2]: https://spectrum-os.org/git/www/commit/?id=6d1ce779016cd84c0413b225355715e5…
[3]: https://spectrum-os.org/git/www/commit/?id=8f37213cfd4d5dbd4a982bfc1899a3b3…
[4]: https://spectrum-os.org/git/www/commit/?id=13c860e1e5bb02fce683012b851c4451…
Developer manual
----------------
Did you know there's a developer manual[5] for Spectrum? There's not a
lot in there yet, so for now it might not be all that useful, but it
will be one day! I added a section about how to hack on Spectrum's
crosvm, because it can be a bit non-obvious how to get up and running
with it. I got some great feedback on it on IRC, and a diff from Cole
with some fixes. Thanks to everybody who gave me feedback and helped
get that documentation into shape. Setting up a crosvm build still
isn't all that easy, but at least it's documented.
[5]: https://spectrum-os.org/doc/developer-manual.html
Infrastructure
--------------
Two(?) people trying to clone all of Nixpkgs from the tiny VPS running
spectrum-os.org caused the server to run out of memory and return 504 to
those clones. It's quite important that people are able to get
Spectrum's code, so I've arranged for the VPS to be upgraded with more
memory, and so now I just need to reboot the machine so it picks it up.
I plan on doing this tomorrow, since there's less likely to be mailing
list traffic then than there is immediately after I send this.
In the medium term I'll host all this stuff on a more sensible machine,
because it's quite clear the workload on this server (which also runs
some other stuff) is starting to get beyond the "tiny VPS" territory.
IRC
---
Activity in #spectrum has really picked up all of a sudden. Hundreds of
messages were sent in the second half of this week, mostly talking
either about Spectrum or about the future of secure computing in
general, and I'm extremely here for all of it. The best thing is that I
wasn't even around for most of these conversations!
It's hugely motivating to me to feel that people are so interested in
Spectrum, and it's great to have people with all sorts of useful
knowledge and questions around.
If you weren't there, I recommend actually checking out the logs from
this period[6], because the conversations being had were just a great
read full of interesting stuff.
#spectrum feels like it's now at the point where you can start a
discussion about secure computing, and be confident that you'll get
at least a couple of extremely knowledgeable people involved in the
conversation, and that's just fantastic.
[6]: https://logs.spectrum-os.org/spectrum/2020-05-06
Wayfire
-------
I replaced Sway in the test VM derivation with our newly packaged
Wayfire. I was worried at first because wf-shell wasn't starting
properly, but strace helped me realise that wf-shell likes HOME to be
set, and I hadn't actually done that in the VMs yet. So now that works.
There hasn't been much movement on my Wayfire PRs, or my PR to add
Wayfire to upstream Nixpkgs, and that's down to me. I felt like after
working on it all week last week, I needed a bit of a break. I'll try
to get back to it next week so all those PRs can be gotten over the
line.
crosvm
------
I merged the latest crosvm changes from Google into Spectrum's crosvm
tree. Doing this at the moment takes quite a while, because my
work-in-progress "interguest" branch has about 50 commits that have to
be rebased onto the new master, and quite a lot of those usually result
in conflicts. This won't be as much of a problem once that code is
ready to be included in master, because updating then will just be a
single merge commit, but for now it's a bit of a pain, especially since
I try to make sure every commit works.
I haven't had much of a look over what's changed yet. There's usually
at least one cool new toy to be excited about when I merge Google's
crosvm, though, so I look forward to finding out what it is this time.
I think that's it for this week, although honestly there have been so
many different things going on I've probably missed something.
Hopefully next week will be a bit more focused and I'll make more
progress with interguest communication. This week, though, we've made
some great improvements in several important areas. I'm optimistic that
the increased IRC and patch volume are a sign of things to come. I'm
looking forward to seeing where things will go from here. :)
A belated Happy International Workers' Day. :)
I had some other stuff going on this week but I'm still happy with what
I got done.
Wayfire
-------
I've been working on packaging Wayfire in Nixpkgs so we can use it in
Spectrum. (See last week's update for why I've chosen Wayfire for
this.) This has required quite a bit of work on Wayfire itself, because
up until now Wayfire has assumed that every component, including
plugins, will be installed into Wayfire's prefix. With Nix, a package's
prefix is only writable by that package, so we need to make some
changes to Wayfire to support each component having its own prefix.
This is necessary even for our simple case because even a basic Wayfire
setup has at least one plugin to provide the desktop shell. Everything
is supposed to be a plugin, after all. And we'll want to add our own
plugins at some point.
To implement plugins in other search paths, I added two environment
variables WAYFIRE_PLUGIN_PATH (for libraries) and
WAYFIRE_PLUGIN_XML_PATH (for the XML files that define Wayfire plugin
options). These are then searched in addition to Wayfire's prefix.
Support for plugins coming from multiple prefixes required some small
changes in several parts of the Wayfire ecosystem.
Wayfire has so far proven to be a very friendly upstream. They
understood the problem I was having, even though it's less of an issue
on traditional distributions, and I always got quick answers on IRC when
I needed to ask for advice on implementing the search paths.
Here's the issue I filed describing the need for search paths:
https://github.com/WayfireWM/wayfire/issues/493
And here are all the pull requests I made to implement them across the
ecosystem:
https://github.com/WayfireWM/wayfire/pull/496https://github.com/WayfireWM/wf-config/pull/25https://github.com/WayfireWM/wayfire/pull/497https://github.com/WayfireWM/wf-shell/pull/52https://github.com/WayfireWM/wf-shell/pull/53https://github.com/WayfireWM/wcm/pull/17https://github.com/WayfireWM/wf-shell/pull/54https://github.com/WayfireWM/wcm/pull/18
With these changes, I can have Wayfire all nicely packaged up. We can
even provide a nice Nix interface for configuring a Wayfire with some
specific plugins:
(wayfireApplications.withPlugins (plugins: with plugins; [ wf-shell ])).wayfire
I made a pull request to add Wayfire, with the interface demonstrated
above, to Nixpkgs:
https://github.com/NixOS/nixpkgs/pull/86606
Unbeknownst to me, while I was working on this, another PR was also
opened to add the latest version of Wayfire to Nixpkgs:
https://github.com/NixOS/nixpkgs/pull/86569
This PR does a good job of making the basic Wayfire installation work,
but it doesn't support extra plugins. Still, it's likely that if this
had been available before I started my work, it would have been good
enough for us for now. This sort of thing is always a risk, and even if
I hadn't done the Wayfire work now I'd have had to come back later
anyway to do plugins, so it's not too bad. The author of that PR and I
have agreed to work together from here to get Wayfire into Nixpkgs,
using my PR as a base because it has the plugins support.
So, hopefully by the time I write my next status update all my Wayfire
pull requests will have been merged, and Wayfire will be in Nixpkgs.
Reviews on the Nixpkgs PR in particular are very much appreciated. :)
From here, I'll start trying to modify Wayfire/wlroots to make it
request host memory rather than allocating its own, as described in last
week's status update. I had a quick look, and it appears that there
aren't many places these allocations are happening, so hopefully that
shouldn't require too much modification. Then, we should be able to run
an application in one VM, displayed on a compositor in another. :)
