The purpose of this book is to assist you in contributing to the Spectrum compartmentalized operating system. If, while using this book, you find information that is outdated, incorrect, incomplete or difficult to understand, I implore you to improve it. Contributing to documentation means that your work will form part of the foundation of the continued development of the project.
Without good documentation, new contributors would not be able to get involved with the project, existing contributors would not be able to keep up, and development would slow to a halt. Spectrum is a complicated system, and nobody can keep everything they need to know to work on every part of the system in their head at one time.
Trust as little as possible. We trust KVM out of necessity, but we don’t necessarily trust the rest of the kernel, especially security features like namespaces that have to be implemented throughout — there’s always the chance somewhere has been missed.
It should be possible to use Spectrum without using any proprietary (non-free) software. Proprietary software is a security issue, because we can’t necessarily know what it does, or fix it if it’s behaving in a way that is incorrect or user-hostile.
virtio_wl, also sometimes called virtwl or virtio-wayland, is a transport for the Wayland protocol over virtio, which is the standard protocol for KVM VMs to communicate with the outside world. It was developed for Chromium OS to allow graphical applications to be run in VMs, but still be part of the standard Chromium OS desktop environment. Currently, this is the only implementation of virtio_wl.
With virtio_wl, it is possible for a graphical application running in a VM to interact with a Wayland compositor running outside it. This allows Spectrum to present multiple applications, running in separate VMs, as part of a unified windowing system, while preventing the applications from being able to interact with each other beyond what is permitted by the compositor on behalf of the user.
virtio_wl requires three components to work together to achieve this:
A kernel driver, to handle the virtio communication from the VM to the outside world. In Spectrum, this is the virtio_wl module from the Chromium OS kernel tree.
A nested Wayland compositor, which handles communication between Wayland clients (applications) running inside a VM, and the virtio_wl device exposed by the host kernel. As far as the clients know, they’re talking to a normal Wayland server, rather than virtio_wl.
This could also have been implemented as part of the kernel module, but the authors at Google found it easier to implement this way.
In Spectrum, the program that does this is Sommelier.
The Virtual Machine Manager (VMM), which connects a Wayland socket to the other end of the virtio queue created by the kernel driver. The Spectrum VMM, which supports virtio_wl, is crosvm.
crosvm is used to run Spectrum’s VMs. There are several reasons for choosing crosvm over other options like Firecracker or QEMU:
virtio_wl is unique to crosvm.
crosvm is written in Rust, a memory-safe language, and so should be much less susceptible to memory safety exploits than a VMM written in a non-memory-safe language like C.
crosvm is very focused on running Linux VMs. This means it is very lightweight compared to more general purpose VMs, both in code volume and runtime efficiency.
crosvm is somewhat documented in its README. The generated Rust documentation for crosvm’s internals is also a very useful tool for working with crosvm’s source code. For easy access, the crosvm documentation for the version currently used by Spectrum is hosted on the Spectrum website.
spectrum-vm is a shell script that allows for testing various Spectrum
VM components, including crosvm, the kernel, and the root filesystem
("rootfs" for short). The simplest way to use this tool is by running
nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum -p
spectrumPackages.spectrum-vm --run spectrum-vm in a terminal, which
will start up an instance of crosvm from Nixpkgs running Wayfire in a
Once you are ready to end your crosvm session, switch back to where you
spectrum-vm and type
poweroff). This window is not
limited to just stopping the VM, however; it is a functional serial
console where you can run commands like
As mentioned above,
spectrum-vm also supports options that allow you
to specify custom components or run arbitrary commands before exiting:
nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum -p spectrumPackages.spectrum-vm # display the currently supported options for spectrum-vm nix-shell$ spectrum-vm --help # run a command that will print out the VM's PATH to the serial terminal and exit nix-shell$ spectrum-vm -c 'echo $PATH' # use the specified crosvm and disable sandboxing nix-shell$ spectrum-vm -C target/debug/crosvm -- --disable-sandbox # use the specified kernel image nix-shell$ spectrum-vm -k result/bzImage # use the specified squashfs as the root filesystem image nix-shell$ spectrum-vm -f squashfs
Hacking on crosvm
There are a few ways to get a crosvm development environment. You can set up a full Chromium OS development chroot, or use the provided Docker scripts. But an approach with less indirection (and therefore probably easier to debug when things to wrong) is to just get everything set up on your system.
Like other Chromium OS components, crosvm is easiest to build when its positioned where it expects to be relative to other Chromium OS components.
So, you should check out the crosvm repository into a hierarchy that looks something like this:
chromiumos/ ├── aosp/ │ └── external/ │ └── minijail/ (git clone https://android.googlesource.com/platform/external/minijail) ├── platform/ │ └── crosvm/ (git clone https://spectrum-os.org/git/crosvm) └── third_party/ └── adhd/ (git clone https://chromium.googlesource.com/chromiumos/third_party/adhd)
The exact other components required by crosvm depend on what build options you’re using and will also likely change over time. So the best thing to do is probably to start with platform/crosvm, and just try building. If you’re missing a required component, you’ll get an error message. You should be able to find the required repository from the list on https://chromium.googlesource.com/.
For a straightforward crosvm build with the default build options, the following packages from Spectrum’s Nixpkgs are required:
You can then build crosvm with cargo build. Spectrum’s Nixpkgs provides a utility program for running a test VM with a custom crosvm build. It can be used as follows:
cargo build nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum \ -p spectrumPackages.spectrum-vm \ --run 'spectrum-vm -C target/debug/crosvm -- --disable-sandbox'
--disable-sandbox is required because crosvm has hardcoded paths to
seccomp policy files. If you need to test sandboxing, your best bet
is to try to modify the crosvm Nix derivation to use your modified
sources. There’s an upstream crosvm bug to
compile the policy files into the crosvm binary so paths don’t have to
The memfd server
Spectrum’s crosvm has been augmented with a feature called the “memfd server” (currently only on the interguest branch). This is a small server that listens on a Unix stream socket. It receives a request consisting of a name and a size, uses those to allocate a memfd, and then sends the resulting memfd file descriptor back over the socket in response, along with a single byte status code.
The purpose of this is that it can be exposed to the guest over virtio_wl. This means that the guest can request and receive allocations of host memory. This is important, because virtio_wl only allows file descriptors pointing to host memory to be sent over virtio_wl — a memfd allocated in a guest cannot be sent over virtio_wl. It is rare that a guest needs to be able to allocate and send shared memory in this way, but it is important for a Wayland compositor running in a guest to be able to do this.
The memfd server is disabled by default, but is enabled at startup
is a nested Wayland compositor developed for Chromium OS that
implements the user-space side of virtio_wl. It interacts with
the device exposed by the kernel virtio_wl driver (usually
/dev/wl0), and exposes a Wayland compositor interface to Wayland
clients running in the VM.
It also supports XWayland, so applications using the legacy X11 protocol can still be used with virtio_wl.
Sommelier is somewhat documented in its README.
The version of Sommelier in Spectrum’s nixpkgs has been modified to support the modern, stable xdg-shell protocol, rather than the obsolete xdg-shell-unstable-v6 still used by Chromium OS at the time of writing. Spectrum’s modifications to Sommelier are managed as patch files applied to the Sommelier package. These are located in the "sommelier" directory in Spectrum’s Nixpkgs tree.
mktuntap is a program that was written for Spectrum to facilitate creating virtual (TAP) network devices for VMs. crosvm expects to be given these devices as file descriptors, rather than as file paths. This means that Spectrum can use non-persistent TAP devices, and have the kernel automatically clean up the device when the last file handle for the device is closed. This saves implementing TAP device lifecycle in Spectrum code.
You can find the development source for mktuntap in its Git repository. If you would like to make bug report, suggest an improvement, or send a patch, please use the Spectrum development mailing list. mktuntap aims to be a general purpose utility program, and improvements are therefore welcome even if not directly required by Spectrum’s use case for the program.
Appendix A: Copyright and License
Permission is granted to copy, distribute and/or modify this document under either the terms of the Creative Commons Attribution-ShareAlike 4.0 International License, or the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.