Introduction

The purpose of this book is to assist you in contributing to the Spectrum compartmentalized operating system. If, while using this book, you find information that is outdated, incorrect, incomplete or difficult to understand, I implore you to improve it. Contributing to documentation means that your work will form part of the foundation of the continued development of the project.

Without good documentation, new contributors would not be able to get involved with the project, existing contributors would not be able to keep up, and development would slow to a halt. Spectrum is a complicated system, and nobody can keep everything they need to know to work on every part of the system in their head at one time.

Principles

  • Trust as little as possible. We trust KVM out of necessity, but we don’t necessarily trust the rest of the kernel, especially security features like namespaces that have to be implemented throughout — there’s always the chance somewhere has been missed.

  • It should be possible to use Spectrum without using any proprietary (non-free) software. Proprietary software is a security issue, because we can’t necessarily know what it does, or fix it if it’s behaving in a way that is incorrect or user-hostile.

Concepts

virtio_wl

virtio_wl, also sometimes called virtwl or virtio-wayland, is a transport for the Wayland protocol over virtio, which is the standard protocol for KVM VMs to communicate with the outside world. It was developed for Chromium OS to allow graphical applications to be run in VMs, but still be part of the standard Chromium OS desktop environment. Currently, this is the only implementation of virtio_wl.

With virtio_wl, it is possible for a graphical application running in a VM to interact with a Wayland compositor running outside it. This allows Spectrum to present multiple applications, running in separate VMs, as part of a unified windowing system, while preventing the applications from being able to interact with each other beyond what is permitted by the compositor on behalf of the user.

virtio_wl requires three components to work together to achieve this:

  • A kernel driver, to handle the virtio communication from the VM to the outside world. In Spectrum, this is the virtio_wl module from the Chromium OS kernel tree.

  • A nested Wayland compositor, which handles communication between Wayland clients (applications) running inside a VM, and the virtio_wl device exposed by the host kernel. As far as the clients know, they’re talking to a normal Wayland server, rather than virtio_wl.

    This could also have been implemented as part of the kernel module, but the authors at Google found it easier to implement this way.

    In Spectrum, the program that does this is Sommelier.

  • The Virtual Machine Manager (VMM), which connects a Wayland socket to the other end of the virtio queue created by the kernel driver. The Spectrum VMM, which supports virtio_wl, is crosvm.

Software

crosvm

crosvm is the Virtual Machine Monitor (VMM) from Chromium OS. It is also the origin of Amazon’s Firecracker, another popular VMM.

crosvm is used to run Spectrum’s VMs. There are several reasons for choosing crosvm over other options like Firecracker or QEMU:

  • virtio_wl is unique to crosvm.

  • crosvm is written in Rust, a memory-safe language, and so should be much less susceptible to memory safety exploits than a VMM written in a non-memory-safe language like C.

  • crosvm is very focused on running Linux VMs. This means it is very lightweight compared to more general purpose VMs, both in code volume and runtime efficiency.

crosvm is somewhat documented in its README. The generated Rust documentation for crosvm’s internals is also a very useful tool for working with crosvm’s source code. For easy access, the crosvm documentation for the version currently used by Spectrum is hosted on the Spectrum website.

Hacking on crosvm

There are a few ways to get a crosvm development environment. You can set up a full Chromium OS development chroot, or use the provided Docker scripts. But an approach with less indirection (and therefore probably easier to debug when things to wrong) is to just get everything set up on your system.

Like other Chromium OS components, crosvm is easiest to build when its positioned where it expects to be relative to other Chromium OS components.

So, you should check out the crosvm repository into a hierarchy that looks something like this:

chromiumos/
├── aosp/
│   └── external/
│       └── minijail/ (git clone https://android.googlesource.com/platform/external/minijail)
├── platform/
│   └── crosvm/ (git clone https://spectrum-os.org/git/crosvm)
└── third_party/
    └── adhd/ (git clone https://chromium.googlesource.com/chromiumos/third_party/adhd)

The exact other components required by crosvm depend on what build options you’re using and will also likely change over time. So the best thing to do is probably to start with platform/crosvm, and just try building. If you’re missing a required component, you’ll get an error message. You should be able to find the required repository from the list on https://chromium.googlesource.com/.

Google have a tool, called "repo", for managing all these checkouts, and keeping them all in sync. It’s a bit heavy for just the couple of repos required by crosvm, though.

For a straightforward crosvm build with the default build options, the following packages from Spectrum’s Nixpkgs are required:

  • cargo

  • pkg-config

  • libcap

  • minijail

  • dtc

You may wonder why we use Minijail from Nixpkgs when crosvm also requires Minijail at ../../aosp/external/minijail. This is because Minijail won’t build out of the box on NixOS, because of hardcoded paths like /bin/echo in its Makefile. Fortunately, when used as a Rust library, Minijail will first check if there’s a Minijail already available in the environment, and if so, it’ll use that rather than building itself.

So when crosvm loads Minijail at ../../aosp/external/minijail, the build.rs there will essentially redirect crosvm to the Minijail from Nixpkgs.

This isn’t great, but it’s the easiest way to get things going. You could also, if you wanted to, apply the same modifications the Minijail Nixpkgs derivation does to fix these hardcoded paths, and then not use the Nixpkgs minijail. You would definitely want to do things this if you were going to be hacking on Minijail.

You can then build crosvm with cargo build. Spectrum’s Nixpkgs provides a utility program for running a test VM with a custom crosvm build. It can be used as follows:

cargo build
nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum \
    -p spectrumPackages.spectrum-vm \
    --run 'spectrum-vm -C target/debug/crosvm -- --disable-sandbox'

--disable-sandbox is required because crosvm has hardcoded paths to seccomp policy files. If you need to test sandboxing, your best bet is to try to modify the crosvm Nix derivation to use your modified sources. There’s an upstream crosvm bug to compile the policy files into the crosvm binary so paths don’t have to be hardcoded.

The memfd server

Spectrum’s crosvm has been augmented with a feature called the “memfd server” (currently only on the interguest branch). This is a small server that listens on a Unix stream socket. It receives a request consisting of a name and a size, uses those to allocate a memfd, and then sends the resulting memfd file descriptor back over the socket in response, along with a single byte status code.

The purpose of this is that it can be exposed to the guest over virtio_wl. This means that the guest can request and receive allocations of host memory. This is important, because virtio_wl only allows file descriptors pointing to host memory to be sent over virtio_wl — a memfd allocated in a guest cannot be sent over virtio_wl. It is rare that a guest needs to be able to allocate and send shared memory in this way, but it is important for a Wayland compositor running in a guest to be able to do this.

The memfd server is disabled by default, but is enabled at startup with the --wl-memfd flag.

Sommelier

Sommelier is a nested Wayland compositor developed for Chromium OS that implements the user-space side of virtio_wl. It interacts with the device exposed by the kernel virtio_wl driver (usually /dev/wl0), and exposes a Wayland compositor interface to Wayland clients running in the VM.

It also supports XWayland, so applications using the legacy X11 protocol can still be used with virtio_wl.

Sommelier is somewhat documented in its README.

The version of Sommelier in Spectrum’s nixpkgs has been modified to support the modern, stable xdg-shell protocol, rather than the obsolete xdg-shell-unstable-v6 still used by Chromium OS at the time of writing. Spectrum’s modifications to Sommelier are managed as patch files applied to the Sommelier package. These are located in the "sommelier" directory in Spectrum’s Nixpkgs tree.

mktuntap

mktuntap is a program that was written for Spectrum to facilitate creating virtual (TAP) network devices for VMs. crosvm expects to be given these devices as file descriptors, rather than as file paths. This means that Spectrum can use non-persistent TAP devices, and have the kernel automatically clean up the device when the last file handle for the device is closed. This saves implementing TAP device lifecycle in Spectrum code.

You can find the development source for mktuntap in its Git repository. If you would like to make bug report, suggest an improvement, or send a patch, please use the Spectrum development mailing list. mktuntap aims to be a general purpose utility program, and improvements are therefore welcome even if not directly required by Spectrum’s use case for the program.

Copyright 2020 Alyssa Ross.

Permission is granted to copy, distribute and/or modify this document under either the terms of the Creative Commons Attribution-ShareAlike 4.0 International License, or the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.