// SPDX-License-Identifier: GFDL-1.3-or-later OR CC-BY-SA-4.0

= Spectrum Developers’ Guide
Alyssa Ross <hi@alyssa.is>
:toc: left
:nofooter:

== Introduction

The purpose of this book is to assist you in contributing to the
Spectrum compartmentalized operating system.  If, while using this
book, you find information that is outdated, incorrect, incomplete or
difficult to understand, *I implore you to improve it*.  Contributing
to documentation means that your work will form part of the foundation
of the continued development of the project.

Without good documentation, new contributors would not be able to get
involved with the project, existing contributors would not be able to
keep up, and development would slow to a halt.  Spectrum is a
complicated system, and nobody can keep everything they need to know
to work on every part of the system in their head at one time.

== Principles

* Trust as little as possible.  We trust KVM out of necessity, but we
  don't necessarily trust the rest of the kernel, especially security
  features like namespaces that have to be implemented throughout --
  there's always the chance somewhere has been missed.

* It should be possible to use Spectrum without using any proprietary
  (non-free) software.  Proprietary software is a security issue,
  because we can't necessarily know what it does, or fix it if it's
  behaving in a way that is incorrect or user-hostile.

== Concepts

=== virtio_wl

virtio_wl, also sometimes called virtwl or virtio-wayland, is a
transport for the https://wayland.freedesktop.org/[Wayland] protocol
over
https://www.ozlabs.org/~rusty/virtio-spec/virtio-paper.pdf[virtio],
which is the standard protocol for KVM VMs to communicate with the
outside world.  It was developed for Chromium OS to allow graphical
applications to be run in VMs, but still be part of the standard
Chromium OS desktop environment.  Currently, this is the only
implementation of virtio_wl.

With virtio_wl, it is possible for a graphical application running in
a VM to interact with a Wayland compositor running outside it.  This
allows Spectrum to present multiple applications, running in separate
VMs, as part of a unified windowing system, while preventing the
applications from being able to interact with each other beyond what
is permitted by the compositor on behalf of the user.

virtio_wl requires three components to work together to achieve this:

- A *kernel driver*, to handle the virtio communication from the VM to
  the outside world.  In Spectrum, this is the
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-4.19/drivers/virtio/virtio_wl.c[virtio_wl]
  module from the Chromium OS kernel tree.

- A *nested Wayland compositor*, which handles communication between
  Wayland clients (applications) running inside a VM, and the
  virtio_wl device exposed by the host kernel.  As far as the clients
  know, they're talking to a normal Wayland server, rather than
  virtio_wl.
+  
This could also have been implemented as part of the kernel module,
but the authors at Google found it easier to implement this way.
+
In Spectrum, the program that does this is <<_sommelier>>.

- The *Virtual Machine Manager (VMM)*, which connects a Wayland socket
  to the other end of the virtio queue created by the kernel driver.
  The Spectrum VMM, which supports virtio_wl, is <<_crosvm>>.

== Software

=== crosvm

https://chromium.googlesource.com/chromiumos/platform/crosvm/[crosvm]
is the Virtual Machine Monitor (VMM) from Chromium OS.  It is also the
origin of Amazon's
https://firecracker-microvm.github.io/[Firecracker], another
popular VMM.

crosvm is used to run Spectrum's VMs.  There are several reasons for
choosing crosvm over other options like Firecracker or QEMU:

- virtio_wl is unique to crosvm.

- crosvm is written in Rust, a memory-safe language, and so should be
  much less susceptible to memory safety exploits than a VMM written
  in a non-memory-safe language like C.

- crosvm is very focused on running Linux VMs.  This means it is very
  lightweight compared to more general purpose VMs, both in code
  volume and runtime efficiency.

crosvm is somewhat documented in its
https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/master/README.md[README].
The generated Rust documentation for crosvm's internals is also a very
useful tool for working with crosvm's source code.  For easy access,
the https://spectrum-os.org/doc/crosvm/crosvm/[crosvm documentation]
for the version currently used by Spectrum is hosted on the Spectrum
website.

==== spectrum-vm

`spectrum-vm` is a shell script that allows for testing various Spectrum
VM components, including crosvm, the kernel, and the root filesystem
("rootfs" for short).  The simplest way to use this tool is by running
`nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum -p
spectrumPackages.spectrum-vm --run spectrum-vm` in a terminal, which
will start up an instance of crosvm from Nixpkgs running Wayfire in a
new window.

****
Be advised that Wayfire's default keymap is set to Dvorak
https://spectrum-os.org/git/nixpkgs/tree/pkgs/os-specific/linux/spectrum/rootfs/default.nix[inside
of the rootfs] -- to change this to your preferred keymap, just set the
`xkb_layout` option (or remove it altogether for a default of QWERTY).
****

Once you are ready to end your crosvm session, switch back to where you
ran `spectrum-vm` and type `reboot` (not `poweroff`).  This window is not
limited to just stopping the VM, however; it is a functional serial
console where you can run commands like `env` and `cat /etc/passwd`.

****
Although it is currently not possible to execute commands as root inside
the graphical environment, one can instead use the serial console to run
privileged commands.
****

===== Examples

As mentioned above, `spectrum-vm` also supports options that allow you
to specify custom components or run arbitrary commands before exiting:

----
nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum -p spectrumPackages.spectrum-vm

# display the currently supported options for spectrum-vm
nix-shell$ spectrum-vm --help

# run a command that will print out the VM's PATH to the serial terminal and exit
nix-shell$ spectrum-vm -c 'echo $PATH'

# use the specified crosvm and disable sandboxing
nix-shell$ spectrum-vm -C target/debug/crosvm -- --disable-sandbox

# use the specified kernel image
nix-shell$ spectrum-vm -k result/bzImage

# use the specified squashfs as the root filesystem image
nix-shell$ spectrum-vm -f squashfs
----

==== Hacking on crosvm

There are a few ways to get a crosvm development environment.  You can
set up a full Chromium OS development chroot, or use the provided
Docker scripts.  But an approach with less indirection (and therefore
probably easier to debug when things to wrong) is to just get
everything set up on your system.

Like other Chromium OS components, crosvm is easiest to build when its
positioned where it expects to be relative to other Chromium OS
components.

So, you should check out the https://spectrum-os.org/git/crosvm[crosvm
repository] into a hierarchy that looks something like this:

....
chromiumos/
├── aosp/
│   └── external/
│       └── minijail/ (git clone https://android.googlesource.com/platform/external/minijail)
├── platform/
│   └── crosvm/ (git clone https://spectrum-os.org/git/crosvm)
└── third_party/
    └── adhd/ (git clone https://chromium.googlesource.com/chromiumos/third_party/adhd)
....

The exact other components required by crosvm depend on what build
options you're using and will also likely change over time.  So the
best thing to do is probably to start with platform/crosvm, and just
try building.  If you're missing a required component, you'll get an
error message.  You should be able to find the required repository
from the list on https://chromium.googlesource.com/.

****
Google have a tool, called "repo", for managing all these checkouts,
and keeping them all in sync.  It's a bit heavy for just the couple of
repos required by crosvm, though.
****

For a straightforward crosvm build with the default build options, the
following packages from https://spectrum-os.org/git/nixpkgs[Spectrum's
Nixpkgs] are required:

- cargo
- pkg-config
- libcap
- minijail
- dtc

****
You may wonder why we use Minijail from Nixpkgs when crosvm also
requires Minijail at ../../aosp/external/minijail.  This is because
Minijail won't build out of the box on NixOS, because of hardcoded
paths like /bin/echo in its Makefile.  Fortunately, when used as a
Rust library, Minijail will first check if there's a Minijail already
available in the environment, and if so, it'll use that rather than
building itself.

So when crosvm loads Minijail at ../../aosp/external/minijail, the
build.rs there will essentially redirect crosvm to the Minijail from
Nixpkgs.

This isn't great, but it's the easiest way to get things going.  You
could also, if you wanted to, apply the same modifications the
Minijail Nixpkgs derivation does to fix these hardcoded paths, and
then not use the Nixpkgs minijail.  You would definitely want to do
things this if you were going to be hacking on Minijail.
****

You can then build crosvm with cargo build.  Spectrum's Nixpkgs
provides a utility program for running a test VM with a custom crosvm
build.  It can be used as follows:

----
cargo build
nix-shell -I nixpkgs=/path/to/nixpkgs-spectrum \
    -p spectrumPackages.spectrum-vm \
    --run 'spectrum-vm -C target/debug/crosvm -- --disable-sandbox'
----

`--disable-sandbox` is required because crosvm has hardcoded paths to
seccomp policy files.  If you need to test sandboxing, your best bet
is to try to modify the crosvm Nix derivation to use your modified
sources.  There's an http://crbug.com/1052126[upstream crosvm bug] to
compile the policy files into the crosvm binary so paths don't have to
be hardcoded.


==== The memfd server

Spectrum's crosvm has been augmented with a feature called the "`memfd
server`" (currently only on the
https://spectrum-os.org/git/crosvm/log/?h=interguest[interguest]
branch).  This is a small server that listens on a Unix stream socket.
It receives a request consisting of a name and a size, uses those to
allocate a
https://man7.org/linux/man-pages/man2/memfd_create.2.html[memfd], and
then sends the resulting memfd file descriptor back over the socket in
response, along with a single byte status code.

The purpose of this is that it can be exposed to the guest over
virtio_wl.  This means that the guest can request and receive
allocations of host memory.  This is important, because virtio_wl only
allows file descriptors pointing to host memory to be sent over
virtio_wl -- a memfd allocated in a guest cannot be sent over virtio_wl.
It is rare that a guest needs to be able to allocate and send shared
memory in this way, but it is important for a Wayland compositor
running in a guest to be able to do this.

The memfd server is disabled by default, but is enabled at startup
with the `--wl-memfd` flag.


=== Sommelier

https://chromium.googlesource.com/chromiumos/platform2/+/master/vm_tools/sommelier[Sommelier]
is a nested Wayland compositor developed for Chromium OS that
implements the user-space side of <<_virtio_wl>>.  It interacts with
the device exposed by the kernel virtio_wl driver (usually
`/dev/wl0`), and exposes a Wayland compositor interface to Wayland
clients running in the VM.

It also supports
https://wayland.freedesktop.org/xserver.html[XWayland], so
applications using the legacy X11 protocol can still be used with
virtio_wl.

Sommelier is somewhat documented in its
https://chromium.googlesource.com/chromiumos/platform2/+/master/vm_tools/sommelier/README.md[README].

The version of Sommelier in Spectrum's nixpkgs has been modified to
support the modern, stable xdg-shell protocol, rather than the
obsolete xdg-shell-unstable-v6 still used by Chromium OS at the time
of writing.  Spectrum's modifications to Sommelier are managed as
patch files applied to the Sommelier package.  These are located in
the
https://spectrum-os.org/git/nixpkgs/tree/pkgs/os-specific/linux/chromium-os/sommelier["sommelier"]
directory in Spectrum's Nixpkgs tree.

=== mktuntap

mktuntap is a program that was written for Spectrum to facilitate
creating virtual (https://en.wikipedia.org/wiki/TUN/TAP[TAP]) network
devices for VMs.  crosvm expects to be given these devices as file
descriptors, rather than as file paths.  This means that Spectrum can
use non-persistent TAP devices, and have the kernel automatically
clean up the device when the last file handle for the device is
closed.  This saves implementing TAP device lifecycle in Spectrum
code.

You can find the development source for mktuntap in its
https://spectrum-os.org/git/mktuntap/[Git repository].  If you would
like to make bug report, suggest an improvement, or send a patch,
please use the
https://spectrum-os.org/participating.html#spectrum-devel[Spectrum
development] mailing list.  mktuntap aims to be a general purpose
utility program, and improvements are therefore welcome even if not
directly required by Spectrum's use case for the program.

[appendix]
== Copyright and License

Copyright 2020 Alyssa Ross.

Permission is granted to copy, distribute and/or modify this document
under either the terms of the
https://creativecommons.org/licenses/by-sa/4.0/[Creative Commons
Attribution-ShareAlike 4.0 International License], or the
https://www.gnu.org/licenses/fdl-1.3.html[GNU Free Documentation
License, Version 1.3] or any later version published by the Free
Software Foundation; with no Invariant Sections, no Front-Cover Texts,
and no Back-Cover Texts.