Hi,
I've been running Qubes for a few years now and I'd like to give
Spectrum a try, as I've been having some hardware and performance
problems with Qubes. Is there some up-to-date guide I can follow? I
found https://alyssa.is/using-virtio-wl/#demo and was able to see the
weston terminal. I also tried updating to the latest commit and was
able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and
increase the memory enough to start firefox, but I haven't managed to
get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems,
to get back to where I was with my Qubes setup, before investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
Thanks!
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
Table of Contents
─────────────────
1. Display controllers and renderers
2. Overview of GPU virtualization technologies
.. 1. Direct passthrough
.. 2. Hardware virtualization and paravirtualization
..... 1. Intel
..... 2. NVIDIA
..... 3. AMD
..... 4. ARM
.. 3. Virgil 3D
3. Seamless windowing
.. 1. virtio-gpu 2d
.. 2. virtio-wl
.. 3. virtio-gpu context types
4. Putting it together — GPUs in Spectrum VMs
This report documents the current state of the GPU virtualization
ecosystem as it relates to Spectrum.
GPU virtualization is a very controversial topic in the world of
compartmentalized operating systems. Qubes doesn't support GPU
acceleration of anything but the GUI VM (the equivalent of Spectrum's
compositor VM), and (I think as a result of this) a way to provide GPU
accelerate to applications is *the* most requested Spectrum feature by a
large margin.
But there are serious security concerns with GPU virtualization. GPUs
have a frankly awful track record when it comes to isolation. This
might improve going forward, especially with SR-IOV-based solutions,
because business demand for secure GPU virtualization is increasing and
GPU vendors are trying to meet that demand. For example, AMD says of
their GPU virtualization implementation[1]:
The hardware-enforced memory isolation logic provides strong
data security among the VFs, which helps prevent one VM from
being able to access another VM’s data.
With security being a bare minimum requirement for any
virtualization solution, AMD’s hardware-based virtualized
GPU solution offers a strong deterrent to unauthorized users
who traverse the software or application layers seeking
means to extract or corrupt GPU user data from the virtual
machines. Although a VF can access full GPU capabilities at
its own GPU partition, it does not have access to the
dedicated local memory of its sibling VFs.
But it remains to be seen whether these claims will stand up in
practice, especially because SR-IOV is not yet widely available. The
alternatives (software multiplexing and GPU emulation) have both seen
exploitable security issues.
Because of this, my current plan is that access to the GPU will be
highly restricted in a default Spectrum system. I do think, though,
that there needs to be a way for users to opt in to using whatever
virtualization features their hardware provides, because I've heard so
many accounts from people who find themselves unable to use
compartmentalized systems /at all/ because of certain tasks where they
need graphics acceleration. My primary goal with Spectrum is to bring
compartmentalized computing to people who are not currently using it,
and to people who're currently using mainstream systems where any
application they run might be stealing their private SSH keys or
ransomwaring all their files, being potentially vulnerable to zero-days
in GPU drivers or hardware is the least of their problems.
1 Display controllers and renderers
═══════════════════════════════════
Standalone GPUs, and Intel integrated GPUs, tend to be display
controllers and renderers all in a single package. The distinction
between them will be important, though, so briefly:
A *display controller* is the hardware that takes care of making a
pixel grid show up on a screen.
A *renderer* does computation (3D renderering, etc.) to create an
image that can be given to a display controller to display.
(Naturally in can also be used for other computations as well.)
2 Overview of GPU virtualization technologies
═════════════════════════════════════════════
2.1 Direct passthrough
──────────────────────
Direct passthrough is the simplest and least interesting option for
accelerated graphics in a VM. The host simply gives the VM control
over a whole GPU. Since we're aiming for one VM per application
instance in Spectrum, direct passthrough might be useful if you have a
dual GPU system, and want to accelerate one particular application,
like a game. Of course, direct passthrough is also the most secure
sort of GPU virtualization — data leaks are far less likely to happen
when each GPU user has their own dedicated hardware! We can
definitely support this.
2.2 Hardware virtualization and paravirtualization
──────────────────────────────────────────────────
By *hardware virtualization*, I mean that the virtualization is
implemented by the GPU itself, and by *paravirtualization* I mean that
the host kernel implements virtual GPUs by multiplexing.
The standard way of doing hardware virtualization is called SR-IOV,
and it's widely used today with server network cards, and to a lesser
extent NVMe drives. It's not widely available on GPUs, especially not
consumer ones, but it seems like that's about to change.
When using this sort of virtualization, all that's being virtualized
is the renderer. The display controller stays attached to the host.
I'm not aware of any implementation for hardware that is both a
display controller and a renderer that supports separately passing
through a display controller to a VM[2].
2.2.1 Intel
╌╌╌╌╌╌╌╌╌╌╌
Intel's GPU paravirtualization technology is called GVT-g. Unlike
solutions from other GPU companies, GVT-g is very widely available —
it's supported for integrated graphics starting from Broadwell
(launched 2014). GVT-g is definitely something we can support in
Spectrum, but one limitation I found is that on my laptop (a Google
Pixelbook), I can create at most two vGPUs, so don't think you're
going to be able to have every application using accelerated graphics
using GVT-g even if you're okay with the security implications.
The most recent Intel GPUs no longer support GVT-g — they have
hardware SR-IOV implementations instead. I don't believe this is
supported in Linux yet, but I imagine it will be at some point.
2.2.2 NVIDIA
╌╌╌╌╌╌╌╌╌╌╌╌
NVIDIA officially supports GPU virtualization for most of their
datacentre and professional GPUs, through their proprietary driver.
Because of NVIDIA's general allergy to publishing code or
documentation, I don't know how this GPU virtualization is
implemented, but my guess is that it's mostly in hardware, because
their product briefs claim their cards support SR-IOV. There also
exists third-party software that can modify NVIDIA's software to
support some consumer cards.
The GPUs that support it tend to support between 16 and 32 virtual
GPUs.
Obviously, running a proprietary driver in the Spectrum host kernel
would have /horrible/ security implications, and I will always
strongly advise against it. But, we probably could make it work if we
wanted to.
2.2.3 AMD
╌╌╌╌╌╌╌╌╌
There are five AMD GPUs that implement hardware (SR-IOV)
virtualization. The Radeon Pro V520 is "only available as a public
cloud offering" and so isn't a product you can actually buy. The
Radeon Pro V340 is only supported by VMware, not AMD's open source
Linux driver. The AMD S7100X, S7150, and S7150 x2 are all supported
with KVM, but none of them have been produced since 2016. It would
certainly be possible to support those last three in Spectrum, but due
to just how obscure AMD GPU virtualization is, I don't see myself
prioritizing it any time soon.
2.2.4 ARM
╌╌╌╌╌╌╌╌╌
There's single ARM GPU that supports hardware virtualization, the
Mali-G78AE. It's intended for "automotive and industrial"
applications, so I doubt it's going to show up in any hardware anybody
wants to run Spectrum on.
2.3 Virgil 3D
─────────────
Virgil 3D takes a different approach, by doing entirely software-based
GPU virtualization. It implements OpenGL in software, in a way that's
still backed by the host system's hardware GPU. This is naturally
less performant than hardware virtualization or paravirtualization due
to the extra overhead, but the advantages are that you can get a lot
of virtual GPUs out of it, since every VM is just another application
using the GPU on the host, and that this approach works even when
there isn't special support for virtualizing a particular GPU.
3 Seamless windowing
════════════════════
One of the earliest features that Spectrum committed to was a seamless
windowing experience, where applications running in different VMs
would appear as native windows in the same Wayland compositor.
3.1 virtio-gpu 2d
─────────────────
The simplest way to do seamless windowing is to run a Wayland
compositor in each VM, and have it render windows to a virtual 2D GPU,
where they can be displayed by a host compositor. To the host
compositor, windows are basically just opaque rectangles. This is
effectively how WSLg, the Windows Subsystem for Linux GUI, works.
This works okay, but you lose the ability to have any integration
between the application windows and the window manager they're being
displayed on. It's not actually seamless. That's why, in screenshots
of WSLg, you see Weston window chrome, instead of Windows ones. The
WSLg developers can either choose to draw Windows window decorations
around every application (which will look bad for GNOME programs that
draw their own decorations), or none of them (which means they have to
use Weston's decorations, or there'd be windows with no decorations),
because they can't directly speak the client-side decoration
negotiation protocol to the applications. This particular use case
doesn't matter for Spectrum, because CSD is incompatible with
unspoofable window decorations, but the point is that if you're not
letting applications and window managers speak Wayland to each other,
you need to come up with ad-hoc protocols every time you /do/ want
them to be able to cooperate.
Fortunately, there's another approach to seamless windowing, that
preserves Wayland as the method of communication between applications
and window manager.
3.2 virtio-wl
─────────────
The original plan for seamless windowing in Spectrum was to use
virtio-wl, a technology from Chromium OS. virtio-wl (plus Sommelier,
a component that runs in guest userspace) presents a socket-like
interface, over which Wayland clients running in VMs can send and
receive arbitrary data, as well as file descriptors, to and from a
Wayland compositor running on the host.
Because implementing support for sending arbitrary file descriptors
out of a guest would be impossible, only certain types needed by the
Wayland protocol are supported — pipes and host-allocated shared
memory. This worked fine for Wayland, but because of how
Wayland-specific it is, virtio-wl was never considered suitable for
upstreaming.
This was a big problem for Spectrum specifically, because in virtio-wl
there is no way for guests to share memory with the host. In
virtio-wl's intended use case, the compositor runs on the host,
clients run in guests, and the compositor handles all memory
allocation, so memory only needs to ever be shared in one direction,
host → guest. But in Spectrum, we'd like to have the compositor in
its own VM, so we'd need some extra mechanism to let the compositor VM
allocate host memory[3].
3.3 virtio-gpu context types
────────────────────────────
Google's second attempt at seamless VM Wayland windowing is a new
mechanism called virtio-gpu context types. Despite the "gpu" in the
name, there's not (necessarily) any GPU involved — it's just
convenient to use virtio-gpu as a transport since it already has
primitives for sharing memory between guest and host.
Context types allow new protocols, like Wayland or Vulkan, to be sent
over virtio-gpu, in addition to the 2D and Virgil 3D protocols it was
designed for. Context types should be available in Linux from v5.15
(the next release at the time of writing). Support for Wayland over
virtio-gpu has already been implemented in crosvm and Sommelier.
Even more excitingly, virtio-gpu has a mechanism by which guest
userspace can do a special memory allocation that can be shared with
the host! So we won't need to add any special memory allocation
mechanism just for Spectrum — we'll just need to make sure the
compositor VM userspace knows how to allocate memory and send it to
the host in this way.
4 Putting it together — GPUs in Spectrum VMs
═════════════════════════════════════════════
After extensively researching all this, my current plan for graphics
in Spectrum is as follows: each VM that runs a graphical application
will get a virtio-gpu device, over which it will be expected to speak
Wayland to the compositor. VMs with accelerated graphics (which will
be opt-in per-VM) will additionally get another GPU device to use for
rendering — this could be, for example, a GVT-g vGPU, or it could be a
Virgil 3D virtio-gpu. They can use that accelerator (or software
rendering if they don't have one) to render window contents into a
buffer that can then be shared with the host (and by extension the
compositor VM) by way of Wayland over virtio-gpu. Rendering on a GPU
while using Wayland over virtio-gpu isn't currently supported by
Sommelier, but Google has it planned[4].
A further problem we need to solve is how the compositor renders to
the display controller. If we can pass the display controller through
to the compositor VM (this might be a possibility on some ARM
systems), all's good. But otherwise, we'd need to render to a virtual
2D GPU, and then have the host display that on the actual display
controller. This is something that could probably be implemented
fairly easily in crosvm's virtio-gpu stack — it already supports
showing output as Wayland or X11 windows. And in the meantime, we
could cheat by running the world's simplest Wayland compositor on the
host, which would just show the crosvm window and take care of talking
to the display controller.
Graphics and windowing is definitely something that we'll be iterating
on for a very long time. We can get started with the happy path
Google already has working, with the compositor on the host. From
there, we can move the compositor into a guest, with direct GPU
passthrough. And then we can start looking at GPU virtualization.
This field is evolving /extremely/ quickly, so by the time we get
there things will undoubtedly have moved on. That means it's very
important not to work too much on it too early, because that work has
a very high chance of being obsoleted shortly afterwards. The purpose
of this report is to demonstrate to the community that GPU accelerated
applications in Spectrum will be possible by presenting how they would
be implemented if I were to do so today. But in reality, it makes
much more sense to focus on almost everything else first, because no
other area of virtualization is moving as fast as this one.
Footnotes
─────────
[1]
<https://www.amd.com/system/files/documents/amd-mxgpu-white-paper.pdf>
[2] (This would be direct passthrough, because if you have a graphics
card with one DisplayPort output, there's physically not much you can
do to share that output between multiple VMs.)
[3] I actually implemented this, before Wayland-over-virtio-gpu
emerged as a viable alternative that didn't need an extra mechanism
for this:
<https://spectrum-os.org/git/crosvm/tree/servers/src/memfd.rs?h=interguest&i…>
[4]
<https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592#note_867838>
