Table of Contents ───────────────── 1. Display controllers and renderers 2. Overview of GPU virtualization technologies .. 1. Direct passthrough .. 2. Hardware virtualization and paravirtualization ..... 1. Intel ..... 2. NVIDIA ..... 3. AMD ..... 4. ARM .. 3. Virgil 3D 3. Seamless windowing .. 1. virtio-gpu 2d .. 2. virtio-wl .. 3. virtio-gpu context types 4. Putting it together — GPUs in Spectrum VMs This report documents the current state of the GPU virtualization ecosystem as it relates to Spectrum. GPU virtualization is a very controversial topic in the world of compartmentalized operating systems. Qubes doesn't support GPU acceleration of anything but the GUI VM (the equivalent of Spectrum's compositor VM), and (I think as a result of this) a way to provide GPU accelerate to applications is *the* most requested Spectrum feature by a large margin. But there are serious security concerns with GPU virtualization. GPUs have a frankly awful track record when it comes to isolation. This might improve going forward, especially with SR-IOV-based solutions, because business demand for secure GPU virtualization is increasing and GPU vendors are trying to meet that demand. For example, AMD says of their GPU virtualization implementation[1]: The hardware-enforced memory isolation logic provides strong data security among the VFs, which helps prevent one VM from being able to access another VM’s data. With security being a bare minimum requirement for any virtualization solution, AMD’s hardware-based virtualized GPU solution offers a strong deterrent to unauthorized users who traverse the software or application layers seeking means to extract or corrupt GPU user data from the virtual machines. Although a VF can access full GPU capabilities at its own GPU partition, it does not have access to the dedicated local memory of its sibling VFs. But it remains to be seen whether these claims will stand up in practice, especially because SR-IOV is not yet widely available. The alternatives (software multiplexing and GPU emulation) have both seen exploitable security issues. Because of this, my current plan is that access to the GPU will be highly restricted in a default Spectrum system. I do think, though, that there needs to be a way for users to opt in to using whatever virtualization features their hardware provides, because I've heard so many accounts from people who find themselves unable to use compartmentalized systems /at all/ because of certain tasks where they need graphics acceleration. My primary goal with Spectrum is to bring compartmentalized computing to people who are not currently using it, and to people who're currently using mainstream systems where any application they run might be stealing their private SSH keys or ransomwaring all their files, being potentially vulnerable to zero-days in GPU drivers or hardware is the least of their problems. 1 Display controllers and renderers ═══════════════════════════════════ Standalone GPUs, and Intel integrated GPUs, tend to be display controllers and renderers all in a single package. The distinction between them will be important, though, so briefly: A *display controller* is the hardware that takes care of making a pixel grid show up on a screen. A *renderer* does computation (3D renderering, etc.) to create an image that can be given to a display controller to display. (Naturally in can also be used for other computations as well.) 2 Overview of GPU virtualization technologies ═════════════════════════════════════════════ 2.1 Direct passthrough ────────────────────── Direct passthrough is the simplest and least interesting option for accelerated graphics in a VM. The host simply gives the VM control over a whole GPU. Since we're aiming for one VM per application instance in Spectrum, direct passthrough might be useful if you have a dual GPU system, and want to accelerate one particular application, like a game. Of course, direct passthrough is also the most secure sort of GPU virtualization — data leaks are far less likely to happen when each GPU user has their own dedicated hardware! We can definitely support this. 2.2 Hardware virtualization and paravirtualization ────────────────────────────────────────────────── By *hardware virtualization*, I mean that the virtualization is implemented by the GPU itself, and by *paravirtualization* I mean that the host kernel implements virtual GPUs by multiplexing. The standard way of doing hardware virtualization is called SR-IOV, and it's widely used today with server network cards, and to a lesser extent NVMe drives. It's not widely available on GPUs, especially not consumer ones, but it seems like that's about to change. When using this sort of virtualization, all that's being virtualized is the renderer. The display controller stays attached to the host. I'm not aware of any implementation for hardware that is both a display controller and a renderer that supports separately passing through a display controller to a VM[2]. 2.2.1 Intel ╌╌╌╌╌╌╌╌╌╌╌ Intel's GPU paravirtualization technology is called GVT-g. Unlike solutions from other GPU companies, GVT-g is very widely available — it's supported for integrated graphics starting from Broadwell (launched 2014). GVT-g is definitely something we can support in Spectrum, but one limitation I found is that on my laptop (a Google Pixelbook), I can create at most two vGPUs, so don't think you're going to be able to have every application using accelerated graphics using GVT-g even if you're okay with the security implications. The most recent Intel GPUs no longer support GVT-g — they have hardware SR-IOV implementations instead. I don't believe this is supported in Linux yet, but I imagine it will be at some point. 2.2.2 NVIDIA ╌╌╌╌╌╌╌╌╌╌╌╌ NVIDIA officially supports GPU virtualization for most of their datacentre and professional GPUs, through their proprietary driver. Because of NVIDIA's general allergy to publishing code or documentation, I don't know how this GPU virtualization is implemented, but my guess is that it's mostly in hardware, because their product briefs claim their cards support SR-IOV. There also exists third-party software that can modify NVIDIA's software to support some consumer cards. The GPUs that support it tend to support between 16 and 32 virtual GPUs. Obviously, running a proprietary driver in the Spectrum host kernel would have /horrible/ security implications, and I will always strongly advise against it. But, we probably could make it work if we wanted to. 2.2.3 AMD ╌╌╌╌╌╌╌╌╌ There are five AMD GPUs that implement hardware (SR-IOV) virtualization. The Radeon Pro V520 is "only available as a public cloud offering" and so isn't a product you can actually buy. The Radeon Pro V340 is only supported by VMware, not AMD's open source Linux driver. The AMD S7100X, S7150, and S7150 x2 are all supported with KVM, but none of them have been produced since 2016. It would certainly be possible to support those last three in Spectrum, but due to just how obscure AMD GPU virtualization is, I don't see myself prioritizing it any time soon. 2.2.4 ARM ╌╌╌╌╌╌╌╌╌ There's single ARM GPU that supports hardware virtualization, the Mali-G78AE. It's intended for "automotive and industrial" applications, so I doubt it's going to show up in any hardware anybody wants to run Spectrum on. 2.3 Virgil 3D ───────────── Virgil 3D takes a different approach, by doing entirely software-based GPU virtualization. It implements OpenGL in software, in a way that's still backed by the host system's hardware GPU. This is naturally less performant than hardware virtualization or paravirtualization due to the extra overhead, but the advantages are that you can get a lot of virtual GPUs out of it, since every VM is just another application using the GPU on the host, and that this approach works even when there isn't special support for virtualizing a particular GPU. 3 Seamless windowing ════════════════════ One of the earliest features that Spectrum committed to was a seamless windowing experience, where applications running in different VMs would appear as native windows in the same Wayland compositor. 3.1 virtio-gpu 2d ───────────────── The simplest way to do seamless windowing is to run a Wayland compositor in each VM, and have it render windows to a virtual 2D GPU, where they can be displayed by a host compositor. To the host compositor, windows are basically just opaque rectangles. This is effectively how WSLg, the Windows Subsystem for Linux GUI, works. This works okay, but you lose the ability to have any integration between the application windows and the window manager they're being displayed on. It's not actually seamless. That's why, in screenshots of WSLg, you see Weston window chrome, instead of Windows ones. The WSLg developers can either choose to draw Windows window decorations around every application (which will look bad for GNOME programs that draw their own decorations), or none of them (which means they have to use Weston's decorations, or there'd be windows with no decorations), because they can't directly speak the client-side decoration negotiation protocol to the applications. This particular use case doesn't matter for Spectrum, because CSD is incompatible with unspoofable window decorations, but the point is that if you're not letting applications and window managers speak Wayland to each other, you need to come up with ad-hoc protocols every time you /do/ want them to be able to cooperate. Fortunately, there's another approach to seamless windowing, that preserves Wayland as the method of communication between applications and window manager. 3.2 virtio-wl ───────────── The original plan for seamless windowing in Spectrum was to use virtio-wl, a technology from Chromium OS. virtio-wl (plus Sommelier, a component that runs in guest userspace) presents a socket-like interface, over which Wayland clients running in VMs can send and receive arbitrary data, as well as file descriptors, to and from a Wayland compositor running on the host. Because implementing support for sending arbitrary file descriptors out of a guest would be impossible, only certain types needed by the Wayland protocol are supported — pipes and host-allocated shared memory. This worked fine for Wayland, but because of how Wayland-specific it is, virtio-wl was never considered suitable for upstreaming. This was a big problem for Spectrum specifically, because in virtio-wl there is no way for guests to share memory with the host. In virtio-wl's intended use case, the compositor runs on the host, clients run in guests, and the compositor handles all memory allocation, so memory only needs to ever be shared in one direction, host → guest. But in Spectrum, we'd like to have the compositor in its own VM, so we'd need some extra mechanism to let the compositor VM allocate host memory[3]. 3.3 virtio-gpu context types ──────────────────────────── Google's second attempt at seamless VM Wayland windowing is a new mechanism called virtio-gpu context types. Despite the "gpu" in the name, there's not (necessarily) any GPU involved — it's just convenient to use virtio-gpu as a transport since it already has primitives for sharing memory between guest and host. Context types allow new protocols, like Wayland or Vulkan, to be sent over virtio-gpu, in addition to the 2D and Virgil 3D protocols it was designed for. Context types should be available in Linux from v5.15 (the next release at the time of writing). Support for Wayland over virtio-gpu has already been implemented in crosvm and Sommelier. Even more excitingly, virtio-gpu has a mechanism by which guest userspace can do a special memory allocation that can be shared with the host! So we won't need to add any special memory allocation mechanism just for Spectrum — we'll just need to make sure the compositor VM userspace knows how to allocate memory and send it to the host in this way. 4 Putting it together — GPUs in Spectrum VMs ═════════════════════════════════════════════ After extensively researching all this, my current plan for graphics in Spectrum is as follows: each VM that runs a graphical application will get a virtio-gpu device, over which it will be expected to speak Wayland to the compositor. VMs with accelerated graphics (which will be opt-in per-VM) will additionally get another GPU device to use for rendering — this could be, for example, a GVT-g vGPU, or it could be a Virgil 3D virtio-gpu. They can use that accelerator (or software rendering if they don't have one) to render window contents into a buffer that can then be shared with the host (and by extension the compositor VM) by way of Wayland over virtio-gpu. Rendering on a GPU while using Wayland over virtio-gpu isn't currently supported by Sommelier, but Google has it planned[4]. A further problem we need to solve is how the compositor renders to the display controller. If we can pass the display controller through to the compositor VM (this might be a possibility on some ARM systems), all's good. But otherwise, we'd need to render to a virtual 2D GPU, and then have the host display that on the actual display controller. This is something that could probably be implemented fairly easily in crosvm's virtio-gpu stack — it already supports showing output as Wayland or X11 windows. And in the meantime, we could cheat by running the world's simplest Wayland compositor on the host, which would just show the crosvm window and take care of talking to the display controller. Graphics and windowing is definitely something that we'll be iterating on for a very long time. We can get started with the happy path Google already has working, with the compositor on the host. From there, we can move the compositor into a guest, with direct GPU passthrough. And then we can start looking at GPU virtualization. This field is evolving /extremely/ quickly, so by the time we get there things will undoubtedly have moved on. That means it's very important not to work too much on it too early, because that work has a very high chance of being obsoleted shortly afterwards. The purpose of this report is to demonstrate to the community that GPU accelerated applications in Spectrum will be possible by presenting how they would be implemented if I were to do so today. But in reality, it makes much more sense to focus on almost everything else first, because no other area of virtualization is moving as fast as this one. Footnotes ───────── [1] https://www.amd.com/system/files/documents/amd-mxgpu-white-paper.pdf [2] (This would be direct passthrough, because if you have a graphics card with one DisplayPort output, there's physically not much you can do to share that output between multiple VMs.) [3] I actually implemented this, before Wayland-over-virtio-gpu emerged as a viable alternative that didn't need an extra mechanism for this: https://spectrum-os.org/git/crosvm/tree/servers/src/memfd.rs?h=interguest&id=e5db778607e330a169e81f6b1d03648e0afaac6c [4] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592#note_867838