- Spectrum Discuss - spectrum-os.org

New documentation site is live!
by Alyssa Ross 19 May '22

19 May '22

We finally have an online render of the new documentation I've been writing: https://spectrum-os.org/doc/ Let me know if anything doesn't look right. This documentation lives in the "Documentation" directory in the Spectrum repository. The online version will be automatically updated whenever changes are pushed. I'm not sure what to put on the index page. I think maybe it would make sense to just merge all the content that's currently on the rest of the website into the documentation, but I'm not sure yet. It's going to be really important for Spectrum to have solid documentation. I recently watched a talk about the Diátaxis system[1], and I think it would be a good model for us to follow. [1]: https://diataxis.fr/

1 0

Documentation and diagrams
by ville.ilvonen＠unikie.com 19 May '22

19 May '22

Hi, I asked thoughts on Spectrum documentation improvements some time back on #spectrum. I'd like to improve documentation in terms of architecture (views, description, diagrams) to support communication with people adopting Spectrum in our research/engineering project(s). Of course I can create presentations for such purposes but those are not supporting the project. There's nothing wrong with asciidoc but as of now it lacks support for diagrams. Another view to diagrams was how we want to create diagrams - writing dsl(diagram-specific-language)-to-diagrams or drawing. Based on our #spectrum discussion I was guided to asciidoc diagram extension - https://docs.asciidoctor.org/diagram-extension/latest/ - so I finally gave it a try last Friday with ditaa - an ascii-to-png-conversion. Long story short - I got it working by adding dependencies to jdk8 etc. but I'm not happy with the result for following reasons: Limitations: - ditaa (and other dsl-to-diagrams) have limited support in either diagram dsl comprehension or available editors. Namely, ascii-based box drawing programs exist (e.g. https://asciiflow.com/#/) but do not directly support ditaa. If ascii-drawings can be modified to ditaa, it's slow, error-prone and worst of all - importing version controlled ditaa-ascii-graph text files back to editors do not work as object imports so in worst case the objects need to be drawn again when one wants to modify the diagrams. - dsls have learning curves with different slopes. Some are easy, some take domain level understanding (like UML) in addition to dsl but it's there. - layout algorithms - many cases go quite ok with this but then there's points where the layout algorithms fail miserably with solutions beyond reasonable effort. - dsls are usually not readable in source format but beyond trivial diagrams always require validation visually Benefits: + Some asciidoc diagram-extension dsl:s (like plantuml, graphviz) work quite ok for both visualizing UML, graphs, dependencies, trees. + Focus on content, not presentation - kind of like LaTeX. Though this is limited by tools like ditaa which can get worst of both worlds. + dsls can be great for code level views. I've used tool where dsl has been combined into generating the diagrams from code or tree like structures. In fact, there's quite a few such tools to do this with nix-store dependency trees and have been useful in spectrum architecture comprehension. What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license. I'd like to hear thoughts/decisions on this before sending any patches to contribute to spectrum architecture documentation. Best regards, -Ville

6 11

Spectrum status update, April 2022
by Alyssa Ross 01 May '22

01 May '22

The big news this month is that Puck is now working with us as a Wayland developer[1]. We also welcomed a bunch of new people into the chat channel who were interested in building Spectrum to try it out for themselves, and contributing to it or related projects, which is absolutely fantastic. [1] https://spectrum-os.org/lists/archives/spectrum-discuss/20220416092718.isdz… Here's what I did in April: Binary Cache ════════════ Using only the cache.nixos.org binary cache, building Spectrum and all its many dependencies could take more than 24 hours on commodity hardware, which was a big turn-off to the people who've been trying to build Spectrum for themselves recently. To mitigate this, this month I put together a binary cache solution for Spectrum[2], and wrote documentation explaining how to use it[3]. Thanks again to Daniel Kuehn (lejonet) for providing the storage hosting for the binary cache. [2] https://spectrum-os.org/lists/archives/spectrum-devel/20220414141547.rafken… [3] https://spectrum-os.org/git/spectrum/tree/Documentation/binary-cache.adoc Hardware support ════════════════ This month, quite a few people were interested in testing Spectrum on their hardware. This surfaced quite a few issues that have now been resolved: • Avoid EFI chainloading in the combined image.[4] • Use a bigger kernel buffer for uevents.[5] As well as one issue still in flight: • Rebroadcast uevents not handled by mdevd.[6] And one issue still being investigated. Additionally, I started to look into what Aarch64 support in Spectrum would look like. I was surprised to discover that Nixpkgs' musl stdenv hadn't even bootstrapped on aarch64 for three years! So I commited some hacks to fix that[7], and put together a follow-up that will make it maintainable going forward[8]. [4] https://spectrum-os.org/git/spectrum/commit/?id=b1ff26eb27ea905357f8bf217d0… [5] https://spectrum-os.org/git/spectrum/commit/?id=0b3b366653030de498e73e51b1c… [6] https://spectrum-os.org/lists/archives/spectrum-devel/20220424080020.32216-… [7] https://github.com/NixOS/nixpkgs/pull/169764 [8] https://github.com/NixOS/nixpkgs/pull/169793 Documentation ═════════════ I spent a lot of this month working on documentation. Having good documentation is going to be really important for bringing on new contributors. I expect to spend a bunch of time on this next month, too. • New documentation: a UUID reference[9], and information about the binary cache[3]. • A proper documentation site, based on Just the Docs[10]. Unfortunately not quite available online yet, but you can see it with nix-shell –run 'jekyll serve' in the Documentation directory. [9] https://spectrum-os.org/git/spectrum/tree/Documentation/uuid-reference.adoc [10] https://just-the-docs.github.io/just-the-docs/ Miscellaneous ═════════════ • I did a short development stream.[11] • Updated Nixpkgs, reducing our patch count by half. • Switched to s6-linux-init from a hand-rolled script, eliminating a longstanding race condition during boot. • Adopted systemd-boot, which will make Spectrum more dual-boot friendly.[12]. • Dropped s6-portable-utils dependency[13], and OVMF development dependency[14]. • Various cleanups to licensing metadata. • Fixed building Spectrum on Ubuntu[15]. • Minor cleanups to VM startup[16][17][18][19]. This is preparatory work in advance of new VM features. [11] https://diode.zone/w/p2Tythup7zJTSAgiGbJ331 [12] https://spectrum-os.org/git/spectrum/commit/?id=b72a136e9ae265f61f5c6dbbd37… [13] https://spectrum-os.org/git/spectrum/commit/?id=04060e0a59c072315b7b5106af2… [14] https://spectrum-os.org/git/spectrum/commit/?id=969842dd0b86922383b3b5e0306… [15] https://spectrum-os.org/git/spectrum/commit/?id=a567476380303fe0ccb0c136ad7… [16] https://spectrum-os.org/git/spectrum/commit/?id=ab9d7bf6d7e7b4cdd77933d5799… [17] https://spectrum-os.org/git/spectrum/commit/?id=9226197f1e9b9f17caf501b2a80… [18] https://spectrum-os.org/git/spectrum/commit/?id=0543017145207d9ea41b6140722… [19] https://spectrum-os.org/git/spectrum/commit/?h=32b701064a15f0045bce2d811a94… Spectrum related upstream Nixpkgs commits ═════════════════════════════════════════ • s6-rc: 0.5.3.0 -> 0.5.3.1 (1ee8b37a851) s6: 2.11.0.1 -> 2.11.1.0 • (bd6d4a6e67f) s6-portable-utils: 2.2.3.4 -> 2.2.4.0 (a3924d70655) • utmps: 0.1.1.0 -> 0.1.2.0 (92b74337544) skalibs: 2.11.1.0 -> • 2.11.2.0 (5eafa6cc7a7) patchelf: clarify license (3a73bf2b6e3) • pkgsMusl.coreutils: fix build on aarch64 (661dfd83478) patchelf: use • 0.13.x on aarch64+musl (e22d0b49a95) patchelf_0_13: init at 0.13.1 • (3838a0a7e7e) cloud-hypervisor: 22.1 -> 23.0 (430c56976fe) busybox: • 1.34.1 -> 1.35.0 (7d83997eb49) crosvm: 99.14468.0.0-rc1 -> • 100.14526.0.0-rc1 (40f140ad872) crosvm.updateScript: generate • Cargo.lock (26d66fbfa9b) squashfs-tools-ng: 1.1.3 -> 1.1.4 • (fd3f39303b4) qemu: remove redundant copy of qemu-ga (665bb90fc3f) • rust: 1.59.0 -> 1.60.0 (6d49a350807) openssh: 8.9p1 -> 9.0p1 • (4cfdd982653) busybox: fix CVE-2022-28391 (ac60e92b15a) Pending Spectrum related Nixpkgs PRs ════════════════════════════════════ • release.nix: build Musl native bootstrap tools (#169793) Spectrum infra related upstream Nixpkgs commits ═══════════════════════════════════════════════ • nixos/mailman: ensure Postfix is started after Mailman (572131c6a94) • ipxe: 1.21.1 -> unstable-2022-04-06 (7d691eadc37) • nixos/postfix: pull setup into its own unit (fb2fa1b50fe) • nixos/mailman: remove obsolete setting (e7c301df52e) • nixos/mailman: improve empty webHosts error (f4a0bb8334d)

1 1

Puck is joining Spectrum as a paid Wayland developer
by Alyssa Ross 16 Apr '22

16 Apr '22

Since I started working on Spectrum, one of the things I've been most excited about has been the potential of Wayland and related technologies to provide a level of integration and cohesion that hasn't been seen before in a compartmentalized operating system. It's the main thing that people who are aware of Spectrum but are not closely following its development know the project for. But people who /have/ been following my work closely will have noticed that I haven't had much to say about it for a while. My original grant agreement covered some basic Wayland functionality. With that funding, I was able to learn about, run (to my knowledge, for the first time outside the context of Chromium OS development) and experiment with virtio wayland. I produced documentation and a portable demo, that helped give other developments like Thomas's wayland-proxy-virtwl[1][2] their start. We're on a solid track to have basic Wayland functionality for applications running in isolated VMs, including things like inter-application copy/paste, and possibly optional GPU acceleration. But to have a system that provides the level of security I want Spectrum to have, there's still a lot of work to do. New protocols need to be designed, discussed, and implemented, not just in Spectrum, but in the Wayland ecosystem in general, to make it possible to do things like identify which application a Wayland window belongs to, or manage access to the clipboard. None of this is so big it's infeasible for Spectrum as a project, but it's a big time investment not covered by my original grant agreement for Spectrum, and so I've been shying away from it in favour of work with a more immediate return on investment, like my recent work creating an installer and live image for Spectrum, or figuring out how VM configuration will work. Recently, though, the stars aligned in a way that mean we should be able to make a lot of progress on these bigger Wayland developments much quicker than I was otherwise expecting. After a couple of months of working to make it happen, I'm so excited to finally be able to announce that we have secured a modest amount of additional funding to enable Puck Meerburg to work on Spectrum with me, focusing entirely on Wayland work. She will be designing the protocols we need, producing reference implementations, discussing them with upstream with an aim to get them standardised, and so on. One of the most exciting things about the work we have planned for Puck is that none of it is Spectrum-specific. In keeping with the general development philosophy for Spectrum, we want to move the ecosystem forward for everyone, rather than developing Spectrum-specific hacks that would be useless if the project ever wasn't able to continue. Puck is the ideal person to be working on this. She was already our resident Wayland expert, being able to answer just about any question I had when doing Wayland work, and even providing some patches of her own[3]. She has experience with standards processes through her work on ActivityPub. She has an amazing talent for finding bugs — just look at her list of Nix discoveries[4], and experience doing security researcher — she recently discovered a "critical"-rated security issue in Mastodon[5]. And she's proven over and over her ability to jump into an unfamiliar problem space and diagnose an issue at an amazing speed. I'm confident that having Puck spending a lot of time working with Spectrum is going to be a huge boon to the project even outside of the Wayland work she's specifically funded to work on. A final note: the grant funding for both Puck and myself expires at the end of September. What happens after that point is yet to be determined. Given the level of donations I currently receive, unless there's a sudden drop in my donation income, I'll should still be able to spend at least most of my time working on Spectrum, regardless of what happens with other funding sources. As I've said before, donation income is also extremely important to provide a level of stability and flexibility that grant funding just can't — it's important to know that I'm not going to be broke because it's taking longer than expected to implement something, or there's something unanticipated I need to do first that I don't have funding for. It would be really helpful if we could get some of that stability and flexibility for Puck as well, especially looking towards what happens once our grant funding is up. So Puck is now also accepting donations, and if you'd like to help Spectrum push forward Wayland security, please consider sponsoring her. https://github.com/sponsors/puckipedia I'm so excited for where things go from here. [1]: https://github.com/talex5/wayland-proxy-virtwl/ [2]: https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ [3]: https://spectrum-os.org/git/nixpkgs/tree/pkgs/os-specific/linux/chromium-os… [4]: https://github.com/NixOS/nix/issues?q=author%3Apuckipedia+is%3Aissue [5]: https://github.com/mastodon/mastodon/releases/tag/v3.4.6

1 0

Spectrum Status Update, March 2022
by Alyssa Ross 02 Apr '22

02 Apr '22

Hi all, I thought I'd try a different format of update. It's difficult to find the time for the big This Week in Spectrum updates I've tried to do before, but I'd like to provide some sort of account of what I've been doing. So, here's an overview of what I did in March. I'm happy to expand on any of it that sounds interesting — just hit Reply All and tell me what you'd like to hear more about! I'm also interested to hear what you think of this status update format. I'd like to get better about communicating what I'm working on, in a sustainable way. Let me know if you have any suggestions! Miscellaneous ------------- • Edited and published demo video[1] • Set up an IRC bot to post incoming mailing list messages • Switched from Busybox's modprobe to kmod • Removed unused dependencies • Various other cleanups and fixes • Started work towards CI for Spectrum • Prototyped a shared base image for application VMs [1]: https://diode.zone/w/dWAWHR38Zu3feRtDKjVEJb virtiofs investigation ---------------------- • Prototyped virtiofs VM filesystem access • Reported a bug: "Can't run unprivileged any more due to setgroups"[2] • Participated in discussion and testing of Musl port[3][4][5] [2]: https://gitlab.com/virtio-fs/virtiofsd/-/issues/36 [3]: https://github.com/slp/capng/pull/2#issuecomment-1059976861 [4]: https://github.com/slp/capng/pull/3 [5]: https://github.com/rust-lang/libc/pull/2713 Spectrum-related upstream Nixpkgs commits ----------------------------------------- • lvm2: don't use targetPlatform (05a6c124e65) • coreutils: add debug output (e30f0f31e8d) • pkgsMusl.systemd: fix build for 250.4 (39eee39fd92) • nghttp2: only run tests on GNU (8685cea963b) • python3.pkgs.importlib-metadata: fix cross (3c7b77e638b) • spidermonkey: use the same LLVM as rustc (3ff5f0eb764) • pkgsStatic.stdenv.cc.cc: put static libs in $lib (12c37aec377) • Revert "gcc: Always pass `--enable-shared` by default" (c6dd11ca39a) • libudev-zero: 1.0.0 -> 1.0.1 (c7b7ad77985) • linux_latest: 5.16.14 -> 5.17 (58ae11758e8) • crosvm: 81.12871.0.0-rc1 -> 99.14468.0.0-rc1 (6aefdafbed9) • shadow: 4.8.1 -> 4.8.11 (8d35d7e2bf1) • pkgsMusl.libnetfilter_conntrack: fix build (2cc5ec86571) • pkgsMusl.systemdMinimal: fix build (b8734c50e29) • linux.configfile: fix alts containing "/m" (fb079c3110d) • cloud-hypervisor: 21.0 -> 22.0 (36a211e1ee3) • edk2: 202108 -> 202202 (9222b68380e) • kmod: add dev and lib outputs (dc1303185f8) • systemd: update patchShebangs comment (a0bfc8e7c1f) • systemd: fix a whole bunch of typos (479b1cb510b) Pending Spectrum related Nixpkgs PRs ------------------------------------ • crosvm: add support for virgl_renderer{,_next} (#165128) • qemu: 6.2.0 -> 7.0.0 (#165291) Spectrum infra related upstream Nixpkgs commits ----------------------------------------------- • irccat: init at 0.4.8 (ce8cbe3c01f) • git: enable debug info (4345b27dedf) • cgit-pink: init at 1.3.0 (deab83e1167) • mailman-web: fix django version check removal (3512f5b7075) Demo video related upstream Nixpkgs commits ------------------------------------------- • ccsymbols: init at 2020-04-19 (cf7556eea5a)

1 0

PlagueOS
by Geblaat 23 Mar '22

23 Mar '22

I recently came across Spectrum. I myself use Qubes OS quite a lot, and I also find Spectrum really interesting. (Though I'm not very technical, so I'll have to wait for development to be in another stage for me to try it.) On the mailing list, I saw an article about FlexOS, which has some similar goals, though it seems to be in very early stages. I thought PlagueOS might be of interest to people here as well. Like Spectrum, it has better hardware compatibility and performance compared to Qubes OS. They use Void OS as hypervisor(looks like standard KVM compared to a more secure Rust alternative) and a lot of hardening. It already has an ISO image available and a graphical desktop environment, so it can already be used by people with not much technical skills. discuss(a)spectrum-os.org

2 2

Spectrum video demo/update
by Alyssa Ross 15 Mar '22

15 Mar '22

Hi all, For the last couple of weeks I've been working on a video demo of some things I've been working on in Spectrum: https://diode.zone/w/dWAWHR38Zu3feRtDKjVEJb Happy to answer any questions, either here, in the comments, or on IRC. Alyssa

1 0

Using virto-gpu for display output for Apps and OS
by Subash SN 10 Jan '22

10 Jan '22

Hi, We tried running GUI apps and OS using virtio-gpu directly. Please check it out here https://blog.openw3b.org/crosvm-for-os-and-app-virtualization-on-linux/. The idea is further reduce the attack surface by avoiding wayland. Please share your views on the same. This mailing list was very helpful along the way! Thank you. - Subash

1 0

Paper review: FlexOS
by Alyssa Ross 03 Jan '22

03 Jan '22

Earlier I was sent a link to a paper called "FlexOS: Towards Flexible OS Isolation"[1]. I thought the paper was great and subscribers to this list might be interested in it and what I thought about it. Like Spectrum, FlexOS is an operating system focused on compartmentalization, but it takes a very different approach. It's a library operating system (LibOS, also known as a unikernel). These operating systems are linked with a single program, and produce an operating system that only runs that single program. This can make for very efficient VMs. As a result, FlexOS isn't about isolation between programs, like Spectrum is, but instead tries to introduce isolation within a single program, generally at library boundaries. Therefore, Spectrum and FlexOS are highly complementary. FlexOS can use several different isolation strategies, including running different libraries in different VMs. Naturally there's a big performance overhead to this but amazingly, it seems like it can be on par with running the program normally on Linux, because the baseline performance is so much faster running on a unikernel than on Linux (not even in a VM AFAICT). Of a SQLite benchmark, the paper says: "Somewhat surprisingly, FlexOS with EPT2 [libraries in different VMs] performs almost identically to Linux. This is because the syscall latency is almost identical to the EPT2 gate latency on this system". Another inspired part of this paper is "Exploration with Partial Safety Ordering". I don't want to go into too much detail about this because the paper does such a good job of explaining it, but essentially you can give FlexOS a performance budget and a benchmark, and it will identify for you five or so of the best security configurations that meet your performance needs, across different combinations of isolation primitive, software hardening, etc. The biggest limitation I can see to FlexOS is that it is based on C source code transformations. Programs need to be specifically ported to it (although it seems like porting an individual program isn't too hard as long as it has good test coverage), and I don't know how easy it would be to integrate with another language. But it's definitely something to keep an eye on that could be useful if we ever find ourselves with a big C program doing important work in Spectrum. The use cases presented in the paper are quite compelling, especially the one about isolating exploitable libraries. And unikernels more generally are definitely worth keeping in mind if the performance results in this paper are anything to go by, although A Linux in Unikernel Clothing[2] is an interesting counterpoint. Finally, reminder that I collect interesting papers, blog posts, talks etc. in the Spectrum bibliography[3], where I've just added a link to this paper. If you come across interesting stuff like this that might be relevant to my work, please send it my way! [1]: https://arxiv.org/abs/2112.06566 [2]: https://doi.org/10.1145/3342195.3387526 [3]: https://spectrum-os.org/bibliography.html

1 0

GPU virtualization in Spectrum
by Alyssa Ross 03 Nov '21

03 Nov '21

Table of Contents ───────────────── 1. Display controllers and renderers 2. Overview of GPU virtualization technologies .. 1. Direct passthrough .. 2. Hardware virtualization and paravirtualization ..... 1. Intel ..... 2. NVIDIA ..... 3. AMD ..... 4. ARM .. 3. Virgil 3D 3. Seamless windowing .. 1. virtio-gpu 2d .. 2. virtio-wl .. 3. virtio-gpu context types 4. Putting it together — GPUs in Spectrum VMs This report documents the current state of the GPU virtualization ecosystem as it relates to Spectrum. GPU virtualization is a very controversial topic in the world of compartmentalized operating systems. Qubes doesn't support GPU acceleration of anything but the GUI VM (the equivalent of Spectrum's compositor VM), and (I think as a result of this) a way to provide GPU accelerate to applications is *the* most requested Spectrum feature by a large margin. But there are serious security concerns with GPU virtualization. GPUs have a frankly awful track record when it comes to isolation. This might improve going forward, especially with SR-IOV-based solutions, because business demand for secure GPU virtualization is increasing and GPU vendors are trying to meet that demand. For example, AMD says of their GPU virtualization implementation[1]: The hardware-enforced memory isolation logic provides strong data security among the VFs, which helps prevent one VM from being able to access another VM’s data. With security being a bare minimum requirement for any virtualization solution, AMD’s hardware-based virtualized GPU solution offers a strong deterrent to unauthorized users who traverse the software or application layers seeking means to extract or corrupt GPU user data from the virtual machines. Although a VF can access full GPU capabilities at its own GPU partition, it does not have access to the dedicated local memory of its sibling VFs. But it remains to be seen whether these claims will stand up in practice, especially because SR-IOV is not yet widely available. The alternatives (software multiplexing and GPU emulation) have both seen exploitable security issues. Because of this, my current plan is that access to the GPU will be highly restricted in a default Spectrum system. I do think, though, that there needs to be a way for users to opt in to using whatever virtualization features their hardware provides, because I've heard so many accounts from people who find themselves unable to use compartmentalized systems /at all/ because of certain tasks where they need graphics acceleration. My primary goal with Spectrum is to bring compartmentalized computing to people who are not currently using it, and to people who're currently using mainstream systems where any application they run might be stealing their private SSH keys or ransomwaring all their files, being potentially vulnerable to zero-days in GPU drivers or hardware is the least of their problems. 1 Display controllers and renderers ═══════════════════════════════════ Standalone GPUs, and Intel integrated GPUs, tend to be display controllers and renderers all in a single package. The distinction between them will be important, though, so briefly: A *display controller* is the hardware that takes care of making a pixel grid show up on a screen. A *renderer* does computation (3D renderering, etc.) to create an image that can be given to a display controller to display. (Naturally in can also be used for other computations as well.) 2 Overview of GPU virtualization technologies ═════════════════════════════════════════════ 2.1 Direct passthrough ────────────────────── Direct passthrough is the simplest and least interesting option for accelerated graphics in a VM. The host simply gives the VM control over a whole GPU. Since we're aiming for one VM per application instance in Spectrum, direct passthrough might be useful if you have a dual GPU system, and want to accelerate one particular application, like a game. Of course, direct passthrough is also the most secure sort of GPU virtualization — data leaks are far less likely to happen when each GPU user has their own dedicated hardware! We can definitely support this. 2.2 Hardware virtualization and paravirtualization ────────────────────────────────────────────────── By *hardware virtualization*, I mean that the virtualization is implemented by the GPU itself, and by *paravirtualization* I mean that the host kernel implements virtual GPUs by multiplexing. The standard way of doing hardware virtualization is called SR-IOV, and it's widely used today with server network cards, and to a lesser extent NVMe drives. It's not widely available on GPUs, especially not consumer ones, but it seems like that's about to change. When using this sort of virtualization, all that's being virtualized is the renderer. The display controller stays attached to the host. I'm not aware of any implementation for hardware that is both a display controller and a renderer that supports separately passing through a display controller to a VM[2]. 2.2.1 Intel ╌╌╌╌╌╌╌╌╌╌╌ Intel's GPU paravirtualization technology is called GVT-g. Unlike solutions from other GPU companies, GVT-g is very widely available — it's supported for integrated graphics starting from Broadwell (launched 2014). GVT-g is definitely something we can support in Spectrum, but one limitation I found is that on my laptop (a Google Pixelbook), I can create at most two vGPUs, so don't think you're going to be able to have every application using accelerated graphics using GVT-g even if you're okay with the security implications. The most recent Intel GPUs no longer support GVT-g — they have hardware SR-IOV implementations instead. I don't believe this is supported in Linux yet, but I imagine it will be at some point. 2.2.2 NVIDIA ╌╌╌╌╌╌╌╌╌╌╌╌ NVIDIA officially supports GPU virtualization for most of their datacentre and professional GPUs, through their proprietary driver. Because of NVIDIA's general allergy to publishing code or documentation, I don't know how this GPU virtualization is implemented, but my guess is that it's mostly in hardware, because their product briefs claim their cards support SR-IOV. There also exists third-party software that can modify NVIDIA's software to support some consumer cards. The GPUs that support it tend to support between 16 and 32 virtual GPUs. Obviously, running a proprietary driver in the Spectrum host kernel would have /horrible/ security implications, and I will always strongly advise against it. But, we probably could make it work if we wanted to. 2.2.3 AMD ╌╌╌╌╌╌╌╌╌ There are five AMD GPUs that implement hardware (SR-IOV) virtualization. The Radeon Pro V520 is "only available as a public cloud offering" and so isn't a product you can actually buy. The Radeon Pro V340 is only supported by VMware, not AMD's open source Linux driver. The AMD S7100X, S7150, and S7150 x2 are all supported with KVM, but none of them have been produced since 2016. It would certainly be possible to support those last three in Spectrum, but due to just how obscure AMD GPU virtualization is, I don't see myself prioritizing it any time soon. 2.2.4 ARM ╌╌╌╌╌╌╌╌╌ There's single ARM GPU that supports hardware virtualization, the Mali-G78AE. It's intended for "automotive and industrial" applications, so I doubt it's going to show up in any hardware anybody wants to run Spectrum on. 2.3 Virgil 3D ───────────── Virgil 3D takes a different approach, by doing entirely software-based GPU virtualization. It implements OpenGL in software, in a way that's still backed by the host system's hardware GPU. This is naturally less performant than hardware virtualization or paravirtualization due to the extra overhead, but the advantages are that you can get a lot of virtual GPUs out of it, since every VM is just another application using the GPU on the host, and that this approach works even when there isn't special support for virtualizing a particular GPU. 3 Seamless windowing ════════════════════ One of the earliest features that Spectrum committed to was a seamless windowing experience, where applications running in different VMs would appear as native windows in the same Wayland compositor. 3.1 virtio-gpu 2d ───────────────── The simplest way to do seamless windowing is to run a Wayland compositor in each VM, and have it render windows to a virtual 2D GPU, where they can be displayed by a host compositor. To the host compositor, windows are basically just opaque rectangles. This is effectively how WSLg, the Windows Subsystem for Linux GUI, works. This works okay, but you lose the ability to have any integration between the application windows and the window manager they're being displayed on. It's not actually seamless. That's why, in screenshots of WSLg, you see Weston window chrome, instead of Windows ones. The WSLg developers can either choose to draw Windows window decorations around every application (which will look bad for GNOME programs that draw their own decorations), or none of them (which means they have to use Weston's decorations, or there'd be windows with no decorations), because they can't directly speak the client-side decoration negotiation protocol to the applications. This particular use case doesn't matter for Spectrum, because CSD is incompatible with unspoofable window decorations, but the point is that if you're not letting applications and window managers speak Wayland to each other, you need to come up with ad-hoc protocols every time you /do/ want them to be able to cooperate. Fortunately, there's another approach to seamless windowing, that preserves Wayland as the method of communication between applications and window manager. 3.2 virtio-wl ───────────── The original plan for seamless windowing in Spectrum was to use virtio-wl, a technology from Chromium OS. virtio-wl (plus Sommelier, a component that runs in guest userspace) presents a socket-like interface, over which Wayland clients running in VMs can send and receive arbitrary data, as well as file descriptors, to and from a Wayland compositor running on the host. Because implementing support for sending arbitrary file descriptors out of a guest would be impossible, only certain types needed by the Wayland protocol are supported — pipes and host-allocated shared memory. This worked fine for Wayland, but because of how Wayland-specific it is, virtio-wl was never considered suitable for upstreaming. This was a big problem for Spectrum specifically, because in virtio-wl there is no way for guests to share memory with the host. In virtio-wl's intended use case, the compositor runs on the host, clients run in guests, and the compositor handles all memory allocation, so memory only needs to ever be shared in one direction, host → guest. But in Spectrum, we'd like to have the compositor in its own VM, so we'd need some extra mechanism to let the compositor VM allocate host memory[3]. 3.3 virtio-gpu context types ──────────────────────────── Google's second attempt at seamless VM Wayland windowing is a new mechanism called virtio-gpu context types. Despite the "gpu" in the name, there's not (necessarily) any GPU involved — it's just convenient to use virtio-gpu as a transport since it already has primitives for sharing memory between guest and host. Context types allow new protocols, like Wayland or Vulkan, to be sent over virtio-gpu, in addition to the 2D and Virgil 3D protocols it was designed for. Context types should be available in Linux from v5.15 (the next release at the time of writing). Support for Wayland over virtio-gpu has already been implemented in crosvm and Sommelier. Even more excitingly, virtio-gpu has a mechanism by which guest userspace can do a special memory allocation that can be shared with the host! So we won't need to add any special memory allocation mechanism just for Spectrum — we'll just need to make sure the compositor VM userspace knows how to allocate memory and send it to the host in this way. 4 Putting it together — GPUs in Spectrum VMs ═════════════════════════════════════════════ After extensively researching all this, my current plan for graphics in Spectrum is as follows: each VM that runs a graphical application will get a virtio-gpu device, over which it will be expected to speak Wayland to the compositor. VMs with accelerated graphics (which will be opt-in per-VM) will additionally get another GPU device to use for rendering — this could be, for example, a GVT-g vGPU, or it could be a Virgil 3D virtio-gpu. They can use that accelerator (or software rendering if they don't have one) to render window contents into a buffer that can then be shared with the host (and by extension the compositor VM) by way of Wayland over virtio-gpu. Rendering on a GPU while using Wayland over virtio-gpu isn't currently supported by Sommelier, but Google has it planned[4]. A further problem we need to solve is how the compositor renders to the display controller. If we can pass the display controller through to the compositor VM (this might be a possibility on some ARM systems), all's good. But otherwise, we'd need to render to a virtual 2D GPU, and then have the host display that on the actual display controller. This is something that could probably be implemented fairly easily in crosvm's virtio-gpu stack — it already supports showing output as Wayland or X11 windows. And in the meantime, we could cheat by running the world's simplest Wayland compositor on the host, which would just show the crosvm window and take care of talking to the display controller. Graphics and windowing is definitely something that we'll be iterating on for a very long time. We can get started with the happy path Google already has working, with the compositor on the host. From there, we can move the compositor into a guest, with direct GPU passthrough. And then we can start looking at GPU virtualization. This field is evolving /extremely/ quickly, so by the time we get there things will undoubtedly have moved on. That means it's very important not to work too much on it too early, because that work has a very high chance of being obsoleted shortly afterwards. The purpose of this report is to demonstrate to the community that GPU accelerated applications in Spectrum will be possible by presenting how they would be implemented if I were to do so today. But in reality, it makes much more sense to focus on almost everything else first, because no other area of virtualization is moving as fast as this one. Footnotes ───────── [1] <https://www.amd.com/system/files/documents/amd-mxgpu-white-paper.pdf> [2] (This would be direct passthrough, because if you have a graphics card with one DisplayPort output, there's physically not much you can do to share that output between multiple VMs.) [3] I actually implemented this, before Wayland-over-virtio-gpu emerged as a viable alternative that didn't need an extra mechanism for this: <https://spectrum-os.org/git/crosvm/tree/servers/src/memfd.rs?h=interguest&i…> [4] <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9592#note_867838>

2 2