- Spectrum Discuss - spectrum-os.org

Documentation of aarch64 port on Spectrum OS upstream
by Ville Ilvonen 23 Nov '22

23 Nov '22

Hi, We've built the wayland demo branch on aarch64 port and added some other apps on it to showcase embedded virtualization with Spectrum OS. The build configuration for the reference device is documented [1] with the out-of-tree build configuration as agreed in our earlier discussion[2]. There's also some additional accompanied documentation on binary cache we have used with aarch64 and could be of general use. I'd like to see that documentation upstreamed as much as possible. Would it make sense to link this work from Spectrum documentation? To indicate work in progress aarch64 support with known issues on a ref. device and avoid forking this documentation? The practical benefit is that anyone interested in building Spectrum OS for an aarch64 device using vendor BSP would have some reference linked from Spectrum OS documentation - e.g. an initial porting guide as even the x86_64 does not yet use the build configuration. Additionally, I don't want to fragment Spectrum OS ports and their documentation further than necessary. Best, -Ville [1] https://github.com/tiiuae/spectrum-config-imx8/blob/main/README.md [2] https://spectrum-os.org/lists/archives/spectrum-discuss/CAP-nJwHTmROzMbyYNt…

2 1

New user getting started questions
by Thomas Leonard 10 Oct '22

10 Oct '22

Hi, I've been running Qubes for a few years now and I'd like to give Spectrum a try, as I've been having some hardware and performance problems with Qubes. Is there some up-to-date guide I can follow? I found https://alyssa.is/using-virtio-wl/#demo and was able to see the weston terminal. I also tried updating to the latest commit and was able to get a nested wayfire window with: nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm (I'm fairly new to Nix, so not sure if this is the right way to do things) I managed to change the keyboard layout, mount a tmpfs for home, and increase the memory enough to start firefox, but I haven't managed to get much further. Things I tried so far: - I tried replacing wayfire with weston-terminal, to avoid the nested session. But sommelier segfaults when I do that. - I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM. - I tried using `-d /dev/mapper/disk` to share an LVM partition, but `mount -t ext4 /dev/vdb /tmp` refused to mount it. - I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it said it couldn't create a tap device. I guess it needs more privileges. Ideally, I'd like to run a VM with each of my old Qubes filesystems, to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-) Thanks! -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC

4 48

Wayland security contexts demo
by Alyssa Ross 03 Oct '22

03 Oct '22

Puck has created a video demonstrating the work she's been doing with the in-development Wayland security-context protocol [1], which allows a Wayland compositor to distinguish between applications running in different sandboxes (e.g. in different VMs). The video is available at https://diode.zone/w/2n3kKNNjXFkSWUwyjT3hgt Or alternatively, magnet:?xt=urn:btih:f340dfd391be0cabbb0638eb8af6659214c5d821&dn=puck%27s%20video%20720p.mp4&tr=https%3A%2F%2Fdiode.zone%2Ftracker%2Fannounce&ws=https%3A%2F%2Fdiode.zone%2Fstatic%2Fstreaming-playlists%2Fhls%2F0b093345-a100-4051-b4c3-37292af48c81%2F176adb94-167a-4cb7-b954-a09b301c4d80-720-fragmented.mp4 As part of this work, she updated the draft wlroots and Sway implementations to support the latest proposed version of the protocol, exposed the security context information to Sway configuration hooks, and created a draft crosvm implementation of exposing security context information to the compositor. There's some more information in Puck's post to the Spectrum development mailing list. [2] Thanks to NLnet and NGI Zero for funding this project. [1]: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/68 [2]: https://spectrum-os.org/lists/archives/spectrum-devel/5cf20f6f-9d89-4cf9-91…

1 0

Puck is doing a talk at the Qubes OS summit tomorrow
by Alyssa Ross 09 Sep '22

09 Sep '22

The Qubes OS summit will be held in Berlin from tomorrow (Friday) until Sunday. I'll be attending, and Puck will be giving a talk at 15:20 tomorrow. Puck's talk is called "Isolating GUIs with the power of Wayland", and the abstract is: Could Qubes OS replace its custom GUI isolation protocol with Wayland while staying as performant and secure? With the advent of Wayland, many strides have been made in the desktop Linux space, limiting the effects a malicious application can have. Gone are the days of every application being able to snoop every keypress! This presentation will dive into the differences between X and Wayland, and why it makes for a great fit in isolating operating systems like Qubes OS and Spectrum. The talk will be live streamed at: https://www.youtube.com/channel/UC_djHbyjuJvhVjfT18nyqmQ iCalendar file for the talk: https://cfp.3mdeb.com/qubes-os-summit-2022/talk/ZY8KHW.ics

1 2

HW identification and configuration on Spectrum
by Ville Ilvonen 19 Aug '22

19 Aug '22

Hi, Now that we've been developing Spectrum ARM (aarch64) support with iMX8 boards, I'd like to get back to Spectrum HW configuration design. On x86 the generic image with kernel supporting most devices as modules can make sense. On ARM, the vendor specific BSP HW quirks are more common. As of now, the spectrum fork for aarch64 just adds another config after rpi configs and replaces the default config to use that to build. With small changes this could be handled like rpi configs. In addition, cloud-hypervisor accepts kernel only in EFI format for aarch64[1]. Anyway, this would allow us to build an aarch64 Spectrum installer - even make it with a more generic kernel. That takes us to ARM vendor/device specific HW quirks which would need to be handled anyway. I'll intentionally leave device specific kernel hardening and disabling kernel module loading for security reasons for now. As of now the vendor/device specifics are not supported unless one builds device specific Spectrum image with all configs build-time and skips installer altogether. The other option that I see. We discussed earlier nix-hardware and device specific modules. That would bring nixos configuration.nix and installation supporting scripts to Spectrum, though. Those could be called from the Spectrum installer but it would change the installer logic from writing an image to dynamically configuring the device during install based on user selections. Any thoughts which would be the preferred way? Maybe some other way? In the end, HW specifics are needed also on x86 as we saw with NUCs and different Lenovo laptops in the spring. I'm not convinced one image to rule them all is realistic or secure. Finally, this is by no means blocking the hardened iMX8 based Spectrum development but will keep that work in Spectrum fork until there's an agreed path to implement this. Integrating this sooner and making it more generic would make Spectrum more useful for a wider audience. Best regards, -Ville [1] https://github.com/tiiuae/spectrum/pull/3#issuecomment-1211834302

2 6

virtio-gpu in cloud-hypervisor: which path do we take?
by Alyssa Ross 17 Aug '22

17 Aug '22

Recently I've been working on making it possible for us to use crosvm's implementation of virtio-gpu (which is necessary for multi-VM Wayland). The approach I was originally planning on was porting crosvm's vhost-user-gpu frontend to cloud-hypervisor. That would allow us to run the crosvm implementation of the device unmodified, with just a small amount of glue code in cloud-hypervisor. But then I discovered some things that made me decide to investigate other approaches: - crosvm does not implement the standard vhost-user-gpu protocol. It implements it its own special way. Perhaps ironically, the crosvm-specific way seems to be closer to how vhost-user works for other devices (like network and block), which should actually make it easier to port the frontend to cloud-hypervisor. But it also changes the potential to upstream that port to cloud-hypervisor from "a hard sell" to "not going to happen". So if I did that, it would commit us to carrying a cloud-hypervisor patch indefinitely. - There's an interesting new protocol called vfio-user, that would be really helpful to us in this situation. Whereas vhost-user requires the VMM to still have some basic per-device knowledge (the glue code I was planning to port), vfio-user operates at the PCI level, so the VMM only needs to know that the device is PCI. So if we could somehow provide a virtio-gpu device to cloud-hypervisor over vfio-user, cloud-hypervisor wouldn't need any GPU-specific code at all. Everything should just work without any changes to cloud-hypervisor, as it already implements a vfio-user client. - The next release of QEMU, 7.1.0, will include support for exporting any virtual device QEMU can provide over vfio-user. So with all this in mind, there are three ways we could try to proceed: 1. Port the crosvm-specific vhost-user-gpu frontend to cloud-hypervisor. 2. Make crosvm speak the standard version of vhost-user-gpu, then use QEMU to act as a bridge between the crosvm GPU device, and cloud-hypervisor, translating vhost-user to vfio-user, but not doing anything else. (So we're not using QEMU to run a VM, just to translate between these two protocols and handle the PCI stuff.) 3. Implement a vfio-user server in crosvm, so crosvm device backends can be used directly with cloud-hypervisor. 3 is the clear best option, because it doesn't require adding QEMU into the system, and it would be entirely upstreamable — I spoke to a crosvm developer about it in their Matrix channel and they said they'd be interested in patches for it. But it's also quite complicated to implement, and beyond my ability, at least if I want results any time soon. 2 would probably be even more complicated as it would require coordinating between crosvm and QEMU to get them to agree on how the protocol should work. So if we want this working soon, 1 is the only feasible option, at least if it's me doing the work. But it means committing to carrying a cloud-hypervisor patch until somebody comes along to implement 3. It gives me bad vibes because it goes against the upstream-first approach I try to take with Spectrum development, and once timely package updates are something we have to take more seriously, the patch no longer applying would block any sort of automatic updates. So I'm posting this to solicit thoughts on what to do here. The ideal scenario is that we are able to find somebody else (with more VMM implementation experience than me) who is able to do 3, and then I would be totally comfortable with doing 1 as a stopgap until that can happen. Otherwise, I can either keep trying to chip away at doing it myself, however long that takes, or we'd have to just accept the consequences of having the patch indefinitely, and hope that Google or somebody else also finds themself wishing crosvm had a vfio-user server and implements it themself. Thoughts welcome. :)

1 0

Unikie has hired me to work on Spectrum
by Alyssa Ross 09 Aug '22

09 Aug '22

If you've been paying close attention recently, you'll have seen patches coming from a few different Unikie[1] email accounts. In addition to contributing to Spectrum, Unikie has hired me to work on it. Unikie is interested in Spectrum for both desktop and embedded use. They're contracting for the TII Secure Systems Research Center[2], working on developing a Spectrum-based reference system. One thing they're currently working on is being able to run Spectrum on the i.MX8 development board, which means we'll hopefully see patches adding ARM and cross-compilation support to Spectrum in the near future. The first big thing I'll be working on for Unikie is finally integrating crosvm's support for graphical application VMs into Spectrum. While I'm working for Unikie, I plan to set aside any donations I receive for my Spectrum work, to be used for project expenses and funding Spectrum work from other people, rather than being used to pay for my general living expenses as has been the case up to now. And just to clarify: Spectrum has not been acquired or anything — I'm still leading the project, and Unikie has the same rights as any other contributor (copyright ownership of contributions, mostly). [1]: https://www.unikie.com/en/ [2]: https://www.tii.ae/secure-systems

3 2

Reportback from MCH 2022
by Alyssa Ross 09 Aug '22

09 Aug '22

I was recently at MCH 2022[1], one of the big European hacker camps. We had some really good conversations about Spectrum, and I thought I'd share my takeaways here: 1. We were praised for our recent documentation efforts, both in implementing Diátaxis[2] and Architecture Decision Records[3]. So big thanks to Ville for spearheading the latter. 2. We talked about the use case of having multiple user data partitions. This would allow very strict separation of security domains, and could also be helpful for data portability — you could have one user data partition in your desktop, and another on a portable disk, for example. And if, way down the line, we want to do really cool things like have live migration of VMs between systems, architecting for multiple user data partitions will be a big help with that too. This is one of those things where it's not difficult to do, as long as we plan for doing it that way from the start. But if we didn't do it that way from the start, and decided we wanted to add it later, I can see how we'd be in for a world of pain. So I think it's a sensible change to make. We're unlikely to regret making it, but reasonably likely to regret not having done it earlier if it becomes really important later on. 3. Something that can apparently be difficult for Qubes is having every VM have a unique, human-readable name in a global namespace. This means that, for example, disposable VMs have to try to generate a name that isn't already in use. This is especially relevant if we end up supporting multiple sources of VMs as described above. So in the short term, we should probably change VMs to be identified with UUIDs, and have human-readable names be a layer on top. Not having a human-readable unique names in a single global namespace will help with thinking about VMs in terms of capabilities. Since points 2 and 3 are architectural changes, I'll write them up and submit them as proper ADRs when I get the chance. [1]: https://mch2022.org/ [2]: https://diataxis.fr/ [3]: https://spectrum-os.org/doc/decisions/

1 0

Workflow preferences - case aarch64 support
by Ville Ilvonen 20 Jun '22

20 Jun '22

Hi all, Good news is that the work on aarch64 support has progressed to the point where the Spectrum host can be built natively and run on aarch64 HW. It's still a bunch of patches in various stages of WIP or review to spectrum or further up the stream. In short, to "nixpkgs-spectrum": 0001-Allow-static-building-of-dtc.patch 0002-Workaround-ld-script-segmentation-fault.patch /* may be not needed */ 0003-Add-imx8-specific-kernel.patch and to "spectrum": 0001-Add-imx8-kernel.patch Should anyone want to reproduce, please let me know and I'll try to find a solution to share them. Probably worth waiting for cross-compilation support unless you have powerful aarch64 at your disposal. In the meanwhile, we try to establish a mechanism to collaborate in branches of required upstream repos. It's fairly impractical to fork "nixpkgs-spectrum"-repo due to its' size, over 2GB, Which takes me to the discussion point - what's the workflow proposal on multiple people working on the same functionality in the same branch? If I'm not mistaken the email workflow creates a new branch for each author? Then someone would have to cherry-pick them to the same branch to integrate them in dev time hydra builds. Best, -Ville

2 1

New documentation site is live!
by Alyssa Ross 19 May '22

19 May '22

We finally have an online render of the new documentation I've been writing: https://spectrum-os.org/doc/ Let me know if anything doesn't look right. This documentation lives in the "Documentation" directory in the Spectrum repository. The online version will be automatically updated whenever changes are pushed. I'm not sure what to put on the index page. I think maybe it would make sense to just merge all the content that's currently on the rest of the website into the documentation, but I'm not sure yet. It's going to be really important for Spectrum to have solid documentation. I recently watched a talk about the Diátaxis system[1], and I think it would be a good model for us to follow. [1]: https://diataxis.fr/

1 0