Qubes-lite With KVM and Wayland

On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:

On Wed, 27 Jan 2021 at 17:31, Thomas Leonard wrote: [...]

...

If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!

I got a bit further (fixed my sommelier problems), but have run out of time for now :-(

I've written up where I got to here:

https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/

I saw this online the other day and started reading it without realising it was you, and then I saw you were using Nix and thought "wow, that's close to what I'm (not) doing", and then I saw the Spectrum section, and then realised who the author was. :) I'll quote a little from it and reply to bits:

When I wanted a newer package (socat with vsock support, only just released) I just told Nix to install it from the latest Git checkout of nixpkgs.

I'm excited to learn that socat has vsock support now! That's going to be very useful. I have a half-done patch somewhere that adds vsock support to strace that I should finish up as well.

True, my squashfs image is getting a bit big. Maybe I should instead make a minimal squashfs boot image, plus a shared directory of hard links to the required files. That would allow sharing the data with the host. I could also just share the whole /nix/store directory, if I wanted to make all host software available to guests.

I think the solution I will end up going with for this will be a custom virtiofsd implementation that can implement some access controls. The even simpler solution would be to seperately expose every store path we want to share as a virtio-fs device, but that's a lot of virtio devices! (I vaguely remember the maximum might be as low as 16, too).

I didn’t have time to write and debug C++ code for every missing Wayland protocol, so I took a short-cut: I wrote my own Wayland library, ocaml-wayland, and then used that to write my own version of sommelier. With that, adding support for copying text was fairly easy.

Well this is interesting! I definitely want to learn more about this.

* One problem with virtwl is that, while we can receive shared memory FDs from the host, we can’t export guest memory to the host. This is unfortunate, because in Wayland the shared memory for window contents is allocated by the application from guest memory, and the proxy therefore has to copy each frame. If the host provided the memory to the guest, this wouldn’t be needed. There is a wl_drm protocol for allocating video memory, which might help here, but I don’t know how that works and, like many Wayland specifications, it seems to be in the process of being replaced by something else.

Yeah, this comes up on the virtio mailing list from time to time. It's a very difficult problem to solve, but there might be a solution some day. I think I've written about my own explorations in this area on this list before.

I’m not sure how guest-to-guest communication works with KVM.

It... doesn't really, at least not the way it does with Xen. virtio-vhost-user[1] is promising, but very early stages. I've talked in quite a lot of detail about how that works on this list before as well. guest-to-guest communication was my main area of work for most of the second half of last year (and what ended up causing me to burn out). [1]: https://wiki.qemu.org/Features/VirtioVhostUser

I hope the SpectrumOS project will resume at some point

Me too! Maybe it's resuming right now! (Although I'm not committing -- just because I'm feeling ready to get back into it today doesn't mean that's going to be sustainable again yet.)