Re: Qubes-lite With KVM and Wayland

On Tue, 9 Mar 2021 at 16:59, Alyssa Ross wrote:

On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:

...

On Wed, 27 Jan 2021 at 17:31, Thomas Leonard wrote: [...]

...

If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!

I got a bit further (fixed my sommelier problems), but have run out of time for now :-(

I've written up where I got to here:

https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/

I saw this online the other day and started reading it without realising it was you, and then I saw you were using Nix and thought "wow, that's close to what I'm (not) doing", and then I saw the Spectrum section, and then realised who the author was. :)

:-)

I'll quote a little from it and reply to bits:

...

When I wanted a newer package (socat with vsock support, only just released) I just told Nix to install it from the latest Git checkout of nixpkgs.

I'm excited to learn that socat has vsock support now! That's going to be very useful. I have a half-done patch somewhere that adds vsock support to strace that I should finish up as well.

Yeah, I'm using it as a hacky replacement for qrexec for now. The fact that it connects to the network system, and allows you to specify the target VM ID, makes it look like it's designed to go between VMs, but it doesn't seem like it does. I worry that they'll enable that at some point and create a sudden security problem...

...

True, my squashfs image is getting a bit big. Maybe I should instead make a minimal squashfs boot image, plus a shared directory of hard links to the required files. That would allow sharing the data with the host. I could also just share the whole /nix/store directory, if I wanted to make all host software available to guests.

I think the solution I will end up going with for this will be a custom virtiofsd implementation that can implement some access controls.

Sounds sensible.

...

I didn’t have time to write and debug C++ code for every missing Wayland protocol, so I took a short-cut: I wrote my own Wayland library, ocaml-wayland, and then used that to write my own version of sommelier. With that, adding support for copying text was fairly easy.

Well this is interesting! I definitely want to learn more about this.

I've put it up here: https://github.com/talex5/wayland-virtwl-proxy There's a default.nix file, so it should build easily enough (make sure to git clone with submodules). I'd be interested to know if it works for other people. I've been using it for about a week now, and it seems fine with firefox, evince and xfce4-terminal (the apps I use). But e.g. kitty won't run because there's no `wl_drm` support. I don't know anything about graphics acceleration. But someone on Hacker News commented that you did panfrost, so I guess you know about that sort of thing.

...

* One problem with virtwl is that, while we can receive shared memory FDs from the host, we can’t export guest memory to the host. This is unfortunate, because in Wayland the shared memory for window contents is allocated by the application from guest memory, and the proxy therefore has to copy each frame. If the host provided the memory to the guest, this wouldn’t be needed. There is a wl_drm protocol for allocating video memory, which might help here, but I don’t know how that works and, like many Wayland specifications, it seems to be in the process of being replaced by something else.

Yeah, this comes up on the virtio mailing list from time to time. It's a very difficult problem to solve, but there might be a solution some day. I think I've written about my own explorations in this area on this list before.

...

I’m not sure how guest-to-guest communication works with KVM.

It... doesn't really, at least not the way it does with Xen. virtio-vhost-user[1] is promising, but very early stages. I've talked in quite a lot of detail about how that works on this list before as well. guest-to-guest communication was my main area of work for most of the second half of last year (and what ended up causing me to burn out).

I guess once you've got shared memory and inter-VM interrupts it might be possible to reuse the Xen protocols and drivers. I made a firewall VM on Qubes that did that a few years ago (https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/). But the virtio protocols will probably be more widely supported in future.

[1]: https://wiki.qemu.org/Features/VirtioVhostUser

...

I hope the SpectrumOS project will resume at some point

Me too! Maybe it's resuming right now! (Although I'm not committing -- just because I'm feeling ready to get back into it today doesn't mean that's going to be sustainable again yet.)

:-) -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC