Thanks for keeping us updated. It's really great to have all this written up to read, even though I'm only getting to it a month and a half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when the clipboard is empty, due to trying to dereference NULL. I've patched it to fix that, and now I can run it directly under sommelier, without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it right (they migrated from C to C++ so the patch didn't apply directly).
This is great to know -- it sounds like maybe they're trying to make Sommelier more widely usable? Will probably be a while before I get to updating but this is very exicting.
- I added xwayland to the VM and sommelier command, allowing X
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open multiple connections. I was able to fix that by using `--parent` mode, and getting rid of PEER_CMD_PREFIX (which just adds some chromium paths preventing it from working).
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application directly it will hang. I used `bash -c 'firefox &'` as the command as a work-around.
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version numbers and that's working now. It needs doing properly, though. e.g. I implemented the new "sl_host_surface_damage_buffer" by simply calling the old damage function, which is obviously not correct but is working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will need updating constantly to proxy every new wayland protocol. Yet it can't add any useful security because it runs inside the VM, and is therefore untrusted.
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
Ah, yes, that would remove a lot of gotchas. I have avoided that so far because I'm hoping to eventually build custom kernels that don't need many modules, to reduce code size in each VM. But it would probably make sense to do for now.
- I switched from `bash` to `bashInteractive` as the VM shell, which
gets cursor keys working.
Good catch! I'll make that change in Spectrum as well.
- I wrote a Nix package to generate one script for each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm shopping instance, with its own /home LVM partition and IP address. It passes the network configuration using some new kernel parameters (alongside spectrumcmd).
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well for my qubes-like VMs, though I guess spectrum will need something more dynamic.
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can access from the host. One problem is that the crosvm driver runs in a minijail with a uidmap that makes every file appear to be owned by root, so only root can write things in the VM. Possibly a newer kernel would help; later versions of the kernel docs say you can include any normal FUSE flags here, so mounting with `uid=1000` might work.
I've only looked into virtio-fs a little bit -- I remember having to make a change to crosvm to make the sandboxing work. Glad to hear it's working well. I'll find out if a later kernel works when I get to updating Nixpkgs (or somebody else does -- someone on IRC was actually offering to try doing this the other day).
- Finally, I added a `vm-halt` command that just calls `reboot`, as I
don't want to develop the habit of typing `reboot` without thinking ;-)
I don't want to think about how many times I've made this mistake, lol.
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
I think it might well be -- the stuff you have going on with networking and filesystems sound great, in particular -- but I'll have to have gotten a bit more back into the project again to know exactly what. Right now I'm focusing on slowly bringing myself back up to speed and remembering what state things are in.
Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very nervous. A secure window manager is very important for Wayland, and I'm not sure how much I trust any of the existing ones to get it right. But with Wayfire I'm hoping it'll at least be easy enough to implement stuff like tagged/coloured windows for the proof of concept (since the plugin API and stuff is Wayfire's niche), and I'm hoping at some point somebody comes up with a security-focused Wayland window manager we can switch to -- I'd love a Rust one, and there's work going on in that area.
Not sure about IPC yet, but I recently read an article about PipeWire, and that's been making me think a bit about audio. With PipeWire, they seem to have cared about security from the start:
To avoid the PulseAudio sandboxing limitations, security was baked-in: a per-client permissions bitfield is attached to every PipeWire node — where one or more SPA nodes are wrapped. This security-aware design allowed easy and safe integration with Flatpak portals; the sandboxed-application permissions interface now promoted to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired by Wayland — without the XML serialization part — was implemented over Unix-domain sockets. Taymans wanted a protocol that is simple and hard-realtime safe.
It goes on to say they use this for sending file descriptors and stuff. The similarity to Wayland is very exciting, because it means we might just be able to run PipeWire over the existing virtio_wl infrastructure very efficiently.
It'll be a while before I get to look at audio in depth, but this all sounds very good -- maybe most of the work will have been done for us! In general I'm feeling very optimistic about a lot of the stuff going on in the ecosystem to try to make Flatpak and co secure -- I don't trust Flatpak itself to provide meaningful security, but it means we're getting standard mechanisms for permissions for standard applications (xdg-desktop-portal is another that comes to mind), and if this goes well it means that all we have to do is provide implementations of those standard interfaces that cross VM boundaries, and applications designed to work in Flatpak etc. should already understand how to interact with them.