Thanks for keeping us updated. It's really great to have all this
written up to read, even though I'm only getting to it a month and a
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when
the clipboard is empty, due to trying to dereference NULL. I've
patched it to fix that, and now I can run it directly under sommelier,
without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of
some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it
right (they migrated from C to C++ so the patch didn't apply
This is great to know -- it sounds like maybe they're trying to make
Sommelier more widely usable? Will probably be a while before I get to
updating but this is very exicting.
- I added xwayland to the VM and sommelier command,
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open
I was able to fix that by using `--parent` mode, and getting rid of
PEER_CMD_PREFIX (which just adds some chromium paths preventing it
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application
directly it will hang. I used `bash -c 'firefox &'` as the command as
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version
numbers and that's working now. It needs doing properly, though. e.g.
I implemented the new "sl_host_surface_damage_buffer" by simply
calling the old damage function, which is obviously not correct but is
working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will
need updating constantly to proxy every new wayland protocol. Yet it
can't add any useful security because it runs inside the VM, and is
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
Ah, yes, that would remove a lot of gotchas. I have avoided that so far
because I'm hoping to eventually build custom kernels that don't need
many modules, to reduce code size in each VM. But it would probably
make sense to do for now.
- I switched from `bash` to `bashInteractive` as the
VM shell, which
gets cursor keys working.
Good catch! I'll make that change in Spectrum as well.
- I wrote a Nix package to generate one script for
each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm
shopping instance, with its own /home LVM partition and IP address. It
passes the network configuration using some new kernel parameters
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well
for my qubes-like VMs, though I guess spectrum will need something
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can
access from the host.
One problem is that the crosvm driver runs in a minijail with a
uidmap that makes every file appear to be owned by root, so only root
can write things in the VM.
Possibly a newer kernel would help; later versions of the kernel
docs say you can include any normal FUSE flags here, so mounting with
`uid=1000` might work.
I've only looked into virtio-fs a little bit -- I remember having to
make a change to crosvm to make the sandboxing work. Glad to hear it's
working well. I'll find out if a later kernel works when I get to
updating Nixpkgs (or somebody else does -- someone on IRC was actually
offering to try doing this the other day).
- Finally, I added a `vm-halt` command that just calls
`reboot`, as I
don't want to develop the habit of typing `reboot` without thinking
I don't want to think about how many times I've made this mistake, lol.
If any of this sounds useful for spectrum let me know.
I can try and
tidy it up; it's all a huge mess at the moment!
I think it might well be -- the stuff you have going on with networking
and filesystems sound great, in particular -- but I'll have to have
gotten a bit more back into the project again to know exactly what.
Right now I'm focusing on slowly bringing myself back up to speed and
remembering what state things are in.
Once this is working more smoothly, I guess the next
issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very
nervous. A secure window manager is very important for Wayland, and I'm
not sure how much I trust any of the existing ones to get it right. But
with Wayfire I'm hoping it'll at least be easy enough to implement stuff
like tagged/coloured windows for the proof of concept (since the
plugin API and stuff is Wayfire's niche), and I'm hoping at some point
somebody comes up with a security-focused Wayland window manager we can
switch to -- I'd love a Rust one, and there's work going on in that
Not sure about IPC yet, but I recently read an article about PipeWire,
and that's been making me think a bit about audio. With PipeWire, they
seem to have cared about security from the start:
To avoid the PulseAudio sandboxing limitations,
baked-in: a per-client permissions bitfield is attached to every
PipeWire node — where one or more SPA nodes are wrapped. This
security-aware design allowed easy and safe integration with Flatpak
portals; the sandboxed-application permissions interface now promoted
to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired
by Wayland —
without the XML serialization part — was implemented over Unix-domain
sockets. Taymans wanted a protocol that is simple and hard-realtime
It goes on to say they use this for sending file descriptors and stuff.
The similarity to Wayland is very exciting, because it means we might
just be able to run PipeWire over the existing virtio_wl infrastructure
It'll be a while before I get to look at audio in depth, but this all
sounds very good -- maybe most of the work will have been done for us!
In general I'm feeling very optimistic about a lot of the stuff going on
in the ecosystem to try to make Flatpak and co secure -- I don't trust
Flatpak itself to provide meaningful security, but it means we're
getting standard mechanisms for permissions for standard applications
(xdg-desktop-portal is another that comes to mind), and if this goes
well it means that all we have to do is provide implementations of those
standard interfaces that cross VM boundaries, and applications designed
to work in Flatpak etc. should already understand how to interact with