Hi,
I've been running Qubes for a few years now and I'd like to give Spectrum a try, as I've been having some hardware and performance problems with Qubes. Is there some up-to-date guide I can follow? I found https://alyssa.is/using-virtio-wl/#demo and was able to see the weston terminal. I also tried updating to the latest commit and was able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and increase the memory enough to start firefox, but I haven't managed to get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested session. But sommelier segfaults when I do that. - I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM. - I tried using `-d /dev/mapper/disk` to share an LVM partition, but `mount -t ext4 /dev/vdb /tmp` refused to mount it. - I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it said it couldn't create a tap device. I guess it needs more privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems, to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-)
Thanks!
I've been running Qubes for a few years now and I'd like to give Spectrum a try, as I've been having some hardware and performance problems with Qubes. Is there some up-to-date guide I can follow? I
Well, there are currently tools as in building blocks, but not yet something announced as a complete solution for a usease.
found https://alyssa.is/using-virtio-wl/#demo and was able to see the weston terminal. I also tried updating to the latest commit and was able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and increase the memory enough to start firefox, but I haven't managed to get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems,
Depending on how the device setup ends up going, that might be practical or not so practical (although hopefully a reasonable amount of driver and window system setup inside each VM should be enough in any case)
to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-)
1) Alyssa is still working on it
2) There is a grant with some milestones still to be reached (and some earlier ones ahve been paid) and some support from people sponsoring Alyssa's work via GitHub
3) Alyssa does so much of the work alone, that a lot of things are just… done without any discussion; some of the discussions are via IRC.
4) Alyssa has mentioned taking a break, and she has not yet announced being back from the break (which is in line with the planned duration of the break)
PS. On the other had, do not forget that SpectrumOS aims at better efficiency in some places, meaning it is almost sure to end up less secure than Qubes, and the first usable version will likely still be a prototype with even lower level of security due to compromises in the component choice.
- Alyssa has mentioned taking a break, and she has not yet announced
being back from the break (which is in line with the planned duration of the break)
Quick update on this -- I'm starting to feel ready to return and have a plan for it, although there's some behind-the-scenes stuff I need to sort out before I'll be back writing code. :)
On 1/6/21 7:04 AM, Alyssa Ross wrote:
- Alyssa has mentioned taking a break, and she has not yet announced
being back from the break (which is in line with the planned duration of the break)
Quick update on this -- I'm starting to feel ready to return and have a plan for it, although there's some behind-the-scenes stuff I need to sort out before I'll be back writing code. :)
Good to hear from you, Alyssa! Happy New Year, and take good care of yourself. The project can wait, self-care is important. :-)
-- Best, r
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael says, I'm currently working towards a proof of concept, so none of what you've tried out so far is really meant for use outside of that proof of concept.
I've been running Qubes for a few years now and I'd like to give Spectrum a try, as I've been having some hardware and performance problems with Qubes. Is there some up-to-date guide I can follow? I found https://alyssa.is/using-virtio-wl/#demo and was able to see the weston terminal. I also tried updating to the latest commit and was able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm(I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of derivations, which is why you end up with lots of numbered result-* symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you a single result symlink pointing to that, and you won't need to go hunting for the right one. :)
I managed to change the keyboard layout, mount a tmpfs for home, and increase the memory enough to start firefox, but I haven't managed to get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
I'm surprised -- this has worked for me before, although it's been a while since I tried this so maybe I changed something.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it to the package since I mostly have been working with my own source builds of crosvm.
[1]: https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100db...
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to do with Nix). You can make a TAP device yourself iproute2 and use --tap-fd to tell crosvm to use it, or you can use the mktuntap program I wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \ sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
Ideally, I'd like to run a VM with each of my old Qubes filesystems, to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready to use like this, but I am working on it (or at least I will be again once my anti-burnout break ends). Once I am, I hope to be more active on the lists again. I used to post weekly status updates, and would like to get into doing that again once I'm back because they were a great way to keep people up to date with the project and for me to have a record of what I'd been doing. Reading some of the old status updates should give you a bit of a feel for where things are, although things are a bit further along than they were when I wrote the last one because I put the status updates on hold to try to chase a funding milestone.
Hope that's all clear -- please ask more questions if you have them, although if it's anything particularly in the weeds I might wait until I'm back from my break to answer. :)
On Wed, 6 Jan 2021 at 07:01, Alyssa Ross hi@alyssa.is wrote:
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael says, I'm currently working towards a proof of concept, so none of what you've tried out so far is really meant for use outside of that proof of concept.
[...]
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm(I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of derivations, which is why you end up with lots of numbered result-* symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you a single result symlink pointing to that, and you won't need to go hunting for the right one. :)
That's better, thanks!
[...]
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it to the package since I mostly have been working with my own source builds of crosvm.
Ah, I didn't realise it was using seccomp too. I'm not sure how to compile specific versions of crosvm. I tried with:
srcs = lib.genAttrs [ "src/third_party/adhd" "src/aosp/external/minijail" ] getSrc // { "src/platform/crosvm" = /home/.../crosvm; };
and blanked out the hash as it requested, but then:
error: failed to sync Caused by: failed to load pkg lockfile Caused by: failed to resolve patches for `https://github.com/rust-lang/crates.io-index%60 Caused by: failed to load source for dependency `libvda` Caused by: Unable to update /build/src/platform2/arc/vm/libvda/rust Caused by: failed to read `/build/src/platform2/arc/vm/libvda/rust/Cargo.toml`
Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f (added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312 just before that gets further, but:
error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be updated but --frozen was passed to prevent this
How do you build it?
(sorry for these basic Nix/Rust questions)
However, I could get 9p to work by running the previous version with --seccomp-log-failures. With that, I can read and write files from the console, but I can't chown things and so can't write from the terminal window, which is running as a user. I guess it needs uidmap set, but I'm not sure how to make that work.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid.
OK, I'll keep trying stuff. I have discovered that if I add the squashfs file as another device (--root "$rootfs" -d "$rootfs") then it shows an error but mounts it anyway!
# ls /tmp # mount /dev/vdb /tmp [ 15.288873] /dev/vdb: Can't open blockdev # ls /tmp bin dev etc nix proc run sbin sys tmp
Sadly, this didn't work with my ext4 partition.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to do with Nix). You can make a TAP device yourself iproute2 and use --tap-fd to tell crosvm to use it, or you can use the mktuntap program I wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \ sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel"
I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31 ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
Ideally, I'd like to run a VM with each of my old Qubes filesystems, to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready to use like this, but I am working on it (or at least I will be again once my anti-burnout break ends).
Great! I'm on a break myself at the moment, which is why I have some time to try all this out.
Once I am, I hope to be more active on the lists again. I used to post weekly status updates, and would like to get into doing that again once I'm back because they were a great way to keep people up to date with the project and for me to have a record of what I'd been doing. Reading some of the old status updates should give you a bit of a feel for where things are, although things are a bit further along than they were when I wrote the last one because I put the status updates on hold to try to chase a funding milestone.
I've read some of them - they're very helpful!
Hope that's all clear -- please ask more questions if you have them, although if it's anything particularly in the weeds I might wait until I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I need to figure out how to make this all work myself anyway, and it's just a bonus if you've already done the work for me...
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard talex5@gmail.com wrote: [...]
exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel"
I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1)
D'oh! I just realised you're not supposed to use the other network options when using `--tap-fd`!
I was then able to browse the web from crosvm, like this:
- Add pkgs/os-specific/linux/spectrum/rootfs/etc/resolv.conf with e.g. "nameserver 8.8.8.8". - Configure the virtual eth0 in the VM setup script: foreground { ifconfig eth0 10.0.0.2 up } foreground { route add default gateway 10.0.0.1 } - Enable NAT in configuration.nix, e.g. networking.nat = { enable = true; externalInterface = "eno2"; internalIPs = [ "10.0.0.0/8" ]; }; - Start the VM. - Run "sudo ifconfig tap0 10.0.0.1 up" on host. - Run firefox in VM to browse the web :-)
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard talex5@gmail.com wrote: [...]
Sadly, this didn't work with my ext4 partition.
This turned out to be just because ext4 is compiled as a module, and /lib isn't in the root filesystem. Adding "EXT4_FS = yes" to the kernel configuration fixed it.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it to the package since I mostly have been working with my own source builds of crosvm.
Ah, I didn't realise it was using seccomp too. I'm not sure how to compile specific versions of crosvm. I tried with:
srcs = lib.genAttrs [ "src/third_party/adhd" "src/aosp/external/minijail" ] getSrc // { "src/platform/crosvm" = /home/.../crosvm; };
and blanked out the hash as it requested, but then:
error: failed to sync Caused by: failed to load pkg lockfile Caused by: failed to resolve patches for `https://github.com/rust-lang/crates.io-index%60 Caused by: failed to load source for dependency `libvda` Caused by: Unable to update /build/src/platform2/arc/vm/libvda/rust Caused by: failed to read `/build/src/platform2/arc/vm/libvda/rust/Cargo.toml`
Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f (added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312 just before that gets further, but:
error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be updated but --frozen was passed to prevent this
How do you build it?
(sorry for these basic Nix/Rust questions)
However, I could get 9p to work by running the previous version with --seccomp-log-failures. With that, I can read and write files from the console, but I can't chown things and so can't write from the terminal window, which is running as a user. I guess it needs uidmap set, but I'm not sure how to make that work.
Yeah, crosvm isn't a very nice program to build or package. :( I tried to get the libvda stuff working some time in the past, but it was very complicated. I think you might be able to disable it with
cargoBuildFlags = [ "--no-default-features" ];
but my knowledge here is a few months out of date. I can have a look in more detail once I get back from my break. :)
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to do with Nix). You can make a TAP device yourself iproute2 and use --tap-fd to tell crosvm to use it, or you can use the mktuntap program I wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \ sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel"
I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31 ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
This is because if you specify --host_ip, --netmask, or --mac, crosvm will try to create its own TAP device. If you omit all those arguments I think it should work.
Hope that's all clear -- please ask more questions if you have them, although if it's anything particularly in the weeds I might wait until I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I need to figure out how to make this all work myself anyway, and it's just a bonus if you've already done the work for me...
Well, my ultimate goal is to provide a distribution so that people don't need to figure this stuff out for themselves, but we are a little while away from that. ;)
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel"
I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31 ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
This is because if you specify --host_ip, --netmask, or --mac, crosvm will try to create its own TAP device. If you omit all those arguments I think it should work.
Oh, whoops, I missed your reply about having worked this out already!
On Thu, 14 Jan 2021 at 12:51, Alyssa Ross hi@alyssa.is wrote: [...]
Oh, whoops, I missed your reply about having worked this out already!
Yeah, disk and networking is OK now.
I also managed to fix the fonts, by using `export FONTCONFIG_FILE /etc/fonts/fonts.conf`. By default, it didn't have a monospace font available, which was pretty hard to read in the terminal.
I want to get wayland forwarding working next. For now, I'm using `ssh -Y` to my VM to forward X. It works, but it's a little slow.
I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
[...] [446067.157] -> wl_region@21.destroy() [446067.481] -> wl_surface@16.set_input_region(wl_region@22) [446068.036] -> wl_region@22.destroy() [446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0) [446069.190] -> wl_surface@16.damage(0, 0, 806, 539) [446070.141] -> wl_surface@16.commit() [446070.531] wl_keyboard@20.keymap(1, fd 8, 48869) [ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp 00007ffe128592c8 error 4 in libwayland-client.so.0.3.0[7fa537604000+6000] [ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00 00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3 66 66
Looking at the crash, I think it's actually the `set_input_region` getting `null`, but not sure why yet. I'm currently reading the Nix manual to get a better understanding of how this is all built, and the wayland docs to find out how that's supposed to work...
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when the clipboard is empty, due to trying to dereference NULL. I've patched it to fix that, and now I can run it directly under sommelier, without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of common-mk for building. Also, they removed the demos and got rid of some bogus dependencies. That simplified the build a lot! - They switched to the stable XDG protocols, but then reverted it again. I unreverted it to get things going again. Not sure if I did it right (they migrated from C to C++ so the patch didn't apply directly). - I added xwayland to the VM and sommelier command, allowing X applications to run in the VM. - By default sommelier runs the program with an already-open socket, which doesn't work if the program (or its children) want to open multiple connections. I was able to fix that by using `--parent` mode, and getting rid of PEER_CMD_PREFIX (which just adds some chromium paths preventing it from working). - Note: in `--parent` mode it waits for the process to exit before processing events on the socket, so if you just run an application directly it will hang. I used `bash -c 'firefox &'` as the command as a work-around. - Some programs (e.g. firefox) refused to start because the protocol versions offered by sommelier were too old. I increased the version numbers and that's working now. It needs doing properly, though. e.g. I implemented the new "sl_host_surface_damage_buffer" by simply calling the old damage function, which is obviously not correct but is working for me so far! - Annoyingly, using `--parent` disables xwayland support. Maybe we should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will need updating constantly to proxy every new wayland protocol. Yet it can't add any useful security because it runs inside the VM, and is therefore untrusted.
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which allows using all the normal features of Linux (e.g. ext4) in the VM. - I switched from `bash` to `bashInteractive` as the VM shell, which gets cursor keys working. - I wrote a Nix package to generate one script for each of my old qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm shopping instance, with its own /home LVM partition and IP address. It passes the network configuration using some new kernel parameters (alongside spectrumcmd). - I put each VM on its own point-to-point virtual network. These networks are set up by /etc/nixos/configuration.nix. That works well for my qubes-like VMs, though I guess spectrum will need something more dynamic. - I enabled the shared filesystem (VIRTIO_FS), which works nicely. I use it to provide a (separate) shared directory to each VM that I can access from the host. One problem is that the crosvm driver runs in a minijail with a uidmap that makes every file appear to be owned by root, so only root can write things in the VM. Possibly a newer kernel would help; later versions of the kernel docs say you can include any normal FUSE flags here, so mounting with `uid=1000` might work. - Finally, I added a `vm-halt` command that just calls `reboot`, as I don't want to develop the habit of typing `reboot` without thinking ;-)
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this?
On Wed, 20 Jan 2021 at 13:04, Thomas Leonard talex5@gmail.com wrote:
On Thu, 14 Jan 2021 at 12:51, Alyssa Ross hi@alyssa.is wrote: [...]
Oh, whoops, I missed your reply about having worked this out already!
Yeah, disk and networking is OK now.
I also managed to fix the fonts, by using `export FONTCONFIG_FILE /etc/fonts/fonts.conf`. By default, it didn't have a monospace font available, which was pretty hard to read in the terminal.
I want to get wayland forwarding working next. For now, I'm using `ssh -Y` to my VM to forward X. It works, but it's a little slow.
I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
[...] [446067.157] -> wl_region@21.destroy() [446067.481] -> wl_surface@16.set_input_region(wl_region@22) [446068.036] -> wl_region@22.destroy() [446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0) [446069.190] -> wl_surface@16.damage(0, 0, 806, 539) [446070.141] -> wl_surface@16.commit() [446070.531] wl_keyboard@20.keymap(1, fd 8, 48869) [ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp 00007ffe128592c8 error 4 in libwayland-client.so.0.3.0[7fa537604000+6000] [ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00 00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3 66 66
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard talex5@gmail.com wrote: [...]
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard talex5@gmail.com wrote: [...]
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
I saw this online the other day and started reading it without realising it was you, and then I saw you were using Nix and thought "wow, that's close to what I'm (not) doing", and then I saw the Spectrum section, and then realised who the author was. :)
I'll quote a little from it and reply to bits:
When I wanted a newer package (socat with vsock support, only just released) I just told Nix to install it from the latest Git checkout of nixpkgs.
I'm excited to learn that socat has vsock support now! That's going to be very useful. I have a half-done patch somewhere that adds vsock support to strace that I should finish up as well.
True, my squashfs image is getting a bit big. Maybe I should instead make a minimal squashfs boot image, plus a shared directory of hard links to the required files. That would allow sharing the data with the host. I could also just share the whole /nix/store directory, if I wanted to make all host software available to guests.
I think the solution I will end up going with for this will be a custom virtiofsd implementation that can implement some access controls. The even simpler solution would be to seperately expose every store path we want to share as a virtio-fs device, but that's a lot of virtio devices! (I vaguely remember the maximum might be as low as 16, too).
I didn’t have time to write and debug C++ code for every missing Wayland protocol, so I took a short-cut: I wrote my own Wayland library, ocaml-wayland, and then used that to write my own version of sommelier. With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this.
- One problem with virtwl is that, while we can receive shared memory FDs from the host, we can’t export guest memory to the host. This is unfortunate, because in Wayland the shared memory for window contents is allocated by the application from guest memory, and the proxy therefore has to copy each frame. If the host provided the memory to the guest, this wouldn’t be needed. There is a wl_drm protocol for allocating video memory, which might help here, but I don’t know how that works and, like many Wayland specifications, it seems to be in the process of being replaced by something else.
Yeah, this comes up on the virtio mailing list from time to time. It's a very difficult problem to solve, but there might be a solution some day. I think I've written about my own explorations in this area on this list before.
I’m not sure how guest-to-guest communication works with KVM.
It... doesn't really, at least not the way it does with Xen. virtio-vhost-user[1] is promising, but very early stages. I've talked in quite a lot of detail about how that works on this list before as well. guest-to-guest communication was my main area of work for most of the second half of last year (and what ended up causing me to burn out).
[1]: https://wiki.qemu.org/Features/VirtioVhostUser
I hope the SpectrumOS project will resume at some point
Me too! Maybe it's resuming right now! (Although I'm not committing -- just because I'm feeling ready to get back into it today doesn't mean that's going to be sustainable again yet.)
On Tue, 9 Mar 2021 at 16:59, Alyssa Ross hi@alyssa.is wrote:
On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard talex5@gmail.com wrote: [...]
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
I saw this online the other day and started reading it without realising it was you, and then I saw you were using Nix and thought "wow, that's close to what I'm (not) doing", and then I saw the Spectrum section, and then realised who the author was. :)
:-)
I'll quote a little from it and reply to bits:
When I wanted a newer package (socat with vsock support, only just released) I just told Nix to install it from the latest Git checkout of nixpkgs.
I'm excited to learn that socat has vsock support now! That's going to be very useful. I have a half-done patch somewhere that adds vsock support to strace that I should finish up as well.
Yeah, I'm using it as a hacky replacement for qrexec for now. The fact that it connects to the network system, and allows you to specify the target VM ID, makes it look like it's designed to go between VMs, but it doesn't seem like it does. I worry that they'll enable that at some point and create a sudden security problem...
True, my squashfs image is getting a bit big. Maybe I should instead make a minimal squashfs boot image, plus a shared directory of hard links to the required files. That would allow sharing the data with the host. I could also just share the whole /nix/store directory, if I wanted to make all host software available to guests.
I think the solution I will end up going with for this will be a custom virtiofsd implementation that can implement some access controls.
Sounds sensible.
I didn’t have time to write and debug C++ code for every missing Wayland protocol, so I took a short-cut: I wrote my own Wayland library, ocaml-wayland, and then used that to write my own version of sommelier. With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this.
I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
There's a default.nix file, so it should build easily enough (make sure to git clone with submodules). I'd be interested to know if it works for other people. I've been using it for about a week now, and it seems fine with firefox, evince and xfce4-terminal (the apps I use).
But e.g. kitty won't run because there's no `wl_drm` support. I don't know anything about graphics acceleration. But someone on Hacker News commented that you did panfrost, so I guess you know about that sort of thing.
- One problem with virtwl is that, while we can receive shared memory FDs from the host, we can’t export guest memory to the host. This is unfortunate, because in Wayland the shared memory for window contents is allocated by the application from guest memory, and the proxy therefore has to copy each frame. If the host provided the memory to the guest, this wouldn’t be needed. There is a wl_drm protocol for allocating video memory, which might help here, but I don’t know how that works and, like many Wayland specifications, it seems to be in the process of being replaced by something else.
Yeah, this comes up on the virtio mailing list from time to time. It's a very difficult problem to solve, but there might be a solution some day. I think I've written about my own explorations in this area on this list before.
I’m not sure how guest-to-guest communication works with KVM.
It... doesn't really, at least not the way it does with Xen. virtio-vhost-user[1] is promising, but very early stages. I've talked in quite a lot of detail about how that works on this list before as well. guest-to-guest communication was my main area of work for most of the second half of last year (and what ended up causing me to burn out).
I guess once you've got shared memory and inter-VM interrupts it might be possible to reuse the Xen protocols and drivers. I made a firewall VM on Qubes that did that a few years ago (https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/). But the virtio protocols will probably be more widely supported in future.
I hope the SpectrumOS project will resume at some point
Me too! Maybe it's resuming right now! (Although I'm not committing -- just because I'm feeling ready to get back into it today doesn't mean that's going to be sustainable again yet.)
:-)
On Wed, Mar 10, 2021 at 02:19:49PM +0000, Thomas Leonard wrote:
I didn’t have time to write and debug C++ code for every missing Wayland protocol, so I took a short-cut: I wrote my own Wayland library, ocaml-wayland, and then used that to write my own version of sommelier. With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this.
I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
There's a default.nix file, so it should build easily enough (make sure to git clone with submodules). I'd be interested to know if it works for other people. I've been using it for about a week now, and it seems fine with firefox, evince and xfce4-terminal (the apps I use).
But e.g. kitty won't run because there's no `wl_drm` support. I don't know anything about graphics acceleration. But someone on Hacker News commented that you did panfrost, so I guess you know about that sort of thing.
Alas, I am not Alyssa Rosenzweig of panfrost (gosh how much easier this would be if I knew as much about Linux graphics as she does!). But it is not uncommon that people get us mixed up. :)
FWIW, wl_drm solves your complaint of having to copy buffers from client-allocated memory -- with wl_drm, the client is given a dmabuf from the server. As I understand it, Chromium OS already supports this with virtio-gpu, but I haven't tried that yet.
I’m not sure how guest-to-guest communication works with KVM.
It... doesn't really, at least not the way it does with Xen. virtio-vhost-user[1] is promising, but very early stages. I've talked in quite a lot of detail about how that works on this list before as well. guest-to-guest communication was my main area of work for most of the second half of last year (and what ended up causing me to burn out).
I guess once you've got shared memory and inter-VM interrupts it might be possible to reuse the Xen protocols and drivers. I made a firewall VM on Qubes that did that a few years ago (https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/). But the virtio protocols will probably be more widely supported in future.
That's an interesting idea I hadn't considered. But I am hoping that virtio gets to the point that we don't need to do that in a reasonable amount of time.
Hi Thomas!
Thanks for keeping us updated. It's really great to have all this written up to read, even though I'm only getting to it a month and a half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when the clipboard is empty, due to trying to dereference NULL. I've patched it to fix that, and now I can run it directly under sommelier, without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it right (they migrated from C to C++ so the patch didn't apply directly).
This is great to know -- it sounds like maybe they're trying to make Sommelier more widely usable? Will probably be a while before I get to updating but this is very exicting.
- I added xwayland to the VM and sommelier command, allowing X
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open multiple connections. I was able to fix that by using `--parent` mode, and getting rid of PEER_CMD_PREFIX (which just adds some chromium paths preventing it from working).
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application directly it will hang. I used `bash -c 'firefox &'` as the command as a work-around.
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version numbers and that's working now. It needs doing properly, though. e.g. I implemented the new "sl_host_surface_damage_buffer" by simply calling the old damage function, which is obviously not correct but is working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will need updating constantly to proxy every new wayland protocol. Yet it can't add any useful security because it runs inside the VM, and is therefore untrusted.
Yeah...
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
Ah, yes, that would remove a lot of gotchas. I have avoided that so far because I'm hoping to eventually build custom kernels that don't need many modules, to reduce code size in each VM. But it would probably make sense to do for now.
- I switched from `bash` to `bashInteractive` as the VM shell, which
gets cursor keys working.
Good catch! I'll make that change in Spectrum as well.
- I wrote a Nix package to generate one script for each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm shopping instance, with its own /home LVM partition and IP address. It passes the network configuration using some new kernel parameters (alongside spectrumcmd).
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well for my qubes-like VMs, though I guess spectrum will need something more dynamic.
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can access from the host. One problem is that the crosvm driver runs in a minijail with a uidmap that makes every file appear to be owned by root, so only root can write things in the VM. Possibly a newer kernel would help; later versions of the kernel docs say you can include any normal FUSE flags here, so mounting with `uid=1000` might work.
I've only looked into virtio-fs a little bit -- I remember having to make a change to crosvm to make the sandboxing work. Glad to hear it's working well. I'll find out if a later kernel works when I get to updating Nixpkgs (or somebody else does -- someone on IRC was actually offering to try doing this the other day).
- Finally, I added a `vm-halt` command that just calls `reboot`, as I
don't want to develop the habit of typing `reboot` without thinking ;-)
I don't want to think about how many times I've made this mistake, lol.
If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment!
I think it might well be -- the stuff you have going on with networking and filesystems sound great, in particular -- but I'll have to have gotten a bit more back into the project again to know exactly what. Right now I'm focusing on slowly bringing myself back up to speed and remembering what state things are in.
Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very nervous. A secure window manager is very important for Wayland, and I'm not sure how much I trust any of the existing ones to get it right. But with Wayfire I'm hoping it'll at least be easy enough to implement stuff like tagged/coloured windows for the proof of concept (since the plugin API and stuff is Wayfire's niche), and I'm hoping at some point somebody comes up with a security-focused Wayland window manager we can switch to -- I'd love a Rust one, and there's work going on in that area[1].
Not sure about IPC yet, but I recently read an article about PipeWire[2], and that's been making me think a bit about audio. With PipeWire, they seem to have cared about security from the start:
To avoid the PulseAudio sandboxing limitations, security was baked-in: a per-client permissions bitfield is attached to every PipeWire node — where one or more SPA nodes are wrapped. This security-aware design allowed easy and safe integration with Flatpak portals; the sandboxed-application permissions interface now promoted to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired by Wayland — without the XML serialization part — was implemented over Unix-domain sockets. Taymans wanted a protocol that is simple and hard-realtime safe.
It goes on to say they use this for sending file descriptors and stuff. The similarity to Wayland is very exciting, because it means we might just be able to run PipeWire over the existing virtio_wl infrastructure very efficiently.
It'll be a while before I get to look at audio in depth, but this all sounds very good -- maybe most of the work will have been done for us! In general I'm feeling very optimistic about a lot of the stuff going on in the ecosystem to try to make Flatpak and co secure -- I don't trust Flatpak itself to provide meaningful security, but it means we're getting standard mechanisms for permissions for standard applications (xdg-desktop-portal is another that comes to mind), and if this goes well it means that all we have to do is provide implementations of those standard interfaces that cross VM boundaries, and applications designed to work in Flatpak etc. should already understand how to interact with them.
[1]: https://smithay.github.io/pages/about.html [2]: https://lwn.net/SubscriberLink/847412/f5595d3e8875ce5d/
On Tue, 9 Mar 2021 at 16:26, Alyssa Ross hi@alyssa.is wrote:
Hi Thomas!
Thanks for keeping us updated. It's really great to have all this written up to read, even though I'm only getting to it a month and a half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
[...]
Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very nervous. A secure window manager is very important for Wayland, and I'm not sure how much I trust any of the existing ones to get it right. But with Wayfire I'm hoping it'll at least be easy enough to implement stuff like tagged/coloured windows for the proof of concept (since the plugin API and stuff is Wayfire's niche), and I'm hoping at some point somebody comes up with a security-focused Wayland window manager we can switch to -- I'd love a Rust one, and there's work going on in that area[1].
For the short-term, it would be fairly easy to make a slight change to the wayland-virtwl-proxy[*] so that a version of it could run on the host. Unlike the guest one, which has to copy frames and deal with virtwl, this would just pass FDs through. And instead of connecting to /dev/wl0, it would just connect to the host compositor socket. It would then block access to screenshots (since it doesn't proxy that), and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going to be busy for the next 6 months at least.
Not sure about IPC yet, but I recently read an article about PipeWire[2], and that's been making me think a bit about audio. With PipeWire, they seem to have cared about security from the start:
Thanks for the link. I hadn't realised PulseAudio was in such a bad state!
To avoid the PulseAudio sandboxing limitations, security was baked-in: a per-client permissions bitfield is attached to every PipeWire node — where one or more SPA nodes are wrapped. This security-aware design allowed easy and safe integration with Flatpak portals; the sandboxed-application permissions interface now promoted to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired by Wayland — without the XML serialization part — was implemented over Unix-domain sockets. Taymans wanted a protocol that is simple and hard-realtime safe.
It goes on to say they use this for sending file descriptors and stuff. The similarity to Wayland is very exciting, because it means we might just be able to run PipeWire over the existing virtio_wl infrastructure very efficiently.
Yes. I wonder why they didn't just use Wayland directly. Removing the XML schema files (I assume that's what they mean) doesn't seem like an improvement. That's what makes it easy to use Wayland from safer languages than C!
[*] https://github.com/talex5/wayland-virtwl-proxy
On Sat, Mar 13, 2021 at 07:21:50AM +0000, Thomas Leonard wrote:
Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very nervous. A secure window manager is very important for Wayland, and I'm not sure how much I trust any of the existing ones to get it right. But with Wayfire I'm hoping it'll at least be easy enough to implement stuff like tagged/coloured windows for the proof of concept (since the plugin API and stuff is Wayfire's niche), and I'm hoping at some point somebody comes up with a security-focused Wayland window manager we can switch to -- I'd love a Rust one, and there's work going on in that area[1].
For the short-term, it would be fairly easy to make a slight change to the wayland-virtwl-proxy[*] so that a version of it could run on the host. Unlike the guest one, which has to copy frames and deal with virtwl, this would just pass FDs through. And instead of connecting to /dev/wl0, it would just connect to the host compositor socket. It would then block access to screenshots (since it doesn't proxy that), and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going to be busy for the next 6 months at least.
A sanitizing proxy of this sort could be a very good way to go indeed, especially early on, because if we tightly control what messages Wayland clients can send, we don't have to worry so much about what the compositor will do if it gets a weird message. Having a proxy rather than investigating in hardening a specific compositor (or writing such a compositor) would also give us more freedom to change compositor later (or allow users to choose their own compositor).
Not sure about IPC yet, but I recently read an article about PipeWire[2], and that's been making me think a bit about audio. With PipeWire, they seem to have cared about security from the start:
Thanks for the link. I hadn't realised PulseAudio was in such a bad state!
I think it's just one of those things where they just didn't think about security very much early on, and retrofitting that after the fact is difficult because every application has already been written to expect not to have to do anything special for security. Very much like the situation with X11 compared to Wayland.
To avoid the PulseAudio sandboxing limitations, security was baked-in: a per-client permissions bitfield is attached to every PipeWire node — where one or more SPA nodes are wrapped. This security-aware design allowed easy and safe integration with Flatpak portals; the sandboxed-application permissions interface now promoted to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired by Wayland — without the XML serialization part — was implemented over Unix-domain sockets. Taymans wanted a protocol that is simple and hard-realtime safe.
It goes on to say they use this for sending file descriptors and stuff. The similarity to Wayland is very exciting, because it means we might just be able to run PipeWire over the existing virtio_wl infrastructure very efficiently.
Yes. I wonder why they didn't just use Wayland directly. Removing the XML schema files (I assume that's what they mean) doesn't seem like an improvement. That's what makes it easy to use Wayland from safer languages than C!
Yeah I was a bit surprised to see that too. Maybe they're not expecting the protocol to grow extensions and stuff as much as Wayland, so automatic code generation is less valuable? But I'm not sure -- I haven't looked into it.
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard talex5@gmail.com wrote: [...]
For the short-term, it would be fairly easy to make a slight change to the wayland-virtwl-proxy so that a version of it could run on the host. Unlike the guest one, which has to copy frames and deal with virtwl, this would just pass FDs through. And instead of connecting to /dev/wl0, it would just connect to the host compositor socket. It would then block access to screenshots (since it doesn't proxy that), and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this, and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up spending my whole holiday getting Xwayland support added (see https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the details - it's surprisingly complicated!).
Thomas Leonard talex5@gmail.com writes:
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard talex5@gmail.com wrote: [...]
For the short-term, it would be fairly easy to make a slight change to the wayland-virtwl-proxy so that a version of it could run on the host. Unlike the guest one, which has to copy frames and deal with virtwl, this would just pass FDs through. And instead of connecting to /dev/wl0, it would just connect to the host compositor socket. It would then block access to screenshots (since it doesn't proxy that), and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this, and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up spending my whole holiday getting Xwayland support added (see https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the details - it's surprisingly complicated!).
That's awesome — thanks for keeping us updated!
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard talex5@gmail.com wrote: [...]
For the short-term, it would be fairly easy to make a slight change to the wayland-virtwl-proxy so that a version of it could run on the host. Unlike the guest one, which has to copy frames and deal with virtwl, this would just pass FDs through. And instead of connecting to /dev/wl0, it would just connect to the host compositor socket. It would then block access to screenshots (since it doesn't proxy that), and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this, and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up spending my whole holiday getting Xwayland support added (see https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the details - it's surprisingly complicated!).
That's awesome — thanks for keeping us updated!
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Ah, is that released already? I saw you mentioned it in your handy graphics write-up, but I thought it wasn't due until Linux 5.16: https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ct...
I don't know any details about it, but unless it's massively more complicated than virtwl it shouldn't be any trouble to switch to it. Will be nice not having to port virtwl to each new kernel!
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing already though? There's no need for any X11 support in the host compositor with them. -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Ah, is that released already? I saw you mentioned it in your handy graphics write-up, but I thought it wasn't due until Linux 5.16: https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ct...
You're right. They've just made it toLinus's git tree. But they should be ready for testing if you're brave! :P
I don't know any details about it, but unless it's massively more complicated than virtwl it shouldn't be any trouble to switch to it. Will be nice not having to port virtwl to each new kernel!
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing already though? There's no need for any X11 support in the host compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to implement an X11 window manager and speak back to the Xserver, like it does in a normal XWayland setup[1]?
[1]: https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Sup...
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
[...]
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing already though? There's no need for any X11 support in the host compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to implement an X11 window manager and speak back to the Xserver, like it does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and just talks plain Wayland to the host. In fact, I managed to work around several problems with Sway's built-in Xwayland integration by using the proxy instead, even on the host (see https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
Thomas Leonard talex5@gmail.com writes:
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
[...]
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing already though? There's no need for any X11 support in the host compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to implement an X11 window manager and speak back to the Xserver, like it does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and just talks plain Wayland to the host. In fact, I managed to work around several problems with Sway's built-in Xwayland integration by using the proxy instead, even on the host (see https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
Ah, I now realise you explained all of this in your blog post. Sorry for the silly questions!
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for email, etc.". _I_ don't even consider myself to be running Spectrum, because as far as I'm concerned there's no integrated system to run yet. So what's in that VM that you consider to be Spectrum? :)
On Thu, 11 Nov 2021 at 11:09, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
[...]
Also, I've been following some Qubes work recently, and it sounds like they might be interested in doing X11 over Wayland in a way that wouldn't need special compositor support (and the corresponding potential for security bugs). It sounds like it would be a lot of work though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing already though? There's no need for any X11 support in the host compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to implement an X11 window manager and speak back to the Xserver, like it does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and just talks plain Wayland to the host. In fact, I managed to work around several problems with Sway's built-in Xwayland integration by using the proxy instead, even on the host (see https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
Ah, I now realise you explained all of this in your blog post. Sorry for the silly questions!
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for email, etc.". _I_ don't even consider myself to be running Spectrum, because as far as I'm concerned there's no integrated system to run yet. So what's in that VM that you consider to be Spectrum? :)
Well, I think of it as a spectrum VM, but I guess it has changed a bit! It's based on your demo VM from early 2021, running Wayfire under Sommelier. Except I replaced Sommelier with my proxy, and I removed Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a loop and I can launch programs inside it from the host by writing to the vsock. e.g. I have a "ff-com" command to launch Firefox in the com VM. It's a bit more Qubes-like than Spectrum really. I also got it to get the applications from the main NixOS repository, as they're more up-to-date, and I replaced the kernel with a newer one because I needed better io_uring support for another project. But your basic build structure and execline boot scripts are still there!
It was a useful skeleton, but now I understand how it works, I should probably replace it with something more standard...
Thomas Leonard talex5@gmail.com writes:
On Thu, 11 Nov 2021 at 11:09, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
Correct; the proxy acts as the X11 window manager to Xwayland, and just talks plain Wayland to the host. In fact, I managed to work around several problems with Sway's built-in Xwayland integration by using the proxy instead, even on the host (see https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for email, etc.". _I_ don't even consider myself to be running Spectrum, because as far as I'm concerned there's no integrated system to run yet. So what's in that VM that you consider to be Spectrum? :)
Well, I think of it as a spectrum VM, but I guess it has changed a bit! It's based on your demo VM from early 2021, running Wayfire under Sommelier. Except I replaced Sommelier with my proxy, and I removed Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a loop and I can launch programs inside it from the host by writing to the vsock. e.g. I have a "ff-com" command to launch Firefox in the com VM. It's a bit more Qubes-like than Spectrum really. I also got it to get the applications from the main NixOS repository, as they're more up-to-date, and I replaced the kernel with a newer one because I needed better io_uring support for another project. But your basic build structure and execline boot scripts are still there!
It was a useful skeleton, but now I understand how it works, I should probably replace it with something more standard...
Wow, I'm amazed by how useful that's been to you! I'm shocked to learn anybody has ever used that for more than running the demo. I guess the only thing you're still getting from it that's non-standard is the Chromium kernel? Which itself can hopefully be dropped soon if virtio-gpu is a suitable replacement.
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Ah, is that released already? I saw you mentioned it in your handy graphics write-up, but I thought it wasn't due until Linux 5.16: https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ct...
You're right. They've just made it toLinus's git tree. But they should be ready for testing if you're brave! :P
I'm happy to test it now with Linux 5.16, but I think I need a newer crosvm too. Looks like support got added in https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f..., so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS only has 81.12871.0.0-rc1, and spectrum seems to be on 91.13904.0.0-rc2.
On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Ah, is that released already? I saw you mentioned it in your handy graphics write-up, but I thought it wasn't due until Linux 5.16: https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ct...
You're right. They've just made it toLinus's git tree. But they should be ready for testing if you're brave! :P
I'm happy to test it now with Linux 5.16, but I think I need a newer crosvm too. Looks like support got added in https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f..., so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS only has 81.12871.0.0-rc1, and spectrum seems to be on 91.13904.0.0-rc2.
I don't know of one. I haven't been keeping it up to date since we've mostly switched over to cloud-hypervisor for Spectrum, but it doesn't have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu implementation is on my list).
I'll have a go at updating it in Nixpkgs and report back.
On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross hi@alyssa.is wrote:
Thomas Leonard talex5@gmail.com writes:
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross hi@alyssa.is wrote:
Have you seen the new mechanism for Wayland over virtio-gpu context types that Google are moving towards? It's supported in mainline Linux now, and in Sommelier. I'd expect virtio-wl to go away at some point in favour of that.
Ah, is that released already? I saw you mentioned it in your handy graphics write-up, but I thought it wasn't due until Linux 5.16: https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ct...
You're right. They've just made it toLinus's git tree. But they should be ready for testing if you're brave! :P
I'm happy to test it now with Linux 5.16, but I think I need a newer crosvm too. Looks like support got added in https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f..., so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS only has 81.12871.0.0-rc1, and spectrum seems to be on 91.13904.0.0-rc2.
I don't know of one. I haven't been keeping it up to date since we've mostly switched over to cloud-hypervisor for Spectrum, but it doesn't have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu implementation is on my list).
I'll have a go at updating it in Nixpkgs and report back.
Okay, here's a package for crosvm 99.14468.0.0-rc1: https://github.com/NixOS/nixpkgs/pull/164306
I've tested some basic stuff, but haven't tested virtio-gpu. I don't see any reason it wouldn't work, though.
Good luck! Let me know if I can help with anything else.
On Tue, 15 Mar 2022 at 20:23, Alyssa Ross hi@alyssa.is wrote:
On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
[ virtio-gpu ]
I'm happy to test it now with Linux 5.16, but I think I need a newer crosvm too. Looks like support got added in https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f..., so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS only has 81.12871.0.0-rc1, and spectrum seems to be on 91.13904.0.0-rc2.
I don't know of one. I haven't been keeping it up to date since we've mostly switched over to cloud-hypervisor for Spectrum, but it doesn't have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu implementation is on my list).
I'll have a go at updating it in Nixpkgs and report back.
Okay, here's a package for crosvm 99.14468.0.0-rc1: https://github.com/NixOS/nixpkgs/pull/164306
Thanks - that's very useful!
Testing it from NixOS stable, it crashed with
[ERROR:src/linux.rs:3264] child pcivirtio-block (pid 47933) died: signo 17, status 11, code 2
In the journal, I saw:
[src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to create an executor: Uring(CreatingContext(Setup(12)))', devices/src/virtio/block/asynchronous.rs:767:50
From a brief look at the code, it looks like crosvm sees that the host is running Linux >= 5.10 and creates a test uring, which works. Then it tries to create a real one, and that fails. I fixed it by increasing the limit on locked memory in configuration.nix:
security.pam.loginLimits = [ { domain = "*"; type = "-"; item = "memlock"; value = "64000"; } ];
Then my VM booted and worked as before.
I've tested some basic stuff, but haven't tested virtio-gpu. I don't see any reason it wouldn't work, though.
I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`, crosvm crashed with:
[ 31.326763] Invalid ELF header magic: != ELF [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0 [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000 [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible [ 31.333984] [drm] features: +context_init [ 31.337289] [drm] number of scanouts: 1 [ 31.337938] [drm] number of cap sets: 1 [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo 17, status 11, code 2 [ERROR:devices/src/proxy.rs:212] failed write to child device process pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
Then I tried with `--gpu=backend=2d` and that didn't crash, but instead opened a window showing some bootloader stuff. I now have /dev/dri/{card0, renderD128} devices, so I guess the next step is figuring out what they do!
Good luck! Let me know if I can help with anything else.
If you happen to know of any documentation, that would be great. But I guess I probably just need to spend a load of time reading crosvm, Linux, and Sommelier source code.
I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`, crosvm crashed with:
[ 31.326763] Invalid ELF header magic: != ELF [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0 [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000 [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible [ 31.333984] [drm] features: +context_init [ 31.337289] [drm] number of scanouts: 1 [ 31.337938] [drm] number of cap sets: 1 [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo 17, status 11, code 2 [ERROR:devices/src/proxy.rs:212] failed write to child device process pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
Then I tried with `--gpu=backend=2d` and that didn't crash, but instead opened a window showing some bootloader stuff. I now have /dev/dri/{card0, renderD128} devices, so I guess the next step is figuring out what they do!
When I've previously attempted this, I had some crosvm thread crash with a seccomp violation because it tried to log something. If you --disable-sandbox, it might work or at least give you a clearer error message.
Good luck! Let me know if I can help with anything else.
If you happen to know of any documentation, that would be great. But I guess I probably just need to spend a load of time reading crosvm, Linux, and Sommelier source code.
There might be some helpful stuff in the Mesa repo as well, because lots of the latest graphics stuff is being done for crosvm. A quick grep found me the Venus (Vulkan over virtio-gpu) documentation, which describes testing with crosvm, and the script they use to run crosvm in CI. It also reminded me that there are two different virgl implementations in crosvm, virgl_renderer and virgl_renderer_next. I think last time I tried I had better luck with virgl_renderer. You can set buildFeatures in the Nix expression to experiment with them.
I hope some of that's helpful. I really wish I'd taken better notes the one time I got this to work.
On Wed, 16 Mar 2022 at 16:18, Thomas Leonard talex5@gmail.com wrote:
On Tue, 15 Mar 2022 at 20:23, Alyssa Ross hi@alyssa.is wrote:
On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
[ virtio-gpu ]
I'm happy to test it now with Linux 5.16, but I think I need a newer crosvm too. Looks like support got added in https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f..., so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS only has 81.12871.0.0-rc1, and spectrum seems to be on 91.13904.0.0-rc2.
I don't know of one. I haven't been keeping it up to date since we've mostly switched over to cloud-hypervisor for Spectrum, but it doesn't have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu implementation is on my list).
I'll have a go at updating it in Nixpkgs and report back.
Okay, here's a package for crosvm 99.14468.0.0-rc1: https://github.com/NixOS/nixpkgs/pull/164306
Thanks - that's very useful!
Testing it from NixOS stable, it crashed with
[...]
[src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to create an executor: Uring(CreatingContext(Setup(12)))', devices/src/virtio/block/asynchronous.rs:767:50
From a brief look at the code, it looks like crosvm sees that the host is running Linux >= 5.10 and creates a test uring, which works. Then it tries to create a real one, and that fails. I fixed it by increasing the limit on locked memory in configuration.nix
However, this led to another problem: after resuming the host from suspend, the crosvm drivers would crash with e.g.
[devices/src/virtio/block/asynchronous.rs:792] An error with a uring source: URing::enter: Failed to enter io uring: 4
So a better fix is to run with "ulimit -l 0". That causes crosvm to decide that uring isn't available and makes it fall back to the old (working) system.
I've tested some basic stuff, but haven't tested virtio-gpu. I don't see any reason it wouldn't work, though.
I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`, crosvm crashed with:
[ 31.326763] Invalid ELF header magic: != ELF [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0 [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000 [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible [ 31.333984] [drm] features: +context_init [ 31.337289] [drm] number of scanouts: 1 [ 31.337938] [drm] number of cap sets: 1 [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo 17, status 11, code 2 [ERROR:devices/src/proxy.rs:212] failed write to child device process pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
Then I tried with `--gpu=backend=2d` and that didn't crash, but instead opened a window showing some bootloader stuff. I now have /dev/dri/{card0, renderD128} devices, so I guess the next step is figuring out what they do!
Good luck! Let me know if I can help with anything else.
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
On Mon, 21 Mar 2022 at 16:05, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Thanks, that is very helpful!
I gave it a try, and it got a little further. But now, doing `modprobe virtio_gpu` in the VM crashes crosvm with:
Stack trace of thread 2: #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6) #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd) #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366) #3 0x00007fa5fcfbf870 egl_single_resolver (libepoxy.so.0 + 0xc0870) #4 0x00007fa5fcfc1d2f epoxy_eglQueryString_global_rewrite_ptr (libepoxy.so.0 + 0xc2d2f) #5 0x0000561703e72ca1 virgl_egl_init (crosvm + 0x3deca1) #6 0x0000561703e72221 vrend_winsys_init (crosvm + 0x3de221) #7 0x0000561703e380dc virgl_renderer_init (crosvm + 0x3a40dc) #8 0x0000561703e35e44 _ZN12rutabaga_gfx14virgl_renderer13VirglRenderer4init17h96573d71589e47fcE (crosvm + 0x3a1e44) #9 0x0000561703e23124 _ZN12rutabaga_gfx13rutabaga_core15RutabagaBuilder5build17h694f3c234f8787ffE (crosvm + 0x38f124) #10 0x0000561703cdfb0f _ZN7devices6virtio3gpu10virtio_gpu9VirtioGpu3new17h43dded1b3497b0f1E (crosvm + 0x24bb0f) #11 0x0000561703d0feb2 _ZN7devices6virtio3gpu5build17hc6e82daf2d41f5feE (crosvm + 0x27beb2) #12 0x0000561703cbee32 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h8f078e46fd25a72dE (crosvm + 0x22ae32) #13 0x0000561703cf5e0d _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h5605b62669a02b38E (crosvm + 0x261e0d) #14 0x0000561703ee8b43 _ZN3std3sys4unix6thread6Thread3new12thread_start17h3f45e1fefa031d31E (crosvm + 0x454b43) #15 0x00007fa5fceb6d40 start_thread (libpthread.so.0 + 0x8d40) #16 0x00007fa5fd16703f __clone (libc.so.6 + 0xfc03f)
Stack trace of thread 1: #0 0x00007fa5fd1684b2 recvfrom (libc.so.6 + 0xfd4b2) #1 0x0000561703ec7b0c _ZN8sys_util3net13UnixSeqpacket20recv_as_vec_with_fds17hcc9cb638a4fdbca9E (crosvm + 0x433b0c) #2 0x0000561703b85b87 _ZN4base4tube4Tube4recv17hd85339bce7434d11E (crosvm + 0xf1b87) #3 0x0000561703b90395 _ZN7devices5proxy10child_proc17hde9579314b1fc020E (crosvm + 0xfc395) #4 0x0000000103001400 n/a (n/a + 0x0)
It looks like it should be printing a message to stderr before calling abort, but I don't see it (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
I also tried with --gpu=backend=virglrenderer,egl=false, which crashes instead with:
Stack trace of thread 2: #0 0x0000000000000000 n/a (n/a + 0x0) #1 0x00007fac70002b00 n/a (n/a + 0x0)
I think I'll give up on virtio_gpu for now and try again after a few months.
On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
On Mon, 21 Mar 2022 at 16:05, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Thanks, that is very helpful!
I gave it a try, and it got a little further. But now, doing `modprobe virtio_gpu` in the VM crashes crosvm with:
Stack trace of thread 2: #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6) #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd) #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366) #3 0x00007fa5fcfbf870 egl_single_resolver (libepoxy.so.0 + 0xc0870) #4 0x00007fa5fcfc1d2f epoxy_eglQueryString_global_rewrite_ptr (libepoxy.so.0 + 0xc2d2f) #5 0x0000561703e72ca1 virgl_egl_init (crosvm + 0x3deca1) #6 0x0000561703e72221 vrend_winsys_init (crosvm + 0x3de221) #7 0x0000561703e380dc virgl_renderer_init (crosvm + 0x3a40dc) #8 0x0000561703e35e44 _ZN12rutabaga_gfx14virgl_renderer13VirglRenderer4init17h96573d71589e47fcE (crosvm + 0x3a1e44) #9 0x0000561703e23124 _ZN12rutabaga_gfx13rutabaga_core15RutabagaBuilder5build17h694f3c234f8787ffE (crosvm + 0x38f124) #10 0x0000561703cdfb0f _ZN7devices6virtio3gpu10virtio_gpu9VirtioGpu3new17h43dded1b3497b0f1E (crosvm + 0x24bb0f) #11 0x0000561703d0feb2 _ZN7devices6virtio3gpu5build17hc6e82daf2d41f5feE (crosvm + 0x27beb2) #12 0x0000561703cbee32 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h8f078e46fd25a72dE (crosvm + 0x22ae32) #13 0x0000561703cf5e0d _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h5605b62669a02b38E (crosvm + 0x261e0d) #14 0x0000561703ee8b43 _ZN3std3sys4unix6thread6Thread3new12thread_start17h3f45e1fefa031d31E (crosvm + 0x454b43) #15 0x00007fa5fceb6d40 start_thread (libpthread.so.0 + 0x8d40) #16 0x00007fa5fd16703f __clone (libc.so.6 + 0xfc03f)
Stack trace of thread 1: #0 0x00007fa5fd1684b2 recvfrom (libc.so.6 + 0xfd4b2) #1 0x0000561703ec7b0c _ZN8sys_util3net13UnixSeqpacket20recv_as_vec_with_fds17hcc9cb638a4fdbca9E (crosvm + 0x433b0c) #2 0x0000561703b85b87 _ZN4base4tube4Tube4recv17hd85339bce7434d11E (crosvm + 0xf1b87) #3 0x0000561703b90395 _ZN7devices5proxy10child_proc17hde9579314b1fc020E (crosvm + 0xfc395) #4 0x0000000103001400 n/a (n/a + 0x0)
It looks like it should be printing a message to stderr before calling abort, but I don't see it (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
Did you try --disable-sandbox, like I suggested in my other mail? The sandbox blocks writing error messages, and is something I frequently trip over when trying to use crosvm.
On Tue, 22 Mar 2022 at 11:16, Alyssa Ross hi@alyssa.is wrote:
On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
On Mon, 21 Mar 2022 at 16:05, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Thanks, that is very helpful!
I gave it a try, and it got a little further. But now, doing `modprobe virtio_gpu` in the VM crashes crosvm with:
Stack trace of thread 2: #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6) #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd) #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
[...]
It looks like it should be printing a message to stderr before calling abort, but I don't see it (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
Did you try --disable-sandbox, like I suggested in my other mail? The sandbox blocks writing error messages, and is something I frequently trip over when trying to use crosvm.
It's not very easy because --disable-sandbox seems to conflict with --shared-dir, which I use for lots of things. But I've now compiled a Linux kernel with "DRM_VIRTIO_GPU = yes" so I don't need the shared /nix to get the kernel module. With that, running with --disabled-sandbox seems to initialise the card successfully! I see:
[ 1.447535] [drm] Host memory window: 0x200000000 +0x200000000 [ 1.447821] [drm] features: +virgl -edid +resource_blob +host_visible [ 1.447821] [drm] features: +context_init [ 1.449292] [drm] number of scanouts: 1 [ 1.449495] [drm] number of cap sets: 4 [DEBUG:rutabaga_gfx/src/virgl_renderer.rs:113] gl_version 32 - es profile enabled [ 1.473662] [drm] cap set 0: id 1, max-version 1, max-size 308 [ 1.474027] [drm] cap set 1: id 2, max-version 2, max-size 764 [ 1.474386] [drm] cap set 2: id 4, max-version 0, max-size 0 [ 1.474702] [drm] cap set 3: id 5, max-version 0, max-size 16 [ 1.475158] [drm] Initialized virtio_gpu 0.1.0 0 for virtio6 on minor 0 [ 1.480946] virtio_gpu virtio6: [drm] drm_plane_enable_fb_damage_clips() not called [ 1.484018] Console: switching to colour frame buffer device 160x64 [ 1.487021] virtio_gpu virtio6: [drm] fb0: virtio_gpudrmfb frame buffer device
Without --disabled-sandbox, it crashes after the "number of cap sets: 4" line.
-- talex5 (GitHub/Twitter) http://roscidus.com/blog/
On Tue, 22 Mar 2022 at 20:05, Thomas Leonard talex5@gmail.com wrote:
On Tue, 22 Mar 2022 at 11:16, Alyssa Ross hi@alyssa.is wrote:
On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
On Mon, 21 Mar 2022 at 16:05, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76...
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Thanks, that is very helpful!
I gave it a try, and it got a little further. But now, doing `modprobe virtio_gpu` in the VM crashes crosvm with:
Stack trace of thread 2: #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6) #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd) #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
[...]
It looks like it should be printing a message to stderr before calling abort, but I don't see it (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
Did you try --disable-sandbox, like I suggested in my other mail? The sandbox blocks writing error messages, and is something I frequently trip over when trying to use crosvm.
It's not very easy because --disable-sandbox seems to conflict with --shared-dir, which I use for lots of things.
I got around this by changing `create_gpu_device` to use `let jail = None;`, so only the GPU device isn't jailed. I suspect the minijail config needs updating for NixOS (e.g. https://github.com/google/crosvm/blob/main/src/linux/gpu.rs#L82).
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
The basic idea seems to be:
1. You allocate a page of memory shared with the crosvm on the host. 2. You tell crosvm to read messages from the host compositor and write them to this page. 3. After doing this, crosvm signals the guest, which reads the data.
The shared page is referred to as a "ring", but it's not used as a ring buffer. The host always writes to the start of it.
Separately, to allocate an image buffer:
1. You tell crosvm the width and height, etc. 2. It assigns a blob_id and writes it to the shared page. 3. You wait for the operation to complete, then use the blob_id to create the buffer.
The problem is that both these operations write to the same page, and they race! So sometimes the image information overwrites the Wayland data, or the Wayland data overwrites the image information, and then it crashes.
This image_query function shows the problem: https://chromium.googlesource.com/chromiumos/platform2/+/refs/heads/main/vm_...
It asks for the image information to be written to "ring_addr_" and then reads it from there. But at the moment when the function is called, ring_addr_ may contain Wayland protocol data that hasn't been read yet. I didn't test it with Sommelier, but that's the problem I had in my code and I don't see how Sommelier's code can work in general. Sommelier caches the results, so it might not hit this case too often. I didn't use a cache, and also added a small sleep to my code to make the problem easier to reproduce.
Anyone have any ideas how this is supposed to work?
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
I don't think it's possible to avoid races completely, but it seems to be working reasonably well so far.
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
I don't think it's possible to avoid races completely, but it seems to be working reasonably well so far.
I wonder if it would be worth asking about the remaining problems on the relevant kernel mailing lists, since they sound like protocol issues rather than anything specific to your implementation.
I tracked down the series where they were added[1], perhaps the From/To/Cc in that posting would be a good starting point?
If you do reach out, please CC me (and this list, if you want)!
[1]: https://lore.kernel.org/dri-devel/20210921232024.817-1-gurchetansingh@chromi...
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross hi@alyssa.is wrote:
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
I don't think it's possible to avoid races completely, but it seems to be working reasonably well so far.
I wonder if it would be worth asking about the remaining problems on the relevant kernel mailing lists, since they sound like protocol issues rather than anything specific to your implementation.
The kernel isn't involved in any of the problems - it just relays opaque messages between the host and the guest. So probably crosvm/sommelier will sort things out by themselves after a while (and I'll need to update the proxy when that happens).
I tracked down the series where they were added[1], perhaps the From/To/Cc in that posting would be a good starting point?
If you do reach out, please CC me (and this list, if you want)!
I don't think it's possible to avoid races completely, but it seems to be working reasonably well so far.
I wonder if it would be worth asking about the remaining problems on the relevant kernel mailing lists, since they sound like protocol issues rather than anything specific to your implementation.
The kernel isn't involved in any of the problems - it just relays opaque messages between the host and the guest. So probably crosvm/sommelier will sort things out by themselves after a while (and I'll need to update the proxy when that happens).
Ah, I see. I wonder if there are any other implementations yet…
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross hi@alyssa.is wrote:
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
A small update on this:
First, I got virtio-gpu working with the jail. It just needs a couple of extra paths for NixOS:
diff --git a/src/linux.rs b/src/linux.rs index ad031749..52d3142f 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) -> Result<Option<Minijail>> { jail_mount_bind_if_exists( &mut jail, &[ + "/run/opengl-driver", + "/nix/store", "/usr/lib", "/usr/lib64", "/lib",
Secondly, I realised that the "video" memory returned by virtio-gpu was just regular host memory (from stracing crosvm). This is because crosvm is compiled without minigbm support and falls back to this.
After enabling minigbm and its amdgpu support (which also required adding a dependency on mesa) it started allocating vram. However, this broke the proxy because vram can't be used with the Wl_shm protocol.
I updated the proxy's test.ml code (which opens a window with a scrolling pattern) to use Linux_dmabuf_unstable_v1, and that almost worked. But some of the pixels are wrong and it often crashes the whole VM. Possibly some of the output is getting stuck in a cache somewhere and is not actually flushed to the graphics card in time? Unfortunately, I don't know anything about GPUs.
Anyway, I merged the changes I had, in case anyone else wants to try to get it working:
https://github.com/talex5/wayland-proxy-virtwl/pull/33
If it did work, it should make things a bit more efficient by avoiding one copy. At the moment, I think the process is:
1. Application renders to guest memory. 2. Proxy copies to host memory. 3. Compositor copies to graphics card.
If the mingbm stuff worked, the proxy could copy directly to the graphics card. And possibly the application could render there directly in the first place, but that would need more work.
Thanks for the update!
On Sun, May 15, 2022 at 03:20:24PM +0000, Thomas Leonard wrote:
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross hi@alyssa.is wrote:
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
A small update on this:
First, I got virtio-gpu working with the jail. It just needs a couple of extra paths for NixOS:
diff --git a/src/linux.rs b/src/linux.rs index ad031749..52d3142f 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) -> Result<Option<Minijail>> { jail_mount_bind_if_exists( &mut jail, &[
"/run/opengl-driver","/nix/store", "/usr/lib", "/usr/lib64", "/lib",Secondly, I realised that the "video" memory returned by virtio-gpu was just regular host memory (from stracing crosvm). This is because crosvm is compiled without minigbm support and falls back to this.
After enabling minigbm and its amdgpu support (which also required adding a dependency on mesa) it started allocating vram. However, this broke the proxy because vram can't be used with the Wl_shm protocol.
Do you have a Nix expression somewhere for crosvm with all this stuff fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you didn't do it with Nix, I can make the changes myself.)
On Mon, 16 May 2022 at 11:55, Alyssa Ross hi@alyssa.is wrote:
Thanks for the update!
On Sun, May 15, 2022 at 03:20:24PM +0000, Thomas Leonard wrote:
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross hi@alyssa.is wrote:
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard talex5@gmail.com wrote: [ converting from virtwl to virtio-gpu ]
I tried, but failed, to figure out the protocol. I did manage to get a test application showing a little animation, but it crashes after a few seconds.
OK, I found a solution to this: you can just open the device file twice and use one instance for Wayland messages and the other for allocating images. This avoids the first race. With that, I got the proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
A small update on this:
First, I got virtio-gpu working with the jail. It just needs a couple of extra paths for NixOS:
diff --git a/src/linux.rs b/src/linux.rs index ad031749..52d3142f 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) -> Result<Option<Minijail>> { jail_mount_bind_if_exists( &mut jail, &[
"/run/opengl-driver","/nix/store", "/usr/lib", "/usr/lib64", "/lib",Secondly, I realised that the "video" memory returned by virtio-gpu was just regular host memory (from stracing crosvm). This is because crosvm is compiled without minigbm support and falls back to this.
After enabling minigbm and its amdgpu support (which also required adding a dependency on mesa) it started allocating vram. However, this broke the proxy because vram can't be used with the Wl_shm protocol.
Do you have a Nix expression somewhere for crosvm with all this stuff fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you didn't do it with Nix, I can make the changes myself.)
No, sorry. I've just been hacking on it in nix-shell. The only change I made to nixpkgs was adding mesa as a dependency. The changes to get minigbm to compile are just a hack for my system. But I've attached the patches just for reference. However, since it stops the proxy from working, these patches currently aren't an improvement from my point of view!
BTW, I did get a little further: using __builtin_ia32_clflush fixed the cache-coherency problem and the scrolling pattern now looks correct. However, it still crashes the VM if you resize the window a bit. I modified crosvm to clear the vram before handing it to the guest (it wasn't doing that before), and that doesn't crash, so the memory is mapped correctly on the host side.
On Wed, 18 May 2022 at 09:55, Thomas Leonard talex5@gmail.com wrote:
On Mon, 16 May 2022 at 11:55, Alyssa Ross hi@alyssa.is wrote:
[...]
Do you have a Nix expression somewhere for crosvm with all this stuff fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you didn't do it with Nix, I can make the changes myself.)
No, sorry. I've just been hacking on it in nix-shell.
I've now collected things together and pushed all my scripts here:
https://gitlab.com/talex5/qubes-lite
I've got quite a few patches to crosvm now (and NixOS 22.05 brought some new problems), but they're hacks and not suitable for upstreaming. They might be a useful starting point for someone else though. They're in the crosvm subdirectory:
fix-gui-jail:
Adds some Nix paths to the jail to allow virtio-gpu to start. Otherwise, the mesa libraries can't be loaded. Note that mesa libraries are loaded dynamically by glvnd using dlopen. If that returns NULL to indicate an error, glvnd just ignores the error and doesn't load any drivers, causing crosvm to segfault later on a NULL pointer. That also hit me when I was accidentally compiling crosvm with a different version of glibc.
fix-keymaps:
Sway/wlroots now sends read-only keymaps, but crosvm wants to map them read/write, which fails. This hack gets crosvm to try a read-only mmap if a read/write one fails. A better solution would be to remember that the resource is a keymap and map it read-only first.
fix-suspend:
After the host resumes from suspend, the disk driver gets EINTR from uring and stops handling disk requests. My previous trick of setting the locked memory limit to zero no longer works for some reason, so this stops uring in the source. A better fix would be to get crosvm to retry on EINTR.
no-gpu-window:
When using virtio-gpu, crosvm opens a pointless console window. This patch forces the use of the Stub backend, which hides it. Possibly this still wastes resources though; maybe disabling the display completely would be better.
share-as-user:
The shared filesystem driver runs in a jail where everything appears to be owned by root, and so this is how it appears in the Linux guest. This patch makes all files appear to be owned by user 1000, and performs all operations as user 0. This allows the regular user in the VM to access shared files. A better fix might be to stop using user namespaces here, but I'm not sure how to make that work.
slow-fences:
While there are GPU operations outstanding, crosvm wakes up 1000 times a second to check whether they're done, which is very wasteful. This patch makes it wake only once per second, which doesn't seem to have any ill effects. I'm not sure why there are outstanding requests all the time - possibly "wait for a user action" counts as an in-progress request?
On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Small update: Nixpkgs unstable's crosvm package is now built with the virgl_renderer and virgl_renderer_next features.
On Tue, 9 Aug 2022 at 12:01, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Small update: Nixpkgs unstable's crosvm package is now built with the virgl_renderer and virgl_renderer_next features.
I got this working eventually, but I had to apply a load of patches. How are you getting it to run?
My patches are here: https://gitlab.com/talex5/crosvm/-/commits/main
In particular: - It failed to start (when using virtio-gpu) because it doesn't have access to /nix/store, and - It failed to send Wayland keymaps because they need to be mapped read-only
Thomas Leonard talex5@gmail.com writes:
On Tue, 9 Aug 2022 at 12:01, Alyssa Ross hi@alyssa.is wrote:
On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds both virgl_renderer and virgl_renderer_next. I think virgl_renderer is on by default with --gpu, and virgl_renderer_next is used with the --gpu-render-server argument. Hopefully at least one of those does the right thing — let me know!
Small update: Nixpkgs unstable's crosvm package is now built with the virgl_renderer and virgl_renderer_next features.
I got this working eventually, but I had to apply a load of patches. How are you getting it to run?
My patches are here: https://gitlab.com/talex5/crosvm/-/commits/main
In particular:
- It failed to start (when using virtio-gpu) because it doesn't have
access to /nix/store, and
I haven't hit this one. I've mostly been testing with vhost-user — maybe crosvm devices don't run sandboxed when run standalone for vhost-user, since running that device is the only thing the crosvm process is doing?
- It failed to send Wayland keymaps because they need to be mapped read-only
The vhost-user-gpu frontend implementation we wrote for cloud-hypervisor[1] will map buffers as read-only if mapping them read-write fails[2]. This is only a workaround for crosvm not setting the flags correctly in the vhost-user message, though. The real fix will be in crosvm.
Also, not all Wayland compositors require keymaps to be mapped read-only. wlroots does, but Weston doesn't. I suspect Chromium doesn't either, hence the bug persisting in crosvm.
[1]: https://spectrum-os.org/lists/archives/spectrum-devel/20220930210906.1696349... [2]: https://spectrum-os.org/lists/archives/spectrum-devel/20b1f9da3af/s/?b=pkgs/...
-
Alyssa Ross -
Michael Raskin -
Michał "rysiek" Woźniak -
Thomas Leonard