5 Jan
2021
5 Jan
'21
7:27 p.m.
Hi,
I've been running Qubes for a few years now and I'd like to give
Spectrum a try, as I've been having some hardware and performance
problems with Qubes. Is there some up-to-date guide I can follow? I
found https://alyssa.is/using-virtio-wl/#demo and was able to see the
weston terminal. I also tried updating to the latest commit and was
able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and
increase the memory enough to start firefox, but I haven't managed to
get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems,
to get back to where I was with my Qubes setup, before investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
Thanks!
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
5 Jan
5 Jan
8:09 p.m.
I've been running Qubes for a few years now and
I'd like to give
Spectrum a try, as I've been having some hardware and performance
problems with Qubes. Is there some up-to-date guide I can follow? I
Well, there are currently tools as in building blocks, but not yet
something announced as a complete solution for a usease.
found https://alyssa.is/using-virtio-wl/#demo and was
able to see the
weston terminal. I also tried updating to the latest commit and was
able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and
increase the memory enough to start firefox, but I haven't managed to
get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems,
Depending on how the device setup ends up going, that might be practical
or not so practical (although hopefully a reasonable amount of driver
and window system setup inside each VM should be enough in any case)
to get back to where I was with my Qubes setup, before
investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
1) Alyssa is still working on it
2) There is a grant with some milestones still to be reached (and some
earlier ones ahve been paid) and some support from people sponsoring
Alyssa's work via GitHub
3) Alyssa does so much of the work alone, that a lot of things are just…
done without any discussion; some of the discussions are via IRC.
4) Alyssa has mentioned taking a break, and she has not yet announced
being back from the break (which is in line with the planned duration of
the break)
PS. On the other had, do not forget that SpectrumOS aims at better
efficiency in some places, meaning it is almost sure to end up less
secure than Qubes, and the first usable version will likely still be
a prototype with even lower level of security due to compromises in the
component choice.
6 Jan
6 Jan
7:04 a.m.
New subject: Alyssa's break
4) Alyssa has mentioned taking a break, and she has
not yet announced
being back from the break (which is in line with the planned duration of
the break)
Quick update on this -- I'm starting to feel ready to return and have a
plan for it, although there's some behind-the-scenes stuff I need to
sort out before I'll be back writing code. :)
9:11 a.m.
New subject: Alyssa's break
On 1/6/21 7:04 AM, Alyssa Ross wrote:
Good to hear from you, Alyssa! Happy New Year, and take good care of yourself.
The project can wait, self-care is important. :-)
--
Best,
r
4) Alyssa has
mentioned taking a break, and she has not yet announced
being back from the break (which is in line with the planned duration of
the break)
Quick update on this -- I'm starting to feel ready to return and have a
plan for it, although there's some behind-the-scenes stuff I need to
sort out before I'll be back writing code. :)
7 a.m.
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael
says, I'm currently working towards a proof of concept, so none of what
you've tried out so far is really meant for use outside of that proof of
concept.
I've been running Qubes for a few years now and
I'd like to give
Spectrum a try, as I've been having some hardware and performance
problems with Qubes. Is there some up-to-date guide I can follow? I
found https://alyssa.is/using-virtio-wl/#demo and was able to see the
weston terminal. I also tried updating to the latest commit and was
able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of
derivations, which is why you end up with lots of numbered result-*
symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you
a single result symlink pointing to that, and you won't need to go
hunting for the right one. :)
I managed to change the keyboard layout, mount a tmpfs
for home, and
increase the memory enough to start firefox, but I haven't managed to
get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
I'm surprised -- this has worked for me before, although it's been a
while since I tried this so maybe I changed something.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to
share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it
to the package since I mostly have been working with my own source
builds of crosvm.
[1]:
https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100d…
- I tried using `-d /dev/mapper/disk` to share an LVM
partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid.
- I tried enabling networking with `--host_ip
10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to
do with Nix). You can make a TAP device yourself iproute2 and use
--tap-fd to tell crosvm to use it, or you can use the mktuntap program I
wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \
sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
Ideally, I'd like to run a VM with each of my old
Qubes filesystems,
to get back to where I was with my Qubes setup, before investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready
to use like this, but I am working on it (or at least I will be again
once my anti-burnout break ends). Once I am, I hope to be more active
on the lists again. I used to post weekly status updates, and would
like to get into doing that again once I'm back because they were a
great way to keep people up to date with the project and for me to have
a record of what I'd been doing. Reading some of the old status updates
should give you a bit of a feel for where things are, although things
are a bit further along than they were when I wrote the last one because
I put the status updates on hold to try to chase a funding milestone.
Hope that's all clear -- please ask more questions if you have them,
although if it's anything particularly in the weeds I might wait until
I'm back from my break to answer. :)
3:56 p.m.
On Wed, 6 Jan 2021 at 07:01, Alyssa Ross <hi(a)alyssa.is> wrote:
That's better, thanks!
[...]
Ah, I didn't realise it was using seccomp too. I'm not sure how to
compile specific versions of crosvm. I tried with:
srcs = lib.genAttrs [
"src/third_party/adhd"
"src/aosp/external/minijail"
] getSrc // { "src/platform/crosvm" = /home/.../crosvm; };
and blanked out the hash as it requested, but then:
error: failed to sync Caused by: failed to load pkg lockfile Caused
by: failed to resolve patches for
`https://github.com/rust-lang/crates.io-index` Caused by: failed to
load source for dependency `libvda` Caused by: Unable to update
/build/src/platform2/arc/vm/libvda/rust Caused by: failed to read
`/build/src/platform2/arc/vm/libvda/rust/Cargo.toml`
Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f
(added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312
just before that gets further, but:
error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be
updated but --frozen was passed to prevent this
How do you build it?
(sorry for these basic Nix/Rust questions)
However, I could get 9p to work by running the previous version with
--seccomp-log-failures. With that, I can read and write files from the
console, but I can't chown things and so can't write from the terminal
window, which is running as a user. I guess it needs uidmap set, but
I'm not sure how to make that work.
OK, I'll keep trying stuff. I have discovered that if I add the
squashfs file as another device (--root "$rootfs" -d "$rootfs") then
it shows an error but mounts it anyway!
# ls /tmp
# mount /dev/vdb /tmp
[ 15.288873] /dev/vdb: Can't open blockdev
# ls /tmp
bin dev etc nix proc run sbin sys tmp
Sadly, this didn't work with my ext4 partition.
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \
sudo -u "$USER" -C 4 \
"$crosvm" run \
-p init=/sbin/init \
-p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
--tap-fd 3 \
--seccomp-log-failures \
--root "$rootfs" \
--host_ip 10.0.0.1 \
--netmask 255.0.0.0 \
--mac c0:ff:ee:c0:ff:ee \
-m 4096 \
"$@" \
"$kernel"
I got "sudo: you are not permitted to use the -C option", which I
fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm:
error creating devices: failed to set up virtio networking: failed to
open tap device: failed to create tap interface: Operation not
permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
Great! I'm on a break myself at the moment, which is why I have some
time to try all this out.
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael
says, I'm currently working towards a proof of concept, so none of what
you've tried out so far is really meant for use outside of that proof of
concept.
[...]
nix-build
. -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of
derivations, which is why you end up with lots of numbered result-*
symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you
a single result symlink pointing to that, and you won't need to go
hunting for the right one. :) - I tried
adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it
to the package since I mostly have been working with my own source
builds of crosvm.
[1]:
https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100d…
- I tried
using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid. - I tried
enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to
do with Nix). You can make a TAP device yourself iproute2 and use
--tap-fd to tell crosvm to use it, or you can use the mktuntap program I
wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \
sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3 Ideally,
I'd like to run a VM with each of my old Qubes filesystems,
to get back to where I was with my Qubes setup, before investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready
to use like this, but I am working on it (or at least I will be again
once my anti-burnout break ends). Once I am, I hope to be more active
on the lists again. I used to post weekly status updates, and would
like to get into doing that again once I'm back because they were a
great way to keep people up to date with the project and for me to have
a record of what I'd been doing. Reading some of the old status updates
should give you a bit of a feel for where things are, although things
are a bit further along than they were when I wrote the last one because
I put the status updates on hold to try to chase a funding milestone.
I've read some of them - they're very helpful!
Hope that's all clear -- please ask more questions
if you have them,
although if it's anything particularly in the weeds I might wait until
I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I
need to figure out how to make this all work myself anyway, and it's
just a bonus if you've already done the work for me...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
7 Jan
7 Jan
11:38 a.m.
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard <talex5(a)gmail.com> wrote:
[...]
exec sudo "$mktuntap" -pvB 3 \
sudo -u "$USER" -C 4 \
"$crosvm" run \
-p init=/sbin/init \
-p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
--tap-fd 3 \
--seccomp-log-failures \
--root "$rootfs" \
--host_ip 10.0.0.1 \
--netmask 255.0.0.0 \
--mac c0:ff:ee:c0:ff:ee \
-m 4096 \
"$@" \
"$kernel"
I got "sudo: you are not permitted to use the -C option", which I
fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm:
error creating devices: failed to set up virtio networking: failed to
open tap device: failed to create tap interface: Operation not
permitted (os error 1)
D'oh! I just realised you're not supposed to use the other network
options when using `--tap-fd`!
I was then able to browse the web from crosvm, like this:
- Add pkgs/os-specific/linux/spectrum/rootfs/etc/resolv.conf with e.g.
"nameserver 8.8.8.8".
- Configure the virtual eth0 in the VM setup script:
foreground { ifconfig eth0 10.0.0.2 up }
foreground { route add default gateway 10.0.0.1 }
- Enable NAT in configuration.nix, e.g.
networking.nat = {
enable = true;
externalInterface = "eno2";
internalIPs = [ "10.0.0.0/8" ];
};
- Start the VM.
- Run "sudo ifconfig tap0 10.0.0.1 up" on host.
- Run firefox in VM to browse the web :-)
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
3:33 p.m.
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard <talex5(a)gmail.com> wrote:
[...]
Sadly, this didn't work with my ext4 partition.
This turned out to be just because ext4 is compiled as a module, and
/lib isn't in the root filesystem.
Adding "EXT4_FS = yes" to the kernel configuration fixed it.
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
14 Jan
14 Jan
12:29 p.m.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to
share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it
to the package since I mostly have been working with my own source
builds of crosvm.
[1]:
https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100d…
Yeah, crosvm
needs to be CAP_NET_ADMIN for that (which is difficult to
do with Nix). You can make a TAP device yourself iproute2 and use
--tap-fd to tell crosvm to use it, or you can use the mktuntap program I
wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \
sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \
sudo -u "$USER" -C 4 \
"$crosvm" run \
-p init=/sbin/init \
-p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
--tap-fd 3 \
--seccomp-log-failures \
--root "$rootfs" \
--host_ip 10.0.0.1 \
--netmask 255.0.0.0 \
--mac c0:ff:ee:c0:ff:ee \
-m 4096 \
"$@" \
"$kernel"
I got "sudo: you are not permitted to use the -C option", which I
fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm:
error creating devices: failed to set up virtio networking: failed to
open tap device: failed to create tap interface: Operation not
permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old? Hope
that's all clear -- please ask more questions if you have them,
although if it's anything particularly in the weeds I might wait until
I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I
need to figure out how to make this all work myself anyway, and it's
just a bonus if you've already done the work for me...
12:51 p.m.
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \
sudo -u "$USER" -C 4 \
"$crosvm" run \
-p init=/sbin/init \
-p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
--tap-fd 3 \
--seccomp-log-failures \
--root "$rootfs" \
--host_ip 10.0.0.1 \
--netmask 255.0.0.0 \
--mac c0:ff:ee:c0:ff:ee \
-m 4096 \
"$@" \
"$kernel"
I got "sudo: you are not permitted to use the -C option", which I
fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm:
error creating devices: failed to set up virtio networking: failed to
open tap device: failed to create tap interface: Operation not
permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
This is because if you specify --host_ip, --netmask, or --mac, crosvm
will try to create its own TAP device. If you omit all those arguments
I think it should work.
20 Jan
20 Jan
1:04 p.m.
On Thu, 14 Jan 2021 at 12:51, Alyssa Ross <hi(a)alyssa.is> wrote:
[...]
Oh, whoops, I missed your reply about having worked
this out already!
Yeah, disk and networking is OK now.
I also managed to fix the fonts, by using `export FONTCONFIG_FILE
/etc/fonts/fonts.conf`. By default, it didn't have a monospace font
available, which was pretty hard to read in the terminal.
I want to get wayland forwarding working next. For now, I'm using `ssh
-Y` to my VM to forward X. It works, but it's a little slow.
I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
[...]
[446067.157] -> wl_region(a)21.destroy()
[446067.481] -> wl_surface@16.set_input_region(wl_region@22)
[446068.036] -> wl_region(a)22.destroy()
[446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0)
[446069.190] -> wl_surface(a)16.damage(0, 0, 806, 539)
[446070.141] -> wl_surface(a)16.commit()
[446070.531] wl_keyboard(a)20.keymap(1, fd 8, 48869)
[ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp
00007ffe128592c8 error 4 in
libwayland-client.so.0.3.0[7fa537604000+6000]
[ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00
00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00
00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3
66 66
Looking at the crash, I think it's actually the `set_input_region`
getting `null`, but not sure why yet. I'm currently reading the Nix
manual to get a better understanding of how this is all built, and the
wayland docs to find out how that's supposed to work...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
27 Jan
27 Jan
5:31 p.m.
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when
the clipboard is empty, due to trying to dereference NULL. I've
patched it to fix that, and now I can run it directly under sommelier,
without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of
some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it
right (they migrated from C to C++ so the patch didn't apply
directly).
- I added xwayland to the VM and sommelier command, allowing X
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open
multiple connections.
I was able to fix that by using `--parent` mode, and getting rid of
PEER_CMD_PREFIX (which just adds some chromium paths preventing it
from working).
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application
directly it will hang. I used `bash -c 'firefox &'` as the command as
a work-around.
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version
numbers and that's working now. It needs doing properly, though. e.g.
I implemented the new "sl_host_surface_damage_buffer" by simply
calling the old damage function, which is obviously not correct but is
working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will
need updating constantly to proxy every new wayland protocol. Yet it
can't add any useful security because it runs inside the VM, and is
therefore untrusted.
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
- I switched from `bash` to `bashInteractive` as the VM shell, which
gets cursor keys working.
- I wrote a Nix package to generate one script for each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm
shopping instance, with its own /home LVM partition and IP address. It
passes the network configuration using some new kernel parameters
(alongside spectrumcmd).
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well
for my qubes-like VMs, though I guess spectrum will need something
more dynamic.
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can
access from the host.
One problem is that the crosvm driver runs in a minijail with a
uidmap that makes every file appear to be owned by root, so only root
can write things in the VM.
Possibly a newer kernel would help; later versions of the kernel
docs say you can include any normal FUSE flags here, so mounting with
`uid=1000` might work.
- Finally, I added a `vm-halt` command that just calls `reboot`, as I
don't want to develop the habit of typing `reboot` without thinking
;-)
If any of this sounds useful for spectrum let me know. I can try and
tidy it up; it's all a huge mess at the moment!
Once this is working more smoothly, I guess the next issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
On Wed, 20 Jan 2021 at 13:04, Thomas Leonard <talex5(a)gmail.com> wrote:
On Thu, 14 Jan 2021 at 12:51, Alyssa Ross <hi(a)alyssa.is> wrote:
[...]
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
Oh, whoops, I missed your reply about having
worked this out already!
Yeah, disk and networking is OK now.
I also managed to fix the fonts, by using `export FONTCONFIG_FILE
/etc/fonts/fonts.conf`. By default, it didn't have a monospace font
available, which was pretty hard to read in the terminal.
I want to get wayland forwarding working next. For now, I'm using `ssh
-Y` to my VM to forward X. It works, but it's a little slow.
I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
[...]
[446067.157] -> wl_region(a)21.destroy()
[446067.481] -> wl_surface@16.set_input_region(wl_region@22)
[446068.036] -> wl_region(a)22.destroy()
[446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0)
[446069.190] -> wl_surface(a)16.damage(0, 0, 806, 539)
[446070.141] -> wl_surface(a)16.commit()
[446070.531] wl_keyboard(a)20.keymap(1, fd 8, 48869)
[ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp
00007ffe128592c8 error 4 in
libwayland-client.so.0.3.0[7fa537604000+6000]
[ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00
00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00
00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3
66 66
7 Mar
7 Mar
12:52 p.m.
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard <talex5(a)gmail.com> wrote:
[...]
If any of this sounds useful for spectrum let me know.
I can try and
tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of
time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
9 Mar
9 Mar
4:59 p.m.
New subject: Qubes-lite With KVM and Wayland
On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard
<talex5(a)gmail.com> wrote:
[...]
I saw this online the other day and started reading it without realising
it was you, and then I saw you were using Nix and thought "wow, that's
close to what I'm (not) doing", and then I saw the Spectrum section, and
then realised who the author was. :)
I'll quote a little from it and reply to bits:
If any of this sounds useful for spectrum let me
know. I can try and
tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of
time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ When I wanted a newer package (socat with vsock
support, only just
released) I just told Nix to install it from the latest Git checkout of
nixpkgs.
I'm excited to learn that socat has vsock support now! That's going to
be very useful. I have a half-done patch somewhere that adds vsock
support to strace that I should finish up as well.
True, my squashfs image is getting a bit big. Maybe I
should instead
make a minimal squashfs boot image, plus a shared directory of hard
links to the required files. That would allow sharing the data with the
host. I could also just share the whole /nix/store directory, if I
wanted to make all host software available to guests.
I think the solution I will end up going with for this will be a custom
virtiofsd implementation that can implement some access controls. The
even simpler solution would be to seperately expose every store path we
want to share as a virtio-fs device, but that's a lot of virtio devices!
(I vaguely remember the maximum might be as low as 16, too).
I didn’t have time to write and debug C++ code for
every missing
Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
ocaml-wayland, and then used that to write my own version of sommelier.
With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this.
* One problem with virtwl is that, while we can
receive shared
memory FDs from the host, we can’t export guest memory to the
host. This is unfortunate, because in Wayland the shared memory for
window contents is allocated by the application from guest memory,
and the proxy therefore has to copy each frame. If the host
provided the memory to the guest, this wouldn’t be needed. There
is a wl_drm protocol for allocating video memory, which might help
here, but I don’t know how that works and, like many Wayland
specifications, it seems to be in the process of being replaced by
something else.
Yeah, this comes up on the virtio mailing list from time to time. It's
a very difficult problem to solve, but there might be a solution some
day. I think I've written about my own explorations in this area on
this list before.
I’m not sure how guest-to-guest communication works
with KVM.
It... doesn't really, at least not the way it does with Xen.
virtio-vhost-user[1] is promising, but very early stages. I've talked
in quite a lot of detail about how that works on this list before as
well. guest-to-guest communication was my main area of work for most of
the second half of last year (and what ended up causing me to burn out).
[1]: https://wiki.qemu.org/Features/VirtioVhostUser
I hope the SpectrumOS project will resume at some
point
Me too! Maybe it's resuming right now! (Although I'm not committing --
just because I'm feeling ready to get back into it today doesn't mean
that's going to be sustainable again yet.)
10 Mar
10 Mar
2:19 p.m.
New subject: Qubes-lite With KVM and Wayland
On Tue, 9 Mar 2021 at 16:59, Alyssa Ross <hi(a)alyssa.is> wrote:
Sounds sensible.
I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
There's a default.nix file, so it should build easily enough (make
sure to git clone with submodules). I'd be interested to know if it
works for other people. I've been using it for about a week now, and
it seems fine with firefox, evince and xfce4-terminal (the apps I
use).
But e.g. kitty won't run because there's no `wl_drm` support. I don't
know anything about graphics acceleration. But someone on Hacker News
commented that you did panfrost, so I guess you know about that sort
of thing.
I guess once you've got shared memory and inter-VM interrupts it might
be possible to reuse the Xen protocols and drivers. I made a firewall
VM on Qubes that did that a few years ago
(https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/).
But the virtio protocols will probably be more widely supported in
future.
On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
:-)
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard
<talex5(a)gmail.com> wrote:
[...]
I saw this online the other day and started reading it without realising
it was you, and then I saw you were using Nix and thought "wow, that's
close to what I'm (not) doing", and then I saw the Spectrum section, and
then realised who the author was. :) If any of this sounds useful for spectrum let me
know. I can try and
tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of
time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ I'll quote a little from it and reply to bits:
Yeah, I'm using it as a hacky replacement for qrexec for now. The fact
that it connects to the network system, and allows you to specify the
target VM ID, makes it look like it's designed to go between VMs, but
it doesn't seem like it does. I worry that they'll enable that at some
point and create a sudden security problem...
When I wanted a newer package (socat with vsock
support, only just
released) I just told Nix to install it from the latest Git checkout of
nixpkgs.
I'm excited to learn that socat has vsock support now! That's going to
be very useful. I have a half-done patch somewhere that adds vsock
support to strace that I should finish up as well. True, my
squashfs image is getting a bit big. Maybe I should instead
make a minimal squashfs boot image, plus a shared directory of hard
links to the required files. That would allow sharing the data with the
host. I could also just share the whole /nix/store directory, if I
wanted to make all host software available to guests.
I think the solution I will end up going with for this will be a custom
virtiofsd implementation that can implement some access controls. I didn’t have
time to write and debug C++ code for every missing
Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
ocaml-wayland, and then used that to write my own version of sommelier.
With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this. * One problem
with virtwl is that, while we can receive shared
memory FDs from the host, we can’t export guest memory to the
host. This is unfortunate, because in Wayland the shared memory for
window contents is allocated by the application from guest memory,
and the proxy therefore has to copy each frame. If the host
provided the memory to the guest, this wouldn’t be needed. There
is a wl_drm protocol for allocating video memory, which might help
here, but I don’t know how that works and, like many Wayland
specifications, it seems to be in the process of being replaced by
something else.
Yeah, this comes up on the virtio mailing list from time to time. It's
a very difficult problem to solve, but there might be a solution some
day. I think I've written about my own explorations in this area on
this list before.
I’m not sure how guest-to-guest communication
works with KVM.
It... doesn't really, at least not the way it does with Xen.
virtio-vhost-user[1] is promising, but very early stages. I've talked
in quite a lot of detail about how that works on this list before as
well. guest-to-guest communication was my main area of work for most of
the second half of last year (and what ended up causing me to burn out). [1]: https://wiki.qemu.org/Features/VirtioVhostUser
:-)
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
I hope the SpectrumOS project will resume at some
point
Me too! Maybe it's resuming right now! (Although I'm not committing --
just because I'm feeling ready to get back into it today doesn't mean
that's going to be sustainable again yet.)
10:34 p.m.
New subject: Qubes-lite With KVM and Wayland
On Wed, Mar 10, 2021 at 02:19:49PM +0000, Thomas Leonard wrote:
I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
There's a default.nix file, so it should build easily enough (make
sure to git clone with submodules). I'd be interested to know if it
works for other people. I've been using it for about a week now, and
it seems fine with firefox, evince and xfce4-terminal (the apps I
use).
But e.g. kitty won't run because there's no `wl_drm` support. I don't
know anything about graphics acceleration. But someone on Hacker News
commented that you did panfrost, so I guess you know about that sort
of thing.
Alas, I am not Alyssa Rosenzweig of panfrost (gosh how much easier this
would be if I knew as much about Linux graphics as she does!). But it
is not uncommon that people get us mixed up. :)
FWIW, wl_drm solves your complaint of having to copy buffers from
client-allocated memory -- with wl_drm, the client is given a dmabuf
from the server. As I understand it, Chromium OS already supports this
with virtio-gpu, but I haven't tried that yet.
I guess once you've got shared memory and inter-VM interrupts it might
be possible to reuse the Xen protocols and drivers. I made a firewall
VM on Qubes that did that a few years ago
(https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/).
But the virtio protocols will probably be more widely supported in
future.
That's an interesting idea I hadn't considered. But I am hoping that
virtio gets to the point that we don't need to do that in a reasonable
amount of time.
I didn’t have time to write and debug C++ code for
every missing
Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
ocaml-wayland, and then used that to write my own version of sommelier.
With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this. I’m not sure how guest-to-guest communication works
with KVM.
It... doesn't really, at least not the way it does with Xen.
virtio-vhost-user[1] is promising, but very early stages. I've talked
in quite a lot of detail about how that works on this list before as
well. guest-to-guest communication was my main area of work for most of
the second half of last year (and what ended up causing me to burn out).
9 Mar
9 Mar
4:25 p.m.
Hi Thomas!
Thanks for keeping us updated. It's really great to have all this
written up to read, even though I'm only getting to it a month and a
half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when
the clipboard is empty, due to trying to dereference NULL. I've
patched it to fix that, and now I can run it directly under sommelier,
without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of
some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it
right (they migrated from C to C++ so the patch didn't apply
directly).
This is great to know -- it sounds like maybe they're trying to make
Sommelier more widely usable? Will probably be a while before I get to
updating but this is very exicting.
- I added xwayland to the VM and sommelier command,
allowing X
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open
multiple connections.
I was able to fix that by using `--parent` mode, and getting rid of
PEER_CMD_PREFIX (which just adds some chromium paths preventing it
from working).
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application
directly it will hang. I used `bash -c 'firefox &'` as the command as
a work-around.
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version
numbers and that's working now. It needs doing properly, though. e.g.
I implemented the new "sl_host_surface_damage_buffer" by simply
calling the old damage function, which is obviously not correct but is
working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will
need updating constantly to proxy every new wayland protocol. Yet it
can't add any useful security because it runs inside the VM, and is
therefore untrusted.
Yeah...
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
Ah, yes, that would remove a lot of gotchas. I have avoided that so far
because I'm hoping to eventually build custom kernels that don't need
many modules, to reduce code size in each VM. But it would probably
make sense to do for now.
- I switched from `bash` to `bashInteractive` as the
VM shell, which
gets cursor keys working.
Good catch! I'll make that change in Spectrum as well.
- I wrote a Nix package to generate one script for
each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm
shopping instance, with its own /home LVM partition and IP address. It
passes the network configuration using some new kernel parameters
(alongside spectrumcmd).
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well
for my qubes-like VMs, though I guess spectrum will need something
more dynamic.
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can
access from the host.
One problem is that the crosvm driver runs in a minijail with a
uidmap that makes every file appear to be owned by root, so only root
can write things in the VM.
Possibly a newer kernel would help; later versions of the kernel
docs say you can include any normal FUSE flags here, so mounting with
`uid=1000` might work.
I've only looked into virtio-fs a little bit -- I remember having to
make a change to crosvm to make the sandboxing work. Glad to hear it's
working well. I'll find out if a later kernel works when I get to
updating Nixpkgs (or somebody else does -- someone on IRC was actually
offering to try doing this the other day).
- Finally, I added a `vm-halt` command that just calls
`reboot`, as I
don't want to develop the habit of typing `reboot` without thinking
;-)
I don't want to think about how many times I've made this mistake, lol.
If any of this sounds useful for spectrum let me know.
I can try and
tidy it up; it's all a huge mess at the moment!
I think it might well be -- the stuff you have going on with networking
and filesystems sound great, in particular -- but I'll have to have
gotten a bit more back into the project again to know exactly what.
Right now I'm focusing on slowly bringing myself back up to speed and
remembering what state things are in.
Once this is working more smoothly, I guess the next
issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very
nervous. A secure window manager is very important for Wayland, and I'm
not sure how much I trust any of the existing ones to get it right. But
with Wayfire I'm hoping it'll at least be easy enough to implement stuff
like tagged/coloured windows for the proof of concept (since the
plugin API and stuff is Wayfire's niche), and I'm hoping at some point
somebody comes up with a security-focused Wayland window manager we can
switch to -- I'd love a Rust one, and there's work going on in that
area[1].
Not sure about IPC yet, but I recently read an article about PipeWire[2],
and that's been making me think a bit about audio. With PipeWire, they
seem to have cared about security from the start:
To avoid the PulseAudio sandboxing limitations,
security was
baked-in: a per-client permissions bitfield is attached to every
PipeWire node — where one or more SPA nodes are wrapped. This
security-aware design allowed easy and safe integration with Flatpak
portals; the sandboxed-application permissions interface now promoted
to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was inspired
by Wayland —
without the XML serialization part — was implemented over Unix-domain
sockets. Taymans wanted a protocol that is simple and hard-realtime
safe.
It goes on to say they use this for sending file descriptors and stuff.
The similarity to Wayland is very exciting, because it means we might
just be able to run PipeWire over the existing virtio_wl infrastructure
very efficiently.
It'll be a while before I get to look at audio in depth, but this all
sounds very good -- maybe most of the work will have been done for us!
In general I'm feeling very optimistic about a lot of the stuff going on
in the ecosystem to try to make Flatpak and co secure -- I don't trust
Flatpak itself to provide meaningful security, but it means we're
getting standard mechanisms for permissions for standard applications
(xdg-desktop-portal is another that comes to mind), and if this goes
well it means that all we have to do is provide implementations of those
standard interfaces that cross VM boundaries, and applications designed
to work in Flatpak etc. should already understand how to interact with
them.
[1]: https://smithay.github.io/pages/about.html
[2]: https://lwn.net/SubscriberLink/847412/f5595d3e8875ce5d/
13 Mar
13 Mar
7:21 a.m.
On Tue, 9 Mar 2021 at 16:26, Alyssa Ross <hi(a)alyssa.is> wrote:
For the short-term, it would be fairly easy to make a slight change to
the wayland-virtwl-proxy[*] so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
Yes. I wonder why they didn't just use Wayland directly. Removing the
XML schema files (I assume that's what they mean) doesn't seem like an
improvement. That's what makes it easy to use Wayland from safer
languages than C!
[*] https://github.com/talex5/wayland-virtwl-proxy
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
Hi Thomas!
Thanks for keeping us updated. It's really great to have all this
written up to read, even though I'm only getting to it a month and a
half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
[...]
Once this is
working more smoothly, I guess the next issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very
nervous. A secure window manager is very important for Wayland, and I'm
not sure how much I trust any of the existing ones to get it right. But
with Wayfire I'm hoping it'll at least be easy enough to implement stuff
like tagged/coloured windows for the proof of concept (since the
plugin API and stuff is Wayfire's niche), and I'm hoping at some point
somebody comes up with a security-focused Wayland window manager we can
switch to -- I'd love a Rust one, and there's work going on in that
area[1]. Not sure about IPC yet, but I recently read an article
about PipeWire[2],
and that's been making me think a bit about audio. With PipeWire, they
seem to have cared about security from the start:
Thanks for the link. I hadn't realised PulseAudio was in such a bad state!
To avoid the
PulseAudio sandboxing limitations, security was
baked-in: a per-client permissions bitfield is attached to every
PipeWire node — where one or more SPA nodes are wrapped. This
security-aware design allowed easy and safe integration with Flatpak
portals; the sandboxed-application permissions interface now promoted
to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was
inspired by Wayland —
without the XML serialization part — was implemented over Unix-domain
sockets. Taymans wanted a protocol that is simple and hard-realtime
safe.
It goes on to say they use this for sending file descriptors and stuff.
The similarity to Wayland is very exciting, because it means we might
just be able to run PipeWire over the existing virtio_wl infrastructure
very efficiently.
1:52 p.m.
On Sat, Mar 13, 2021 at 07:21:50AM +0000, Thomas Leonard wrote:
For the short-term, it would be fairly easy to make a slight change to
the wayland-virtwl-proxy[*] so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
A sanitizing proxy of this sort could be a very good way to go indeed,
especially early on, because if we tightly control what messages Wayland
clients can send, we don't have to worry so much about what the
compositor will do if it gets a weird message. Having a proxy rather
than investigating in hardening a specific compositor (or writing such a
compositor) would also give us more freedom to change compositor later
(or allow users to choose their own compositor).
I think it's just one of those things where they just didn't think about
security very much early on, and retrofitting that after the fact is
difficult because every application has already been written to expect
not to have to do anything special for security. Very much like the
situation with X11 compared to Wayland.
Yes. I wonder why they didn't just use Wayland directly. Removing the
XML schema files (I assume that's what they mean) doesn't seem like an
improvement. That's what makes it easy to use Wayland from safer
languages than C!
Yeah I was a bit surprised to see that too. Maybe they're not expecting
the protocol to grow extensions and stuff as much as Wayland, so
automatic code generation is less valuable? But I'm not sure -- I
haven't looked into it.
Once this is working more smoothly, I guess the next
issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very
nervous. A secure window manager is very important for Wayland, and I'm
not sure how much I trust any of the existing ones to get it right. But
with Wayfire I'm hoping it'll at least be easy enough to implement stuff
like tagged/coloured windows for the proof of concept (since the
plugin API and stuff is Wayfire's niche), and I'm hoping at some point
somebody comes up with a security-focused Wayland window manager we can
switch to -- I'd love a Rust one, and there's work going on in that
area[1]. Not sure about
IPC yet, but I recently read an article about PipeWire[2],
and that's been making me think a bit about audio. With PipeWire, they
seem to have cared about security from the start:
Thanks for the link. I hadn't realised PulseAudio was in such a bad state! To avoid the PulseAudio sandboxing limitations,
security was
baked-in: a per-client permissions bitfield is attached to every
PipeWire node — where one or more SPA nodes are wrapped. This
security-aware design allowed easy and safe integration with Flatpak
portals; the sandboxed-application permissions interface now promoted
to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
a native fully asynchronous protocol that was
inspired by Wayland —
without the XML serialization part — was implemented over Unix-domain
sockets. Taymans wanted a protocol that is simple and hard-realtime
safe.
It goes on to say they use this for sending file descriptors and stuff.
The similarity to Wayland is very exciting, because it means we might
just be able to run PipeWire over the existing virtio_wl infrastructure
very efficiently.
30 Oct
30 Oct
12:58 p.m.
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard <talex5(a)gmail.com> wrote:
[...]
For the short-term, it would be fairly easy to make a
slight change to
the wayland-virtwl-proxy so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this,
and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up
spending my whole holiday getting Xwayland support added (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
details - it's surprisingly complicated!).
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
3 Nov
3 Nov
11:36 a.m.
Thomas Leonard <talex5(a)gmail.com> writes:
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard
<talex5(a)gmail.com> wrote:
[...]
That's awesome — thanks for keeping us updated!
Have you seen the new mechanism for Wayland over virtio-gpu context
types that Google are moving towards? It's supported in mainline Linux
now, and in Sommelier. I'd expect virtio-wl to go away at some point in
favour of that.
Also, I've been following some Qubes work recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
For the short-term, it would be fairly easy to
make a slight change to
the wayland-virtwl-proxy so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this,
and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up
spending my whole holiday getting Xwayland support added (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
details - it's surprisingly complicated!).
6:27 p.m.
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi(a)alyssa.is> wrote:
Thomas Leonard <talex5(a)gmail.com> writes:
Ah, is that released already? I saw you mentioned it in your handy
graphics write-up, but I thought it wasn't due until Linux 5.16:
https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-C…
I don't know any details about it, but unless it's massively more
complicated than virtwl it shouldn't be any trouble to switch to it.
Will be nice not having to port virtwl to each new kernel!
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard
<talex5(a)gmail.com> wrote:
[...]
That's awesome — thanks for keeping us updated!
Have you seen the new mechanism for Wayland over virtio-gpu context
types that Google are moving towards? It's supported in mainline Linux
now, and in Sommelier. I'd expect virtio-wl to go away at some point in
favour of that. For the short-term, it would be fairly easy to
make a slight change to
the wayland-virtwl-proxy so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this,
and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up
spending my whole holiday getting Xwayland support added (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
details - it's surprisingly complicated!). Also, I've been following some Qubes work
recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
10 Nov
10 Nov
12:58 p.m.
Thomas Leonard <talex5(a)gmail.com> writes:
Oh I didn't realise that. So the host compositor doesn't have to
implement an X11 window manager and speak back to the Xserver, like it
does in a normal XWayland setup[1]?
[1]:
https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Su…
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross
<hi(a)alyssa.is> wrote:
You're right. They've just made it toLinus's git tree. But they should
be ready for testing if you're brave! :P
Have you seen the new mechanism for Wayland over virtio-gpu context
types that Google are moving towards? It's supported in mainline Linux
now, and in Sommelier. I'd expect virtio-wl to go away at some point in
favour of that.
Ah, is that released already? I saw you mentioned it in your handy
graphics write-up, but I thought it wasn't due until Linux 5.16:
https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-C…
I don't know any details about it, but unless
it's massively more
complicated than virtwl it shouldn't be any trouble to switch to it.
Will be nice not having to port virtwl to each new kernel!
Also, I've
been following some Qubes work recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
noon
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi(a)alyssa.is> wrote:
Oh I didn't realise that. So the host compositor doesn't have to
implement an X11 window manager and speak back to the Xserver, like it
does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and
just talks plain Wayland to the host. In fact, I managed to work
around several problems with Sway's built-in Xwayland integration by
using the proxy instead, even on the host (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
Thomas Leonard <talex5(a)gmail.com> writes:
> On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi(a)alyssa.is> wrote:
[...]
Also, I've been following some Qubes work
recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
11 Nov
11 Nov
11:09 a.m.
Thomas Leonard <talex5(a)gmail.com> writes:
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross
<hi(a)alyssa.is> wrote:
Oh I didn't realise that. So the host compositor doesn't have to
implement an X11 window manager and speak back to the Xserver, like it
does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and
just talks plain Wayland to the host. In fact, I managed to work
around several problems with Sway's built-in Xwayland integration by
using the proxy instead, even on the host (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
> [1]:
https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Su…
Ah, I now realise you explained all of this in your blog post. Sorry
for the silly questions!
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
email, etc.". _I_ don't even consider myself to be running Spectrum,
because as far as I'm concerned there's no integrated system to run yet.
So what's in that VM that you consider to be Spectrum? :)
Thomas Leonard <talex5(a)gmail.com> writes:
> On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi(a)alyssa.is> wrote:
[...]
Also, I've been following some Qubes work
recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
4:12 p.m.
On Thu, 11 Nov 2021 at 11:09, Alyssa Ross <hi(a)alyssa.is> wrote:
Thomas Leonard <talex5(a)gmail.com> writes:
Well, I think of it as a spectrum VM, but I guess it has changed a
bit! It's based on your demo VM from early 2021, running Wayfire under
Sommelier. Except I replaced Sommelier with my proxy, and I removed
Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a
loop and I can launch programs inside it from the host by writing to
the vsock. e.g. I have a "ff-com" command to launch Firefox in the com
VM. It's a bit more Qubes-like than Spectrum really. I also got it to
get the applications from the main NixOS repository, as they're more
up-to-date, and I replaced the kernel with a newer one because I
needed better io_uring support for another project. But your basic
build structure and execline boot scripts are still there!
It was a useful skeleton, but now I understand how it works, I should
probably replace it with something more standard...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross
<hi(a)alyssa.is> wrote:
Correct; the proxy acts as the X11 window manager to Xwayland, and
just talks plain Wayland to the host. In fact, I managed to work
around several problems with Sway's built-in Xwayland integration by
using the proxy instead, even on the host (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
> [1]:
https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Su…
Ah, I now realise you explained all of this in your blog post. Sorry
for the silly questions!
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
email, etc.". _I_ don't even consider myself to be running Spectrum,
because as far as I'm concerned there's no integrated system to run yet.
So what's in that VM that you consider to be Spectrum? :)
Thomas Leonard <talex5(a)gmail.com> writes:
> On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi(a)alyssa.is> wrote:
[...]
> Also,
I've been following some Qubes work recently, and it sounds like
> they might be interested in doing X11 over Wayland in a way that
> wouldn't need special compositor support (and the corresponding
> potential for security bugs). It sounds like it would be a lot of work
> though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to
implement an X11 window manager and speak back to the Xserver, like it
does in a normal XWayland setup[1]?
12 Nov
12 Nov
10:47 a.m.
Thomas Leonard <talex5(a)gmail.com> writes:
On Thu, 11 Nov 2021 at 11:09, Alyssa Ross
<hi(a)alyssa.is> wrote:
Wow, I'm amazed by how useful that's been to you! I'm shocked to learn
anybody has ever used that for more than running the demo. I guess the
only thing you're still getting from it that's non-standard is the
Chromium kernel? Which itself can hopefully be dropped soon if
virtio-gpu is a suitable replacement.
Thomas Leonard <talex5(a)gmail.com> writes:
Well, I think of it as a spectrum VM, but I guess it has changed a
bit! It's based on your demo VM from early 2021, running Wayfire under
Sommelier. Except I replaced Sommelier with my proxy, and I removed
Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a
loop and I can launch programs inside it from the host by writing to
the vsock. e.g. I have a "ff-com" command to launch Firefox in the com
VM. It's a bit more Qubes-like than Spectrum really. I also got it to
get the applications from the main NixOS repository, as they're more
up-to-date, and I replaced the kernel with a newer one because I
needed better io_uring support for another project. But your basic
build structure and execline boot scripts are still there!
It was a useful skeleton, but now I understand how it works, I should
probably replace it with something more standard... Correct; the proxy acts as the X11 window manager
to Xwayland, and
just talks plain Wayland to the host. In fact, I managed to work
around several problems with Sway's built-in Xwayland integration by
using the proxy instead, even on the host (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
email, etc.". _I_ don't even consider myself to be running Spectrum,
because as far as I'm concerned there's no integrated system to run yet.
So what's in that VM that you consider to be Spectrum? :)
16
days inactive
327
days old
26 comments
4 participants
participants (4)
-
Alyssa Ross -
Michael Raskin -
Michał "rysiek" Woźniak -
Thomas Leonard