* New user getting started questions
@ 2021-01-05 19:27 Thomas Leonard
2021-01-05 20:09 ` Michael Raskin
2021-01-06 7:00 ` New user getting started questions Alyssa Ross
0 siblings, 2 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-01-05 19:27 UTC (permalink / raw)
To: discuss
Hi,
I've been running Qubes for a few years now and I'd like to give
Spectrum a try, as I've been having some hardware and performance
problems with Qubes. Is there some up-to-date guide I can follow? I
found https://alyssa.is/using-virtio-wl/#demo and was able to see the
weston terminal. I also tried updating to the latest commit and was
able to get a nested wayfire window with:
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
I managed to change the keyboard layout, mount a tmpfs for home, and
increase the memory enough to start firefox, but I haven't managed to
get much further. Things I tried so far:
- I tried replacing wayfire with weston-terminal, to avoid the nested
session. But sommelier segfaults when I do that.
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
in the VM seemed to work, but `ls /tmp` crashed the VM.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
`mount -t ext4 /dev/vdb /tmp` refused to mount it.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
said it couldn't create a tap device. I guess it needs more
privileges.
Ideally, I'd like to run a VM with each of my old Qubes filesystems,
to get back to where I was with my Qubes setup, before investigating
new spectrum stuff (e.g. one app per VM). Do you have any advice on
this? I see these lists are a bit quiet - I hope someone is still
working on this because it sounds great :-)
Thanks!
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* New user getting started questions
2021-01-05 19:27 New user getting started questions Thomas Leonard
@ 2021-01-05 20:09 ` Michael Raskin
2021-01-06 7:04 ` Alyssa's break Alyssa Ross
2021-01-06 7:00 ` New user getting started questions Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Michael Raskin @ 2021-01-05 20:09 UTC (permalink / raw)
To: talex5, discuss
>I've been running Qubes for a few years now and I'd like to give
>Spectrum a try, as I've been having some hardware and performance
>problems with Qubes. Is there some up-to-date guide I can follow? I
Well, there are currently tools as in building blocks, but not yet
something announced as a complete solution for a usease.
>found https://alyssa.is/using-virtio-wl/#demo and was able to see the
>weston terminal. I also tried updating to the latest commit and was
>able to get a nested wayfire window with:
>
> nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
>
>(I'm fairly new to Nix, so not sure if this is the right way to do things)
>
>I managed to change the keyboard layout, mount a tmpfs for home, and
>increase the memory enough to start firefox, but I haven't managed to
>get much further. Things I tried so far:
>
>- I tried replacing wayfire with weston-terminal, to avoid the nested
>session. But sommelier segfaults when I do that.
>- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
>directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
>in the VM seemed to work, but `ls /tmp` crashed the VM.
>- I tried using `-d /dev/mapper/disk` to share an LVM partition, but
>`mount -t ext4 /dev/vdb /tmp` refused to mount it.
>- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
>said it couldn't create a tap device. I guess it needs more
>privileges.
>
>Ideally, I'd like to run a VM with each of my old Qubes filesystems,
Depending on how the device setup ends up going, that might be practical
or not so practical (although hopefully a reasonable amount of driver
and window system setup inside each VM should be enough in any case)
>to get back to where I was with my Qubes setup, before investigating
>new spectrum stuff (e.g. one app per VM). Do you have any advice on
>this? I see these lists are a bit quiet - I hope someone is still
>working on this because it sounds great :-)
1) Alyssa is still working on it
2) There is a grant with some milestones still to be reached (and some
earlier ones ahve been paid) and some support from people sponsoring
Alyssa's work via GitHub
3) Alyssa does so much of the work alone, that a lot of things are just…
done without any discussion; some of the discussions are via IRC.
4) Alyssa has mentioned taking a break, and she has not yet announced
being back from the break (which is in line with the planned duration of
the break)
PS. On the other had, do not forget that SpectrumOS aims at better
efficiency in some places, meaning it is almost sure to end up less
secure than Qubes, and the first usable version will likely still be
a prototype with even lower level of security due to compromises in the
component choice.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-05 19:27 New user getting started questions Thomas Leonard
2021-01-05 20:09 ` Michael Raskin
@ 2021-01-06 7:00 ` Alyssa Ross
2021-01-06 15:56 ` Thomas Leonard
1 sibling, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-01-06 7:00 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 3924 bytes --]
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael
says, I'm currently working towards a proof of concept, so none of what
you've tried out so far is really meant for use outside of that proof of
concept.
> I've been running Qubes for a few years now and I'd like to give
> Spectrum a try, as I've been having some hardware and performance
> problems with Qubes. Is there some up-to-date guide I can follow? I
> found https://alyssa.is/using-virtio-wl/#demo and was able to see the
> weston terminal. I also tried updating to the latest commit and was
> able to get a nested wayfire window with:
>
> nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
>
> (I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of
derivations, which is why you end up with lots of numbered result-*
symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you
a single result symlink pointing to that, and you won't need to go
hunting for the right one. :)
> I managed to change the keyboard layout, mount a tmpfs for home, and
> increase the memory enough to start firefox, but I haven't managed to
> get much further. Things I tried so far:
>
> - I tried replacing wayfire with weston-terminal, to avoid the nested
> session. But sommelier segfaults when I do that.
I'm surprised -- this has worked for me before, although it's been a
while since I tried this so maybe I changed something.
> - I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
> directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
> in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it
to the package since I mostly have been working with my own source
builds of crosvm.
[1]: https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100dbe4ec287ac5
> - I tried using `-d /dev/mapper/disk` to share an LVM partition, but
> `mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid.
> - I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
> said it couldn't create a tap device. I guess it needs more
> privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to
do with Nix). You can make a TAP device yourself iproute2 and use
--tap-fd to tell crosvm to use it, or you can use the mktuntap program I
wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \
sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
> Ideally, I'd like to run a VM with each of my old Qubes filesystems,
> to get back to where I was with my Qubes setup, before investigating
> new spectrum stuff (e.g. one app per VM). Do you have any advice on
> this? I see these lists are a bit quiet - I hope someone is still
> working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready
to use like this, but I am working on it (or at least I will be again
once my anti-burnout break ends). Once I am, I hope to be more active
on the lists again. I used to post weekly status updates, and would
like to get into doing that again once I'm back because they were a
great way to keep people up to date with the project and for me to have
a record of what I'd been doing. Reading some of the old status updates
should give you a bit of a feel for where things are, although things
are a bit further along than they were when I wrote the last one because
I put the status updates on hold to try to chase a funding milestone.
Hope that's all clear -- please ask more questions if you have them,
although if it's anything particularly in the weeds I might wait until
I'm back from my break to answer. :)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Alyssa's break
2021-01-05 20:09 ` Michael Raskin
@ 2021-01-06 7:04 ` Alyssa Ross
2021-01-06 9:11 ` Michał "rysiek" Woźniak
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-01-06 7:04 UTC (permalink / raw)
To: 7c6f434c, discuss; +Cc: talex5
[-- Attachment #1: Type: text/plain, Size: 348 bytes --]
> 4) Alyssa has mentioned taking a break, and she has not yet announced
> being back from the break (which is in line with the planned duration of
> the break)
Quick update on this -- I'm starting to feel ready to return and have a
plan for it, although there's some behind-the-scenes stuff I need to
sort out before I'll be back writing code. :)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Alyssa's break
2021-01-06 7:04 ` Alyssa's break Alyssa Ross
@ 2021-01-06 9:11 ` Michał "rysiek" Woźniak
0 siblings, 0 replies; 49+ messages in thread
From: Michał "rysiek" Woźniak @ 2021-01-06 9:11 UTC (permalink / raw)
To: discuss
On 1/6/21 7:04 AM, Alyssa Ross wrote:
>> 4) Alyssa has mentioned taking a break, and she has not yet announced
>> being back from the break (which is in line with the planned duration of
>> the break)
>
> Quick update on this -- I'm starting to feel ready to return and have a
> plan for it, although there's some behind-the-scenes stuff I need to
> sort out before I'll be back writing code. :)
>
Good to hear from you, Alyssa! Happy New Year, and take good care of yourself.
The project can wait, self-care is important. :-)
--
Best,
r
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-06 7:00 ` New user getting started questions Alyssa Ross
@ 2021-01-06 15:56 ` Thomas Leonard
2021-01-07 11:38 ` Thomas Leonard
` (2 more replies)
0 siblings, 3 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-01-06 15:56 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 6 Jan 2021 at 07:01, Alyssa Ross <hi@alyssa.is> wrote:
>
> Hi! Thanks for getting in touch. :)
>
> To be honest, I'm surprised you got as far as you did -- like Michael
> says, I'm currently working towards a proof of concept, so none of what
> you've tried out so far is really meant for use outside of that proof of
> concept.
[...]
> > nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
> >
> > (I'm fairly new to Nix, so not sure if this is the right way to do things)
>
> Pretty close -- spectrumPackages is an attribute set containing lots of
> derivations, which is why you end up with lots of numbered result-*
> symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you
> a single result symlink pointing to that, and you won't need to go
> hunting for the right one. :)
That's better, thanks!
[...]
> > - I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
> > directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
> > in the VM seemed to work, but `ls /tmp` crashed the VM.
>
> Yeah, this is a known issue. I have a patch[1] for it but didn't add it
> to the package since I mostly have been working with my own source
> builds of crosvm.
>
> [1]: https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100dbe4ec287ac5
Ah, I didn't realise it was using seccomp too. I'm not sure how to
compile specific versions of crosvm. I tried with:
srcs = lib.genAttrs [
"src/third_party/adhd"
"src/aosp/external/minijail"
] getSrc // { "src/platform/crosvm" = /home/.../crosvm; };
and blanked out the hash as it requested, but then:
error: failed to sync Caused by: failed to load pkg lockfile Caused
by: failed to resolve patches for
`https://github.com/rust-lang/crates.io-index` Caused by: failed to
load source for dependency `libvda` Caused by: Unable to update
/build/src/platform2/arc/vm/libvda/rust Caused by: failed to read
`/build/src/platform2/arc/vm/libvda/rust/Cargo.toml`
Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f
(added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312
just before that gets further, but:
error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be
updated but --frozen was passed to prevent this
How do you build it?
(sorry for these basic Nix/Rust questions)
However, I could get 9p to work by running the previous version with
--seccomp-log-failures. With that, I can read and write files from the
console, but I can't chown things and so can't write from the terminal
window, which is running as a user. I guess it needs uidmap set, but
I'm not sure how to make that work.
> > - I tried using `-d /dev/mapper/disk` to share an LVM partition, but
> > `mount -t ext4 /dev/vdb /tmp` refused to mount it.
>
> Never tried that, so I don't know anything about it I'm afraid.
OK, I'll keep trying stuff. I have discovered that if I add the
squashfs file as another device (--root "$rootfs" -d "$rootfs") then
it shows an error but mounts it anyway!
# ls /tmp
# mount /dev/vdb /tmp
[ 15.288873] /dev/vdb: Can't open blockdev
# ls /tmp
bin dev etc nix proc run sbin sys tmp
Sadly, this didn't work with my ext4 partition.
> > - I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it
> > said it couldn't create a tap device. I guess it needs more
> > privileges.
>
> Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to
> do with Nix). You can make a TAP device yourself iproute2 and use
> --tap-fd to tell crosvm to use it, or you can use the mktuntap program I
> wrote (with a privelege drop after running mktuntap), like this:
>
> sudo mktuntap -pvB 3 \
> sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
OK, I tried like this:
exec sudo "$mktuntap" -pvB 3 \
sudo -u "$USER" -C 4 \
"$crosvm" run \
-p init=/sbin/init \
-p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
--tap-fd 3 \
--seccomp-log-failures \
--root "$rootfs" \
--host_ip 10.0.0.1 \
--netmask 255.0.0.0 \
--mac c0:ff:ee:c0:ff:ee \
-m 4096 \
"$@" \
"$kernel"
I got "sudo: you are not permitted to use the -C option", which I
fixed by editing the sudoers file. Then it fails with:
[ERROR:src/main.rs:1351] The architecture failed to build the vm:
error creating devices: failed to set up virtio networking: failed to
open tap device: failed to create tap interface: Operation not
permitted (os error 1)
Strace shows:
openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
Maybe it's just because my crosvm is too old?
> > Ideally, I'd like to run a VM with each of my old Qubes filesystems,
> > to get back to where I was with my Qubes setup, before investigating
> > new spectrum stuff (e.g. one app per VM). Do you have any advice on
> > this? I see these lists are a bit quiet - I hope someone is still
> > working on this because it sounds great :-)
>
> Like Michael said, there's a lot I need to do before it's really ready
> to use like this, but I am working on it (or at least I will be again
> once my anti-burnout break ends).
Great! I'm on a break myself at the moment, which is why I have some
time to try all this out.
> Once I am, I hope to be more active
> on the lists again. I used to post weekly status updates, and would
> like to get into doing that again once I'm back because they were a
> great way to keep people up to date with the project and for me to have
> a record of what I'd been doing. Reading some of the old status updates
> should give you a bit of a feel for where things are, although things
> are a bit further along than they were when I wrote the last one because
> I put the status updates on hold to try to chase a funding milestone.
I've read some of them - they're very helpful!
> Hope that's all clear -- please ask more questions if you have them,
> although if it's anything particularly in the weeds I might wait until
> I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I
need to figure out how to make this all work myself anyway, and it's
just a bonus if you've already done the work for me...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-06 15:56 ` Thomas Leonard
@ 2021-01-07 11:38 ` Thomas Leonard
2021-01-07 15:33 ` Thomas Leonard
2021-01-14 12:29 ` Alyssa Ross
2 siblings, 0 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-01-07 11:38 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard <talex5@gmail.com> wrote:
[...]
> exec sudo "$mktuntap" -pvB 3 \
> sudo -u "$USER" -C 4 \
> "$crosvm" run \
> -p init=/sbin/init \
> -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
> --tap-fd 3 \
> --seccomp-log-failures \
> --root "$rootfs" \
> --host_ip 10.0.0.1 \
> --netmask 255.0.0.0 \
> --mac c0:ff:ee:c0:ff:ee \
> -m 4096 \
> "$@" \
> "$kernel"
>
> I got "sudo: you are not permitted to use the -C option", which I
> fixed by editing the sudoers file. Then it fails with:
>
> [ERROR:src/main.rs:1351] The architecture failed to build the vm:
> error creating devices: failed to set up virtio networking: failed to
> open tap device: failed to create tap interface: Operation not
> permitted (os error 1)
D'oh! I just realised you're not supposed to use the other network
options when using `--tap-fd`!
I was then able to browse the web from crosvm, like this:
- Add pkgs/os-specific/linux/spectrum/rootfs/etc/resolv.conf with e.g.
"nameserver 8.8.8.8".
- Configure the virtual eth0 in the VM setup script:
foreground { ifconfig eth0 10.0.0.2 up }
foreground { route add default gateway 10.0.0.1 }
- Enable NAT in configuration.nix, e.g.
networking.nat = {
enable = true;
externalInterface = "eno2";
internalIPs = [ "10.0.0.0/8" ];
};
- Start the VM.
- Run "sudo ifconfig tap0 10.0.0.1 up" on host.
- Run firefox in VM to browse the web :-)
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-06 15:56 ` Thomas Leonard
2021-01-07 11:38 ` Thomas Leonard
@ 2021-01-07 15:33 ` Thomas Leonard
2021-01-14 12:29 ` Alyssa Ross
2 siblings, 0 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-01-07 15:33 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Wed, 6 Jan 2021 at 15:56, Thomas Leonard <talex5@gmail.com> wrote:
[...]
> Sadly, this didn't work with my ext4 partition.
This turned out to be just because ext4 is compiled as a module, and
/lib isn't in the root filesystem.
Adding "EXT4_FS = yes" to the kernel configuration fixed it.
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-06 15:56 ` Thomas Leonard
2021-01-07 11:38 ` Thomas Leonard
2021-01-07 15:33 ` Thomas Leonard
@ 2021-01-14 12:29 ` Alyssa Ross
2021-01-14 12:51 ` Alyssa Ross
2 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-01-14 12:29 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 4354 bytes --]
>> > - I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host
>> > directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp`
>> > in the VM seemed to work, but `ls /tmp` crashed the VM.
>>
>> Yeah, this is a known issue. I have a patch[1] for it but didn't add it
>> to the package since I mostly have been working with my own source
>> builds of crosvm.
>>
>> [1]: https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100dbe4ec287ac5
>
> Ah, I didn't realise it was using seccomp too. I'm not sure how to
> compile specific versions of crosvm. I tried with:
>
> srcs = lib.genAttrs [
> "src/third_party/adhd"
> "src/aosp/external/minijail"
> ] getSrc // { "src/platform/crosvm" = /home/.../crosvm; };
>
> and blanked out the hash as it requested, but then:
>
> error: failed to sync Caused by: failed to load pkg lockfile Caused
> by: failed to resolve patches for
> `https://github.com/rust-lang/crates.io-index` Caused by: failed to
> load source for dependency `libvda` Caused by: Unable to update
> /build/src/platform2/arc/vm/libvda/rust Caused by: failed to read
> `/build/src/platform2/arc/vm/libvda/rust/Cargo.toml`
>
> Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f
> (added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312
> just before that gets further, but:
>
> error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be
> updated but --frozen was passed to prevent this
>
> How do you build it?
>
> (sorry for these basic Nix/Rust questions)
>
> However, I could get 9p to work by running the previous version with
> --seccomp-log-failures. With that, I can read and write files from the
> console, but I can't chown things and so can't write from the terminal
> window, which is running as a user. I guess it needs uidmap set, but
> I'm not sure how to make that work.
Yeah, crosvm isn't a very nice program to build or package. :( I tried
to get the libvda stuff working some time in the past, but it was very
complicated. I think you might be able to disable it with
cargoBuildFlags = [ "--no-default-features" ];
but my knowledge here is a few months out of date. I can have a look in
more detail once I get back from my break. :)
>> Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to
>> do with Nix). You can make a TAP device yourself iproute2 and use
>> --tap-fd to tell crosvm to use it, or you can use the mktuntap program I
>> wrote (with a privelege drop after running mktuntap), like this:
>>
>> sudo mktuntap -pvB 3 \
>> sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
>
> OK, I tried like this:
>
> exec sudo "$mktuntap" -pvB 3 \
> sudo -u "$USER" -C 4 \
> "$crosvm" run \
> -p init=/sbin/init \
> -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
> --tap-fd 3 \
> --seccomp-log-failures \
> --root "$rootfs" \
> --host_ip 10.0.0.1 \
> --netmask 255.0.0.0 \
> --mac c0:ff:ee:c0:ff:ee \
> -m 4096 \
> "$@" \
> "$kernel"
>
> I got "sudo: you are not permitted to use the -C option", which I
> fixed by editing the sudoers file. Then it fails with:
>
> [ERROR:src/main.rs:1351] The architecture failed to build the vm:
> error creating devices: failed to set up virtio networking: failed to
> open tap device: failed to create tap interface: Operation not
> permitted (os error 1)
>
> Strace shows:
>
> openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
> ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
>
> Maybe it's just because my crosvm is too old?
This is because if you specify --host_ip, --netmask, or --mac, crosvm
will try to create its own TAP device. If you omit all those arguments
I think it should work.
>> Hope that's all clear -- please ask more questions if you have them,
>> although if it's anything particularly in the weeds I might wait until
>> I'm back from my break to answer. :)
>
> I have many questions :-) But don't feel pressured to answer them; I
> need to figure out how to make this all work myself anyway, and it's
> just a bonus if you've already done the work for me...
Well, my ultimate goal is to provide a distribution so that people don't
need to figure this stuff out for themselves, but we are a little while
away from that. ;)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-14 12:29 ` Alyssa Ross
@ 2021-01-14 12:51 ` Alyssa Ross
2021-01-20 13:04 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-01-14 12:51 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]
>>
>> OK, I tried like this:
>>
>> exec sudo "$mktuntap" -pvB 3 \
>> sudo -u "$USER" -C 4 \
>> "$crosvm" run \
>> -p init=/sbin/init \
>> -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \
>> --tap-fd 3 \
>> --seccomp-log-failures \
>> --root "$rootfs" \
>> --host_ip 10.0.0.1 \
>> --netmask 255.0.0.0 \
>> --mac c0:ff:ee:c0:ff:ee \
>> -m 4096 \
>> "$@" \
>> "$kernel"
>>
>> I got "sudo: you are not permitted to use the -C option", which I
>> fixed by editing the sudoers file. Then it fails with:
>>
>> [ERROR:src/main.rs:1351] The architecture failed to build the vm:
>> error creating devices: failed to set up virtio networking: failed to
>> open tap device: failed to create tap interface: Operation not
>> permitted (os error 1)
>>
>> Strace shows:
>>
>> openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31
>> ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted)
>>
>> Maybe it's just because my crosvm is too old?
>
> This is because if you specify --host_ip, --netmask, or --mac, crosvm
> will try to create its own TAP device. If you omit all those arguments
> I think it should work.
Oh, whoops, I missed your reply about having worked this out already!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-14 12:51 ` Alyssa Ross
@ 2021-01-20 13:04 ` Thomas Leonard
2021-01-27 17:31 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-01-20 13:04 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Thu, 14 Jan 2021 at 12:51, Alyssa Ross <hi@alyssa.is> wrote:
[...]
> Oh, whoops, I missed your reply about having worked this out already!
Yeah, disk and networking is OK now.
I also managed to fix the fonts, by using `export FONTCONFIG_FILE
/etc/fonts/fonts.conf`. By default, it didn't have a monospace font
available, which was pretty hard to read in the terminal.
I want to get wayland forwarding working next. For now, I'm using `ssh
-Y` to my VM to forward X. It works, but it's a little slow.
I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
[...]
[446067.157] -> wl_region@21.destroy()
[446067.481] -> wl_surface@16.set_input_region(wl_region@22)
[446068.036] -> wl_region@22.destroy()
[446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0)
[446069.190] -> wl_surface@16.damage(0, 0, 806, 539)
[446070.141] -> wl_surface@16.commit()
[446070.531] wl_keyboard@20.keymap(1, fd 8, 48869)
[ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp
00007ffe128592c8 error 4 in
libwayland-client.so.0.3.0[7fa537604000+6000]
[ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00
00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00
00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3
66 66
Looking at the crash, I think it's actually the `set_input_region`
getting `null`, but not sure why yet. I'm currently reading the Nix
manual to get a better understanding of how this is all built, and the
wayland docs to find out how that's supposed to work...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-20 13:04 ` Thomas Leonard
@ 2021-01-27 17:31 ` Thomas Leonard
2021-03-07 12:52 ` Thomas Leonard
2021-03-09 16:25 ` New user getting started questions Alyssa Ross
0 siblings, 2 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-01-27 17:31 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
I've made a bit of progress this week:
It turns out that weston-terminal crashes sommelier if started when
the clipboard is empty, due to trying to dereference NULL. I've
patched it to fix that, and now I can run it directly under sommelier,
without wayfire. I made a few other changes to sommelier too:
- I switched to the latest version, which provides meson instead of
common-mk for building. Also, they removed the demos and got rid of
some bogus dependencies. That simplified the build a lot!
- They switched to the stable XDG protocols, but then reverted it
again. I unreverted it to get things going again. Not sure if I did it
right (they migrated from C to C++ so the patch didn't apply
directly).
- I added xwayland to the VM and sommelier command, allowing X
applications to run in the VM.
- By default sommelier runs the program with an already-open socket,
which doesn't work if the program (or its children) want to open
multiple connections.
I was able to fix that by using `--parent` mode, and getting rid of
PEER_CMD_PREFIX (which just adds some chromium paths preventing it
from working).
- Note: in `--parent` mode it waits for the process to exit before
processing events on the socket, so if you just run an application
directly it will hang. I used `bash -c 'firefox &'` as the command as
a work-around.
- Some programs (e.g. firefox) refused to start because the protocol
versions offered by sommelier were too old. I increased the version
numbers and that's working now. It needs doing properly, though. e.g.
I implemented the new "sl_host_surface_damage_buffer" by simply
calling the old damage function, which is obviously not correct but is
working for me so far!
- Annoyingly, using `--parent` disables xwayland support. Maybe we
should run xwayland manually, or use a second sommelier instance?
In general, sommelier seems quite buggy and annoying. I guess it will
need updating constantly to proxy every new wayland protocol. Yet it
can't add any useful security because it runs inside the VM, and is
therefore untrusted.
Some other changes that I found useful:
- I added the generated kernel modules directory to rootfs, which
allows using all the normal features of Linux (e.g. ext4) in the VM.
- I switched from `bash` to `bashInteractive` as the VM shell, which
gets cursor keys working.
- I wrote a Nix package to generate one script for each of my old
qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm
shopping instance, with its own /home LVM partition and IP address. It
passes the network configuration using some new kernel parameters
(alongside spectrumcmd).
- I put each VM on its own point-to-point virtual network. These
networks are set up by /etc/nixos/configuration.nix. That works well
for my qubes-like VMs, though I guess spectrum will need something
more dynamic.
- I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
use it to provide a (separate) shared directory to each VM that I can
access from the host.
One problem is that the crosvm driver runs in a minijail with a
uidmap that makes every file appear to be owned by root, so only root
can write things in the VM.
Possibly a newer kernel would help; later versions of the kernel
docs say you can include any normal FUSE flags here, so mounting with
`uid=1000` might work.
- Finally, I added a `vm-halt` command that just calls `reboot`, as I
don't want to develop the habit of typing `reboot` without thinking
;-)
If any of this sounds useful for spectrum let me know. I can try and
tidy it up; it's all a huge mess at the moment!
Once this is working more smoothly, I guess the next issues will be
setting up some kind of secure window manager on the host (e.g.
labelling windows with the VM they come from, not allowing
screenshots, etc). Would also be good to get sound forwarding working
somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
to control the levels for each, but I don't know how that worked). It
also needs some kind of VM manager to keep track of which VMs are
running. And some kind of IPC system like qrexec would be useful. Do
you have thoughts or plans about how to do any of this?
On Wed, 20 Jan 2021 at 13:04, Thomas Leonard <talex5@gmail.com> wrote:
>
> On Thu, 14 Jan 2021 at 12:51, Alyssa Ross <hi@alyssa.is> wrote:
> [...]
> > Oh, whoops, I missed your reply about having worked this out already!
>
> Yeah, disk and networking is OK now.
>
> I also managed to fix the fonts, by using `export FONTCONFIG_FILE
> /etc/fonts/fonts.conf`. By default, it didn't have a monospace font
> available, which was pretty hard to read in the terminal.
>
> I want to get wayland forwarding working next. For now, I'm using `ssh
> -Y` to my VM to forward X. It works, but it's a little slow.
>
> I set `export WAYLAND_DEBUG 1`, and tried weston-terminal again. That produced:
>
> [...]
> [446067.157] -> wl_region@21.destroy()
> [446067.481] -> wl_surface@16.set_input_region(wl_region@22)
> [446068.036] -> wl_region@22.destroy()
> [446068.412] -> wl_surface@16.attach(wl_buffer@24, 0, 0)
> [446069.190] -> wl_surface@16.damage(0, 0, 806, 539)
> [446070.141] -> wl_surface@16.commit()
> [446070.531] wl_keyboard@20.keymap(1, fd 8, 48869)
> [ 1.796076] sommelier[88]: segfault at 30 ip 00007fa5376062c0 sp
> 00007ffe128592c8 error 4 in
> libwayland-client.so.0.3.0[7fa537604000+6000]
> [ 1.798026] Code: ff ff ff 5d 41 5c c3 0f 1f 00 48 8d b7 d0 00 00
> 00 e9 e4 df ff ff 0f 1f 40 00 48 89 77 30 c3 66 66 2e 0f 1f 84 00 00
> 00 00 00 <48> 8b 47 30 c3 66 66 2e 0f 1f 84 00 00 00 00 00 8b 47 40 c3
> 66 66
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-27 17:31 ` Thomas Leonard
@ 2021-03-07 12:52 ` Thomas Leonard
2021-03-09 16:59 ` Qubes-lite With KVM and Wayland Alyssa Ross
2021-03-09 16:25 ` New user getting started questions Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-03-07 12:52 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 27 Jan 2021 at 17:31, Thomas Leonard <talex5@gmail.com> wrote:
[...]
> If any of this sounds useful for spectrum let me know. I can try and
> tidy it up; it's all a huge mess at the moment!
I got a bit further (fixed my sommelier problems), but have run out of
time for now :-(
I've written up where I got to here:
https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-01-27 17:31 ` Thomas Leonard
2021-03-07 12:52 ` Thomas Leonard
@ 2021-03-09 16:25 ` Alyssa Ross
2021-03-13 7:21 ` Thomas Leonard
1 sibling, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-03-09 16:25 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 8409 bytes --]
Hi Thomas!
Thanks for keeping us updated. It's really great to have all this
written up to read, even though I'm only getting to it a month and a
half later.
On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
> I've made a bit of progress this week:
>
> It turns out that weston-terminal crashes sommelier if started when
> the clipboard is empty, due to trying to dereference NULL. I've
> patched it to fix that, and now I can run it directly under sommelier,
> without wayfire. I made a few other changes to sommelier too:
>
> - I switched to the latest version, which provides meson instead of
> common-mk for building. Also, they removed the demos and got rid of
> some bogus dependencies. That simplified the build a lot!
> - They switched to the stable XDG protocols, but then reverted it
> again. I unreverted it to get things going again. Not sure if I did it
> right (they migrated from C to C++ so the patch didn't apply
> directly).
This is great to know -- it sounds like maybe they're trying to make
Sommelier more widely usable? Will probably be a while before I get to
updating but this is very exicting.
> - I added xwayland to the VM and sommelier command, allowing X
> applications to run in the VM.
> - By default sommelier runs the program with an already-open socket,
> which doesn't work if the program (or its children) want to open
> multiple connections.
> I was able to fix that by using `--parent` mode, and getting rid of
> PEER_CMD_PREFIX (which just adds some chromium paths preventing it
> from working).
> - Note: in `--parent` mode it waits for the process to exit before
> processing events on the socket, so if you just run an application
> directly it will hang. I used `bash -c 'firefox &'` as the command as
> a work-around.
> - Some programs (e.g. firefox) refused to start because the protocol
> versions offered by sommelier were too old. I increased the version
> numbers and that's working now. It needs doing properly, though. e.g.
> I implemented the new "sl_host_surface_damage_buffer" by simply
> calling the old damage function, which is obviously not correct but is
> working for me so far!
> - Annoyingly, using `--parent` disables xwayland support. Maybe we
> should run xwayland manually, or use a second sommelier instance?
>
> In general, sommelier seems quite buggy and annoying. I guess it will
> need updating constantly to proxy every new wayland protocol. Yet it
> can't add any useful security because it runs inside the VM, and is
> therefore untrusted.
Yeah...
> Some other changes that I found useful:
>
> - I added the generated kernel modules directory to rootfs, which
> allows using all the normal features of Linux (e.g. ext4) in the VM.
Ah, yes, that would remove a lot of gotchas. I have avoided that so far
because I'm hoping to eventually build custom kernels that don't need
many modules, to reduce code size in each VM. But it would probably
make sense to do for now.
> - I switched from `bash` to `bashInteractive` as the VM shell, which
> gets cursor keys working.
Good catch! I'll make that change in Spectrum as well.
> - I wrote a Nix package to generate one script for each of my old
> qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm
> shopping instance, with its own /home LVM partition and IP address. It
> passes the network configuration using some new kernel parameters
> (alongside spectrumcmd).
> - I put each VM on its own point-to-point virtual network. These
> networks are set up by /etc/nixos/configuration.nix. That works well
> for my qubes-like VMs, though I guess spectrum will need something
> more dynamic.
> - I enabled the shared filesystem (VIRTIO_FS), which works nicely. I
> use it to provide a (separate) shared directory to each VM that I can
> access from the host.
> One problem is that the crosvm driver runs in a minijail with a
> uidmap that makes every file appear to be owned by root, so only root
> can write things in the VM.
> Possibly a newer kernel would help; later versions of the kernel
> docs say you can include any normal FUSE flags here, so mounting with
> `uid=1000` might work.
I've only looked into virtio-fs a little bit -- I remember having to
make a change to crosvm to make the sandboxing work. Glad to hear it's
working well. I'll find out if a later kernel works when I get to
updating Nixpkgs (or somebody else does -- someone on IRC was actually
offering to try doing this the other day).
> - Finally, I added a `vm-halt` command that just calls `reboot`, as I
> don't want to develop the habit of typing `reboot` without thinking
> ;-)
I don't want to think about how many times I've made this mistake, lol.
> If any of this sounds useful for spectrum let me know. I can try and
> tidy it up; it's all a huge mess at the moment!
I think it might well be -- the stuff you have going on with networking
and filesystems sound great, in particular -- but I'll have to have
gotten a bit more back into the project again to know exactly what.
Right now I'm focusing on slowly bringing myself back up to speed and
remembering what state things are in.
> Once this is working more smoothly, I guess the next issues will be
> setting up some kind of secure window manager on the host (e.g.
> labelling windows with the VM they come from, not allowing
> screenshots, etc). Would also be good to get sound forwarding working
> somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
> to control the levels for each, but I don't know how that worked). It
> also needs some kind of VM manager to keep track of which VMs are
> running. And some kind of IPC system like qrexec would be useful. Do
> you have thoughts or plans about how to do any of this?
The window manager is a part of this whole thing that makes me very
nervous. A secure window manager is very important for Wayland, and I'm
not sure how much I trust any of the existing ones to get it right. But
with Wayfire I'm hoping it'll at least be easy enough to implement stuff
like tagged/coloured windows for the proof of concept (since the
plugin API and stuff is Wayfire's niche), and I'm hoping at some point
somebody comes up with a security-focused Wayland window manager we can
switch to -- I'd love a Rust one, and there's work going on in that
area[1].
Not sure about IPC yet, but I recently read an article about PipeWire[2],
and that's been making me think a bit about audio. With PipeWire, they
seem to have cared about security from the start:
> To avoid the PulseAudio sandboxing limitations, security was
> baked-in: a per-client permissions bitfield is attached to every
> PipeWire node — where one or more SPA nodes are wrapped. This
> security-aware design allowed easy and safe integration with Flatpak
> portals; the sandboxed-application permissions interface now promoted
> to a freedesktop XDG standard.
And it gets better! In particular, this sounds very promising:
> a native fully asynchronous protocol that was inspired by Wayland —
> without the XML serialization part — was implemented over Unix-domain
> sockets. Taymans wanted a protocol that is simple and hard-realtime
> safe.
It goes on to say they use this for sending file descriptors and stuff.
The similarity to Wayland is very exciting, because it means we might
just be able to run PipeWire over the existing virtio_wl infrastructure
very efficiently.
It'll be a while before I get to look at audio in depth, but this all
sounds very good -- maybe most of the work will have been done for us!
In general I'm feeling very optimistic about a lot of the stuff going on
in the ecosystem to try to make Flatpak and co secure -- I don't trust
Flatpak itself to provide meaningful security, but it means we're
getting standard mechanisms for permissions for standard applications
(xdg-desktop-portal is another that comes to mind), and if this goes
well it means that all we have to do is provide implementations of those
standard interfaces that cross VM boundaries, and applications designed
to work in Flatpak etc. should already understand how to interact with
them.
[1]: https://smithay.github.io/pages/about.html
[2]: https://lwn.net/SubscriberLink/847412/f5595d3e8875ce5d/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Qubes-lite With KVM and Wayland
2021-03-07 12:52 ` Thomas Leonard
@ 2021-03-09 16:59 ` Alyssa Ross
2021-03-10 14:19 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-03-09 16:59 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 3845 bytes --]
On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
> On Wed, 27 Jan 2021 at 17:31, Thomas Leonard <talex5@gmail.com> wrote:
> [...]
> > If any of this sounds useful for spectrum let me know. I can try and
> > tidy it up; it's all a huge mess at the moment!
>
> I got a bit further (fixed my sommelier problems), but have run out of
> time for now :-(
>
> I've written up where I got to here:
>
> https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
I saw this online the other day and started reading it without realising
it was you, and then I saw you were using Nix and thought "wow, that's
close to what I'm (not) doing", and then I saw the Spectrum section, and
then realised who the author was. :)
I'll quote a little from it and reply to bits:
> When I wanted a newer package (socat with vsock support, only just
> released) I just told Nix to install it from the latest Git checkout of
> nixpkgs.
I'm excited to learn that socat has vsock support now! That's going to
be very useful. I have a half-done patch somewhere that adds vsock
support to strace that I should finish up as well.
> True, my squashfs image is getting a bit big. Maybe I should instead
> make a minimal squashfs boot image, plus a shared directory of hard
> links to the required files. That would allow sharing the data with the
> host. I could also just share the whole /nix/store directory, if I
> wanted to make all host software available to guests.
I think the solution I will end up going with for this will be a custom
virtiofsd implementation that can implement some access controls. The
even simpler solution would be to seperately expose every store path we
want to share as a virtio-fs device, but that's a lot of virtio devices!
(I vaguely remember the maximum might be as low as 16, too).
> I didn’t have time to write and debug C++ code for every missing
> Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
> ocaml-wayland, and then used that to write my own version of sommelier.
> With that, adding support for copying text was fairly easy.
Well this is interesting! I definitely want to learn more about this.
> * One problem with virtwl is that, while we can receive shared
> memory FDs from the host, we can’t export guest memory to the
> host. This is unfortunate, because in Wayland the shared memory for
> window contents is allocated by the application from guest memory,
> and the proxy therefore has to copy each frame. If the host
> provided the memory to the guest, this wouldn’t be needed. There
> is a wl_drm protocol for allocating video memory, which might help
> here, but I don’t know how that works and, like many Wayland
> specifications, it seems to be in the process of being replaced by
> something else.
Yeah, this comes up on the virtio mailing list from time to time. It's
a very difficult problem to solve, but there might be a solution some
day. I think I've written about my own explorations in this area on
this list before.
> I’m not sure how guest-to-guest communication works with KVM.
It... doesn't really, at least not the way it does with Xen.
virtio-vhost-user[1] is promising, but very early stages. I've talked
in quite a lot of detail about how that works on this list before as
well. guest-to-guest communication was my main area of work for most of
the second half of last year (and what ended up causing me to burn out).
[1]: https://wiki.qemu.org/Features/VirtioVhostUser
> I hope the SpectrumOS project will resume at some point
Me too! Maybe it's resuming right now! (Although I'm not committing --
just because I'm feeling ready to get back into it today doesn't mean
that's going to be sustainable again yet.)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Qubes-lite With KVM and Wayland
2021-03-09 16:59 ` Qubes-lite With KVM and Wayland Alyssa Ross
@ 2021-03-10 14:19 ` Thomas Leonard
2021-03-10 22:34 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-03-10 14:19 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Tue, 9 Mar 2021 at 16:59, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Sun, Mar 07, 2021 at 12:52:36PM +0000, Thomas Leonard wrote:
> > On Wed, 27 Jan 2021 at 17:31, Thomas Leonard <talex5@gmail.com> wrote:
> > [...]
> > > If any of this sounds useful for spectrum let me know. I can try and
> > > tidy it up; it's all a huge mess at the moment!
> >
> > I got a bit further (fixed my sommelier problems), but have run out of
> > time for now :-(
> >
> > I've written up where I got to here:
> >
> > https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
>
> I saw this online the other day and started reading it without realising
> it was you, and then I saw you were using Nix and thought "wow, that's
> close to what I'm (not) doing", and then I saw the Spectrum section, and
> then realised who the author was. :)
:-)
> I'll quote a little from it and reply to bits:
>
> > When I wanted a newer package (socat with vsock support, only just
> > released) I just told Nix to install it from the latest Git checkout of
> > nixpkgs.
>
> I'm excited to learn that socat has vsock support now! That's going to
> be very useful. I have a half-done patch somewhere that adds vsock
> support to strace that I should finish up as well.
Yeah, I'm using it as a hacky replacement for qrexec for now. The fact
that it connects to the network system, and allows you to specify the
target VM ID, makes it look like it's designed to go between VMs, but
it doesn't seem like it does. I worry that they'll enable that at some
point and create a sudden security problem...
> > True, my squashfs image is getting a bit big. Maybe I should instead
> > make a minimal squashfs boot image, plus a shared directory of hard
> > links to the required files. That would allow sharing the data with the
> > host. I could also just share the whole /nix/store directory, if I
> > wanted to make all host software available to guests.
>
> I think the solution I will end up going with for this will be a custom
> virtiofsd implementation that can implement some access controls.
Sounds sensible.
> > I didn’t have time to write and debug C++ code for every missing
> > Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
> > ocaml-wayland, and then used that to write my own version of sommelier.
> > With that, adding support for copying text was fairly easy.
>
> Well this is interesting! I definitely want to learn more about this.
I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
There's a default.nix file, so it should build easily enough (make
sure to git clone with submodules). I'd be interested to know if it
works for other people. I've been using it for about a week now, and
it seems fine with firefox, evince and xfce4-terminal (the apps I
use).
But e.g. kitty won't run because there's no `wl_drm` support. I don't
know anything about graphics acceleration. But someone on Hacker News
commented that you did panfrost, so I guess you know about that sort
of thing.
> > * One problem with virtwl is that, while we can receive shared
> > memory FDs from the host, we can’t export guest memory to the
> > host. This is unfortunate, because in Wayland the shared memory for
> > window contents is allocated by the application from guest memory,
> > and the proxy therefore has to copy each frame. If the host
> > provided the memory to the guest, this wouldn’t be needed. There
> > is a wl_drm protocol for allocating video memory, which might help
> > here, but I don’t know how that works and, like many Wayland
> > specifications, it seems to be in the process of being replaced by
> > something else.
>
> Yeah, this comes up on the virtio mailing list from time to time. It's
> a very difficult problem to solve, but there might be a solution some
> day. I think I've written about my own explorations in this area on
> this list before.
>
> > I’m not sure how guest-to-guest communication works with KVM.
>
> It... doesn't really, at least not the way it does with Xen.
> virtio-vhost-user[1] is promising, but very early stages. I've talked
> in quite a lot of detail about how that works on this list before as
> well. guest-to-guest communication was my main area of work for most of
> the second half of last year (and what ended up causing me to burn out).
I guess once you've got shared memory and inter-VM interrupts it might
be possible to reuse the Xen protocols and drivers. I made a firewall
VM on Qubes that did that a few years ago
(https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/).
But the virtio protocols will probably be more widely supported in
future.
> [1]: https://wiki.qemu.org/Features/VirtioVhostUser
>
> > I hope the SpectrumOS project will resume at some point
>
> Me too! Maybe it's resuming right now! (Although I'm not committing --
> just because I'm feeling ready to get back into it today doesn't mean
> that's going to be sustainable again yet.)
:-)
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Qubes-lite With KVM and Wayland
2021-03-10 14:19 ` Thomas Leonard
@ 2021-03-10 22:34 ` Alyssa Ross
0 siblings, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2021-03-10 22:34 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]
On Wed, Mar 10, 2021 at 02:19:49PM +0000, Thomas Leonard wrote:
> > > I didn’t have time to write and debug C++ code for every missing
> > > Wayland protocol, so I took a short-cut: I wrote my own Wayland library,
> > > ocaml-wayland, and then used that to write my own version of sommelier.
> > > With that, adding support for copying text was fairly easy.
> >
> > Well this is interesting! I definitely want to learn more about this.
>
> I've put it up here: https://github.com/talex5/wayland-virtwl-proxy
>
> There's a default.nix file, so it should build easily enough (make
> sure to git clone with submodules). I'd be interested to know if it
> works for other people. I've been using it for about a week now, and
> it seems fine with firefox, evince and xfce4-terminal (the apps I
> use).
>
> But e.g. kitty won't run because there's no `wl_drm` support. I don't
> know anything about graphics acceleration. But someone on Hacker News
> commented that you did panfrost, so I guess you know about that sort
> of thing.
Alas, I am not Alyssa Rosenzweig of panfrost (gosh how much easier this
would be if I knew as much about Linux graphics as she does!). But it
is not uncommon that people get us mixed up. :)
FWIW, wl_drm solves your complaint of having to copy buffers from
client-allocated memory -- with wl_drm, the client is given a dmabuf
from the server. As I understand it, Chromium OS already supports this
with virtio-gpu, but I haven't tried that yet.
> > > I’m not sure how guest-to-guest communication works with KVM.
> >
> > It... doesn't really, at least not the way it does with Xen.
> > virtio-vhost-user[1] is promising, but very early stages. I've talked
> > in quite a lot of detail about how that works on this list before as
> > well. guest-to-guest communication was my main area of work for most of
> > the second half of last year (and what ended up causing me to burn out).
>
> I guess once you've got shared memory and inter-VM interrupts it might
> be possible to reuse the Xen protocols and drivers. I made a firewall
> VM on Qubes that did that a few years ago
> (https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/).
> But the virtio protocols will probably be more widely supported in
> future.
That's an interesting idea I hadn't considered. But I am hoping that
virtio gets to the point that we don't need to do that in a reasonable
amount of time.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-03-09 16:25 ` New user getting started questions Alyssa Ross
@ 2021-03-13 7:21 ` Thomas Leonard
2021-03-13 13:52 ` Alyssa Ross
2021-10-30 12:58 ` Thomas Leonard
0 siblings, 2 replies; 49+ messages in thread
From: Thomas Leonard @ 2021-03-13 7:21 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Tue, 9 Mar 2021 at 16:26, Alyssa Ross <hi@alyssa.is> wrote:
>
> Hi Thomas!
>
> Thanks for keeping us updated. It's really great to have all this
> written up to read, even though I'm only getting to it a month and a
> half later.
>
> On Wed, Jan 27, 2021 at 05:31:08PM +0000, Thomas Leonard wrote:
[...]
> > Once this is working more smoothly, I guess the next issues will be
> > setting up some kind of secure window manager on the host (e.g.
> > labelling windows with the VM they come from, not allowing
> > screenshots, etc). Would also be good to get sound forwarding working
> > somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
> > to control the levels for each, but I don't know how that worked). It
> > also needs some kind of VM manager to keep track of which VMs are
> > running. And some kind of IPC system like qrexec would be useful. Do
> > you have thoughts or plans about how to do any of this?
>
> The window manager is a part of this whole thing that makes me very
> nervous. A secure window manager is very important for Wayland, and I'm
> not sure how much I trust any of the existing ones to get it right. But
> with Wayfire I'm hoping it'll at least be easy enough to implement stuff
> like tagged/coloured windows for the proof of concept (since the
> plugin API and stuff is Wayfire's niche), and I'm hoping at some point
> somebody comes up with a security-focused Wayland window manager we can
> switch to -- I'd love a Rust one, and there's work going on in that
> area[1].
For the short-term, it would be fairly easy to make a slight change to
the wayland-virtwl-proxy[*] so that a version of it could run on the
host. Unlike the guest one, which has to copy frames and deal with
virtwl, this would just pass FDs through. And instead of connecting to
/dev/wl0, it would just connect to the host compositor socket. It
would then block access to screenshots (since it doesn't proxy that),
and would add the VM's name to each window's title.
Eventually I'd like to turn it into a full compositor, but I'm going
to be busy for the next 6 months at least.
> Not sure about IPC yet, but I recently read an article about PipeWire[2],
> and that's been making me think a bit about audio. With PipeWire, they
> seem to have cared about security from the start:
Thanks for the link. I hadn't realised PulseAudio was in such a bad state!
> > To avoid the PulseAudio sandboxing limitations, security was
> > baked-in: a per-client permissions bitfield is attached to every
> > PipeWire node — where one or more SPA nodes are wrapped. This
> > security-aware design allowed easy and safe integration with Flatpak
> > portals; the sandboxed-application permissions interface now promoted
> > to a freedesktop XDG standard.
>
> And it gets better! In particular, this sounds very promising:
>
> > a native fully asynchronous protocol that was inspired by Wayland —
> > without the XML serialization part — was implemented over Unix-domain
> > sockets. Taymans wanted a protocol that is simple and hard-realtime
> > safe.
>
> It goes on to say they use this for sending file descriptors and stuff.
> The similarity to Wayland is very exciting, because it means we might
> just be able to run PipeWire over the existing virtio_wl infrastructure
> very efficiently.
Yes. I wonder why they didn't just use Wayland directly. Removing the
XML schema files (I assume that's what they mean) doesn't seem like an
improvement. That's what makes it easy to use Wayland from safer
languages than C!
[*] https://github.com/talex5/wayland-virtwl-proxy
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-03-13 7:21 ` Thomas Leonard
@ 2021-03-13 13:52 ` Alyssa Ross
2021-10-30 12:58 ` Thomas Leonard
1 sibling, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2021-03-13 13:52 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 4514 bytes --]
On Sat, Mar 13, 2021 at 07:21:50AM +0000, Thomas Leonard wrote:
> > > Once this is working more smoothly, I guess the next issues will be
> > > setting up some kind of secure window manager on the host (e.g.
> > > labelling windows with the VM they come from, not allowing
> > > screenshots, etc). Would also be good to get sound forwarding working
> > > somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer
> > > to control the levels for each, but I don't know how that worked). It
> > > also needs some kind of VM manager to keep track of which VMs are
> > > running. And some kind of IPC system like qrexec would be useful. Do
> > > you have thoughts or plans about how to do any of this?
> >
> > The window manager is a part of this whole thing that makes me very
> > nervous. A secure window manager is very important for Wayland, and I'm
> > not sure how much I trust any of the existing ones to get it right. But
> > with Wayfire I'm hoping it'll at least be easy enough to implement stuff
> > like tagged/coloured windows for the proof of concept (since the
> > plugin API and stuff is Wayfire's niche), and I'm hoping at some point
> > somebody comes up with a security-focused Wayland window manager we can
> > switch to -- I'd love a Rust one, and there's work going on in that
> > area[1].
>
> For the short-term, it would be fairly easy to make a slight change to
> the wayland-virtwl-proxy[*] so that a version of it could run on the
> host. Unlike the guest one, which has to copy frames and deal with
> virtwl, this would just pass FDs through. And instead of connecting to
> /dev/wl0, it would just connect to the host compositor socket. It
> would then block access to screenshots (since it doesn't proxy that),
> and would add the VM's name to each window's title.
>
> Eventually I'd like to turn it into a full compositor, but I'm going
> to be busy for the next 6 months at least.
A sanitizing proxy of this sort could be a very good way to go indeed,
especially early on, because if we tightly control what messages Wayland
clients can send, we don't have to worry so much about what the
compositor will do if it gets a weird message. Having a proxy rather
than investigating in hardening a specific compositor (or writing such a
compositor) would also give us more freedom to change compositor later
(or allow users to choose their own compositor).
> > Not sure about IPC yet, but I recently read an article about PipeWire[2],
> > and that's been making me think a bit about audio. With PipeWire, they
> > seem to have cared about security from the start:
>
> Thanks for the link. I hadn't realised PulseAudio was in such a bad state!
I think it's just one of those things where they just didn't think about
security very much early on, and retrofitting that after the fact is
difficult because every application has already been written to expect
not to have to do anything special for security. Very much like the
situation with X11 compared to Wayland.
> > > To avoid the PulseAudio sandboxing limitations, security was
> > > baked-in: a per-client permissions bitfield is attached to every
> > > PipeWire node — where one or more SPA nodes are wrapped. This
> > > security-aware design allowed easy and safe integration with Flatpak
> > > portals; the sandboxed-application permissions interface now promoted
> > > to a freedesktop XDG standard.
> >
> > And it gets better! In particular, this sounds very promising:
> >
> > > a native fully asynchronous protocol that was inspired by Wayland —
> > > without the XML serialization part — was implemented over Unix-domain
> > > sockets. Taymans wanted a protocol that is simple and hard-realtime
> > > safe.
> >
> > It goes on to say they use this for sending file descriptors and stuff.
> > The similarity to Wayland is very exciting, because it means we might
> > just be able to run PipeWire over the existing virtio_wl infrastructure
> > very efficiently.
>
> Yes. I wonder why they didn't just use Wayland directly. Removing the
> XML schema files (I assume that's what they mean) doesn't seem like an
> improvement. That's what makes it easy to use Wayland from safer
> languages than C!
Yeah I was a bit surprised to see that too. Maybe they're not expecting
the protocol to grow extensions and stuff as much as Wayland, so
automatic code generation is less valuable? But I'm not sure -- I
haven't looked into it.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-03-13 7:21 ` Thomas Leonard
2021-03-13 13:52 ` Alyssa Ross
@ 2021-10-30 12:58 ` Thomas Leonard
2021-11-03 11:36 ` Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-10-30 12:58 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Sat, 13 Mar 2021 at 07:21, Thomas Leonard <talex5@gmail.com> wrote:
[...]
> For the short-term, it would be fairly easy to make a slight change to
> the wayland-virtwl-proxy so that a version of it could run on the
> host. Unlike the guest one, which has to copy frames and deal with
> virtwl, this would just pass FDs through. And instead of connecting to
> /dev/wl0, it would just connect to the host compositor socket. It
> would then block access to screenshots (since it doesn't proxy that),
> and would add the VM's name to each window's title.
>
> Eventually I'd like to turn it into a full compositor, but I'm going
> to be busy for the next 6 months at least.
The 6 months passed and I had a bit more free time to work on this,
and now the proxy runs on the host too!
I didn't have time to write a compositor though, because I ended up
spending my whole holiday getting Xwayland support added (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
details - it's surprisingly complicated!).
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-10-30 12:58 ` Thomas Leonard
@ 2021-11-03 11:36 ` Alyssa Ross
2021-11-03 18:27 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-11-03 11:36 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]
Thomas Leonard <talex5@gmail.com> writes:
> On Sat, 13 Mar 2021 at 07:21, Thomas Leonard <talex5@gmail.com> wrote:
> [...]
>> For the short-term, it would be fairly easy to make a slight change to
>> the wayland-virtwl-proxy so that a version of it could run on the
>> host. Unlike the guest one, which has to copy frames and deal with
>> virtwl, this would just pass FDs through. And instead of connecting to
>> /dev/wl0, it would just connect to the host compositor socket. It
>> would then block access to screenshots (since it doesn't proxy that),
>> and would add the VM's name to each window's title.
>>
>> Eventually I'd like to turn it into a full compositor, but I'm going
>> to be busy for the next 6 months at least.
>
> The 6 months passed and I had a bit more free time to work on this,
> and now the proxy runs on the host too!
>
> I didn't have time to write a compositor though, because I ended up
> spending my whole holiday getting Xwayland support added (see
> https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
> details - it's surprisingly complicated!).
That's awesome — thanks for keeping us updated!
Have you seen the new mechanism for Wayland over virtio-gpu context
types that Google are moving towards? It's supported in mainline Linux
now, and in Sommelier. I'd expect virtio-wl to go away at some point in
favour of that.
Also, I've been following some Qubes work recently, and it sounds like
they might be interested in doing X11 over Wayland in a way that
wouldn't need special compositor support (and the corresponding
potential for security bugs). It sounds like it would be a lot of work
though, so we'll see…
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-03 11:36 ` Alyssa Ross
@ 2021-11-03 18:27 ` Thomas Leonard
2021-11-10 12:58 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-11-03 18:27 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
>
> Thomas Leonard <talex5@gmail.com> writes:
>
> > On Sat, 13 Mar 2021 at 07:21, Thomas Leonard <talex5@gmail.com> wrote:
> > [...]
> >> For the short-term, it would be fairly easy to make a slight change to
> >> the wayland-virtwl-proxy so that a version of it could run on the
> >> host. Unlike the guest one, which has to copy frames and deal with
> >> virtwl, this would just pass FDs through. And instead of connecting to
> >> /dev/wl0, it would just connect to the host compositor socket. It
> >> would then block access to screenshots (since it doesn't proxy that),
> >> and would add the VM's name to each window's title.
> >>
> >> Eventually I'd like to turn it into a full compositor, but I'm going
> >> to be busy for the next 6 months at least.
> >
> > The 6 months passed and I had a bit more free time to work on this,
> > and now the proxy runs on the host too!
> >
> > I didn't have time to write a compositor though, because I ended up
> > spending my whole holiday getting Xwayland support added (see
> > https://roscidus.com/blog/blog/2021/10/30/xwayland/ if you want the
> > details - it's surprisingly complicated!).
>
> That's awesome — thanks for keeping us updated!
>
> Have you seen the new mechanism for Wayland over virtio-gpu context
> types that Google are moving towards? It's supported in mainline Linux
> now, and in Sommelier. I'd expect virtio-wl to go away at some point in
> favour of that.
Ah, is that released already? I saw you mentioned it in your handy
graphics write-up, but I thought it wasn't due until Linux 5.16:
https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ctx-Type
I don't know any details about it, but unless it's massively more
complicated than virtwl it shouldn't be any trouble to switch to it.
Will be nice not having to port virtwl to each new kernel!
> Also, I've been following some Qubes work recently, and it sounds like
> they might be interested in doing X11 over Wayland in a way that
> wouldn't need special compositor support (and the corresponding
> potential for security bugs). It sounds like it would be a lot of work
> though, so we'll see…
Isn't that what Sommelier and wayland-proxy-virtwl are both doing
already though? There's no need for any X11 support in the host
compositor with them.
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-10 12:58 ` Alyssa Ross
@ 2021-11-10 12:00 ` Thomas Leonard
2021-11-11 11:09 ` Alyssa Ross
2022-03-13 15:08 ` Thomas Leonard
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-11-10 12:00 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
>
> Thomas Leonard <talex5@gmail.com> writes:
>
> > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
[...]
> >> Also, I've been following some Qubes work recently, and it sounds like
> >> they might be interested in doing X11 over Wayland in a way that
> >> wouldn't need special compositor support (and the corresponding
> >> potential for security bugs). It sounds like it would be a lot of work
> >> though, so we'll see…
> >
> > Isn't that what Sommelier and wayland-proxy-virtwl are both doing
> > already though? There's no need for any X11 support in the host
> > compositor with them.
>
> Oh I didn't realise that. So the host compositor doesn't have to
> implement an X11 window manager and speak back to the Xserver, like it
> does in a normal XWayland setup[1]?
Correct; the proxy acts as the X11 window manager to Xwayland, and
just talks plain Wayland to the host. In fact, I managed to work
around several problems with Sway's built-in Xwayland integration by
using the proxy instead, even on the host (see
https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
> [1]: https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Support-architecture
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-03 18:27 ` Thomas Leonard
@ 2021-11-10 12:58 ` Alyssa Ross
2021-11-10 12:00 ` Thomas Leonard
2022-03-13 15:08 ` Thomas Leonard
0 siblings, 2 replies; 49+ messages in thread
From: Alyssa Ross @ 2021-11-10 12:58 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]
Thomas Leonard <talex5@gmail.com> writes:
> On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> Have you seen the new mechanism for Wayland over virtio-gpu context
>> types that Google are moving towards? It's supported in mainline Linux
>> now, and in Sommelier. I'd expect virtio-wl to go away at some point in
>> favour of that.
>
> Ah, is that released already? I saw you mentioned it in your handy
> graphics write-up, but I thought it wasn't due until Linux 5.16:
> https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ctx-Type
You're right. They've just made it toLinus's git tree. But they should
be ready for testing if you're brave! :P
> I don't know any details about it, but unless it's massively more
> complicated than virtwl it shouldn't be any trouble to switch to it.
> Will be nice not having to port virtwl to each new kernel!
>> Also, I've been following some Qubes work recently, and it sounds like
>> they might be interested in doing X11 over Wayland in a way that
>> wouldn't need special compositor support (and the corresponding
>> potential for security bugs). It sounds like it would be a lot of work
>> though, so we'll see…
>
> Isn't that what Sommelier and wayland-proxy-virtwl are both doing
> already though? There's no need for any X11 support in the host
> compositor with them.
Oh I didn't realise that. So the host compositor doesn't have to
implement an X11 window manager and speak back to the Xserver, like it
does in a normal XWayland setup[1]?
[1]: https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Support-architecture
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-10 12:00 ` Thomas Leonard
@ 2021-11-11 11:09 ` Alyssa Ross
2021-11-11 16:12 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2021-11-11 11:09 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1758 bytes --]
Thomas Leonard <talex5@gmail.com> writes:
> On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> Thomas Leonard <talex5@gmail.com> writes:
>>
>> > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
> [...]
>> >> Also, I've been following some Qubes work recently, and it sounds like
>> >> they might be interested in doing X11 over Wayland in a way that
>> >> wouldn't need special compositor support (and the corresponding
>> >> potential for security bugs). It sounds like it would be a lot of work
>> >> though, so we'll see…
>> >
>> > Isn't that what Sommelier and wayland-proxy-virtwl are both doing
>> > already though? There's no need for any X11 support in the host
>> > compositor with them.
>>
>> Oh I didn't realise that. So the host compositor doesn't have to
>> implement an X11 window manager and speak back to the Xserver, like it
>> does in a normal XWayland setup[1]?
>
> Correct; the proxy acts as the X11 window manager to Xwayland, and
> just talks plain Wayland to the host. In fact, I managed to work
> around several problems with Sway's built-in Xwayland integration by
> using the proxy instead, even on the host (see
> https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
>
>> [1]: https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Support-architecture
Ah, I now realise you explained all of this in your blog post. Sorry
for the silly questions!
By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
email, etc.". _I_ don't even consider myself to be running Spectrum,
because as far as I'm concerned there's no integrated system to run yet.
So what's in that VM that you consider to be Spectrum? :)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-11 11:09 ` Alyssa Ross
@ 2021-11-11 16:12 ` Thomas Leonard
2021-11-12 10:47 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2021-11-11 16:12 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Thu, 11 Nov 2021 at 11:09, Alyssa Ross <hi@alyssa.is> wrote:
>
> Thomas Leonard <talex5@gmail.com> writes:
>
> > On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
> >>
> >> Thomas Leonard <talex5@gmail.com> writes:
> >>
> >> > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
> > [...]
> >> >> Also, I've been following some Qubes work recently, and it sounds like
> >> >> they might be interested in doing X11 over Wayland in a way that
> >> >> wouldn't need special compositor support (and the corresponding
> >> >> potential for security bugs). It sounds like it would be a lot of work
> >> >> though, so we'll see…
> >> >
> >> > Isn't that what Sommelier and wayland-proxy-virtwl are both doing
> >> > already though? There's no need for any X11 support in the host
> >> > compositor with them.
> >>
> >> Oh I didn't realise that. So the host compositor doesn't have to
> >> implement an X11 window manager and speak back to the Xserver, like it
> >> does in a normal XWayland setup[1]?
> >
> > Correct; the proxy acts as the X11 window manager to Xwayland, and
> > just talks plain Wayland to the host. In fact, I managed to work
> > around several problems with Sway's built-in Xwayland integration by
> > using the proxy instead, even on the host (see
> > https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
> >
> >> [1]: https://wayland.freedesktop.org/docs/html/ch05.html#sect-X11-Application-Support-architecture
>
> Ah, I now realise you explained all of this in your blog post. Sorry
> for the silly questions!
>
> By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
> email, etc.". _I_ don't even consider myself to be running Spectrum,
> because as far as I'm concerned there's no integrated system to run yet.
> So what's in that VM that you consider to be Spectrum? :)
Well, I think of it as a spectrum VM, but I guess it has changed a
bit! It's based on your demo VM from early 2021, running Wayfire under
Sommelier. Except I replaced Sommelier with my proxy, and I removed
Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a
loop and I can launch programs inside it from the host by writing to
the vsock. e.g. I have a "ff-com" command to launch Firefox in the com
VM. It's a bit more Qubes-like than Spectrum really. I also got it to
get the applications from the main NixOS repository, as they're more
up-to-date, and I replaced the kernel with a newer one because I
needed better io_uring support for another project. But your basic
build structure and execline boot scripts are still there!
It was a useful skeleton, but now I understand how it works, I should
probably replace it with something more standard...
--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-11 16:12 ` Thomas Leonard
@ 2021-11-12 10:47 ` Alyssa Ross
0 siblings, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2021-11-12 10:47 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]
Thomas Leonard <talex5@gmail.com> writes:
> On Thu, 11 Nov 2021 at 11:09, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> Thomas Leonard <talex5@gmail.com> writes:
>>
>> > Correct; the proxy acts as the X11 window manager to Xwayland, and
>> > just talks plain Wayland to the host. In fact, I managed to work
>> > around several problems with Sway's built-in Xwayland integration by
>> > using the proxy instead, even on the host (see
>> > https://roscidus.com/blog/blog/2021/10/30/xwayland/#bonus-features).
>>
>> By the way, I'm very curious about "[com] is a SpectrumOS VM I use for
>> email, etc.". _I_ don't even consider myself to be running Spectrum,
>> because as far as I'm concerned there's no integrated system to run yet.
>> So what's in that VM that you consider to be Spectrum? :)
>
> Well, I think of it as a spectrum VM, but I guess it has changed a
> bit! It's based on your demo VM from early 2021, running Wayfire under
> Sommelier. Except I replaced Sommelier with my proxy, and I removed
> Wayfire. Instead, it runs "socat vsock-listen:5000 exec:dash" in a
> loop and I can launch programs inside it from the host by writing to
> the vsock. e.g. I have a "ff-com" command to launch Firefox in the com
> VM. It's a bit more Qubes-like than Spectrum really. I also got it to
> get the applications from the main NixOS repository, as they're more
> up-to-date, and I replaced the kernel with a newer one because I
> needed better io_uring support for another project. But your basic
> build structure and execline boot scripts are still there!
>
> It was a useful skeleton, but now I understand how it works, I should
> probably replace it with something more standard...
Wow, I'm amazed by how useful that's been to you! I'm shocked to learn
anybody has ever used that for more than running the demo. I guess the
only thing you're still getting from it that's non-standard is the
Chromium kernel? Which itself can hopefully be dropped soon if
virtio-gpu is a suitable replacement.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2021-11-10 12:58 ` Alyssa Ross
2021-11-10 12:00 ` Thomas Leonard
@ 2022-03-13 15:08 ` Thomas Leonard
2022-03-15 14:06 ` Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-03-13 15:08 UTC (permalink / raw)
To: Alyssa Ross; +Cc: Michael Raskin, discuss
On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
>
> Thomas Leonard <talex5@gmail.com> writes:
>
> > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
> >>
> >> Have you seen the new mechanism for Wayland over virtio-gpu context
> >> types that Google are moving towards? It's supported in mainline Linux
> >> now, and in Sommelier. I'd expect virtio-wl to go away at some point in
> >> favour of that.
> >
> > Ah, is that released already? I saw you mentioned it in your handy
> > graphics write-up, but I thought it wasn't due until Linux 5.16:
> > https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ctx-Type
>
> You're right. They've just made it toLinus's git tree. But they should
> be ready for testing if you're brave! :P
I'm happy to test it now with Linux 5.16, but I think I need a newer
crosvm too. Looks like support got added in
https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
so it should be in 94.14150.
Is there a new-enough package for crosvm available somewhere? NixOS
only has 81.12871.0.0-rc1, and spectrum seems to be on
91.13904.0.0-rc2.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2022-03-13 15:08 ` Thomas Leonard
@ 2022-03-15 14:06 ` Alyssa Ross
2022-03-15 20:23 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2022-03-15 14:06 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1564 bytes --]
On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
> On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > Thomas Leonard <talex5@gmail.com> writes:
> >
> > > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
> > >>
> > >> Have you seen the new mechanism for Wayland over virtio-gpu context
> > >> types that Google are moving towards? It's supported in mainline Linux
> > >> now, and in Sommelier. I'd expect virtio-wl to go away at some point in
> > >> favour of that.
> > >
> > > Ah, is that released already? I saw you mentioned it in your handy
> > > graphics write-up, but I thought it wasn't due until Linux 5.16:
> > > https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ctx-Type
> >
> > You're right. They've just made it toLinus's git tree. But they should
> > be ready for testing if you're brave! :P
>
> I'm happy to test it now with Linux 5.16, but I think I need a newer
> crosvm too. Looks like support got added in
> https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
> so it should be in 94.14150.
>
> Is there a new-enough package for crosvm available somewhere? NixOS
> only has 81.12871.0.0-rc1, and spectrum seems to be on
> 91.13904.0.0-rc2.
I don't know of one. I haven't been keeping it up to date since we've
mostly switched over to cloud-hypervisor for Spectrum, but it doesn't
have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu
implementation is on my list).
I'll have a go at updating it in Nixpkgs and report back.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: New user getting started questions
2022-03-15 14:06 ` Alyssa Ross
@ 2022-03-15 20:23 ` Alyssa Ross
2022-03-16 16:18 ` Using virtio-gpu instead of virtwl Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2022-03-15 20:23 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Michael Raskin, discuss
[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]
On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
> On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
> > On Wed, 10 Nov 2021 at 12:59, Alyssa Ross <hi@alyssa.is> wrote:
> > >
> > > Thomas Leonard <talex5@gmail.com> writes:
> > >
> > > > On Wed, 3 Nov 2021 at 11:37, Alyssa Ross <hi@alyssa.is> wrote:
> > > >>
> > > >> Have you seen the new mechanism for Wayland over virtio-gpu context
> > > >> types that Google are moving towards? It's supported in mainline Linux
> > > >> now, and in Sommelier. I'd expect virtio-wl to go away at some point in
> > > >> favour of that.
> > > >
> > > > Ah, is that released already? I saw you mentioned it in your handy
> > > > graphics write-up, but I thought it wasn't due until Linux 5.16:
> > > > https://www.phoronix.com/scan.php?page=news_item&px=VirtIO-Linux-5.16-Ctx-Type
> > >
> > > You're right. They've just made it toLinus's git tree. But they should
> > > be ready for testing if you're brave! :P
> >
> > I'm happy to test it now with Linux 5.16, but I think I need a newer
> > crosvm too. Looks like support got added in
> > https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
> > so it should be in 94.14150.
> >
> > Is there a new-enough package for crosvm available somewhere? NixOS
> > only has 81.12871.0.0-rc1, and spectrum seems to be on
> > 91.13904.0.0-rc2.
>
> I don't know of one. I haven't been keeping it up to date since we've
> mostly switched over to cloud-hypervisor for Spectrum, but it doesn't
> have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu
> implementation is on my list).
>
> I'll have a go at updating it in Nixpkgs and report back.
Okay, here's a package for crosvm 99.14468.0.0-rc1:
https://github.com/NixOS/nixpkgs/pull/164306
I've tested some basic stuff, but haven't tested virtio-gpu. I don't
see any reason it wouldn't work, though.
Good luck! Let me know if I can help with anything else.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Using virtio-gpu instead of virtwl
2022-03-15 20:23 ` Alyssa Ross
@ 2022-03-16 16:18 ` Thomas Leonard
2022-03-16 16:54 ` Alyssa Ross
2022-03-21 12:10 ` Thomas Leonard
0 siblings, 2 replies; 49+ messages in thread
From: Thomas Leonard @ 2022-03-16 16:18 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Tue, 15 Mar 2022 at 20:23, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
> > On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
[ virtio-gpu ]
> > > I'm happy to test it now with Linux 5.16, but I think I need a newer
> > > crosvm too. Looks like support got added in
> > > https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
> > > so it should be in 94.14150.
> > >
> > > Is there a new-enough package for crosvm available somewhere? NixOS
> > > only has 81.12871.0.0-rc1, and spectrum seems to be on
> > > 91.13904.0.0-rc2.
> >
> > I don't know of one. I haven't been keeping it up to date since we've
> > mostly switched over to cloud-hypervisor for Spectrum, but it doesn't
> > have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu
> > implementation is on my list).
> >
> > I'll have a go at updating it in Nixpkgs and report back.
>
> Okay, here's a package for crosvm 99.14468.0.0-rc1:
> https://github.com/NixOS/nixpkgs/pull/164306
Thanks - that's very useful!
Testing it from NixOS stable, it crashed with
[ERROR:src/linux.rs:3264] child pcivirtio-block (pid 47933) died:
signo 17, status 11, code 2
In the journal, I saw:
[src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to
create an executor: Uring(CreatingContext(Setup(12)))',
devices/src/virtio/block/asynchronous.rs:767:50
From a brief look at the code, it looks like crosvm sees that the host
is running Linux >= 5.10 and creates a test uring, which works. Then
it tries to create a real one, and that fails. I fixed it by
increasing the limit on locked memory in configuration.nix:
security.pam.loginLimits = [
{
domain = "*";
type = "-";
item = "memlock";
value = "64000";
}
];
Then my VM booted and worked as before.
> I've tested some basic stuff, but haven't tested virtio-gpu. I don't
> see any reason it wouldn't work, though.
I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`,
crosvm crashed with:
[ 31.326763] Invalid ELF header magic: != ELF
[ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0
[ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000
[ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible
[ 31.333984] [drm] features: +context_init
[ 31.337289] [drm] number of scanouts: 1
[ 31.337938] [drm] number of cap sets: 1
[ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo
17, status 11, code 2
[ERROR:devices/src/proxy.rs:212] failed write to child device process
pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
Then I tried with `--gpu=backend=2d` and that didn't crash, but
instead opened a window showing some bootloader stuff. I now have
/dev/dri/{card0, renderD128} devices, so I guess the next step is
figuring out what they do!
> Good luck! Let me know if I can help with anything else.
If you happen to know of any documentation, that would be great. But I
guess I probably just need to spend a load of time reading crosvm,
Linux, and Sommelier source code.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-16 16:18 ` Using virtio-gpu instead of virtwl Thomas Leonard
@ 2022-03-16 16:54 ` Alyssa Ross
2022-03-21 12:10 ` Thomas Leonard
1 sibling, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2022-03-16 16:54 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]
> I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`,
> crosvm crashed with:
>
> [ 31.326763] Invalid ELF header magic: != ELF
> [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0
> [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000
> [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible
> [ 31.333984] [drm] features: +context_init
> [ 31.337289] [drm] number of scanouts: 1
> [ 31.337938] [drm] number of cap sets: 1
> [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo
> 17, status 11, code 2
> [ERROR:devices/src/proxy.rs:212] failed write to child device process
> pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
>
> Then I tried with `--gpu=backend=2d` and that didn't crash, but
> instead opened a window showing some bootloader stuff. I now have
> /dev/dri/{card0, renderD128} devices, so I guess the next step is
> figuring out what they do!
When I've previously attempted this, I had some crosvm thread crash with
a seccomp violation because it tried to log something. If you
--disable-sandbox, it might work or at least give you a clearer error
message.
> > Good luck! Let me know if I can help with anything else.
>
> If you happen to know of any documentation, that would be great. But I
> guess I probably just need to spend a load of time reading crosvm,
> Linux, and Sommelier source code.
There might be some helpful stuff in the Mesa repo as well, because lots
of the latest graphics stuff is being done for crosvm. A quick grep
found me the Venus (Vulkan over virtio-gpu) documentation, which
describes testing with crosvm, and the script they use to run crosvm in
CI. It also reminded me that there are two different virgl
implementations in crosvm, virgl_renderer and virgl_renderer_next. I
think last time I tried I had better luck with virgl_renderer. You can
set buildFeatures in the Nix expression to experiment with them.
I hope some of that's helpful. I really wish I'd taken better notes
the one time I got this to work.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-16 16:18 ` Using virtio-gpu instead of virtwl Thomas Leonard
2022-03-16 16:54 ` Alyssa Ross
@ 2022-03-21 12:10 ` Thomas Leonard
2022-03-21 16:05 ` Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-03-21 12:10 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Wed, 16 Mar 2022 at 16:18, Thomas Leonard <talex5@gmail.com> wrote:
>
> On Tue, 15 Mar 2022 at 20:23, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote:
> > > On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote:
> [ virtio-gpu ]
> > > > I'm happy to test it now with Linux 5.16, but I think I need a newer
> > > > crosvm too. Looks like support got added in
> > > > https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676,
> > > > so it should be in 94.14150.
> > > >
> > > > Is there a new-enough package for crosvm available somewhere? NixOS
> > > > only has 81.12871.0.0-rc1, and spectrum seems to be on
> > > > 91.13904.0.0-rc2.
> > >
> > > I don't know of one. I haven't been keeping it up to date since we've
> > > mostly switched over to cloud-hypervisor for Spectrum, but it doesn't
> > > have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu
> > > implementation is on my list).
> > >
> > > I'll have a go at updating it in Nixpkgs and report back.
> >
> > Okay, here's a package for crosvm 99.14468.0.0-rc1:
> > https://github.com/NixOS/nixpkgs/pull/164306
>
> Thanks - that's very useful!
>
> Testing it from NixOS stable, it crashed with
[...]
> [src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to
> create an executor: Uring(CreatingContext(Setup(12)))',
> devices/src/virtio/block/asynchronous.rs:767:50
>
> From a brief look at the code, it looks like crosvm sees that the host
> is running Linux >= 5.10 and creates a test uring, which works. Then
> it tries to create a real one, and that fails. I fixed it by
> increasing the limit on locked memory in configuration.nix
However, this led to another problem: after resuming the host from
suspend, the crosvm drivers would crash with e.g.
[devices/src/virtio/block/asynchronous.rs:792] An error with a uring
source: URing::enter: Failed to enter io uring: 4
So a better fix is to run with "ulimit -l 0". That causes crosvm to
decide that uring isn't available and makes it fall back to the old
(working) system.
> > I've tested some basic stuff, but haven't tested virtio-gpu. I don't
> > see any reason it wouldn't work, though.
>
> I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`,
> crosvm crashed with:
>
> [ 31.326763] Invalid ELF header magic: != ELF
> [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0
> [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000
> [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible
> [ 31.333984] [drm] features: +context_init
> [ 31.337289] [drm] number of scanouts: 1
> [ 31.337938] [drm] number of cap sets: 1
> [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo
> 17, status 11, code 2
> [ERROR:devices/src/proxy.rs:212] failed write to child device process
> pcivirtio-gpu: failed to send packet: Broken pipe (os error 32)
>
> Then I tried with `--gpu=backend=2d` and that didn't crash, but
> instead opened a window showing some bootloader stuff. I now have
> /dev/dri/{card0, renderD128} devices, so I guess the next step is
> figuring out what they do!
>
> > Good luck! Let me know if I can help with anything else.
Looking at the Linux virtio_gpu driver, it seems that using contexts
requires virgl:
static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
*data, struct drm_file *file)
{
...
if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
return -EINVAL;
https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
I think perhaps that crosvm is compiled without the "virgl_renderer"
feature (it's not in the default set), and this is causing it to crash
because that's also "self.default_component". I don't know how to
compile crosvm with virgl enabled, though.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-21 12:10 ` Thomas Leonard
@ 2022-03-21 16:05 ` Alyssa Ross
2022-03-22 11:08 ` Thomas Leonard
2022-08-09 12:00 ` Alyssa Ross
0 siblings, 2 replies; 49+ messages in thread
From: Alyssa Ross @ 2022-03-21 16:05 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]
On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> Looking at the Linux virtio_gpu driver, it seems that using contexts
> requires virgl:
>
> static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
> *data, struct drm_file *file)
> {
> ...
> if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
> return -EINVAL;
>
> https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
>
> I think perhaps that crosvm is compiled without the "virgl_renderer"
> feature (it's not in the default set), and this is causing it to crash
> because that's also "self.default_component". I don't know how to
> compile crosvm with virgl enabled, though.
It wasn't easy, but I got it to build[1]. I hope that helps. It adds
both virgl_renderer and virgl_renderer_next. I think virgl_renderer
is on by default with --gpu, and virgl_renderer_next is used with the
--gpu-render-server argument. Hopefully at least one of those does the
right thing — let me know!
[1]: https://github.com/NixOS/nixpkgs/pull/165128
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-21 16:05 ` Alyssa Ross
@ 2022-03-22 11:08 ` Thomas Leonard
2022-03-22 11:16 ` Alyssa Ross
2022-08-09 12:00 ` Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-03-22 11:08 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Mon, 21 Mar 2022 at 16:05, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > Looking at the Linux virtio_gpu driver, it seems that using contexts
> > requires virgl:
> >
> > static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
> > *data, struct drm_file *file)
> > {
> > ...
> > if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
> > return -EINVAL;
> >
> > https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
> >
> > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > feature (it's not in the default set), and this is causing it to crash
> > because that's also "self.default_component". I don't know how to
> > compile crosvm with virgl enabled, though.
>
> It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> is on by default with --gpu, and virgl_renderer_next is used with the
> --gpu-render-server argument. Hopefully at least one of those does the
> right thing — let me know!
>
> [1]: https://github.com/NixOS/nixpkgs/pull/165128
Thanks, that is very helpful!
I gave it a try, and it got a little further. But now, doing `modprobe
virtio_gpu` in the VM crashes crosvm with:
Stack trace of thread 2:
#0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6)
#1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd)
#2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
#3 0x00007fa5fcfbf870 egl_single_resolver (libepoxy.so.0 + 0xc0870)
#4 0x00007fa5fcfc1d2f epoxy_eglQueryString_global_rewrite_ptr
(libepoxy.so.0 + 0xc2d2f)
#5 0x0000561703e72ca1 virgl_egl_init (crosvm + 0x3deca1)
#6 0x0000561703e72221 vrend_winsys_init (crosvm + 0x3de221)
#7 0x0000561703e380dc virgl_renderer_init (crosvm + 0x3a40dc)
#8 0x0000561703e35e44
_ZN12rutabaga_gfx14virgl_renderer13VirglRenderer4init17h96573d71589e47fcE
(crosvm + 0x3a1e44)
#9 0x0000561703e23124
_ZN12rutabaga_gfx13rutabaga_core15RutabagaBuilder5build17h694f3c234f8787ffE
(crosvm + 0x38f124)
#10 0x0000561703cdfb0f
_ZN7devices6virtio3gpu10virtio_gpu9VirtioGpu3new17h43dded1b3497b0f1E
(crosvm + 0x24bb0f)
#11 0x0000561703d0feb2
_ZN7devices6virtio3gpu5build17hc6e82daf2d41f5feE (crosvm + 0x27beb2)
#12 0x0000561703cbee32
_ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h8f078e46fd25a72dE
(crosvm + 0x22ae32)
#13 0x0000561703cf5e0d
_ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h5605b62669a02b38E
(crosvm + 0x261e0d)
#14 0x0000561703ee8b43
_ZN3std3sys4unix6thread6Thread3new12thread_start17h3f45e1fefa031d31E
(crosvm + 0x454b43)
#15 0x00007fa5fceb6d40 start_thread (libpthread.so.0 + 0x8d40)
#16 0x00007fa5fd16703f __clone (libc.so.6 + 0xfc03f)
Stack trace of thread 1:
#0 0x00007fa5fd1684b2 recvfrom (libc.so.6 + 0xfd4b2)
#1 0x0000561703ec7b0c
_ZN8sys_util3net13UnixSeqpacket20recv_as_vec_with_fds17hcc9cb638a4fdbca9E
(crosvm + 0x433b0c)
#2 0x0000561703b85b87 _ZN4base4tube4Tube4recv17hd85339bce7434d11E
(crosvm + 0xf1b87)
#3 0x0000561703b90395
_ZN7devices5proxy10child_proc17hde9579314b1fc020E (crosvm + 0xfc395)
#4 0x0000000103001400 n/a (n/a + 0x0)
It looks like it should be printing a message to stderr before calling
abort, but I don't see it
(https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
I also tried with --gpu=backend=virglrenderer,egl=false, which crashes
instead with:
Stack trace of thread 2:
#0 0x0000000000000000 n/a (n/a + 0x0)
#1 0x00007fac70002b00 n/a (n/a + 0x0)
I think I'll give up on virtio_gpu for now and try again after a few months.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-22 11:08 ` Thomas Leonard
@ 2022-03-22 11:16 ` Alyssa Ross
2022-03-22 20:05 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2022-03-22 11:16 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 3911 bytes --]
On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
> On Mon, 21 Mar 2022 at 16:05, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > > Looking at the Linux virtio_gpu driver, it seems that using contexts
> > > requires virgl:
> > >
> > > static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
> > > *data, struct drm_file *file)
> > > {
> > > ...
> > > if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
> > > return -EINVAL;
> > >
> > > https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
> > >
> > > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > > feature (it's not in the default set), and this is causing it to crash
> > > because that's also "self.default_component". I don't know how to
> > > compile crosvm with virgl enabled, though.
> >
> > It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> > both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> > is on by default with --gpu, and virgl_renderer_next is used with the
> > --gpu-render-server argument. Hopefully at least one of those does the
> > right thing — let me know!
> >
> > [1]: https://github.com/NixOS/nixpkgs/pull/165128
>
> Thanks, that is very helpful!
>
> I gave it a try, and it got a little further. But now, doing `modprobe
> virtio_gpu` in the VM crashes crosvm with:
>
> Stack trace of thread 2:
> #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6)
> #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd)
> #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
> #3 0x00007fa5fcfbf870 egl_single_resolver (libepoxy.so.0 + 0xc0870)
> #4 0x00007fa5fcfc1d2f epoxy_eglQueryString_global_rewrite_ptr
> (libepoxy.so.0 + 0xc2d2f)
> #5 0x0000561703e72ca1 virgl_egl_init (crosvm + 0x3deca1)
> #6 0x0000561703e72221 vrend_winsys_init (crosvm + 0x3de221)
> #7 0x0000561703e380dc virgl_renderer_init (crosvm + 0x3a40dc)
> #8 0x0000561703e35e44
> _ZN12rutabaga_gfx14virgl_renderer13VirglRenderer4init17h96573d71589e47fcE
> (crosvm + 0x3a1e44)
> #9 0x0000561703e23124
> _ZN12rutabaga_gfx13rutabaga_core15RutabagaBuilder5build17h694f3c234f8787ffE
> (crosvm + 0x38f124)
> #10 0x0000561703cdfb0f
> _ZN7devices6virtio3gpu10virtio_gpu9VirtioGpu3new17h43dded1b3497b0f1E
> (crosvm + 0x24bb0f)
> #11 0x0000561703d0feb2
> _ZN7devices6virtio3gpu5build17hc6e82daf2d41f5feE (crosvm + 0x27beb2)
> #12 0x0000561703cbee32
> _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h8f078e46fd25a72dE
> (crosvm + 0x22ae32)
> #13 0x0000561703cf5e0d
> _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h5605b62669a02b38E
> (crosvm + 0x261e0d)
> #14 0x0000561703ee8b43
> _ZN3std3sys4unix6thread6Thread3new12thread_start17h3f45e1fefa031d31E
> (crosvm + 0x454b43)
> #15 0x00007fa5fceb6d40 start_thread (libpthread.so.0 + 0x8d40)
> #16 0x00007fa5fd16703f __clone (libc.so.6 + 0xfc03f)
>
> Stack trace of thread 1:
> #0 0x00007fa5fd1684b2 recvfrom (libc.so.6 + 0xfd4b2)
> #1 0x0000561703ec7b0c
> _ZN8sys_util3net13UnixSeqpacket20recv_as_vec_with_fds17hcc9cb638a4fdbca9E
> (crosvm + 0x433b0c)
> #2 0x0000561703b85b87 _ZN4base4tube4Tube4recv17hd85339bce7434d11E
> (crosvm + 0xf1b87)
> #3 0x0000561703b90395
> _ZN7devices5proxy10child_proc17hde9579314b1fc020E (crosvm + 0xfc395)
> #4 0x0000000103001400 n/a (n/a + 0x0)
>
> It looks like it should be printing a message to stderr before calling
> abort, but I don't see it
> (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
Did you try --disable-sandbox, like I suggested in my other mail?
The sandbox blocks writing error messages, and is something I frequently
trip over when trying to use crosvm.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-22 11:16 ` Alyssa Ross
@ 2022-03-22 20:05 ` Thomas Leonard
2022-04-06 12:19 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-03-22 20:05 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Tue, 22 Mar 2022 at 11:16, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
> > On Mon, 21 Mar 2022 at 16:05, Alyssa Ross <hi@alyssa.is> wrote:
> > >
> > > On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > > > Looking at the Linux virtio_gpu driver, it seems that using contexts
> > > > requires virgl:
> > > >
> > > > static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
> > > > *data, struct drm_file *file)
> > > > {
> > > > ...
> > > > if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
> > > > return -EINVAL;
> > > >
> > > > https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
> > > >
> > > > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > > > feature (it's not in the default set), and this is causing it to crash
> > > > because that's also "self.default_component". I don't know how to
> > > > compile crosvm with virgl enabled, though.
> > >
> > > It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> > > both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> > > is on by default with --gpu, and virgl_renderer_next is used with the
> > > --gpu-render-server argument. Hopefully at least one of those does the
> > > right thing — let me know!
> > >
> > > [1]: https://github.com/NixOS/nixpkgs/pull/165128
> >
> > Thanks, that is very helpful!
> >
> > I gave it a try, and it got a little further. But now, doing `modprobe
> > virtio_gpu` in the VM crashes crosvm with:
> >
> > Stack trace of thread 2:
> > #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6)
> > #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd)
> > #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
[...]
> >
> > It looks like it should be printing a message to stderr before calling
> > abort, but I don't see it
> > (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
>
> Did you try --disable-sandbox, like I suggested in my other mail?
> The sandbox blocks writing error messages, and is something I frequently
> trip over when trying to use crosvm.
It's not very easy because --disable-sandbox seems to conflict with
--shared-dir, which I use for lots of things. But I've now compiled a
Linux kernel with "DRM_VIRTIO_GPU = yes" so I don't need the shared
/nix to get the kernel module. With that, running with
--disabled-sandbox seems to initialise the card successfully! I see:
[ 1.447535] [drm] Host memory window: 0x200000000 +0x200000000
[ 1.447821] [drm] features: +virgl -edid +resource_blob +host_visible
[ 1.447821] [drm] features: +context_init
[ 1.449292] [drm] number of scanouts: 1
[ 1.449495] [drm] number of cap sets: 4
[DEBUG:rutabaga_gfx/src/virgl_renderer.rs:113] gl_version 32 - es
profile enabled
[ 1.473662] [drm] cap set 0: id 1, max-version 1, max-size 308
[ 1.474027] [drm] cap set 1: id 2, max-version 2, max-size 764
[ 1.474386] [drm] cap set 2: id 4, max-version 0, max-size 0
[ 1.474702] [drm] cap set 3: id 5, max-version 0, max-size 16
[ 1.475158] [drm] Initialized virtio_gpu 0.1.0 0 for virtio6 on minor 0
[ 1.480946] virtio_gpu virtio6: [drm]
drm_plane_enable_fb_damage_clips() not called
[ 1.484018] Console: switching to colour frame buffer device 160x64
[ 1.487021] virtio_gpu virtio6: [drm] fb0: virtio_gpudrmfb frame
buffer device
Without --disabled-sandbox, it crashes after the "number of cap sets: 4" line.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-22 20:05 ` Thomas Leonard
@ 2022-04-06 12:19 ` Thomas Leonard
2022-04-13 17:12 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-04-06 12:19 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Tue, 22 Mar 2022 at 20:05, Thomas Leonard <talex5@gmail.com> wrote:
>
> On Tue, 22 Mar 2022 at 11:16, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Tue, Mar 22, 2022 at 11:08:15AM +0000, Thomas Leonard wrote:
> > > On Mon, 21 Mar 2022 at 16:05, Alyssa Ross <hi@alyssa.is> wrote:
> > > >
> > > > On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > > > > Looking at the Linux virtio_gpu driver, it seems that using contexts
> > > > > requires virgl:
> > > > >
> > > > > static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void
> > > > > *data, struct drm_file *file)
> > > > > {
> > > > > ...
> > > > > if (!vgdev->has_context_init || !vgdev->has_virgl_3d)
> > > > > return -EINVAL;
> > > > >
> > > > > https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732
> > > > >
> > > > > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > > > > feature (it's not in the default set), and this is causing it to crash
> > > > > because that's also "self.default_component". I don't know how to
> > > > > compile crosvm with virgl enabled, though.
> > > >
> > > > It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> > > > both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> > > > is on by default with --gpu, and virgl_renderer_next is used with the
> > > > --gpu-render-server argument. Hopefully at least one of those does the
> > > > right thing — let me know!
> > > >
> > > > [1]: https://github.com/NixOS/nixpkgs/pull/165128
> > >
> > > Thanks, that is very helpful!
> > >
> > > I gave it a try, and it got a little further. But now, doing `modprobe
> > > virtio_gpu` in the VM crashes crosvm with:
> > >
> > > Stack trace of thread 2:
> > > #0 0x00007fa5fd0915f6 abort (libc.so.6 + 0x265f6)
> > > #1 0x00007fa5fcfc6bfd get_dlopen_handle.part.0 (libepoxy.so.0 + 0xc7bfd)
> > > #2 0x00007fa5fcfc7366 epoxy_egl_dlsym (libepoxy.so.0 + 0xc8366)
> [...]
> > >
> > > It looks like it should be printing a message to stderr before calling
> > > abort, but I don't see it
> > > (https://github.com/anholt/libepoxy/blob/1.5.9/src/dispatch_common.c#L315).
> >
> > Did you try --disable-sandbox, like I suggested in my other mail?
> > The sandbox blocks writing error messages, and is something I frequently
> > trip over when trying to use crosvm.
>
> It's not very easy because --disable-sandbox seems to conflict with
> --shared-dir, which I use for lots of things.
I got around this by changing `create_gpu_device` to use `let jail =
None;`, so only the GPU device isn't jailed.
I suspect the minijail config needs updating for NixOS (e.g.
https://github.com/google/crosvm/blob/main/src/linux/gpu.rs#L82).
I tried, but failed, to figure out the protocol. I did manage to get a
test application showing a little animation, but it crashes after a
few seconds.
The basic idea seems to be:
1. You allocate a page of memory shared with the crosvm on the host.
2. You tell crosvm to read messages from the host compositor and write
them to this page.
3. After doing this, crosvm signals the guest, which reads the data.
The shared page is referred to as a "ring", but it's not used as a
ring buffer. The host always writes to the start of it.
Separately, to allocate an image buffer:
1. You tell crosvm the width and height, etc.
2. It assigns a blob_id and writes it to the shared page.
3. You wait for the operation to complete, then use the blob_id to
create the buffer.
The problem is that both these operations write to the same page, and
they race! So sometimes the image information overwrites the Wayland
data, or the Wayland data overwrites the image information, and then
it crashes.
This image_query function shows the problem:
https://chromium.googlesource.com/chromiumos/platform2/+/refs/heads/main/vm_tools/sommelier/virtualization/virtgpu_channel.cc#499
It asks for the image information to be written to "ring_addr_" and
then reads it from there. But at the moment when the function is
called, ring_addr_ may contain Wayland protocol data that hasn't been
read yet. I didn't test it with Sommelier, but that's the problem I
had in my code and I don't see how Sommelier's code can work in
general. Sommelier caches the results, so it might not hit this case
too often. I didn't use a cache, and also added a small sleep to my
code to make the problem easier to reproduce.
Anyone have any ideas how this is supposed to work?
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-04-06 12:19 ` Thomas Leonard
@ 2022-04-13 17:12 ` Thomas Leonard
2022-04-14 13:57 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-04-13 17:12 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
[ converting from virtwl to virtio-gpu ]
> I tried, but failed, to figure out the protocol. I did manage to get a
> test application showing a little animation, but it crashes after a
> few seconds.
OK, I found a solution to this: you can just open the device file
twice and use one instance for Wayland messages and the other for
allocating images. This avoids the first race. With that, I got the
proxy converted:
https://github.com/talex5/wayland-proxy-virtwl/pull/28
Though I'm not sure it's an improvement: +1,819 −577 lines!
Instructions for configuring crosvm to use it:
https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
And I wrote up my guesses about the protocol here:
https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
I don't think it's possible to avoid races completely, but it seems to
be working reasonably well so far.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-04-13 17:12 ` Thomas Leonard
@ 2022-04-14 13:57 ` Alyssa Ross
2022-04-19 12:58 ` Thomas Leonard
2022-05-15 15:20 ` Thomas Leonard
0 siblings, 2 replies; 49+ messages in thread
From: Alyssa Ross @ 2022-04-14 13:57 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 1634 bytes --]
On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
> On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
> [ converting from virtwl to virtio-gpu ]
> > I tried, but failed, to figure out the protocol. I did manage to get a
> > test application showing a little animation, but it crashes after a
> > few seconds.
>
> OK, I found a solution to this: you can just open the device file
> twice and use one instance for Wayland messages and the other for
> allocating images. This avoids the first race. With that, I got the
> proxy converted:
>
> https://github.com/talex5/wayland-proxy-virtwl/pull/28
>
> Though I'm not sure it's an improvement: +1,819 −577 lines!
>
> Instructions for configuring crosvm to use it:
>
> https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
>
> And I wrote up my guesses about the protocol here:
>
> https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
That's extremely helpful, thanks for writing it up!
> I don't think it's possible to avoid races completely, but it seems to
> be working reasonably well so far.
I wonder if it would be worth asking about the remaining problems on the
relevant kernel mailing lists, since they sound like protocol issues
rather than anything specific to your implementation.
I tracked down the series where they were added[1], perhaps the
From/To/Cc in that posting would be a good starting point?
If you do reach out, please CC me (and this list, if you want)!
[1]: https://lore.kernel.org/dri-devel/20210921232024.817-1-gurchetansingh@chromium.org/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-04-19 12:58 ` Thomas Leonard
@ 2022-04-19 12:01 ` Alyssa Ross
0 siblings, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2022-04-19 12:01 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 668 bytes --]
> > > I don't think it's possible to avoid races completely, but it seems to
> > > be working reasonably well so far.
> >
> > I wonder if it would be worth asking about the remaining problems on the
> > relevant kernel mailing lists, since they sound like protocol issues
> > rather than anything specific to your implementation.
>
> The kernel isn't involved in any of the problems - it just relays
> opaque messages between the host and the guest. So probably
> crosvm/sommelier will sort things out by themselves after a while (and
> I'll need to update the proxy when that happens).
Ah, I see. I wonder if there are any other implementations yet…
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-04-14 13:57 ` Alyssa Ross
@ 2022-04-19 12:58 ` Thomas Leonard
2022-04-19 12:01 ` Alyssa Ross
2022-05-15 15:20 ` Thomas Leonard
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-04-19 12:58 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
> > On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
> > [ converting from virtwl to virtio-gpu ]
> > > I tried, but failed, to figure out the protocol. I did manage to get a
> > > test application showing a little animation, but it crashes after a
> > > few seconds.
> >
> > OK, I found a solution to this: you can just open the device file
> > twice and use one instance for Wayland messages and the other for
> > allocating images. This avoids the first race. With that, I got the
> > proxy converted:
> >
> > https://github.com/talex5/wayland-proxy-virtwl/pull/28
> >
> > Though I'm not sure it's an improvement: +1,819 −577 lines!
> >
> > Instructions for configuring crosvm to use it:
> >
> > https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
> >
> > And I wrote up my guesses about the protocol here:
> >
> > https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
>
> That's extremely helpful, thanks for writing it up!
>
> > I don't think it's possible to avoid races completely, but it seems to
> > be working reasonably well so far.
>
> I wonder if it would be worth asking about the remaining problems on the
> relevant kernel mailing lists, since they sound like protocol issues
> rather than anything specific to your implementation.
The kernel isn't involved in any of the problems - it just relays
opaque messages between the host and the guest. So probably
crosvm/sommelier will sort things out by themselves after a while (and
I'll need to update the proxy when that happens).
> I tracked down the series where they were added[1], perhaps the
> From/To/Cc in that posting would be a good starting point?
>
> If you do reach out, please CC me (and this list, if you want)!
>
> [1]: https://lore.kernel.org/dri-devel/20210921232024.817-1-gurchetansingh@chromium.org/
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-04-14 13:57 ` Alyssa Ross
2022-04-19 12:58 ` Thomas Leonard
@ 2022-05-15 15:20 ` Thomas Leonard
2022-05-16 11:55 ` Alyssa Ross
1 sibling, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-05-15 15:20 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Thu, 14 Apr 2022 at 13:57, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
> > On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
> > [ converting from virtwl to virtio-gpu ]
> > > I tried, but failed, to figure out the protocol. I did manage to get a
> > > test application showing a little animation, but it crashes after a
> > > few seconds.
> >
> > OK, I found a solution to this: you can just open the device file
> > twice and use one instance for Wayland messages and the other for
> > allocating images. This avoids the first race. With that, I got the
> > proxy converted:
> >
> > https://github.com/talex5/wayland-proxy-virtwl/pull/28
> >
> > Though I'm not sure it's an improvement: +1,819 −577 lines!
> >
> > Instructions for configuring crosvm to use it:
> >
> > https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
> >
> > And I wrote up my guesses about the protocol here:
> >
> > https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
>
> That's extremely helpful, thanks for writing it up!
A small update on this:
First, I got virtio-gpu working with the jail. It just needs a couple
of extra paths for NixOS:
diff --git a/src/linux.rs b/src/linux.rs
index ad031749..52d3142f 100644
--- a/src/linux.rs
+++ b/src/linux.rs
@@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) ->
Result<Option<Minijail>> {
jail_mount_bind_if_exists(
&mut jail,
&[
+ "/run/opengl-driver",
+ "/nix/store",
"/usr/lib",
"/usr/lib64",
"/lib",
Secondly, I realised that the "video" memory returned by virtio-gpu
was just regular host memory (from stracing crosvm). This is because
crosvm is compiled without minigbm support and falls back to this.
After enabling minigbm and its amdgpu support (which also required
adding a dependency on mesa) it started allocating vram. However, this
broke the proxy because vram can't be used with the Wl_shm protocol.
I updated the proxy's test.ml code (which opens a window with a
scrolling pattern) to use Linux_dmabuf_unstable_v1, and that almost
worked. But some of the pixels are wrong and it often crashes the
whole VM. Possibly some of the output is getting stuck in a cache
somewhere and is not actually flushed to the graphics card in time?
Unfortunately, I don't know anything about GPUs.
Anyway, I merged the changes I had, in case anyone else wants to try
to get it working:
https://github.com/talex5/wayland-proxy-virtwl/pull/33
If it did work, it should make things a bit more efficient by avoiding
one copy. At the moment, I think the process is:
1. Application renders to guest memory.
2. Proxy copies to host memory.
3. Compositor copies to graphics card.
If the mingbm stuff worked, the proxy could copy directly to the
graphics card. And possibly the application could render there
directly in the first place, but that would need more work.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-05-15 15:20 ` Thomas Leonard
@ 2022-05-16 11:55 ` Alyssa Ross
2022-05-18 9:55 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2022-05-16 11:55 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 2572 bytes --]
Thanks for the update!
On Sun, May 15, 2022 at 03:20:24PM +0000, Thomas Leonard wrote:
> On Thu, 14 Apr 2022 at 13:57, Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
> > > On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
> > > [ converting from virtwl to virtio-gpu ]
> > > > I tried, but failed, to figure out the protocol. I did manage to get a
> > > > test application showing a little animation, but it crashes after a
> > > > few seconds.
> > >
> > > OK, I found a solution to this: you can just open the device file
> > > twice and use one instance for Wayland messages and the other for
> > > allocating images. This avoids the first race. With that, I got the
> > > proxy converted:
> > >
> > > https://github.com/talex5/wayland-proxy-virtwl/pull/28
> > >
> > > Though I'm not sure it's an improvement: +1,819 −577 lines!
> > >
> > > Instructions for configuring crosvm to use it:
> > >
> > > https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
> > >
> > > And I wrote up my guesses about the protocol here:
> > >
> > > https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
> >
> > That's extremely helpful, thanks for writing it up!
>
> A small update on this:
>
> First, I got virtio-gpu working with the jail. It just needs a couple
> of extra paths for NixOS:
>
> diff --git a/src/linux.rs b/src/linux.rs
> index ad031749..52d3142f 100644
> --- a/src/linux.rs
> +++ b/src/linux.rs
> @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) ->
> Result<Option<Minijail>> {
> jail_mount_bind_if_exists(
> &mut jail,
> &[
> + "/run/opengl-driver",
> + "/nix/store",
> "/usr/lib",
> "/usr/lib64",
> "/lib",
>
> Secondly, I realised that the "video" memory returned by virtio-gpu
> was just regular host memory (from stracing crosvm). This is because
> crosvm is compiled without minigbm support and falls back to this.
>
> After enabling minigbm and its amdgpu support (which also required
> adding a dependency on mesa) it started allocating vram. However, this
> broke the proxy because vram can't be used with the Wl_shm protocol.
Do you have a Nix expression somewhere for crosvm with all this stuff
fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you
didn't do it with Nix, I can make the changes myself.)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-05-16 11:55 ` Alyssa Ross
@ 2022-05-18 9:55 ` Thomas Leonard
2022-06-05 16:29 ` Thomas Leonard
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-05-18 9:55 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 3576 bytes --]
On Mon, 16 May 2022 at 11:55, Alyssa Ross <hi@alyssa.is> wrote:
>
> Thanks for the update!
>
> On Sun, May 15, 2022 at 03:20:24PM +0000, Thomas Leonard wrote:
> > On Thu, 14 Apr 2022 at 13:57, Alyssa Ross <hi@alyssa.is> wrote:
> > >
> > > On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote:
> > > > On Wed, 6 Apr 2022 at 12:19, Thomas Leonard <talex5@gmail.com> wrote:
> > > > [ converting from virtwl to virtio-gpu ]
> > > > > I tried, but failed, to figure out the protocol. I did manage to get a
> > > > > test application showing a little animation, but it crashes after a
> > > > > few seconds.
> > > >
> > > > OK, I found a solution to this: you can just open the device file
> > > > twice and use one instance for Wayland messages and the other for
> > > > allocating images. This avoids the first race. With that, I got the
> > > > proxy converted:
> > > >
> > > > https://github.com/talex5/wayland-proxy-virtwl/pull/28
> > > >
> > > > Though I'm not sure it's an improvement: +1,819 −577 lines!
> > > >
> > > > Instructions for configuring crosvm to use it:
> > > >
> > > > https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support
> > > >
> > > > And I wrote up my guesses about the protocol here:
> > > >
> > > > https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spec.md
> > >
> > > That's extremely helpful, thanks for writing it up!
> >
> > A small update on this:
> >
> > First, I got virtio-gpu working with the jail. It just needs a couple
> > of extra paths for NixOS:
> >
> > diff --git a/src/linux.rs b/src/linux.rs
> > index ad031749..52d3142f 100644
> > --- a/src/linux.rs
> > +++ b/src/linux.rs
> > @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) ->
> > Result<Option<Minijail>> {
> > jail_mount_bind_if_exists(
> > &mut jail,
> > &[
> > + "/run/opengl-driver",
> > + "/nix/store",
> > "/usr/lib",
> > "/usr/lib64",
> > "/lib",
> >
> > Secondly, I realised that the "video" memory returned by virtio-gpu
> > was just regular host memory (from stracing crosvm). This is because
> > crosvm is compiled without minigbm support and falls back to this.
> >
> > After enabling minigbm and its amdgpu support (which also required
> > adding a dependency on mesa) it started allocating vram. However, this
> > broke the proxy because vram can't be used with the Wl_shm protocol.
>
> Do you have a Nix expression somewhere for crosvm with all this stuff
> fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you
> didn't do it with Nix, I can make the changes myself.)
No, sorry. I've just been hacking on it in nix-shell. The only change
I made to nixpkgs was adding mesa as a dependency. The changes to get
minigbm to compile are just a hack for my system. But I've attached
the patches just for reference. However, since it stops the proxy from
working, these patches currently aren't an improvement from my point
of view!
BTW, I did get a little further: using __builtin_ia32_clflush fixed
the cache-coherency problem and the scrolling pattern now looks
correct. However, it still crashes the VM if you resize the window a
bit. I modified crosvm to clear the vram before handing it to the
guest (it wasn't doing that before), and that doesn't crash, so the
memory is mapped correctly on the host side.
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
[-- Attachment #2: mesa.patch --]
[-- Type: application/octet-stream, Size: 577 bytes --]
index 26e89e0cf20..06baf0bb352 100644
--- a/pkgs/applications/virtualization/crosvm/default.nix
+++ b/pkgs/applications/virtualization/crosvm/default.nix
@@ -1,7 +1,7 @@
{ stdenv, lib, rust, rustPlatform, fetchgit
, meson, ninja, pkg-config, python3, wayland-scanner
, libcap, libdrm, libepoxy, libglvnd, minijail, wayland, wayland-protocols, xorg
-, linux
+, linux, mesa
}:
let
@@ -40,7 +40,7 @@ in
buildInputs = [
libcap libdrm libepoxy libglvnd minijail wayland wayland-protocols
- xorg.libX11
+ xorg.libX11 mesa
];
postPatch = ''
[-- Attachment #3: minigbm.patch --]
[-- Type: application/octet-stream, Size: 1180 bytes --]
diff --git a/Makefile b/Makefile
index 987708f..9442911 100644
--- a/Makefile
+++ b/Makefile
@@ -10,10 +10,10 @@ PC_LIBS := $(shell $(PKG_CONFIG) --libs $(PC_DEPS))
CPPFLAGS += -D_GNU_SOURCE=1
CFLAGS += -std=c99 -Wall -Wsign-compare -Wpointer-arith -Wcast-qual \
- -Wcast-align -D_GNU_SOURCE=1 -D_FILE_OFFSET_BITS=64
+ -Wcast-align -D_GNU_SOURCE=1 -D_FILE_OFFSET_BITS=64 -DDRV_AMDGPU -DDRI_DRIVER_DIR=/run/opengl-driver/lib/dri
ifdef DRV_AMDGPU
- CFLAGS += $(shell $(PKG_CONFIG) --cflags libdrm_amdgpu)
+ CFLAGS += $(shell $(PKG_CONFIG) --cflags libdrm_amdgpu osmesa)
LDLIBS += -ldrm_amdgpu -ldl
endif
ifdef DRV_I915
diff --git a/dri.c b/dri.c
index 8b55c32..ac8794b 100644
--- a/dri.c
+++ b/dri.c
@@ -452,7 +452,7 @@ size_t dri_num_planes_from_modifier(struct driver *drv, uint32_t format, uint64_
}
uint64_t planes;
- GLboolean ret = dri->image_extension->queryDmaBufFormatModifierAttribs(
+ int ret = dri->image_extension->queryDmaBufFormatModifierAttribs(
dri->device, format, modifier, __DRI_IMAGE_FORMAT_MODIFIER_ATTRIB_PLANE_COUNT, &planes);
if (!ret)
return 0;
[-- Attachment #4: crosvm.patch --]
[-- Type: application/octet-stream, Size: 1003 bytes --]
diff --git a/Cargo.toml b/Cargo.toml
index e98c7daa..8c1dfca0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -134,7 +134,7 @@ disk = { path = "disk" }
enumn = "0.1.0"
gdbstub = { version = "0.5.0", optional = true }
gdbstub_arch = { version = "0.1.0", optional = true }
-rutabaga_gfx = { path = "rutabaga_gfx"}
+rutabaga_gfx = { path = "rutabaga_gfx", features = ["minigbm"]}
hypervisor = { path = "hypervisor" }
kernel_cmdline = { path = "kernel_cmdline" }
kernel_loader = { path = "kernel_loader" }
diff --git a/rutabaga_gfx/build.rs b/rutabaga_gfx/build.rs
index eb109a29..366d8953 100644
--- a/rutabaga_gfx/build.rs
+++ b/rutabaga_gfx/build.rs
@@ -72,6 +72,7 @@ fn build_minigbm(out_dir: &Path) -> Result<()> {
let make_flags = env::var("CARGO_MAKEFLAGS").unwrap();
let status = Command::new("make")
+ .env("DRV_AMDGPU", "1")
.env("MAKEFLAGS", make_flags)
.env("CROSS_COMPILE", get_cross_compile_prefix())
.arg(format!("OUT={}", out_dir.display()))
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-05-18 9:55 ` Thomas Leonard
@ 2022-06-05 16:29 ` Thomas Leonard
0 siblings, 0 replies; 49+ messages in thread
From: Thomas Leonard @ 2022-06-05 16:29 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Wed, 18 May 2022 at 09:55, Thomas Leonard <talex5@gmail.com> wrote:
>
> On Mon, 16 May 2022 at 11:55, Alyssa Ross <hi@alyssa.is> wrote:
[...]
> > Do you have a Nix expression somewhere for crosvm with all this stuff
> > fixed? I'd like to integrate it into my draft Nixpkgs PR. (If you
> > didn't do it with Nix, I can make the changes myself.)
>
> No, sorry. I've just been hacking on it in nix-shell.
I've now collected things together and pushed all my scripts here:
https://gitlab.com/talex5/qubes-lite
I've got quite a few patches to crosvm now (and NixOS 22.05 brought
some new problems), but they're hacks and not suitable for
upstreaming. They might be a useful starting point for someone else
though. They're in the crosvm subdirectory:
fix-gui-jail:
Adds some Nix paths to the jail to allow virtio-gpu to start.
Otherwise, the mesa libraries can't be loaded. Note that mesa
libraries are loaded dynamically by glvnd using dlopen. If that
returns NULL to indicate an error, glvnd just ignores the error and
doesn't load any drivers, causing crosvm to segfault later on a NULL
pointer. That also hit me when I was accidentally compiling crosvm
with a different version of glibc.
fix-keymaps:
Sway/wlroots now sends read-only keymaps, but crosvm wants to map them
read/write, which fails. This hack gets crosvm to try a read-only mmap
if a read/write one fails. A better solution would be to remember that
the resource is a keymap and map it read-only first.
fix-suspend:
After the host resumes from suspend, the disk driver gets EINTR from
uring and stops handling disk requests. My previous trick of setting
the locked memory limit to zero no longer works for some reason, so
this stops uring in the source. A better fix would be to get crosvm to
retry on EINTR.
no-gpu-window:
When using virtio-gpu, crosvm opens a pointless console window. This
patch forces the use of the Stub backend, which hides it. Possibly
this still wastes resources though; maybe disabling the display
completely would be better.
share-as-user:
The shared filesystem driver runs in a jail where everything appears
to be owned by root, and so this is how it appears in the Linux guest.
This patch makes all files appear to be owned by user 1000, and
performs all operations as user 0. This allows the regular user in the
VM to access shared files. A better fix might be to stop using user
namespaces here, but I'm not sure how to make that work.
slow-fences:
While there are GPU operations outstanding, crosvm wakes up 1000 times
a second to check whether they're done, which is very wasteful. This
patch makes it wake only once per second, which doesn't seem to have
any ill effects. I'm not sure why there are outstanding requests all
the time - possibly "wait for a user action" counts as an in-progress
request?
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-03-21 16:05 ` Alyssa Ross
2022-03-22 11:08 ` Thomas Leonard
@ 2022-08-09 12:00 ` Alyssa Ross
2022-10-10 15:16 ` Thomas Leonard
1 sibling, 1 reply; 49+ messages in thread
From: Alyssa Ross @ 2022-08-09 12:00 UTC (permalink / raw)
To: Thomas Leonard; +Cc: discuss
[-- Attachment #1: Type: text/plain, Size: 906 bytes --]
On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
> On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > feature (it's not in the default set), and this is causing it to crash
> > because that's also "self.default_component". I don't know how to
> > compile crosvm with virgl enabled, though.
>
> It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> is on by default with --gpu, and virgl_renderer_next is used with the
> --gpu-render-server argument. Hopefully at least one of those does the
> right thing — let me know!
>
> [1]: https://github.com/NixOS/nixpkgs/pull/165128
Small update: Nixpkgs unstable's crosvm package is now built with the
virgl_renderer and virgl_renderer_next features.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-08-09 12:00 ` Alyssa Ross
@ 2022-10-10 15:16 ` Thomas Leonard
2022-10-10 16:53 ` Alyssa Ross
0 siblings, 1 reply; 49+ messages in thread
From: Thomas Leonard @ 2022-10-10 15:16 UTC (permalink / raw)
To: Alyssa Ross; +Cc: discuss
On Tue, 9 Aug 2022 at 12:01, Alyssa Ross <hi@alyssa.is> wrote:
>
> On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
> > On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
> > > I think perhaps that crosvm is compiled without the "virgl_renderer"
> > > feature (it's not in the default set), and this is causing it to crash
> > > because that's also "self.default_component". I don't know how to
> > > compile crosvm with virgl enabled, though.
> >
> > It wasn't easy, but I got it to build[1]. I hope that helps. It adds
> > both virgl_renderer and virgl_renderer_next. I think virgl_renderer
> > is on by default with --gpu, and virgl_renderer_next is used with the
> > --gpu-render-server argument. Hopefully at least one of those does the
> > right thing — let me know!
> >
> > [1]: https://github.com/NixOS/nixpkgs/pull/165128
>
> Small update: Nixpkgs unstable's crosvm package is now built with the
> virgl_renderer and virgl_renderer_next features.
I got this working eventually, but I had to apply a load of patches.
How are you getting it to run?
My patches are here: https://gitlab.com/talex5/crosvm/-/commits/main
In particular:
- It failed to start (when using virtio-gpu) because it doesn't have
access to /nix/store, and
- It failed to send Wayland keymaps because they need to be mapped read-only
--
talex5 (GitHub/Twitter)
http://roscidus.com/blog/
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Using virtio-gpu instead of virtwl
2022-10-10 15:16 ` Thomas Leonard
@ 2022-10-10 16:53 ` Alyssa Ross
0 siblings, 0 replies; 49+ messages in thread
From: Alyssa Ross @ 2022-10-10 16:53 UTC (permalink / raw)
To: Thomas Leonard; +Cc: Puck Meerburg, discuss
[-- Attachment #1: Type: text/plain, Size: 2444 bytes --]
Thomas Leonard <talex5@gmail.com> writes:
> On Tue, 9 Aug 2022 at 12:01, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> On Mon, Mar 21, 2022 at 04:05:34PM +0000, Alyssa Ross wrote:
>> > On Mon, Mar 21, 2022 at 12:10:43PM +0000, Thomas Leonard wrote:
>> > > I think perhaps that crosvm is compiled without the "virgl_renderer"
>> > > feature (it's not in the default set), and this is causing it to crash
>> > > because that's also "self.default_component". I don't know how to
>> > > compile crosvm with virgl enabled, though.
>> >
>> > It wasn't easy, but I got it to build[1]. I hope that helps. It adds
>> > both virgl_renderer and virgl_renderer_next. I think virgl_renderer
>> > is on by default with --gpu, and virgl_renderer_next is used with the
>> > --gpu-render-server argument. Hopefully at least one of those does the
>> > right thing — let me know!
>> >
>> > [1]: https://github.com/NixOS/nixpkgs/pull/165128
>>
>> Small update: Nixpkgs unstable's crosvm package is now built with the
>> virgl_renderer and virgl_renderer_next features.
>
> I got this working eventually, but I had to apply a load of patches.
> How are you getting it to run?
>
> My patches are here: https://gitlab.com/talex5/crosvm/-/commits/main
>
> In particular:
> - It failed to start (when using virtio-gpu) because it doesn't have
> access to /nix/store, and
I haven't hit this one. I've mostly been testing with vhost-user —
maybe crosvm devices don't run sandboxed when run standalone for
vhost-user, since running that device is the only thing the crosvm
process is doing?
> - It failed to send Wayland keymaps because they need to be mapped read-only
The vhost-user-gpu frontend implementation we wrote for cloud-hypervisor[1]
will map buffers as read-only if mapping them read-write fails[2]. This
is only a workaround for crosvm not setting the flags correctly in the
vhost-user message, though. The real fix will be in crosvm.
Also, not all Wayland compositors require keymaps to be mapped
read-only. wlroots does, but Weston doesn't. I suspect Chromium
doesn't either, hence the bug persisting in crosvm.
[1]: https://spectrum-os.org/lists/archives/spectrum-devel/20220930210906.1696349-8-alyssa.ross@unikie.com/
[2]: https://spectrum-os.org/lists/archives/spectrum-devel/20b1f9da3af/s/?b=pkgs/applications/virtualization/cloud-hypervisor/0003-virtio-devices-add-a-GPU-device.patch#n247
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
end of thread, other threads:[~2022-10-10 16:54 UTC | newest]
Thread overview: 49+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-05 19:27 New user getting started questions Thomas Leonard
2021-01-05 20:09 ` Michael Raskin
2021-01-06 7:04 ` Alyssa's break Alyssa Ross
2021-01-06 9:11 ` Michał "rysiek" Woźniak
2021-01-06 7:00 ` New user getting started questions Alyssa Ross
2021-01-06 15:56 ` Thomas Leonard
2021-01-07 11:38 ` Thomas Leonard
2021-01-07 15:33 ` Thomas Leonard
2021-01-14 12:29 ` Alyssa Ross
2021-01-14 12:51 ` Alyssa Ross
2021-01-20 13:04 ` Thomas Leonard
2021-01-27 17:31 ` Thomas Leonard
2021-03-07 12:52 ` Thomas Leonard
2021-03-09 16:59 ` Qubes-lite With KVM and Wayland Alyssa Ross
2021-03-10 14:19 ` Thomas Leonard
2021-03-10 22:34 ` Alyssa Ross
2021-03-09 16:25 ` New user getting started questions Alyssa Ross
2021-03-13 7:21 ` Thomas Leonard
2021-03-13 13:52 ` Alyssa Ross
2021-10-30 12:58 ` Thomas Leonard
2021-11-03 11:36 ` Alyssa Ross
2021-11-03 18:27 ` Thomas Leonard
2021-11-10 12:58 ` Alyssa Ross
2021-11-10 12:00 ` Thomas Leonard
2021-11-11 11:09 ` Alyssa Ross
2021-11-11 16:12 ` Thomas Leonard
2021-11-12 10:47 ` Alyssa Ross
2022-03-13 15:08 ` Thomas Leonard
2022-03-15 14:06 ` Alyssa Ross
2022-03-15 20:23 ` Alyssa Ross
2022-03-16 16:18 ` Using virtio-gpu instead of virtwl Thomas Leonard
2022-03-16 16:54 ` Alyssa Ross
2022-03-21 12:10 ` Thomas Leonard
2022-03-21 16:05 ` Alyssa Ross
2022-03-22 11:08 ` Thomas Leonard
2022-03-22 11:16 ` Alyssa Ross
2022-03-22 20:05 ` Thomas Leonard
2022-04-06 12:19 ` Thomas Leonard
2022-04-13 17:12 ` Thomas Leonard
2022-04-14 13:57 ` Alyssa Ross
2022-04-19 12:58 ` Thomas Leonard
2022-04-19 12:01 ` Alyssa Ross
2022-05-15 15:20 ` Thomas Leonard
2022-05-16 11:55 ` Alyssa Ross
2022-05-18 9:55 ` Thomas Leonard
2022-06-05 16:29 ` Thomas Leonard
2022-08-09 12:00 ` Alyssa Ross
2022-10-10 15:16 ` Thomas Leonard
2022-10-10 16:53 ` Alyssa Ross
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).