On 1/6/21 7:04 AM, Alyssa Ross wrote: Good to hear from you, Alyssa! Happy New Year, and take good care of yourself. The project can wait, self-care is important. :-) -- Best, r

On Wed, 6 Jan 2021 at 07:01, Alyssa Ross <hi(a)alyssa.is> wrote: [...] That's better, thanks! [...] Ah, I didn't realise it was using seccomp too. I'm not sure how to compile specific versions of crosvm. I tried with: srcs = lib.genAttrs [ "src/third_party/adhd" "src/aosp/external/minijail" ] getSrc // { "src/platform/crosvm" = /home/.../crosvm; }; and blanked out the hash as it requested, but then: error: failed to sync Caused by: failed to load pkg lockfile Caused by: failed to resolve patches for `https://github.com/rust-lang/crates.io-index` Caused by: failed to load source for dependency `libvda` Caused by: Unable to update /build/src/platform2/arc/vm/libvda/rust Caused by: failed to read `/build/src/platform2/arc/vm/libvda/rust/Cargo.toml` Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f (added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312 just before that gets further, but: error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be updated but --frozen was passed to prevent this How do you build it? (sorry for these basic Nix/Rust questions) However, I could get 9p to work by running the previous version with --seccomp-log-failures. With that, I can read and write files from the console, but I can't chown things and so can't write from the terminal window, which is running as a user. I guess it needs uidmap set, but I'm not sure how to make that work. OK, I'll keep trying stuff. I have discovered that if I add the squashfs file as another device (--root "$rootfs" -d "$rootfs") then it shows an error but mounts it anyway! # ls /tmp # mount /dev/vdb /tmp [ 15.288873] /dev/vdb: Can't open blockdev # ls /tmp bin dev etc nix proc run sbin sys tmp Sadly, this didn't work with my ext4 partition. OK, I tried like this: exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel" I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with: [ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1) Strace shows: openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31 ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted) Maybe it's just because my crosvm is too old? Great! I'm on a break myself at the moment, which is why I have some time to try all this out. I've read some of them - they're very helpful! I have many questions :-) But don't feel pressured to answer them; I need to figure out how to make this all work myself anyway, and it's just a bonus if you've already done the work for me... -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC

On Wed, 6 Jan 2021 at 15:56, Thomas Leonard <talex5(a)gmail.com> wrote: [...] This turned out to be just because ext4 is compiled as a module, and /lib isn't in the root filesystem. Adding "EXT4_FS = yes" to the kernel configuration fixed it. -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC

Oh, whoops, I missed your reply about having worked this out already!

I've made a bit of progress this week: It turns out that weston-terminal crashes sommelier if started when the clipboard is empty, due to trying to dereference NULL. I've patched it to fix that, and now I can run it directly under sommelier, without wayfire. I made a few other changes to sommelier too: - I switched to the latest version, which provides meson instead of common-mk for building. Also, they removed the demos and got rid of some bogus dependencies. That simplified the build a lot! - They switched to the stable XDG protocols, but then reverted it again. I unreverted it to get things going again. Not sure if I did it right (they migrated from C to C++ so the patch didn't apply directly). - I added xwayland to the VM and sommelier command, allowing X applications to run in the VM. - By default sommelier runs the program with an already-open socket, which doesn't work if the program (or its children) want to open multiple connections. I was able to fix that by using `--parent` mode, and getting rid of PEER_CMD_PREFIX (which just adds some chromium paths preventing it from working). - Note: in `--parent` mode it waits for the process to exit before processing events on the socket, so if you just run an application directly it will hang. I used `bash -c 'firefox &'` as the command as a work-around. - Some programs (e.g. firefox) refused to start because the protocol versions offered by sommelier were too old. I increased the version numbers and that's working now. It needs doing properly, though. e.g. I implemented the new "sl_host_surface_damage_buffer" by simply calling the old damage function, which is obviously not correct but is working for me so far! - Annoyingly, using `--parent` disables xwayland support. Maybe we should run xwayland manually, or use a second sommelier instance? In general, sommelier seems quite buggy and annoying. I guess it will need updating constantly to proxy every new wayland protocol. Yet it can't add any useful security because it runs inside the VM, and is therefore untrusted. Some other changes that I found useful: - I added the generated kernel modules directory to rootfs, which allows using all the normal features of Linux (e.g. ext4) in the VM. - I switched from `bash` to `bashInteractive` as the VM shell, which gets cursor keys working. - I wrote a Nix package to generate one script for each of my old qubes. So e.g. I can now run `qvm-start-shopping` to start my crosvm shopping instance, with its own /home LVM partition and IP address. It passes the network configuration using some new kernel parameters (alongside spectrumcmd). - I put each VM on its own point-to-point virtual network. These networks are set up by /etc/nixos/configuration.nix. That works well for my qubes-like VMs, though I guess spectrum will need something more dynamic. - I enabled the shared filesystem (VIRTIO_FS), which works nicely. I use it to provide a (separate) shared directory to each VM that I can access from the host. One problem is that the crosvm driver runs in a minijail with a uidmap that makes every file appear to be owned by root, so only root can write things in the VM. Possibly a newer kernel would help; later versions of the kernel docs say you can include any normal FUSE flags here, so mounting with `uid=1000` might work. - Finally, I added a `vm-halt` command that just calls `reboot`, as I don't want to develop the habit of typing `reboot` without thinking ;-) If any of this sounds useful for spectrum let me know. I can try and tidy it up; it's all a huge mess at the moment! Once this is working more smoothly, I guess the next issues will be setting up some kind of secure window manager on the host (e.g. labelling windows with the VM they come from, not allowing screenshots, etc). Would also be good to get sound forwarding working somehow (Qubes routes pulseaudio to all the VMs and gives you a mixer to control the levels for each, but I don't know how that worked). It also needs some kind of VM manager to keep track of which VMs are running. And some kind of IPC system like qrexec would be useful. Do you have thoughts or plans about how to do any of this? On Wed, 20 Jan 2021 at 13:04, Thomas Leonard <talex5(a)gmail.com> wrote: -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC