patches and low-level development discussion
 help / color / mirror / code / Atom feed
* [PATCH nixpkgs 00/16] Inter-guest networking
@ 2021-04-11 11:57 Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 01/16] linux: enable Xen everywhere it can be Alyssa Ross
                   ` (16 more replies)
  0 siblings, 17 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

In Spectrum, we want the host kernel to include as few drivers as
possible, to reduce attack service.  To accomplish this, we need to
move as much hardware interaction as possible into VMs.  This series
introduces proof-of-concept network hardware isolation by passing
through network devices to a VM, and having that VM handle all
interaction with that hardware instead of the host system.


Background
----------

Ideally, the Spectrum host system wouldn't have to have support for
networking at all.  Network hardware could be handled by a VM, which
would act a bit like a router, and export virtual network devices
directly to other VMs.  The hard part of that ideal is the bit where a
VM exports a virtual device to another VM.  There's work going on in
the Linux virtualisation ecosystem to make that possible, through a
protocol called virtio-vhost-user[1].  But it looks like it's going to
be a long time before virtio-vhost-user is baked enough to be useful
to us, and inter-guest networking is so fundamentally important to
Spectrum that we can't afford to wait that long.  So we need a plan B.

Because we have no way to have a VM provide a virtual device, the host
is going to need to do it.  And this means that some amount of
networking is going to have to happen on the host.  But we can still
get most of what we were going for:

 * Pass hardware network devices through to a dedicated VM.
 * Attach a virtual network device to the same VM.
 * Connect htat virtual network device to virtual network devices for
   other VMs on the host.

With this approach, the host is still doing networking, but there's a
lot less code involved, because instead of having loads of drivers for
every kind of Wi-Fi card and so on available on the host, the host
only ever needs to use the driver for the virtual network devices.  If
we connect each network client VM to the router VM using a bridge
(i.e. a virtual Ethernet switch), we don't even need to worry about IP
addresses or routing tables on the host.  (There are some drawbacks to
this approach, which are elaborated later in the series.)

So that's what's implemented here.

[1]: https://wiki.qemu.org/Features/VirtioVhostUser


Implementation
--------------

This series starts with some cherry-picks from upstream Nixpkgs.  (We
should sync with upstream soon, but I didn't want to delay this any
longer by blocking it on a Nixpkgs sync.)  Then there's some important
refactoring to make it possible to fit multiple VM definitions into
spectrumPackages.  (Until now, there's only been one.)  Finally, I
implement two VMs -- one to act as the router and one to be the client
-- as well as a service manager that can set everything up on the host
to be able to run those VMs, and run them.

Don't read too much into the structure of the Nix code for the VMs.  I
just needed _some_ structure, and this was what I came up with.  As we
explore the configuration aspect of Spectrum more, I expect it to
change dramatically.  There's quite a bit of duplication between VM
definitions, but I don't think it's worth spending time getting rid of
that for now when the entire structure could change before it becomes
an issue.  The important thing here is what's going on at runtime on
the host and in the VMs, not the Nix code used to build that.

Everything in the router VM is set up to be able to handle multiple
clients coming and going at runtime (with the exception of a small
issue with clients going away that I'll get into later in the series).
The only reason the client VM is defined up front is because that's
what fits our Nix code.  There's no reason from the networking point
of view that client VMs couldn't be instantiated entirely at runtime.

One thing that might jump out to anybody skimming the series is that
we're using cloud-hypervisor here.  This is because there's a feature
it supports that crosvm doesn't (I'll go into detail about what
exactly later in the series), and it's more expedient to just use both
VMMs as required by the needs of the VMs they're running than it would
be to port features from cloud-hypervisor to crosvm or vice versa.
Obviously using multiple different VMMs on the host is not good in the
long term, but things are so in flux at the moment that it's very
likely that any porting work will no longer be in use by the time we
need to pare down to a single VMM anyway.


Testing
-------

If you want to try this out for yourself (and I encourage you to do so
and reply to me with a Tested-by!), here's what you need to do:

 1. Identify the PCI location of a physical Ethernet device on your
    system.  `lspci -n' should help with this.

 2. Modify pkgs/os-specific/linux/spectrum/testhost/default.nix to
    define PCI_LOCATION to be your one.  In future, we can either have
    a configuration option for this sort of thing, or try to figure it
    out at runtime.

 3. Start the service manager:

    	sudo env XDG_RUNTIME_DIR=/run $(nix-build -A spectrumPackages.spectrum-testhost)/bin/spectrum-testhost

    A temporary directory will be created for service manager state.
    Its location will be printed to the terminal.

 4. At another terminal, tell the service manager to start the
    "application VM".  This is the client that will be connected to
    the router VM.

    	s6-rc -u -l /run/spectrum.sy2huQuC3x/s6-rc/live change vm-app

    Remember to substitute in your temporary directory.

    You'll need the s6-rc command available (it's in Nixpkgs).  This
    tells the service manager to start the service named vm-app.  This
    service has a dependency on another VM called vm-net, which is the
    router VM that deals with the network hardware, so both will be
    started when you ask s6-rc to start vm-app.

 5. The terminal running the service manager will be connected to the
    serial console of client VM.  After a few seconds (because the
    router VM will probably have to sort out DHCP first), you should
    be able to ping hosts on the internet.  Note that the client VM
    doesn't come with DNS set up.

 6. Once you're done, you can tell the service manager to stop the
    VMs:

    	s6-rc -da -l /run/spectrum.sy2huQuC3x/s6-rc/live change

    Again, remember to substitute your temporary directory.

    The service manager itself won't respond to ^C.  I believe this is
    handled better in more recent versions of s6, which we'll get when
    we sync Nixpkgs.

    After this, it's a good idea to reboot, to restore all the network
    devices to their default state.


Alyssa Ross (16):
  linux: enable Xen everywhere it can be
  cloud-hypervisor: 0.8.0 -> 0.14.1
  mdevd: init at 0.1.3.0
  spectrumPackages.linux_vm: fix cloud-hypervisor hotplug
  spectrumPackages.linux_vm: allow config overrides
  crosvm: support setting guest MAC from --tap-fd
  spectrumPackages: export makeRootfs
  spectrumPackages.rootfs: add s6-rc support
  spectrumPackages.rootfs: make /var/lib and /var/run
  spectrumPackages.rootfs: add dbus configuration
  spectrumPackages.rootfs: add connman dbus services
  spectrumPackages.sys-vms.comp: init
  spectrumPackages.makeRootfs: move to default.nix
  spectrumPackages.sys-vms.net: init
  spectrumPackages.sys-vms.app: init
  spectrumPackages.spectrum-testhost: init

 .../cargo-lock-vendor-fix.patch               |  53 ----
 .../cloud-hypervisor/default.nix              |  15 +-
 ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++
 .../linux/chromium-os/crosvm/default.nix      |   1 +
 .../linux/kernel/common-config.nix            |  13 +-
 pkgs/os-specific/linux/kernel/patches.nix     |   9 +
 pkgs/os-specific/linux/mdevd/default.nix      |  28 ++
 pkgs/os-specific/linux/spectrum/default.nix   |   6 +-
 pkgs/os-specific/linux/spectrum/linux/vm.nix  |   7 +-
 .../linux/spectrum/rootfs/default.nix         |  92 +++---
 .../linux/spectrum/rootfs/etc/group           |   1 +
 .../linux/spectrum/rootfs/etc/passwd          |   1 +
 .../linux/spectrum/rootfs/generic.nix         |  48 ---
 .../linux/spectrum/rootfs/rc-services.nix     |  26 ++
 .../linux/spectrum/rootfs/stage1.nix          |  25 +-
 .../linux/spectrum/spectrum-vm/default.nix    |   6 +-
 .../linux/spectrum/testhost/default.nix       | 205 ++++++++++++
 .../linux/spectrum/vm/app/default.nix         |  63 ++++
 .../linux/spectrum/vm/comp/default.nix        |  86 +++++
 .../os-specific/linux/spectrum/vm/default.nix |   9 +
 .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++
 pkgs/top-level/aliases.nix                    |   6 +
 pkgs/top-level/all-packages.nix               |  12 +-
 23 files changed, 976 insertions(+), 195 deletions(-)
 delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch
 create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
 create mode 100644 pkgs/os-specific/linux/mdevd/default.nix
 delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/generic.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix

-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 01/16] linux: enable Xen everywhere it can be
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 02/16] cloud-hypervisor: 0.8.0 -> 0.14.1 Alyssa Ross
                   ` (15 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

I don't think there's any reason to have a seperate kernel variant
because of this, with all the maintenance burden that imposes.  Debian
and Fedora both enable all these options on their normal kernels.

Alias the Linux Xen attributes, so this change should be seemless for
people who were using the Xen kernels up to now.

All the Xen options are marked as optional anyway, so it should be
fine to try to enable them on non-x86 platforms as well.

Cherry-picking this for Spectrum because PVH is cloud-hypervisor's
preferred boot entry point on x86, and it'll complain if our kernel
doesn't support it.

Fixes: https://github.com/NixOS/nixpkgs/issues/115182

(cherry picked from commit 5f24024de6f6405bb9ae9fc4c960b3fba84613b9)
---
 pkgs/os-specific/linux/kernel/common-config.nix | 13 ++++---------
 pkgs/top-level/aliases.nix                      |  6 ++++++
 pkgs/top-level/all-packages.nix                 |  9 ---------
 3 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/pkgs/os-specific/linux/kernel/common-config.nix b/pkgs/os-specific/linux/kernel/common-config.nix
index e80e29aad3d..c0da19dd391 100644
--- a/pkgs/os-specific/linux/kernel/common-config.nix
+++ b/pkgs/os-specific/linux/kernel/common-config.nix
@@ -12,7 +12,7 @@
 # Configuration
 { stdenv, version
 
-, features ? { grsecurity = false; xen_dom0 = false; }
+, features ? { grsecurity = false; }
 }:
 
 with stdenv.lib;
@@ -505,13 +505,8 @@ let
       VBOXGUEST = option no;
       DRM_VBOXVIDEO = option no;
 
-    } // optionalAttrs (stdenv.isx86_64 || stdenv.isi686) ({
-      XEN = option yes;
-
-      # XXX: why isn't this in the xen-dom0 conditional section below?
-      XEN_DOM0 = option yes;
-
-    } // optionalAttrs features.xen_dom0 {
+      XEN                         = option yes;
+      XEN_DOM0                    = option yes;
       PCI_XEN                     = option yes;
       HVC_XEN                     = option yes;
       HVC_XEN_FRONTEND            = option yes;
@@ -530,7 +525,7 @@ let
       XEN_SELFBALLOONING          = option yes;
       XEN_STUB                    = option yes;
       XEN_TMEM                    = option yes;
-    });
+    };
 
     media = {
       MEDIA_DIGITAL_TV_SUPPORT = yes;
diff --git a/pkgs/top-level/aliases.nix b/pkgs/top-level/aliases.nix
index b020d6fac55..75dd4ab76ca 100644
--- a/pkgs/top-level/aliases.nix
+++ b/pkgs/top-level/aliases.nix
@@ -276,6 +276,12 @@ mapAliases ({
   linuxPackages_testing_hardened = throw "linuxPackages_testing_hardened has been removed, please use linuxPackages_latest_hardened";
   linux_testing_hardened = throw "linux_testing_hardened has been removed, please use linux_latest_hardened";
 
+  # added 2021-04-04
+  linuxPackages_xen_dom0 = linuxPackages;
+  linuxPackages_latest_xen_dom0 = linuxPackages_latest;
+  linuxPackages_xen_dom0_hardened = linuxPackages_hardened;
+  linuxPackages_latest_xen_dom0_hardened = linuxPackages_latest_hardened;
+
   linux-steam-integration = throw "linux-steam-integration has been removed, as the upstream project has been abandoned"; # added 2020-05-22
   loadcaffe = throw "loadcaffe has been removed, as the upstream project has been abandoned"; # added 2020-03-28
   lttngTools = lttng-tools;  # added 2014-07-31
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index eb0f87ab3f2..0003282d18a 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -17807,11 +17807,6 @@ in
   # Build a kernel with bcachefs module
   linuxPackages_testing_bcachefs = recurseIntoAttrs (linuxPackagesFor pkgs.linux_testing_bcachefs);
 
-  # Build a kernel for Xen dom0
-  linuxPackages_xen_dom0 = recurseIntoAttrs (linuxPackagesFor (pkgs.linux.override { features.xen_dom0=true; }));
-
-  linuxPackages_latest_xen_dom0 = recurseIntoAttrs (linuxPackagesFor (pkgs.linux_latest.override { features.xen_dom0=true; }));
-
   # Hardened Linux
   hardenedLinuxPackagesFor = kernel': overrides:
     let # Note: We use this hack since the hardened patches can lag behind and we don't want to delay updates:
@@ -17835,10 +17830,6 @@ in
   linuxPackages_latest_hardened = recurseIntoAttrs (hardenedLinuxPackagesFor pkgs.linux_latest { });
   linux_latest_hardened = linuxPackages_latest_hardened.kernel;
 
-  linuxPackages_xen_dom0_hardened = recurseIntoAttrs (hardenedLinuxPackagesFor pkgs.linux { features.xen_dom0=true; });
-
-  linuxPackages_latest_xen_dom0_hardened = recurseIntoAttrs (hardenedLinuxPackagesFor pkgs.linux_latest { features.xen_dom0=true; });
-
   # Hardkernel (Odroid) kernels.
   linuxPackages_hardkernel_4_14 = recurseIntoAttrs (linuxPackagesFor pkgs.linux_hardkernel_4_14);
   linuxPackages_hardkernel_latest = linuxPackages_hardkernel_4_14;
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 02/16] cloud-hypervisor: 0.8.0 -> 0.14.1
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 01/16] linux: enable Xen everywhere it can be Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 03/16] mdevd: init at 0.1.3.0 Alyssa Ross
                   ` (14 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

(cherry picked from commit 5fa10aa313f16e2afedefc58806d1c0a40ed08df)
---
 .../cargo-lock-vendor-fix.patch               | 53 -------------------
 .../cloud-hypervisor/default.nix              | 15 +++---
 2 files changed, 7 insertions(+), 61 deletions(-)
 delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch

diff --git a/pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch b/pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch
deleted file mode 100644
index 7a7378faa58..00000000000
--- a/pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch
+++ /dev/null
@@ -1,53 +0,0 @@
-diff --git a/Cargo.lock b/Cargo.lock
-index e566ed25..a661a963 100644
---- a/Cargo.lock
-+++ b/Cargo.lock
-@@ -689,7 +689,7 @@ dependencies = [
-  "serde",
-  "serde_derive",
-  "serde_json",
-- "vfio-bindings 0.2.0 (git+https://github.com/rust-vmm/vfio-bindings)",
-+ "vfio-bindings",
-  "vfio-ioctls",
-  "vm-allocator",
-  "vm-device",
-@@ -1346,17 +1346,12 @@ checksum = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191"
- [[package]]
- name = "vfio-bindings"
- version = "0.2.0"
--source = "git+https://github.com/rust-vmm/vfio-bindings#f08cbcbf4041c981441d9c036c49ebad5098ed1c"
-+source = "registry+https://github.com/rust-lang/crates.io-index"
-+checksum = "4a21f546f2bda37f5a8cfb138c87f95b8e34d2d78d6a7a92ba3785f4e08604a7"
- dependencies = [
-  "vmm-sys-util",
- ]
- 
--[[package]]
--name = "vfio-bindings"
--version = "0.2.0"
--source = "registry+https://github.com/rust-lang/crates.io-index"
--checksum = "4a21f546f2bda37f5a8cfb138c87f95b8e34d2d78d6a7a92ba3785f4e08604a7"
--
- [[package]]
- name = "vfio-ioctls"
- version = "0.1.0"
-@@ -1366,7 +1361,7 @@ dependencies = [
-  "kvm-bindings",
-  "kvm-ioctls",
-  "log 0.4.8",
-- "vfio-bindings 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
-+ "vfio-bindings",
-  "vm-memory",
-  "vmm-sys-util",
- ]
-diff --git a/pci/Cargo.toml b/pci/Cargo.toml
-index 9c6955c7..4ecf8e6b 100644
---- a/pci/Cargo.toml
-+++ b/pci/Cargo.toml
-@@ -23,5 +23,5 @@ vm-memory = "0.2.1"
- vm-migration = { path = "../vm-migration" }
- 
- [dependencies.vfio-bindings]
--git = "https://github.com/rust-vmm/vfio-bindings"
-+version = "0.2.0"
- features = ["fam-wrappers"]
diff --git a/pkgs/applications/virtualization/cloud-hypervisor/default.nix b/pkgs/applications/virtualization/cloud-hypervisor/default.nix
index 59c97c4a8de..4abd9fe1ab9 100644
--- a/pkgs/applications/virtualization/cloud-hypervisor/default.nix
+++ b/pkgs/applications/virtualization/cloud-hypervisor/default.nix
@@ -1,28 +1,27 @@
-{ lib, fetchFromGitHub, rustPlatform, pkgconfig, openssl }:
+{ lib, stdenv, fetchFromGitHub, rustPlatform, pkgconfig, dtc, openssl }:
 
 rustPlatform.buildRustPackage rec {
   pname = "cloud-hypervisor";
-  version = "0.8.0";
+  version = "0.14.1";
 
   src = fetchFromGitHub {
     owner = "cloud-hypervisor";
     repo = pname;
     rev = "v${version}";
-    sha256 = "h2aWWjycTm84TS89/vhqnAvwOqeeSDtvvCt+Is6I0eI=";
+    sha256 = "0pnfg6dzpz8v40cwg3dmlj52x8pblavv7mkczar814dwbk01y7vr";
   };
 
   nativeBuildInputs = [ pkgconfig ];
-  buildInputs = [ openssl ];
+  buildInputs = [ openssl ] ++ lib.optional stdenv.isAarch64 dtc;
 
-  cargoPatches = [ ./cargo-lock-vendor-fix.patch ];
-  cargoSha256 = "fOIB+qVDqAAgQPW3bK2NfST24GzYJeRXgaMFXyNPcPQ=";
+  cargoSha256 = "0dbjds40znly11i0ssfv66w82ynxp00ixw1349m5ln9i9ms94sr4";
 
   meta = with lib; {
     homepage = "https://github.com/cloud-hypervisor/cloud-hypervisor";
     description = "Open source Virtual Machine Monitor (VMM) that runs on top of KVM";
     changelog = "https://github.com/cloud-hypervisor/cloud-hypervisor/releases/tag/v${version}";
     license = with licenses; [ asl20 bsd3 ];
-    maintainers = with maintainers; [ offline ];
-    platforms = [ "x86_64-linux" ];
+    maintainers = with maintainers; [ offline qyliss ];
+    platforms = [ "aarch64-linux" "x86_64-linux" ];
   };
 }
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 03/16] mdevd: init at 0.1.3.0
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 01/16] linux: enable Xen everywhere it can be Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 02/16] cloud-hypervisor: 0.8.0 -> 0.14.1 Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 04/16] spectrumPackages.linux_vm: fix cloud-hypervisor hotplug Alyssa Ross
                   ` (13 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

By cherry-picking this for Spectrum, we're actually building mdevd in
an unsupported configuration, because it is documented as requiring a
newer version of skalibs than we have.  But it seems to work fine, and
I think we'll be updating soon anyway.

(cherry picked from commit 441f0c894aae2846190724828054e4bad9284579)
---
 pkgs/os-specific/linux/mdevd/default.nix | 28 ++++++++++++++++++++++++
 pkgs/top-level/all-packages.nix          |  3 +++
 2 files changed, 31 insertions(+)
 create mode 100644 pkgs/os-specific/linux/mdevd/default.nix

diff --git a/pkgs/os-specific/linux/mdevd/default.nix b/pkgs/os-specific/linux/mdevd/default.nix
new file mode 100644
index 00000000000..b88e3ad1e6f
--- /dev/null
+++ b/pkgs/os-specific/linux/mdevd/default.nix
@@ -0,0 +1,28 @@
+{ lib, skawarePackages }:
+
+with skawarePackages;
+
+buildPackage {
+  pname = "mdevd";
+  version = "0.1.3.0";
+  sha256 = "0spvw27xxd0m6j8bl8xysmgsx18fl769smr6dsh25s2d5h3sp2dy";
+
+  description = "mdev-compatible Linux hotplug manager daemon";
+  platforms = lib.platforms.linux;
+
+  outputs = [ "bin" "out" "dev" "doc" ];
+
+  configureFlags = [
+    "--with-sysdeps=${skalibs.lib}/lib/skalibs/sysdeps"
+    "--with-include=${skalibs.dev}/include"
+    "--with-lib=${skalibs.lib}/lib"
+  ];
+
+  postInstall = ''
+    # remove all mdevd executables from build directory
+    rm $(find -type f -mindepth 1 -maxdepth 1 -executable)
+
+    mv doc $doc/share/doc/mdevd/html
+    mv examples $doc/share/doc/mdevd/examples
+  '';
+}
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 0003282d18a..c22a62456db 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -15145,6 +15145,7 @@ in
     s6-portable-utils = callPackage ../tools/misc/s6-portable-utils { };
     s6-rc = callPackage ../tools/system/s6-rc { };
 
+    mdevd = callPackage ../os-specific/linux/mdevd { };
     nsss = callPackage ../development/libraries/nsss { };
     utmps = callPackage ../development/libraries/utmps { };
     sdnotify-wrapper = callPackage ../os-specific/linux/sdnotify-wrapper { };
@@ -17946,6 +17947,8 @@ in
   mdadm = mdadm4;
   mdadm4 = callPackage ../os-specific/linux/mdadm { };
 
+  inherit (skawarePackages) mdevd;
+
   metastore = callPackage ../os-specific/linux/metastore { };
 
   mingetty = callPackage ../os-specific/linux/mingetty { };
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 04/16] spectrumPackages.linux_vm: fix cloud-hypervisor hotplug
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (2 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 03/16] mdevd: init at 0.1.3.0 Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 05/16] spectrumPackages.linux_vm: allow config overrides Alyssa Ross
                   ` (12 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This patch (backported from Linux 5.5) fixes hotplugging with
cloud-hypervisor.

See <https://github.com/cloud-hypervisor/cloud-hypervisor/blob/8b7aafad16ee8a23d7178a7f7aa6d2c1ec509b5d/docs/hotplug.md#kernel-support>.
---
 pkgs/os-specific/linux/kernel/patches.nix    | 9 +++++++++
 pkgs/os-specific/linux/spectrum/linux/vm.nix | 5 ++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/pkgs/os-specific/linux/kernel/patches.nix b/pkgs/os-specific/linux/kernel/patches.nix
index 8ce1ac2b587..a802a0a24d7 100644
--- a/pkgs/os-specific/linux/kernel/patches.nix
+++ b/pkgs/os-specific/linux/kernel/patches.nix
@@ -92,4 +92,13 @@
     name = "mac_nvme_t2";
     patch = ./mac-nvme-t2.patch;
   };
+
+  # https://github.com/cloud-hypervisor/cloud-hypervisor/blob/8b7aafad16ee8a23d7178a7f7aa6d2c1ec509b5d/docs/hotplug.md#kernel-support
+  evged = {
+    name = "evged";
+    patch = fetchpatch {
+      url = "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0";
+      sha256 = "0hzl3s2fmsvz54hq478lqi14dwry8a5g2b2awsssxf0r2ga15jk5";
+    };
+  };
 }
diff --git a/pkgs/os-specific/linux/spectrum/linux/vm.nix b/pkgs/os-specific/linux/spectrum/linux/vm.nix
index 9c9ba78b691..f456053dc99 100644
--- a/pkgs/os-specific/linux/spectrum/linux/vm.nix
+++ b/pkgs/os-specific/linux/spectrum/linux/vm.nix
@@ -1,8 +1,11 @@
-{ lib, linux }:
+{ lib, linux, kernelPatches }:
 
 with lib.kernel;
 
 linux.override {
+  # Fix hotplugging with cloud-hypervisor.
+  kernelPatches = linux.kernelPatches ++ [ kernelPatches.evged ];
+
   structuredExtraConfig = {
     VIRTIO_PCI = yes;
     VIRTIO_BLK = yes;
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 05/16] spectrumPackages.linux_vm: allow config overrides
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (3 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 04/16] spectrumPackages.linux_vm: fix cloud-hypervisor hotplug Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 06/16] crosvm: support setting guest MAC from --tap-fd Alyssa Ross
                   ` (11 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This way, linux_vm's configuration can be overridden further when it's
used, for example for VM-specific configuration.
---
 pkgs/os-specific/linux/spectrum/linux/vm.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pkgs/os-specific/linux/spectrum/linux/vm.nix b/pkgs/os-specific/linux/spectrum/linux/vm.nix
index f456053dc99..d6911cf4a81 100644
--- a/pkgs/os-specific/linux/spectrum/linux/vm.nix
+++ b/pkgs/os-specific/linux/spectrum/linux/vm.nix
@@ -1,4 +1,4 @@
-{ lib, linux, kernelPatches }:
+{ lib, linux, kernelPatches, structuredExtraConfig ? {} }:
 
 with lib.kernel;
 
@@ -24,5 +24,5 @@ linux.override {
     NET_9P = yes;
     NET_9P_VIRTIO = yes;
     "9P_FS" = yes;
-  };
+  } // structuredExtraConfig;
 }
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 06/16] crosvm: support setting guest MAC from --tap-fd
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (4 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 05/16] spectrumPackages.linux_vm: allow config overrides Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 07/16] spectrumPackages: export makeRootfs Alyssa Ross
                   ` (10 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This will be important for host-based networking in Spectrum.
---
 ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++
 .../linux/chromium-os/crosvm/default.nix      |   1 +
 2 files changed, 295 insertions(+)
 create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch

diff --git a/pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch b/pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
new file mode 100644
index 00000000000..df0f16ee23a
--- /dev/null
+++ b/pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
@@ -0,0 +1,294 @@
+From 2db1db4e42e87f05e414384a0c09be340e81d94d Mon Sep 17 00:00:00 2001
+From: Alyssa Ross <hi@alyssa.is>
+Date: Sun, 27 Sep 2020 15:34:02 +0000
+Subject: [PATCH] crosvm: support setting guest MAC from tap-fd
+
+This adds a mac= option to crosvm's --tap-fd option.  The virtio-net
+driver in the guest will read the desired MAC from virtio
+configuration space.
+
+See the documentation for VIRTIO_NET_F_MAC in the Virtio spec[1].
+
+[1]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
+
+Thanks-to: Puck Meerburg <puck@puckipedia.com>
+
+---
+ devices/src/virtio/net.rs | 31 ++++++++++++++++---
+ src/crosvm.rs             |  8 +++--
+ src/linux.rs              | 20 +++++++-----
+ src/main.rs               | 64 ++++++++++++++++++++++++++++++---------
+ 4 files changed, 96 insertions(+), 27 deletions(-)
+
+diff --git a/devices/src/virtio/net.rs b/devices/src/virtio/net.rs
+index 44a39abd..fe71371f 100644
+--- a/devices/src/virtio/net.rs
++++ b/devices/src/virtio/net.rs
+@@ -22,7 +22,9 @@ use virtio_sys::virtio_net::{
+ };
+ use virtio_sys::{vhost, virtio_net};
+ 
+-use super::{DescriptorError, Interrupt, Queue, Reader, VirtioDevice, Writer, TYPE_NET};
++use super::{
++    copy_config, DescriptorError, Interrupt, Queue, Reader, VirtioDevice, Writer, TYPE_NET,
++};
+ 
+ const QUEUE_SIZE: u16 = 256;
+ const NUM_QUEUES: usize = 3;
+@@ -373,7 +375,13 @@ where
+     }
+ }
+ 
++#[derive(Default)]
++pub struct NetOptions {
++    pub guest_mac: Option<net_util::MacAddress>,
++}
++
+ pub struct Net<T: TapT> {
++    config: Vec<u8>,
+     workers_kill_evt: Option<EventFd>,
+     kill_evt: EventFd,
+     worker_thread: Option<thread::JoinHandle<Worker<T>>>,
+@@ -392,6 +400,7 @@ where
+         ip_addr: Ipv4Addr,
+         netmask: Ipv4Addr,
+         mac_addr: MacAddress,
++        options: NetOptions,
+     ) -> Result<Net<T>, NetError> {
+         let tap: T = T::new(true).map_err(NetError::TapOpen)?;
+         tap.set_ip_addr(ip_addr).map_err(NetError::TapSetIp)?;
+@@ -401,18 +410,18 @@ where
+ 
+         tap.enable().map_err(NetError::TapEnable)?;
+ 
+-        Net::from(tap)
++        Net::with_tap(tap, options)
+     }
+ 
+     /// Creates a new virtio network device from a tap device that has already been
+     /// configured.
+-    pub fn from(tap: T) -> Result<Net<T>, NetError> {
++    pub fn with_tap(tap: T, options: NetOptions) -> Result<Net<T>, NetError> {
+         // This would also validate a tap created by Self::new(), but that's a good thing as it
+         // would ensure that any changes in the creation procedure are matched in the validation.
+         // Plus we still need to set the offload and vnet_hdr_size values.
+         validate_and_configure_tap(&tap)?;
+ 
+-        let avail_features = 1 << virtio_net::VIRTIO_NET_F_GUEST_CSUM
++        let mut avail_features = 1 << virtio_net::VIRTIO_NET_F_GUEST_CSUM
+             | 1 << virtio_net::VIRTIO_NET_F_CSUM
+             | 1 << virtio_net::VIRTIO_NET_F_CTRL_VQ
+             | 1 << virtio_net::VIRTIO_NET_F_CTRL_GUEST_OFFLOADS
+@@ -422,8 +431,18 @@ where
+             | 1 << virtio_net::VIRTIO_NET_F_HOST_UFO
+             | 1 << vhost::VIRTIO_F_VERSION_1;
+ 
++        if options.guest_mac.is_some() {
++            avail_features |= 1 << virtio_net::VIRTIO_NET_F_MAC;
++        }
++
++        let config = options
++            .guest_mac
++            .map(|mac| mac.octets().to_vec())
++            .unwrap_or_default();
++
+         let kill_evt = EventFd::new().map_err(NetError::CreateKillEventFd)?;
+         Ok(Net {
++            config,
+             workers_kill_evt: Some(kill_evt.try_clone().map_err(NetError::CloneKillEventFd)?),
+             kill_evt,
+             worker_thread: None,
+@@ -545,6 +564,10 @@ where
+         }
+     }
+ 
++    fn read_config(&self, offset: u64, data: &mut [u8]) {
++        copy_config(data, 0, self.config.as_slice(), offset);
++    }
++
+     fn activate(
+         &mut self,
+         mem: GuestMemory,
+diff --git a/src/crosvm.rs b/src/crosvm.rs
+index 81344c32..e69f2dfc 100644
+--- a/src/crosvm.rs
++++ b/src/crosvm.rs
+@@ -157,6 +157,10 @@ impl Default for SharedDir {
+     }
+ }
+ 
++pub struct TapFdOptions {
++    pub mac: Option<net_util::MacAddress>,
++}
++
+ /// Aggregate of all configurable options for a running VM.
+ pub struct Config {
+     pub vcpu_count: Option<u32>,
+@@ -177,7 +181,7 @@ pub struct Config {
+     pub netmask: Option<net::Ipv4Addr>,
+     pub mac_address: Option<net_util::MacAddress>,
+     pub vhost_net: bool,
+-    pub tap_fd: Vec<RawFd>,
++    pub tap_fd: BTreeMap<RawFd, TapFdOptions>,
+     pub cid: Option<u64>,
+     pub wayland_socket_paths: BTreeMap<String, PathBuf>,
+     pub wayland_dmabuf: bool,
+@@ -224,7 +228,7 @@ impl Default for Config {
+             netmask: None,
+             mac_address: None,
+             vhost_net: false,
+-            tap_fd: Vec::new(),
++            tap_fd: BTreeMap::new(),
+             cid: None,
+             #[cfg(feature = "gpu")]
+             gpu_parameters: None,
+diff --git a/src/linux.rs b/src/linux.rs
+index 3370c1e1..f7f78ad2 100644
+--- a/src/linux.rs
++++ b/src/linux.rs
+@@ -60,7 +60,9 @@ use vm_control::{
+     VmMemoryRequest, VmMemoryResponse, VmRunMode,
+ };
+ 
+-use crate::{Config, DiskOption, Executable, SharedDir, SharedDirKind, TouchDeviceOption};
++use crate::{
++    Config, DiskOption, Executable, SharedDir, SharedDirKind, TapFdOptions, TouchDeviceOption,
++};
+ use arch::{self, LinuxArch, RunnableLinuxVm, VirtioDeviceStub, VmComponents, VmImage};
+ 
+ #[cfg(any(target_arch = "arm", target_arch = "aarch64"))]
+@@ -586,14 +588,18 @@ fn create_balloon_device(cfg: &Config, socket: BalloonControlResponseSocket) ->
+     })
+ }
+ 
+-fn create_tap_net_device(cfg: &Config, tap_fd: RawFd) -> DeviceResult {
++fn create_tap_net_device(cfg: &Config, tap_fd: RawFd, options: &TapFdOptions) -> DeviceResult {
+     // Safe because we ensure that we get a unique handle to the fd.
+     let tap = unsafe {
+         Tap::from_raw_fd(validate_raw_fd(tap_fd).map_err(Error::ValidateRawFd)?)
+             .map_err(Error::CreateTapDevice)?
+     };
+ 
+-    let dev = virtio::Net::from(tap).map_err(Error::NetDeviceNew)?;
++    let net_opts = virtio::NetOptions {
++        guest_mac: options.mac,
++    };
++
++    let dev = virtio::Net::with_tap(tap, net_opts).map_err(Error::NetDeviceNew)?;
+ 
+     Ok(VirtioDeviceStub {
+         dev: Box::new(dev),
+@@ -614,8 +620,8 @@ fn create_net_device(
+                 .map_err(Error::VhostNetDeviceNew)?;
+         Box::new(dev) as Box<dyn VirtioDevice>
+     } else {
+-        let dev =
+-            virtio::Net::<Tap>::new(host_ip, netmask, mac_address).map_err(Error::NetDeviceNew)?;
++        let dev = virtio::Net::<Tap>::new(host_ip, netmask, mac_address, Default::default())
++            .map_err(Error::NetDeviceNew)?;
+         Box::new(dev) as Box<dyn VirtioDevice>
+     };
+ 
+@@ -1006,8 +1012,8 @@ fn create_virtio_devices(
+     devs.push(create_balloon_device(cfg, balloon_device_socket)?);
+ 
+     // We checked above that if the IP is defined, then the netmask is, too.
+-    for tap_fd in &cfg.tap_fd {
+-        devs.push(create_tap_net_device(cfg, *tap_fd)?);
++    for (tap_fd, options) in &cfg.tap_fd {
++        devs.push(create_tap_net_device(cfg, *tap_fd, options)?);
+     }
+ 
+     if let (Some(host_ip), Some(netmask), Some(mac_address)) =
+diff --git a/src/main.rs b/src/main.rs
+index 3afca8e0..053af465 100644
+--- a/src/main.rs
++++ b/src/main.rs
+@@ -21,7 +21,8 @@ use arch::Pstore;
+ use audio_streams::StreamEffect;
+ use crosvm::{
+     argument::{self, print_help, set_arguments, Argument},
+-    linux, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TouchDeviceOption,
++    linux, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TapFdOptions,
++    TouchDeviceOption,
+ };
+ #[cfg(feature = "gpu")]
+ use devices::virtio::gpu::{GpuMode, GpuParameters};
+@@ -1041,17 +1042,52 @@ fn set_argument(cfg: &mut Config, name: &str, value: Option<&str>) -> argument::
+         }
+         "vhost-net" => cfg.vhost_net = true,
+         "tap-fd" => {
+-            cfg.tap_fd.push(
+-                value
+-                    .unwrap()
+-                    .parse()
+-                    .map_err(|_| argument::Error::InvalidValue {
+-                        value: value.unwrap().to_owned(),
+-                        expected: String::from(
+-                            "this value for `tap-fd` must be an unsigned integer",
+-                        ),
+-                    })?,
+-            );
++            let mut components = value.unwrap().split(',');
++
++            let fd: RawFd = components
++                .next()
++                .and_then(|x| x.parse().ok())
++                .ok_or_else(|| argument::Error::InvalidValue {
++                    value: value.unwrap().to_owned(),
++                    expected: String::from("this value for `tap-fd` must be an unsigned integer"),
++                })?;
++
++            let mut mac = None;
++            for c in components {
++                let mut kv = c.splitn(2, '=');
++                let (kind, value) = match (kv.next(), kv.next()) {
++                    (Some(kind), Some(value)) => (kind, value),
++                    _ => {
++                        return Err(argument::Error::InvalidValue {
++                            value: c.to_owned(),
++                            expected: String::from("option must be of the form `kind=value`"),
++                        })
++                    }
++                };
++                match kind {
++                    "mac" => {
++                        mac = Some(value.parse().map_err(|_| argument::Error::InvalidValue {
++                            value: value.to_owned(),
++                            expected: String::from(
++                                "`mac` needs to be in the form \"XX:XX:XX:XX:XX:XX\"",
++                            ),
++                        })?)
++                    }
++                    _ => {
++                        return Err(argument::Error::InvalidValue {
++                            value: kind.to_owned(),
++                            expected: String::from("unrecognized option"),
++                        })
++                    }
++                }
++            }
++            if cfg.tap_fd.contains_key(&fd) {
++                return Err(argument::Error::TooManyArguments(format!(
++                    "TAP FD already used: '{}'",
++                    name
++                )));
++            }
++            cfg.tap_fd.insert(fd, TapFdOptions { mac });
+         }
+         #[cfg(feature = "gpu")]
+         "gpu" => {
+@@ -1295,8 +1331,8 @@ writeback=BOOL - Indicates whether the VM can use writeback caching (default: fa
+           Argument::value("plugin-gid-map-file", "PATH", "Path to the file listing supplemental GIDs that should be mapped in plugin jail.  Can be given more than once."),
+           Argument::flag("vhost-net", "Use vhost for networking."),
+           Argument::value("tap-fd",
+-                          "fd",
+-                          "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given."),
++                          "FD[,mac=MAC]",
++                          "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given. MAC is the MAC address that will be set in the guest."),
+           #[cfg(feature = "gpu")]
+           Argument::flag_or_value("gpu",
+                                   "[width=INT,height=INT]",
+-- 
+2.27.0
+
diff --git a/pkgs/os-specific/linux/chromium-os/crosvm/default.nix b/pkgs/os-specific/linux/chromium-os/crosvm/default.nix
index d64dc316772..d54c050d308 100644
--- a/pkgs/os-specific/linux/chromium-os/crosvm/default.nix
+++ b/pkgs/os-specific/linux/chromium-os/crosvm/default.nix
@@ -39,6 +39,7 @@ in
 
     patches = [
       ./default-seccomp-policy-dir.diff
+      ./0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
     ];
 
     cargoSha256 = "0wzqn2n4vyv3bk39079yg1zbnriagi5xns928bzdqmq9djdcj21i";
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 07/16] spectrumPackages: export makeRootfs
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (5 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 06/16] crosvm: support setting guest MAC from --tap-fd Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 08/16] spectrumPackages.rootfs: add s6-rc support Alyssa Ross
                   ` (9 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

We'll want service VMs to be able to call this themselves to construct
their own root filesystems.
---
 pkgs/os-specific/linux/spectrum/default.nix        |  2 ++
 pkgs/os-specific/linux/spectrum/rootfs/default.nix | 13 +++----------
 2 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/pkgs/os-specific/linux/spectrum/default.nix b/pkgs/os-specific/linux/spectrum/default.nix
index ea4fa902e95..ea86dc25c2b 100644
--- a/pkgs/os-specific/linux/spectrum/default.nix
+++ b/pkgs/os-specific/linux/spectrum/default.nix
@@ -8,6 +8,8 @@ let
 
     linux_vm = callPackage ./linux/vm.nix { linux = linux_cros; };
 
+    makeRootfs = callPackage ./rootfs/generic.nix { };
+
     rootfs = callPackage ./rootfs { };
   };
 in
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/default.nix b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
index cd1cccc1a87..6f46ad8054b 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/default.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
@@ -1,16 +1,9 @@
-{ runCommandNoCC, writeScript, writeText, makeFontsConf, writeReferencesToFile
-, lib, dash, busybox, execline, s6, s6-portable-utils, s6-linux-utils
-, s6-linux-init, mesa, squashfs-tools-ng
-, source-code-pro, zsh, emacs26-nox, gcc, wayfire, sommelier, westonLite
+{ writeScript, writeText, lib, makeRootfs
+, busybox, execline, s6, sommelier, source-code-pro, wayfire, zsh
+, gcc, emacs26-nox, westonLite
 }:
 
 let
-  makeRootfs = import ./generic.nix {
-    inherit runCommandNoCC writeScript writeReferencesToFile makeFontsConf lib
-      dash execline s6 s6-portable-utils s6-linux-utils s6-linux-init busybox
-      mesa squashfs-tools-ng;
-  };
-
   path = [
     zsh emacs26-nox gcc wayfire sommelier westonLite busybox s6 execline
   ];
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 08/16] spectrumPackages.rootfs: add s6-rc support
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (6 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 07/16] spectrumPackages: export makeRootfs Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 09/16] spectrumPackages.rootfs: make /var/lib and /var/run Alyssa Ross
                   ` (8 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This allows specifying s6-rc services (which can have dependencies,
and be taken up and down) in the same way that s6 services (which are
merely supervised) are declared.

This removes the spectrumcmd mechanism to clean up the boot process --
it could be reintroduced as an s6-rc service if required.
---
 .../linux/spectrum/rootfs/default.nix         | 43 +++++++++++++++----
 .../linux/spectrum/rootfs/generic.nix         | 15 ++++---
 .../linux/spectrum/rootfs/rc-services.nix     | 26 +++++++++++
 .../linux/spectrum/rootfs/stage1.nix          | 22 +++-------
 4 files changed, 77 insertions(+), 29 deletions(-)
 create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix

diff --git a/pkgs/os-specific/linux/spectrum/rootfs/default.nix b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
index 6f46ad8054b..70ddac4c545 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/default.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
@@ -35,17 +35,44 @@ makeRootfs {
     ${busybox}/bin/getty -i -n -l ${login} 38400 ttyS0
   '';
 
-  run = ''
-    if { chown user /dev/wl0 }
+  rcServices.ok-all = {
+    type = writeText "ok-all-type" ''
+      bundle
+    '';
+    contents = writeText "ok-all-contents" ''
+      wayfire
+    '';
+  };
 
-    s6-applyuidgid -u 1000 -g 1000
-    export XDG_RUNTIME_DIR /run/user/1000
+  rcServices.wayfire = {
+    type = writeText "wayfire-type" ''
+      longrun
+    '';
+    run = writeScript "wayfire-run" ''
+      #! ${execline}/bin/execlineb -S0
 
-    export PATH ${lib.makeBinPath path}
+      s6-applyuidgid -u 1000 -g 1000
 
-    ${sommelier}/bin/sommelier
-    wayfire -c ${wayfireConfig}
-  '';
+      export HOME /
+      export PATH ${lib.makeBinPath path}
+      export XDG_RUNTIME_DIR /run/user/1000
+
+      ${sommelier}/bin/sommelier
+      wayfire -c ${wayfireConfig}
+    '';
+    dependencies = writeText "wayfire-dependencies" ''
+      wl0
+    '';
+  };
+
+  rcServices.wl0 = {
+    type = writeText "wl0-type" ''
+      oneshot
+    '';
+    up = writeText "wl0-run" ''
+      chown user /dev/wl0
+    '';
+  };
 
   fonts = [ source-code-pro ];
 }
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
index 81fb80a614f..4122abf80c6 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
@@ -1,14 +1,14 @@
 { runCommandNoCC, writeScript, writeReferencesToFile, makeFontsConf, lib
-, dash, execline, s6, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox
+, dash, execline, s6, s6-rc, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox
 , mesa, squashfs-tools-ng
 }:
 
-{ services, run, fonts ? [], path ? [] }:
+{ services, rcServices ? {}, fonts ? [], path ? [] }:
 
 let
-  makeStage1 = import ./stage1.nix {
+  stage1 = import ./stage1.nix {
     inherit writeScript lib
-      execline s6 s6-portable-utils s6-linux-utils s6-linux-init busybox mesa
+      execline s6 s6-rc s6-portable-utils s6-linux-utils s6-linux-init busybox mesa
       path;
   };
 
@@ -16,6 +16,10 @@ let
     inherit runCommandNoCC writeScript lib execline;
   };
 
+  makeRcServicesDir = import ./rc-services.nix {
+    inherit runCommandNoCC lib s6-rc;
+  };
+
   fontsConf = makeFontsConf { fontDirectories = fonts; };
 
   squashfs = runCommandNoCC "root-squashfs" {} ''
@@ -34,7 +38,7 @@ let
 
     mkdir bin sbin dev proc run sys tmp
     ln -s ${dash}/bin/dash bin/sh
-    ln -s ${makeStage1 { inherit run; }} sbin/init
+    ln -s ${stage1} sbin/init
     cp -r ${./etc} etc
     chmod u+w etc
 
@@ -43,6 +47,7 @@ let
 
     touch etc/login.defs
     cp -r ${makeServicesDir { inherit services; }} etc/service
+    cp -r ${makeRcServicesDir { services = rcServices; }} etc/s6-rc
   '';
 in
 rootfs
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix b/pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix
new file mode 100644
index 00000000000..4c942189c5e
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix
@@ -0,0 +1,26 @@
+{ runCommandNoCC, lib, s6-rc }:
+
+{ services ? [] }:
+
+let
+  inherit (lib) concatStrings escapeShellArg mapAttrsToList optionalString;
+
+  source = runCommandNoCC "s6-services-source" {} ''
+    mkdir $out
+    ${concatStrings (mapAttrsToList (name: attrs: ''
+      mkdir $out/${name}
+      ${concatStrings (mapAttrsToList (key: value: ''
+        cp ${value} $out/${name}/${key}
+      '') attrs)}
+    '') services)}
+  '';
+
+  s6RcCompile = { fdhuser ? null }: source:
+    runCommandNoCC "s6-rc-compile" {} ''
+      ${s6-rc}/bin/s6-rc-compile \
+        ${optionalString (fdhuser != null) "-h ${escapeShellArg fdhuser}"} \
+        $out ${source}
+    '';
+in
+
+s6RcCompile {} source
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix b/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
index 6caf9ff93b8..13de2d09876 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
@@ -1,13 +1,11 @@
 { writeScript, lib
-, execline, s6, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox, mesa
+, execline, s6, s6-rc, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox, mesa
 , path ? []
 }:
 
-{ run ? "true" }:
-
 let
   path' = path ++ [
-    s6 s6-portable-utils s6-linux-utils s6-linux-init busybox execline
+    s6 s6-rc s6-portable-utils s6-linux-utils s6-linux-init busybox execline
   ];
 in
 
@@ -16,8 +14,6 @@ writeScript "init-stage1" ''
   export PATH ${lib.makeBinPath path'}
   ${s6}/bin/s6-setsid -qb --
 
-  importas -i spectrumcmd spectrumcmd
-
   umask 022
   if { s6-mount -t tmpfs -o mode=0755 tmpfs /run }
   if { s6-hiercopy /etc/service /run/service }
@@ -25,6 +21,9 @@ writeScript "init-stage1" ''
 
   background {
     s6-setsid --
+
+    if { s6-rc-init -c /etc/s6-rc /run/service }
+
     if { s6-mkdir -p /run/user/0 /dev/pts /dev/shm }
     if { install -o user -g user -d /run/user/1000 }
     if { s6-mount -t devpts -o gid=4,mode=620 none /dev/pts }
@@ -33,16 +32,7 @@ writeScript "init-stage1" ''
     if { s6-mount -t sysfs none /sys }
     if { s6-ln -s ${mesa.drivers} /run/opengl-driver }
 
-    export HOME /
-    export XDG_RUNTIME_DIR /run/user/0
-    foreground {
-      ifelse { test -n $spectrumcmd }
-        { pipeline { heredoc 0 $spectrumcmd base64 -d } /bin/sh }
-        ${run}
-    }
-    importas -i ? ?
-    if { s6-echo STATUS: $? }
-    s6-svscanctl -6 /run/service
+    s6-rc change ok-all
   }
 
   unexport !
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 09/16] spectrumPackages.rootfs: make /var/lib and /var/run
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (7 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 08/16] spectrumPackages.rootfs: add s6-rc support Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 10/16] spectrumPackages.rootfs: add dbus configuration Alyssa Ross
                   ` (7 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

Needed by some programs.  There's no persistance in /var/lib for
now (or anywhere else for that matter).
---
 pkgs/os-specific/linux/spectrum/rootfs/generic.nix | 3 ++-
 pkgs/os-specific/linux/spectrum/rootfs/stage1.nix  | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
index 4122abf80c6..ac628de13a3 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
@@ -36,7 +36,8 @@ let
     mkdir $out
     cd $out
 
-    mkdir bin sbin dev proc run sys tmp
+    mkdir -p bin sbin dev proc run sys tmp var/lib
+    ln -s /run var/run
     ln -s ${dash}/bin/dash bin/sh
     ln -s ${stage1} sbin/init
     cp -r ${./etc} etc
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix b/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
index 13de2d09876..de10d60ffb6 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/stage1.nix
@@ -26,10 +26,13 @@ writeScript "init-stage1" ''
 
     if { s6-mkdir -p /run/user/0 /dev/pts /dev/shm }
     if { install -o user -g user -d /run/user/1000 }
+
     if { s6-mount -t devpts -o gid=4,mode=620 none /dev/pts }
     if { s6-mount -t tmpfs none /dev/shm }
+    if { s6-mount -t tmpfs none /var/lib }
     if { s6-mount -t proc none /proc }
     if { s6-mount -t sysfs none /sys }
+
     if { s6-ln -s ${mesa.drivers} /run/opengl-driver }
 
     s6-rc change ok-all
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 10/16] spectrumPackages.rootfs: add dbus configuration
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (8 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 09/16] spectrumPackages.rootfs: make /var/lib and /var/run Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 11/16] spectrumPackages.rootfs: add connman dbus services Alyssa Ross
                   ` (6 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This doesn't actually include dbus, just sets up its user, group and
configuration files.  We'll need dbus for connman or NetworkManager.
---
 pkgs/os-specific/linux/spectrum/rootfs/etc/group   | 1 +
 pkgs/os-specific/linux/spectrum/rootfs/etc/passwd  | 1 +
 pkgs/os-specific/linux/spectrum/rootfs/generic.nix | 6 +++++-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/pkgs/os-specific/linux/spectrum/rootfs/etc/group b/pkgs/os-specific/linux/spectrum/rootfs/etc/group
index e1f50c66958..df4940a5516 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/etc/group
+++ b/pkgs/os-specific/linux/spectrum/rootfs/etc/group
@@ -1,2 +1,3 @@
 root:x:0:root
+messagebus:x:4:messagebus
 user:x:1000:user
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd b/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd
index 467ffc9e42c..c76fb8f2a99 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd
+++ b/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd
@@ -1,2 +1,3 @@
 root:x:0:0:System administrator:/:/bin/sh
+messagebus:x:1:1:D-Bus system message bus daemon user:/run/dbus:/bin/sh
 user:x:1000:1000:User:/:/bin/sh
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
index ac628de13a3..ae4dd6579f5 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
@@ -1,6 +1,6 @@
 { runCommandNoCC, writeScript, writeReferencesToFile, makeFontsConf, lib
 , dash, execline, s6, s6-rc, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox
-, mesa, squashfs-tools-ng
+, mesa, squashfs-tools-ng, makeDBusConf
 }:
 
 { services, rcServices ? {}, fonts ? [], path ? [] }:
@@ -42,6 +42,10 @@ let
     ln -s ${stage1} sbin/init
     cp -r ${./etc} etc
     chmod u+w etc
+    ln -s ${makeDBusConf {
+      suidHelper = "/run/dbus-daemon-launch-helper";
+      serviceDirectories = [];
+    }} etc/dbus-1
 
     mkdir etc/fonts
     ln -s ${fontsConf} etc/fonts/fonts.conf
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 11/16] spectrumPackages.rootfs: add connman dbus services
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (9 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 10/16] spectrumPackages.rootfs: add dbus configuration Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 12/16] spectrumPackages.sys-vms.comp: init Alyssa Ross
                   ` (5 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

At some point we'll want to make it so different VMs don't share this,
because only VMs that talk to network hardware will need connman.
But this'll do for now.
---
 pkgs/os-specific/linux/spectrum/rootfs/generic.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
index ae4dd6579f5..56f2d15b103 100644
--- a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
+++ b/pkgs/os-specific/linux/spectrum/rootfs/generic.nix
@@ -1,6 +1,6 @@
 { runCommandNoCC, writeScript, writeReferencesToFile, makeFontsConf, lib
 , dash, execline, s6, s6-rc, s6-portable-utils, s6-linux-utils, s6-linux-init, busybox
-, mesa, squashfs-tools-ng, makeDBusConf
+, mesa, squashfs-tools-ng, makeDBusConf, connman
 }:
 
 { services, rcServices ? {}, fonts ? [], path ? [] }:
@@ -44,7 +44,7 @@ let
     chmod u+w etc
     ln -s ${makeDBusConf {
       suidHelper = "/run/dbus-daemon-launch-helper";
-      serviceDirectories = [];
+      serviceDirectories = [ connman ];
     }} etc/dbus-1
 
     mkdir etc/fonts
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 12/16] spectrumPackages.sys-vms.comp: init
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (10 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 11/16] spectrumPackages.rootfs: add connman dbus services Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 13/16] spectrumPackages.makeRootfs: move to default.nix Alyssa Ross
                   ` (4 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This removes the default rootfs, and creates a VM called "comp" that
runs all the Wayfire stuff the default rootfs did previously.

This is in a new namespace called sys-vms, which I have a very vague
intention of being a place to put VMs that run system services like a
Wayland compositor or hardware drivers.  I don't think this will be
the final structure, but it's _a_ structure that supports more than
one VM, so it's an improvement over what we had before.
---
 pkgs/os-specific/linux/spectrum/default.nix   |  4 +-
 .../linux/spectrum/rootfs/default.nix         | 78 -----------------
 .../linux/spectrum/spectrum-vm/default.nix    |  6 +-
 .../linux/spectrum/vm/comp/default.nix        | 86 +++++++++++++++++++
 .../os-specific/linux/spectrum/vm/default.nix |  5 ++
 5 files changed, 96 insertions(+), 83 deletions(-)
 delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix

diff --git a/pkgs/os-specific/linux/spectrum/default.nix b/pkgs/os-specific/linux/spectrum/default.nix
index ea86dc25c2b..f8a9813d16a 100644
--- a/pkgs/os-specific/linux/spectrum/default.nix
+++ b/pkgs/os-specific/linux/spectrum/default.nix
@@ -4,13 +4,13 @@ let
   self = with self; {
     callPackage = newScope self;
 
+    sys-vms = callPackage ./vm { };
+
     spectrum-vm = callPackage ./spectrum-vm { linux = linux_vm; };
 
     linux_vm = callPackage ./linux/vm.nix { linux = linux_cros; };
 
     makeRootfs = callPackage ./rootfs/generic.nix { };
-
-    rootfs = callPackage ./rootfs { };
   };
 in
 self
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/default.nix b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
deleted file mode 100644
index 70ddac4c545..00000000000
--- a/pkgs/os-specific/linux/spectrum/rootfs/default.nix
+++ /dev/null
@@ -1,78 +0,0 @@
-{ writeScript, writeText, lib, makeRootfs
-, busybox, execline, s6, sommelier, source-code-pro, wayfire, zsh
-, gcc, emacs26-nox, westonLite
-}:
-
-let
-  path = [
-    zsh emacs26-nox gcc wayfire sommelier westonLite busybox s6 execline
-  ];
-
-  login = writeScript "login" ''
-    #! ${execline}/bin/execlineb -s0
-    unexport !
-    ${busybox}/bin/login -p -f root $@
-  '';
-
-  # This can't be /etc/wayfire/defaults.ini because autostart entries
-  # from that file aren't applied.
-  wayfireConfig = writeText "wayfire-config" ''
-    [core]
-    xwayland = false
-
-    [input]
-    xkb_layout = us
-    xkb_variant = dvorak
-
-    [autostart]
-    terminal = weston-terminal --shell $(command -v zsh)
-  '';
-in
-
-makeRootfs {
-  services.getty.run = writeScript "getty-run" ''
-    #! ${execline}/bin/execlineb -P
-    ${busybox}/bin/getty -i -n -l ${login} 38400 ttyS0
-  '';
-
-  rcServices.ok-all = {
-    type = writeText "ok-all-type" ''
-      bundle
-    '';
-    contents = writeText "ok-all-contents" ''
-      wayfire
-    '';
-  };
-
-  rcServices.wayfire = {
-    type = writeText "wayfire-type" ''
-      longrun
-    '';
-    run = writeScript "wayfire-run" ''
-      #! ${execline}/bin/execlineb -S0
-
-      s6-applyuidgid -u 1000 -g 1000
-
-      export HOME /
-      export PATH ${lib.makeBinPath path}
-      export XDG_RUNTIME_DIR /run/user/1000
-
-      ${sommelier}/bin/sommelier
-      wayfire -c ${wayfireConfig}
-    '';
-    dependencies = writeText "wayfire-dependencies" ''
-      wl0
-    '';
-  };
-
-  rcServices.wl0 = {
-    type = writeText "wl0-type" ''
-      oneshot
-    '';
-    up = writeText "wl0-run" ''
-      chown user /dev/wl0
-    '';
-  };
-
-  fonts = [ source-code-pro ];
-}
diff --git a/pkgs/os-specific/linux/spectrum/spectrum-vm/default.nix b/pkgs/os-specific/linux/spectrum/spectrum-vm/default.nix
index 56f1eadea99..c56d2537c63 100644
--- a/pkgs/os-specific/linux/spectrum/spectrum-vm/default.nix
+++ b/pkgs/os-specific/linux/spectrum/spectrum-vm/default.nix
@@ -1,4 +1,4 @@
-{ stdenv, lib, makeWrapper, utillinux, crosvm, linux, rootfs }:
+{ stdenv, lib, makeWrapper, utillinux, crosvm, linux, sys-vms }:
 
 stdenv.mkDerivation {
   name = "spectrum-vm";
@@ -18,8 +18,8 @@ stdenv.mkDerivation {
 
   getopt = "${lib.getBin utillinux}/bin/getopt";
   crosvm = "${lib.getBin crosvm}/bin/crosvm";
-  kernel = "${linux}/bzImage";
-  rootfs = rootfs.squashfs;
+  kernel = "${sys-vms.comp.linux}/bzImage";
+  rootfs = sys-vms.comp.rootfs.squashfs;
 
   installPhase = ''
     mkdir -p $out/bin
diff --git a/pkgs/os-specific/linux/spectrum/vm/comp/default.nix b/pkgs/os-specific/linux/spectrum/vm/comp/default.nix
new file mode 100644
index 00000000000..988fad5fba6
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/vm/comp/default.nix
@@ -0,0 +1,86 @@
+{ lib, makeRootfs, runCommand, writeScript, writeText
+, busybox, emacs26-nox, execline, gcc, linux_vm, s6, sommelier, source-code-pro
+, wayfire, westonLite, zsh
+}:
+
+runCommand "vm-comp" rec {
+  linux = linux_vm;
+
+  path = [
+    busybox emacs26-nox execline gcc s6 sommelier wayfire westonLite zsh
+  ];
+
+  login = writeScript "login" ''
+    #! ${execline}/bin/execlineb -s0
+    unexport !
+    ${busybox}/bin/login -p -f root $@
+  '';
+
+  # This can't be /etc/wayfire/defaults.ini because autostart entries
+  # from that file aren't applied.
+  wayfireConfig = writeText "wayfire-config" ''
+    [core]
+    xwayland = false
+
+    [input]
+    xkb_layout = us
+    xkb_variant = dvorak
+
+    [autostart]
+    terminal = weston-terminal --shell $(command -v zsh)
+  '';
+
+  rootfs = makeRootfs {
+    services.getty.run = writeScript "getty-run" ''
+      #! ${execline}/bin/execlineb -P
+      ${busybox}/bin/getty -i -n -l ${login} 38400 ttyS0
+    '';
+
+    rcServices.ok-all = {
+      type = writeText "ok-all-type" ''
+        bundle
+      '';
+      contents = writeText "ok-all-contents" ''
+        wayfire
+      '';
+    };
+
+    rcServices.wayfire = {
+      type = writeText "wayfire-type" ''
+        longrun
+      '';
+      run = writeScript "wayfire-run" ''
+        #! ${execline}/bin/execlineb -S0
+
+        s6-applyuidgid -u 1000 -g 1000
+
+        export HOME /
+        export PATH ${lib.makeBinPath path}
+        export XDG_RUNTIME_DIR /run/user/1000
+
+        ${sommelier}/bin/sommelier
+        wayfire -c ${wayfireConfig}
+      '';
+      dependencies = writeText "wayfire-dependencies" ''
+        wl0
+      '';
+    };
+
+    rcServices.wl0 = {
+      type = writeText "wl0-type" ''
+        oneshot
+      '';
+      up = writeText "wl0-run" ''
+        chown user /dev/wl0
+      '';
+    };
+
+    fonts = [ source-code-pro ];
+  };
+
+  inherit (rootfs) squashfs;
+} ''
+  mkdir $out
+  ln -s $linux/bzImage $out/kernel
+  ln -s $squashfs $out/squashfs
+''
diff --git a/pkgs/os-specific/linux/spectrum/vm/default.nix b/pkgs/os-specific/linux/spectrum/vm/default.nix
new file mode 100644
index 00000000000..582a0c2e9c3
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/vm/default.nix
@@ -0,0 +1,5 @@
+{ callPackage }:
+
+{
+  comp = callPackage ./comp { };
+}
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 13/16] spectrumPackages.makeRootfs: move to default.nix
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (11 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 12/16] spectrumPackages.sys-vms.comp: init Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init Alyssa Ross
                   ` (3 subsequent siblings)
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

There is no default rootfs any more, so we might as well repurpose the
blessed filename.
---
 pkgs/os-specific/linux/spectrum/default.nix                     | 2 +-
 .../linux/spectrum/rootfs/{generic.nix => default.nix}          | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename pkgs/os-specific/linux/spectrum/rootfs/{generic.nix => default.nix} (100%)

diff --git a/pkgs/os-specific/linux/spectrum/default.nix b/pkgs/os-specific/linux/spectrum/default.nix
index f8a9813d16a..7e07ee60f43 100644
--- a/pkgs/os-specific/linux/spectrum/default.nix
+++ b/pkgs/os-specific/linux/spectrum/default.nix
@@ -10,7 +10,7 @@ let
 
     linux_vm = callPackage ./linux/vm.nix { linux = linux_cros; };
 
-    makeRootfs = callPackage ./rootfs/generic.nix { };
+    makeRootfs = callPackage ./rootfs { };
   };
 in
 self
diff --git a/pkgs/os-specific/linux/spectrum/rootfs/generic.nix b/pkgs/os-specific/linux/spectrum/rootfs/default.nix
similarity index 100%
rename from pkgs/os-specific/linux/spectrum/rootfs/generic.nix
rename to pkgs/os-specific/linux/spectrum/rootfs/default.nix
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (12 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 13/16] spectrumPackages.makeRootfs: move to default.nix Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-14 20:49   ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 15/16] spectrumPackages.sys-vms.app: init Alyssa Ross
                   ` (2 subsequent siblings)
  16 siblings, 1 reply; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This is a VM that acts as a router.  It handles talking to network
hardware, and other VMs can be connected to it by attaching virtual
ethernet devices.

It expects to get a physical ethernet device passed through to it
using VFIO.  Wi-Fi should work too, but would need to be configured so
I've stuck with Ethernet for now.  We use ConnMan[1] to configure
physical network interfaces, and it automatically takes care of DHCP
and stuff for us.  I chose ConnMan over NetworkManager because it was
easier to get set up.

Virtual ethernet devices are identified by a specific OUI in their MAC
address.  The NIC part of the MAC address is used to encode the last
three octets of the IPv4 address this VM should assign to the
interface.  This way, the host can tell this VM what the address of
each virtual interface is without having to resort to a secondary
communication channel.  The first octet will always be 100, as the
intention is to use the IPv4 shared address space (aka the CGNAT
space) for inter-VM networks to match the behaviour of Chromium OS[2].

Every networking client will be connected to a router VM with a /31,
where the low address is for the router, and the high address is for
the client.  This way the host's job is as simple as possible -- it
just has to connect two TAPs together, without worrying about any
routing rules or anything -- that can all happen in the VM.  This does
mean that the router has to have a virtio-net device for every client,
though.  We may run into scaling limitations with this approach, in
which case we might have to revisit how this works on the host.

[1]: https://git.kernel.org/pub/scm/network/connman/connman.git/about/
[2]: https://chromium.googlesource.com/chromiumos/platform2/+/39e48f668a937d266638f3f7d31d3427a4966464/patchpanel/address_manager.cc#13
---
 .../os-specific/linux/spectrum/vm/default.nix |   2 +
 .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++++++++++
 2 files changed, 167 insertions(+)
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix

diff --git a/pkgs/os-specific/linux/spectrum/vm/default.nix b/pkgs/os-specific/linux/spectrum/vm/default.nix
index 582a0c2e9c3..c4ff729cb8a 100644
--- a/pkgs/os-specific/linux/spectrum/vm/default.nix
+++ b/pkgs/os-specific/linux/spectrum/vm/default.nix
@@ -2,4 +2,6 @@
 
 {
   comp = callPackage ./comp { };
+
+  net = callPackage ./net { };
 }
diff --git a/pkgs/os-specific/linux/spectrum/vm/net/default.nix b/pkgs/os-specific/linux/spectrum/vm/net/default.nix
new file mode 100644
index 00000000000..079311c80e6
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/vm/net/default.nix
@@ -0,0 +1,165 @@
+{ lib, makeRootfs, runCommand, writeScript, writeText
+, busybox, connman, dbus, execline, iptables, iproute, jq, linux_vm, mdevd
+}:
+
+runCommand "vm-net" rec {
+  linux = linux_vm.override {
+    structuredExtraConfig = with lib.kernel; {
+      E1000E = yes;
+      PACKET = yes;
+
+      IP_NF_NAT = yes;
+      IP_NF_IPTABLES = yes;
+      IP_NF_TARGET_MASQUERADE = yes;
+      NF_CONNTRACK = yes;
+    };
+  };
+
+  login = writeScript "login" ''
+    #! ${execline}/bin/execlineb -s0
+    unexport !
+    ${busybox}/bin/login -p -f root $@
+  '';
+
+  rootfs = makeRootfs {
+    rcServices.ok-all = {
+      type = writeText "ok-all-type" ''
+        bundle
+      '';
+      contents = writeText "ok-all-contents" ''
+        mdevd-coldplug
+      '';
+    };
+
+    rcServices.mdevd = {
+      type = writeText "mdevd-type" ''
+        longrun
+      '';
+      run = writeScript "mdevd-run" ''
+        #! ${execline}/bin/execlineb -P
+        ${mdevd}/bin/mdevd -D3 -f ${writeText "mdevd.conf" ''
+          $INTERFACE=.* 0:0 660 ! @${writeScript "interface" ''
+            #! ${execline}/bin/execlineb -S0
+
+            multisubstitute {
+              importas -i DEVPATH DEVPATH
+              importas -i INTERFACE INTERFACE
+            }
+
+            ifte
+
+            {
+              # This interface is connected to another VM.
+
+              # Our IP is encoded in the NIC-specific portion of the
+              # interface's MAC address.
+              backtick -i LOCAL_IP {
+                pipeline { ip -j link show $INTERFACE }
+                pipeline { jq -r ".[0].address | split(\":\") | .[3:6] | \"0x\" + .[]" }
+                xargs printf "100.%d.%d.%d"
+              }
+              importas -iu LOCAL_IP LOCAL_IP
+
+              if { ip address add ''${LOCAL_IP}/31 dev $INTERFACE }
+              ip link set $INTERFACE up
+            }
+
+            {
+              if { test $INTERFACE != lo }
+              # This is a physical connection to a network device.
+              if { iptables -t nat -A POSTROUTING -o $INTERFACE -j MASQUERADE }
+              s6-rc -u change connman
+            }
+
+            grep -iq ^0A:B3:EC: /sys/class/net/''${INTERFACE}/address
+          ''}
+        ''}
+      '';
+      notification-fd = writeText "mdevd-notification-fd" ''
+        3
+      '';
+      dependencies = writeText "mdevd-dependencies" ''
+        sysctl
+      '';
+    };
+
+    rcServices.mdevd-coldplug = {
+      type = writeText "mdevd-coldplug-type" ''
+        oneshot
+      '';
+      up = writeText "mdevd-run" ''
+        ${mdevd}/bin/mdevd-coldplug
+      '';
+      dependencies = writeText "mdevd-coldplug-dependencies" ''
+        mdevd
+      '';
+    };
+
+    rcServices.dbus = {
+      type = writeText "dbus-daemon" ''
+        longrun
+      '';
+      run = writeScript "dbus-daemon-run" ''
+        #! ${execline}/bin/execlineb -S0
+        foreground { mkdir /run/dbus }
+        # Busybox cp doesn't have -n to avoid copying to paths that
+        # already exist, but we can abuse -u for the same effect,
+        # since every file in the store is from Jan 1 1970.
+        foreground { cp -u ${dbus}/libexec/dbus-daemon-launch-helper /run }
+        foreground { chgrp messagebus /run/dbus-daemon-launch-helper }
+        foreground { chmod 4550 /run/dbus-daemon-launch-helper }
+        ${dbus}/bin/dbus-daemon
+          --nofork --nosyslog --nopidfile --config-file=/etc/dbus-1/system.conf
+      '';
+    };
+
+    rcServices.connman = {
+      type = writeText "connman-type" ''
+        longrun
+      '';
+      run = writeScript "connman-run" ''
+        #! ${execline}/bin/execlineb -S0
+        backtick -in HARDWARE_INTERFACES {
+          pipeline {
+            find -L /sys/class/net -mindepth 2 -maxdepth 2 -name address -print0
+          }
+
+          # Filter out other VMs and the loopback device.
+          pipeline { xargs -0 grep -iL ^\\(0A:B3:EC:\\|00:00:00:00:00:00$\\) }
+
+          # Extract the interface names from the address file paths.
+          awk -F/ "{if (NR > 1) printf \",\"; printf \"%s\", $5}"
+        }
+        importas -iu HARDWARE_INTERFACES HARDWARE_INTERFACES
+
+        ${connman}/bin/connmand -ni $HARDWARE_INTERFACES
+      '';
+      dependencies = writeText "connman-dependencies" ''
+        dbus
+      '';
+    };
+
+    rcServices.sysctl = {
+      type = writeText "sysctl-type" ''
+        oneshot
+      '';
+      up = writeText "sysctl-up" ''
+        redirfd -w 1 /proc/sys/net/ipv4/ip_forward
+        echo 1
+      '';
+    };
+
+    services.getty.run = writeScript "getty-run" ''
+      #! ${execline}/bin/execlineb -P
+      ${busybox}/bin/getty -i -n -l ${login} 38400 ttyS0
+    '';
+
+    path = [ iproute iptables jq ];
+  };
+
+  inherit (rootfs) squashfs;
+} ''
+  mkdir $out
+  ln -s $linux/bzImage $out/kernel
+  ln -s $squashfs $out/squashfs
+''
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 15/16] spectrumPackages.sys-vms.app: init
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (13 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-11 11:57 ` [PATCH nixpkgs 16/16] spectrumPackages.spectrum-testhost: init Alyssa Ross
  2021-04-14 22:15 ` [PATCH nixpkgs 00/16] Inter-guest networking Cole Helbling
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This is a very barebones VM.  It's purpose is just to be a
demonstration that other VMs can connect to the router and have their
packets reach the network.  So all it does is infer its IPv4 address,
and the IPv4 address of the router, from the MAC address of its
virtual ethernet device, and configure the network interfaces and
routes appropriately.

vmID is an integer seed we can use to derive things for the VM like IP
and VSOCK addresses.  I don't foresee this sitting around, because I
think it would make more sense for this to be assigned at runtime,
since starting arbitrary VMs at runtime is a goal.  But we'll need
some way to ensure unique addresses.
---
 .../linux/spectrum/vm/app/default.nix         | 63 +++++++++++++++++++
 .../os-specific/linux/spectrum/vm/default.nix |  2 +
 2 files changed, 65 insertions(+)
 create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix

diff --git a/pkgs/os-specific/linux/spectrum/vm/app/default.nix b/pkgs/os-specific/linux/spectrum/vm/app/default.nix
new file mode 100644
index 00000000000..65dbb51f5e1
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/vm/app/default.nix
@@ -0,0 +1,63 @@
+{ runCommand, writeScript, writeText, makeRootfs
+, busybox, execline, linux_vm, jq, iproute
+}:
+
+runCommand "vm-app" rec {
+  linux = linux_vm;
+
+  login = writeScript "login" ''
+    #! ${execline}/bin/execlineb -s0
+    unexport !
+    ${busybox}/bin/login -p -f root $@
+  '';
+
+  rootfs = makeRootfs {
+    rcServices.ok-all = {
+      type = writeText "ok-all-type" ''
+        bundle
+      '';
+      contents = writeText "ok-all-contents" ''
+        net
+      '';
+    };
+
+    rcServices.net = {
+      type = writeText "net-type" ''
+        oneshot
+      '';
+      up = writeText "net-up" ''
+        backtick -i LOCAL_IP {
+          pipeline { ip -j link show eth0 }
+          pipeline { jq -r ".[0].address | split(\":\") | .[3:6] | \"0x\" + .[]" }
+          xargs printf "100.%d.%d.%d"
+        }
+        importas -iu LOCAL_IP LOCAL_IP
+
+        backtick -i REMOTE_IP {
+          jq -jn --arg localip $LOCAL_IP
+            "$localip | split(\".\") | .[3] |= tonumber - 1 | join(\".\")"
+        }
+        importas -iu REMOTE_IP REMOTE_IP
+
+        if { ip address add ''${LOCAL_IP}/31 dev eth0 }
+        if { ip link set eth0 up }
+        ip route add default via $REMOTE_IP
+      '';
+    };
+
+    services.getty.run = writeScript "getty-run" ''
+      #! ${execline}/bin/execlineb -P
+      ${busybox}/bin/getty -i -n -l ${login} 38400 ttyS0
+    '';
+
+    path = [ iproute jq ];
+  };
+
+  inherit (rootfs) squashfs;
+  vmID = 0;
+} ''
+  mkdir $out
+  echo "$vmID" > $out/vm-id
+  ln -s $linux/bzImage $out/kernel
+  ln -s $squashfs $out/squashfs
+''
diff --git a/pkgs/os-specific/linux/spectrum/vm/default.nix b/pkgs/os-specific/linux/spectrum/vm/default.nix
index c4ff729cb8a..f5d591a960a 100644
--- a/pkgs/os-specific/linux/spectrum/vm/default.nix
+++ b/pkgs/os-specific/linux/spectrum/vm/default.nix
@@ -1,6 +1,8 @@
 { callPackage }:
 
 {
+  app = callPackage ./app { };
+
   comp = callPackage ./comp { };
 
   net = callPackage ./net { };
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH nixpkgs 16/16] spectrumPackages.spectrum-testhost: init
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (14 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 15/16] spectrumPackages.sys-vms.app: init Alyssa Ross
@ 2021-04-11 11:57 ` Alyssa Ross
  2021-04-14 22:15 ` [PATCH nixpkgs 00/16] Inter-guest networking Cole Helbling
  16 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-11 11:57 UTC (permalink / raw)
  To: devel

This produces a shell script that sets up a host system for running
the VMs under sys-vms, and then starts an s6-rc service manager that
can run the VMs.  This mirrors how I imagine things working on the
eventual Spectrum host system.

With s6-rc, we can express dependencies between services, so when
vm-app is started, vm-net will automatically be started too if it
isn't already running.

One thing I haven't implemented yet is readiness notification.
Ideally, the cloud-hypervisor instance running the router VM would be
able to tell s6 once its control socket was listening, and s6 wouldn't
consider it to be up until that point.  But it can't do that at the
moment, so s6 considers it to be up immediately, and starts vm-app
right away.  This means that vm-app will usually fail once due to
vm-net's socket not existing, and then immediately be restarted and
work the second time.  I think that's fine for now.

The serial console for vm-app will be connected to the terminal.  To
interact with vm-net instead, the serial console for vm-app can be
disconnected uncommenting the two commented out lines in its
definition, and the serial console for vm-net can be enabled by
commenting the redirfd that disables its stdin, and uncommenting the
--serial line.

We're using cloud-hypervisor instead of crosvm for vm-net because
cloud-hypervisor supports adding virtual ethernet devices at runtime
and crosvm doesn't, and it'll be important to be able to add
connections to new VMs as applications are started on a running
system.

The TODO about removing the device from cloud-hypervisor is going to
stay a TODO for now, because the solution is quite complicated: if we
tell cloud-hypervisor to remove the device from the VM using its API,
it'll still be trying to read from it for a short time after
returning, so we'll still get read errors in cloud-hypervisor after we
delete the TAP.  I think the only good way to handle this will be to
use a non-persistent TAP device, so that it automatically gets cleaned
up by the kernel when cloud-hypervisor is done with it.  But to do
that, cloud-hypervisor will have to be able to add TAPs from file
descriptors at runtime, which will probably be quite difficult to fit
into its HTTP API -- at least it's over a Unix socket.
---
 pkgs/os-specific/linux/spectrum/default.nix   |   2 +
 .../linux/spectrum/testhost/default.nix       | 205 ++++++++++++++++++
 2 files changed, 207 insertions(+)
 create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix

diff --git a/pkgs/os-specific/linux/spectrum/default.nix b/pkgs/os-specific/linux/spectrum/default.nix
index 7e07ee60f43..c4cccab3787 100644
--- a/pkgs/os-specific/linux/spectrum/default.nix
+++ b/pkgs/os-specific/linux/spectrum/default.nix
@@ -8,6 +8,8 @@ let
 
     spectrum-vm = callPackage ./spectrum-vm { linux = linux_vm; };
 
+    spectrum-testhost = callPackage ./testhost { };
+
     linux_vm = callPackage ./linux/vm.nix { linux = linux_cros; };
 
     makeRootfs = callPackage ./rootfs { };
diff --git a/pkgs/os-specific/linux/spectrum/testhost/default.nix b/pkgs/os-specific/linux/spectrum/testhost/default.nix
new file mode 100644
index 00000000000..7e1a973e8c6
--- /dev/null
+++ b/pkgs/os-specific/linux/spectrum/testhost/default.nix
@@ -0,0 +1,205 @@
+{ lib, runCommandNoCC, writeScript, writeScriptBin, writeShellScript, writeText
+, coreutils, cloud-hypervisor, crosvm, curl, execline, gnutar, gnused, iproute
+, iptables, jq, kmod, mktuntap, rsync, s6, s6-rc, sys-vms, utillinux
+}:
+
+let
+  inherit (lib) concatStrings escapeShellArg makeBinPath mapAttrsToList
+    optionalString;
+
+  compose2 = f: g: a: b: f (g a b);
+
+  concatMapAttrs = compose2 concatStrings mapAttrsToList;
+
+  makeServicesDir = { services }:
+    runCommandNoCC "services" {} ''
+      mkdir $out
+      ${concatMapAttrs (name: attrs: ''
+        mkdir $out/${name}
+        ${concatMapAttrs (key: value: ''
+          cp -r ${value} $out/${name}/${key}
+        '') attrs}
+      '') services}
+    '';
+
+  s6RcCompile = { fdhuser ? null }: source:
+    runCommandNoCC "s6-rc-compile" {} ''
+      ${s6-rc}/bin/s6-rc-compile \
+        ${optionalString (fdhuser != null) "-h ${escapeShellArg fdhuser}"} \
+        dest ${source}
+      tar -C dest -cf $out .
+    '';
+
+  compiledRcServicesDir = s6RcCompile {} (makeServicesDir {
+    services = {
+      vm-app = {
+        run = writeScript "app-run" ''
+          #! ${execline}/bin/execlineb -S0
+          # fdclose 0
+
+          # Checking the return value of the bridge creation is
+          # important, because if it fails due to the bridge already
+          # existing that means something else could already be using
+          # this bridge.
+          if { ip link add name br0 type bridge }
+          if { ip link set br0 up }
+
+          # Calculate the MACs for our TAP and the router's TAP.
+          backtick -in router_nic_dec {
+            expr ${toString sys-vms.app.vmID} * 2 + 64 * 256 * 256
+          }
+          backtick -in client_nic_dec {
+            expr ${toString sys-vms.app.vmID} * 2 + 64 * 256 * 256 + 1
+          }
+          multisubstitute {
+            importas -iu router_nic_dec router_nic_dec
+            importas -iu client_nic_dec client_nic_dec
+          }
+          backtick -i router_mac {
+            pipeline { printf %x $router_nic_dec }
+            sed s/^\\(..\\)\\(..\\)\\(..\\)$/0A:B3:EC:\\1:\\2:\\3/
+          }
+          backtick -i client_mac {
+            pipeline { printf %x $client_nic_dec }
+            sed s/^\\(..\\)\\(..\\)\\(..\\)$/0A:B3:EC:\\1:\\2:\\3/
+          }
+          multisubstitute {
+            importas -iu router_mac router_mac
+            importas -iu client_mac client_mac
+          }
+
+          # Create the net VM end, and attach it to the net VM.
+          #
+          # Use a hardcoded name for now because if we use a dynamic
+          # one iproute2 has no way of telling us the name that was
+          # chosen:
+          # https://lore.kernel.org/netdev/20210406134240.wwumpnrzfjbttnmd@eve.qyliss.net/
+          define other_tap_name vmtapnet
+          # Try to delete the device in case the VM was powered off
+          # (as the finish script wouldn't have been run in that
+          # case.)  Since we check the return value of ip tuntap add,
+          # in the case of a race condition between deleting the
+          # device and creating it again, we'll just fail and try
+          # again.
+          foreground { ip link delete $other_tap_name }
+          if { ip tuntap add name $other_tap_name mode tap }
+          if { ip link set $other_tap_name master br0 }
+          if { ip link set $other_tap_name up }
+          if {
+            pipeline {
+              jq -n "$ARGS.named"
+                --arg tap $other_tap_name
+                --arg mac $router_mac
+            }
+            curl -iX PUT
+              -H "Accept: application/json"
+              -H "Content-Type: application/json"
+              --data-binary @-
+              --unix-socket ../vm-net/env/cloud-hypervisor.sock
+              http://localhost/api/v1/vm.add-net
+          }
+
+          mktuntap -pvBi vmtap%d 6
+          importas -iu tap_name TUNTAP_NAME
+          if { ip link set $tap_name master br0 }
+          if { ip link set $tap_name up }
+          if { iptables -t nat -A POSTROUTING -o $tap_name -j MASQUERADE }
+
+          ${crosvm}/bin/crosvm run -p init=/sbin/init -p notifyport=''${port}
+            # --serial type=file,path=/tmp/app.log
+            --cid 4
+            --tap-fd 6,mac=''${client_mac}
+            --root ${sys-vms.app.rootfs.squashfs} ${sys-vms.app.linux}/bzImage
+        '';
+        finish = writeScript "app-finish" ''
+          #! ${execline}/bin/execlineb -S0
+          # TODO: remove from vm-net
+          foreground { ip link delete vmtapnet }
+          ip link delete br0
+        '';
+        type = writeText "app-type" ''
+          longrun
+        '';
+        dependencies = writeText "app-dependencies" ''
+          vm-net
+        '';
+      };
+
+      vm-net = {
+        run = writeScript "net-run" ''
+          #! ${execline}/bin/execlineb -S0
+          # This is only necessary for when running s6 from a tty.
+          # (i.e. when debugging or running the demo).
+          redirfd -w 0 /dev/null
+
+          define PCI_LOCATION 0000:00:19.0
+          define PCI_PATH /sys/bus/pci/devices/''${PCI_LOCATION}
+
+          # Unbind the network device from the driver it's already
+          # attached to, if any.
+          foreground {
+            redirfd -w 1 ''${PCI_PATH}/driver/unbind
+            printf "%s" $PCI_LOCATION
+          }
+
+          # (Re)bind the device to the VFIO PCI driver.
+          if { modprobe vfio-pci }
+          backtick -in device_id {
+            if { dd bs=2 skip=1 count=2 status=none if=''${PCI_PATH}/vendor }
+            if { printf " " }
+            dd bs=2 skip=1 count=2 status=none if=''${PCI_PATH}/device
+          }
+          importas -iu device_id device_id
+          foreground {
+            redirfd -w 1 /sys/bus/pci/drivers/vfio-pci/new_id
+            printf "%s" $device_id
+          }
+
+          foreground { mkdir env }
+
+          ${cloud-hypervisor}/bin/cloud-hypervisor
+            --api-socket env/cloud-hypervisor.sock
+            --console off
+            # --serial tty
+            --cmdline "console=ttyS0 panic=30 root=/dev/vda"
+            --device path=''${PCI_PATH}
+            --disk path=${sys-vms.net.rootfs.squashfs},readonly=on
+            --kernel ${sys-vms.net.linux.dev}/vmlinux
+        '';
+        type = writeText "net-type" ''
+          longrun
+        '';
+      };
+    };
+  });
+
+  servicesDir = makeServicesDir {
+    services = {
+      ".s6-svscan" = {
+        finish = writeShellScript ".s6-svscan-finish" "";
+      };
+    };
+  };
+in
+
+writeScriptBin "spectrum-testhost" ''
+  #! ${execline}/bin/execlineb -S0
+  export PATH ${makeBinPath [
+    coreutils curl execline gnused gnutar iproute iptables jq kmod mktuntap rsync
+    s6 s6-rc
+  ]}
+
+  if { redirfd -w 1 /proc/sys/net/ipv4/ip_forward echo 1 }
+
+  importas -iu runtime_dir XDG_RUNTIME_DIR
+  backtick -in TOP { mktemp -dp $runtime_dir spectrum.XXXXXXXXXX }
+  importas -iu top TOP
+  if { echo $top }
+  if { rsync -r --chmod=Du+w ${servicesDir}/ ''${top}/service }
+  background {
+    if { mkdir -p ''${top}/s6-rc/compiled }
+    if { tar -C ''${top}/s6-rc/compiled -xf ${compiledRcServicesDir} }
+    s6-rc-init -c ''${top}/s6-rc/compiled -l ''${top}/s6-rc/live ''${top}/service
+  }
+  s6-svscan ''${top}/service
+''
-- 
2.30.0

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init
  2021-04-11 11:57 ` [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init Alyssa Ross
@ 2021-04-14 20:49   ` Alyssa Ross
  0 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-14 20:49 UTC (permalink / raw)
  To: Cole Helbling; +Cc: devel

[-- Attachment #1: Type: text/plain, Size: 2913 bytes --]

On Sun, Apr 11, 2021 at 11:57:38AM +0000, Alyssa Ross wrote:
> This is a VM that acts as a router.  It handles talking to network
> hardware, and other VMs can be connected to it by attaching virtual
> ethernet devices.
>
> It expects to get a physical ethernet device passed through to it
> using VFIO.  Wi-Fi should work too, but would need to be configured so
> I've stuck with Ethernet for now.  We use ConnMan[1] to configure
> physical network interfaces, and it automatically takes care of DHCP
> and stuff for us.  I chose ConnMan over NetworkManager because it was
> easier to get set up.
>
> Virtual ethernet devices are identified by a specific OUI in their MAC
> address.  The NIC part of the MAC address is used to encode the last
> three octets of the IPv4 address this VM should assign to the
> interface.  This way, the host can tell this VM what the address of
> each virtual interface is without having to resort to a secondary
> communication channel.  The first octet will always be 100, as the
> intention is to use the IPv4 shared address space (aka the CGNAT
> space) for inter-VM networks to match the behaviour of Chromium OS[2].
>
> Every networking client will be connected to a router VM with a /31,
> where the low address is for the router, and the high address is for
> the client.  This way the host's job is as simple as possible -- it
> just has to connect two TAPs together, without worrying about any
> routing rules or anything -- that can all happen in the VM.  This does
> mean that the router has to have a virtio-net device for every client,
> though.  We may run into scaling limitations with this approach, in
> which case we might have to revisit how this works on the host.
>
> [1]: https://git.kernel.org/pub/scm/network/connman/connman.git/about/
> [2]: https://chromium.googlesource.com/chromiumos/platform2/+/39e48f668a937d266638f3f7d31d3427a4966464/patchpanel/address_manager.cc#13
> ---
>  .../os-specific/linux/spectrum/vm/default.nix |   2 +
>  .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++++++++++
>  2 files changed, 167 insertions(+)
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix

Cole, if you want to test this, you'll probably need the following
additional diff to make it work with your hardware.

It's quite clearly time to support loadable modules in this VM, but I'd
like to leave that for a follow up after this series is sorted.

diff --git i/pkgs/os-specific/linux/spectrum/vm/net/default.nix w/pkgs/os-specific/linux/spectrum/vm/net/default.nix
index 079311c80e6..1deb7031caf 100644
--- i/pkgs/os-specific/linux/spectrum/vm/net/default.nix
+++ w/pkgs/os-specific/linux/spectrum/vm/net/default.nix
@@ -6,6 +6,7 @@ runCommand "vm-net" rec {
   linux = linux_vm.override {
     structuredExtraConfig = with lib.kernel; {
       E1000E = yes;
+      IGB = yes;
       PACKET = yes;

       IP_NF_NAT = yes;

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH nixpkgs 00/16] Inter-guest networking
  2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
                   ` (15 preceding siblings ...)
  2021-04-11 11:57 ` [PATCH nixpkgs 16/16] spectrumPackages.spectrum-testhost: init Alyssa Ross
@ 2021-04-14 22:15 ` Cole Helbling
  2021-04-14 23:56   ` Alyssa Ross
  16 siblings, 1 reply; 20+ messages in thread
From: Cole Helbling @ 2021-04-14 22:15 UTC (permalink / raw)
  To: Alyssa Ross, devel

On Sun Apr 11, 2021 at 4:57 AM PDT, Alyssa Ross wrote:
> In Spectrum, we want the host kernel to include as few drivers as
> possible, to reduce attack service.  To accomplish this, we need to
> move as much hardware interaction as possible into VMs.  This series
> introduces proof-of-concept network hardware isolation by passing
> through network devices to a VM, and having that VM handle all
> interaction with that hardware instead of the host system.

[snip]

> Alyssa Ross (16):
>   linux: enable Xen everywhere it can be
>   cloud-hypervisor: 0.8.0 -> 0.14.1
>   mdevd: init at 0.1.3.0
>   spectrumPackages.linux_vm: fix cloud-hypervisor hotplug
>   spectrumPackages.linux_vm: allow config overrides
>   crosvm: support setting guest MAC from --tap-fd
>   spectrumPackages: export makeRootfs
>   spectrumPackages.rootfs: add s6-rc support
>   spectrumPackages.rootfs: make /var/lib and /var/run
>   spectrumPackages.rootfs: add dbus configuration
>   spectrumPackages.rootfs: add connman dbus services
>   spectrumPackages.sys-vms.comp: init
>   spectrumPackages.makeRootfs: move to default.nix
>   spectrumPackages.sys-vms.net: init
>   spectrumPackages.sys-vms.app: init
>   spectrumPackages.spectrum-testhost: init
>
>  .../cargo-lock-vendor-fix.patch               |  53 ----
>  .../cloud-hypervisor/default.nix              |  15 +-
>  ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++
>  .../linux/chromium-os/crosvm/default.nix      |   1 +
>  .../linux/kernel/common-config.nix            |  13 +-
>  pkgs/os-specific/linux/kernel/patches.nix     |   9 +
>  pkgs/os-specific/linux/mdevd/default.nix      |  28 ++
>  pkgs/os-specific/linux/spectrum/default.nix   |   6 +-
>  pkgs/os-specific/linux/spectrum/linux/vm.nix  |   7 +-
>  .../linux/spectrum/rootfs/default.nix         |  92 +++---
>  .../linux/spectrum/rootfs/etc/group           |   1 +
>  .../linux/spectrum/rootfs/etc/passwd          |   1 +
>  .../linux/spectrum/rootfs/generic.nix         |  48 ---
>  .../linux/spectrum/rootfs/rc-services.nix     |  26 ++
>  .../linux/spectrum/rootfs/stage1.nix          |  25 +-
>  .../linux/spectrum/spectrum-vm/default.nix    |   6 +-
>  .../linux/spectrum/testhost/default.nix       | 205 ++++++++++++
>  .../linux/spectrum/vm/app/default.nix         |  63 ++++
>  .../linux/spectrum/vm/comp/default.nix        |  86 +++++
>  .../os-specific/linux/spectrum/vm/default.nix |   9 +
>  .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++
>  pkgs/top-level/aliases.nix                    |   6 +
>  pkgs/top-level/all-packages.nix               |  12 +-
>  23 files changed, 976 insertions(+), 195 deletions(-)
>  delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch
>  create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
>  create mode 100644 pkgs/os-specific/linux/mdevd/default.nix
>  delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/generic.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix
>
> -- 
> 2.30.0

Thanks for the beautiful cover letter. Such a great amount of detail
and information brings a tear to my eye!

Each individual patch reviewed-by me, nothing stood out as weird or
wrong or bad or whatever. I only tested the final result, not each
individual patch.

Reviewed-by: Cole Helbling <cole.e.helbling@outlook.com>
Tested-by: Cole Helbling <cole.e.helbling@outlook.com>

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH nixpkgs 00/16] Inter-guest networking
  2021-04-14 22:15 ` [PATCH nixpkgs 00/16] Inter-guest networking Cole Helbling
@ 2021-04-14 23:56   ` Alyssa Ross
  0 siblings, 0 replies; 20+ messages in thread
From: Alyssa Ross @ 2021-04-14 23:56 UTC (permalink / raw)
  To: Cole Helbling; +Cc: devel

[-- Attachment #1: Type: text/plain, Size: 4029 bytes --]

"Cole Helbling" <cole.e.helbling@outlook.com> writes:

> On Sun Apr 11, 2021 at 4:57 AM PDT, Alyssa Ross wrote:
>> In Spectrum, we want the host kernel to include as few drivers as
>> possible, to reduce attack service.  To accomplish this, we need to
>> move as much hardware interaction as possible into VMs.  This series
>> introduces proof-of-concept network hardware isolation by passing
>> through network devices to a VM, and having that VM handle all
>> interaction with that hardware instead of the host system.
>
> [snip]
>
>> Alyssa Ross (16):
>>   linux: enable Xen everywhere it can be
>>   cloud-hypervisor: 0.8.0 -> 0.14.1
>>   mdevd: init at 0.1.3.0
>>   spectrumPackages.linux_vm: fix cloud-hypervisor hotplug
>>   spectrumPackages.linux_vm: allow config overrides
>>   crosvm: support setting guest MAC from --tap-fd
>>   spectrumPackages: export makeRootfs
>>   spectrumPackages.rootfs: add s6-rc support
>>   spectrumPackages.rootfs: make /var/lib and /var/run
>>   spectrumPackages.rootfs: add dbus configuration
>>   spectrumPackages.rootfs: add connman dbus services
>>   spectrumPackages.sys-vms.comp: init
>>   spectrumPackages.makeRootfs: move to default.nix
>>   spectrumPackages.sys-vms.net: init
>>   spectrumPackages.sys-vms.app: init
>>   spectrumPackages.spectrum-testhost: init
>>
>>  .../cargo-lock-vendor-fix.patch               |  53 ----
>>  .../cloud-hypervisor/default.nix              |  15 +-
>>  ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++
>>  .../linux/chromium-os/crosvm/default.nix      |   1 +
>>  .../linux/kernel/common-config.nix            |  13 +-
>>  pkgs/os-specific/linux/kernel/patches.nix     |   9 +
>>  pkgs/os-specific/linux/mdevd/default.nix      |  28 ++
>>  pkgs/os-specific/linux/spectrum/default.nix   |   6 +-
>>  pkgs/os-specific/linux/spectrum/linux/vm.nix  |   7 +-
>>  .../linux/spectrum/rootfs/default.nix         |  92 +++---
>>  .../linux/spectrum/rootfs/etc/group           |   1 +
>>  .../linux/spectrum/rootfs/etc/passwd          |   1 +
>>  .../linux/spectrum/rootfs/generic.nix         |  48 ---
>>  .../linux/spectrum/rootfs/rc-services.nix     |  26 ++
>>  .../linux/spectrum/rootfs/stage1.nix          |  25 +-
>>  .../linux/spectrum/spectrum-vm/default.nix    |   6 +-
>>  .../linux/spectrum/testhost/default.nix       | 205 ++++++++++++
>>  .../linux/spectrum/vm/app/default.nix         |  63 ++++
>>  .../linux/spectrum/vm/comp/default.nix        |  86 +++++
>>  .../os-specific/linux/spectrum/vm/default.nix |   9 +
>>  .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++
>>  pkgs/top-level/aliases.nix                    |   6 +
>>  pkgs/top-level/all-packages.nix               |  12 +-
>>  23 files changed, 976 insertions(+), 195 deletions(-)
>>  delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch
>>  create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch
>>  create mode 100644 pkgs/os-specific/linux/mdevd/default.nix
>>  delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/generic.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix
>>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix
>>
>> --
>> 2.30.0
>
> Thanks for the beautiful cover letter. Such a great amount of detail
> and information brings a tear to my eye!

<3

> Each individual patch reviewed-by me, nothing stood out as weird or
> wrong or bad or whatever. I only tested the final result, not each
> individual patch.
>
> Reviewed-by: Cole Helbling <cole.e.helbling@outlook.com>
> Tested-by: Cole Helbling <cole.e.helbling@outlook.com>

Thanks!  Committed as 583eb604ce3.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2021-04-14 23:57 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-11 11:57 [PATCH nixpkgs 00/16] Inter-guest networking Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 01/16] linux: enable Xen everywhere it can be Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 02/16] cloud-hypervisor: 0.8.0 -> 0.14.1 Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 03/16] mdevd: init at 0.1.3.0 Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 04/16] spectrumPackages.linux_vm: fix cloud-hypervisor hotplug Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 05/16] spectrumPackages.linux_vm: allow config overrides Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 06/16] crosvm: support setting guest MAC from --tap-fd Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 07/16] spectrumPackages: export makeRootfs Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 08/16] spectrumPackages.rootfs: add s6-rc support Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 09/16] spectrumPackages.rootfs: make /var/lib and /var/run Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 10/16] spectrumPackages.rootfs: add dbus configuration Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 11/16] spectrumPackages.rootfs: add connman dbus services Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 12/16] spectrumPackages.sys-vms.comp: init Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 13/16] spectrumPackages.makeRootfs: move to default.nix Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 14/16] spectrumPackages.sys-vms.net: init Alyssa Ross
2021-04-14 20:49   ` Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 15/16] spectrumPackages.sys-vms.app: init Alyssa Ross
2021-04-11 11:57 ` [PATCH nixpkgs 16/16] spectrumPackages.spectrum-testhost: init Alyssa Ross
2021-04-14 22:15 ` [PATCH nixpkgs 00/16] Inter-guest networking Cole Helbling
2021-04-14 23:56   ` Alyssa Ross

Code repositories for project(s) associated with this public inbox

	https://spectrum-os.org/git/crosvm
	https://spectrum-os.org/git/doc
	https://spectrum-os.org/git/mktuntap
	https://spectrum-os.org/git/nixpkgs
	https://spectrum-os.org/git/spectrum
	https://spectrum-os.org/git/ucspi-vsock
	https://spectrum-os.org/git/www

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).