From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-4.5 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.4 Received: by atuin.qyliss.net (Postfix, from userid 496) id 800C95B4C; Sun, 11 Apr 2021 11:58:14 +0000 (UTC) Received: from [127.0.0.1] (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id D748F5B8D; Sun, 11 Apr 2021 11:57:56 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id 710605AF0; Sun, 11 Apr 2021 11:57:54 +0000 (UTC) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by atuin.qyliss.net (Postfix) with ESMTPS id E6F315AEE for ; Sun, 11 Apr 2021 11:57:49 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 2528D5C0056 for ; Sun, 11 Apr 2021 07:57:48 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sun, 11 Apr 2021 07:57:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h= from:to:subject:date:message-id:mime-version :content-transfer-encoding; s=fm2; bh=7YqdPbR10KY/dUH7umdg8Npc4v Rwnj8iI/zAnN3fuSM=; b=RsuSgmEpNsBtlgYoMpBVBR8ppmUjRn7QkDU4tMDIkX 0+W2RdZjQ3oSbQzzWzgUUbuA7Xa+HsdukMnrdvO+zoAWCeX6OCKYtLPEyeFYbLuO gwW6tI0XVYl+JP0WSvNh4p4HE+gss6/IMIemtcUKh5RkA8az8FMUyOCTKjQW9hn8 HJrWiIoCDbDMLzL/KMFB0Gxvg5YqEXu5Uyecp1eEqly9RsSgEMJ6xbhgfD/rtDss fxipyuH8snzV+He+RkMSDHDhcwbmDz/yOpCDeVQ1JL1o+2jNVfBwpTSOSnu52ooO HUbTLV/tp3u1PtQS2x4BkijdkEkTPCUqL2qLbp3+sAjw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:date:from :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=7YqdPbR10KY/dUH7u mdg8Npc4vRwnj8iI/zAnN3fuSM=; b=aAb+QWgPUlSqiWUxv6/QDChxG8QOnfjZG gbfQ2HRcvamWKBcN5vUqLwjmFdjVxzQkIkMQ+IrPUNtHaTnIFPFOlAfYo6mNvwEI U0TEoqGMAiBIypxUaWe0NcpP83diqYMo9lr9gVArZAaS2q1jARAbgDFYLB3j2H8z 3FXi/+s/C4E2Xau5o6kvA0rTFHytCe+V2YCFEYdHfk0AkKV0Ti4FxGl+NC3o0p+Y ApW+JjFg1zJEUp4hlMGQow9JheVSmKfBwuk5WxPasdTLiOzv/omNJBYhrmZ7D9cC +3M8pdHVONiLITMBRr6Zc2yvCekoAd08bQxxtQzSUUn/bj9HNilOQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekgedgudejhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffvufffkffoggfgsedtkeertd ertddtnecuhfhrohhmpeetlhihshhsrgcutfhoshhsuceohhhisegrlhihshhsrgdrihhs qeenucggtffrrghtthgvrhhnpeeuieefieeiheefiedtgfduueevueehgeejvdduvddufe ejvefhieeuteevhffgvdenucffohhmrghinhepqhgvmhhurdhorhhgpdhshihsqdhvmhhs rdhnvghtpdhshihsqdhvmhhsrdgrphhpnecukfhppeejledrvdefhedrudduledruddtje enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehqhihl ihhsshesgidvvddtrdhqhihlihhsshdrnhgvth X-ME-Proxy: Received: from x220.qyliss.net (p4feb776b.dip0.t-ipconnect.de [79.235.119.107]) by mail.messagingengine.com (Postfix) with ESMTPA id 82DB71080063 for ; Sun, 11 Apr 2021 07:57:47 -0400 (EDT) Received: by x220.qyliss.net (Postfix, from userid 1000) id 6165618F2; Sun, 11 Apr 2021 11:57:44 +0000 (UTC) From: Alyssa Ross To: devel@spectrum-os.org Subject: [PATCH nixpkgs 00/16] Inter-guest networking Date: Sun, 11 Apr 2021 11:57:24 +0000 Message-Id: <20210411115740.29615-1-hi@alyssa.is> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: RSVDIXQ536E7IY4P3FI66ZIMIYC65V7K X-Message-ID-Hash: RSVDIXQ536E7IY4P3FI66ZIMIYC65V7K X-MailFrom: qyliss@x220.qyliss.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.3.1 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: In Spectrum, we want the host kernel to include as few drivers as possible, to reduce attack service. To accomplish this, we need to move as much hardware interaction as possible into VMs. This series introduces proof-of-concept network hardware isolation by passing through network devices to a VM, and having that VM handle all interaction with that hardware instead of the host system. Background ---------- Ideally, the Spectrum host system wouldn't have to have support for networking at all. Network hardware could be handled by a VM, which would act a bit like a router, and export virtual network devices directly to other VMs. The hard part of that ideal is the bit where a VM exports a virtual device to another VM. There's work going on in the Linux virtualisation ecosystem to make that possible, through a protocol called virtio-vhost-user[1]. But it looks like it's going to be a long time before virtio-vhost-user is baked enough to be useful to us, and inter-guest networking is so fundamentally important to Spectrum that we can't afford to wait that long. So we need a plan B. Because we have no way to have a VM provide a virtual device, the host is going to need to do it. And this means that some amount of networking is going to have to happen on the host. But we can still get most of what we were going for: * Pass hardware network devices through to a dedicated VM. * Attach a virtual network device to the same VM. * Connect htat virtual network device to virtual network devices for other VMs on the host. With this approach, the host is still doing networking, but there's a lot less code involved, because instead of having loads of drivers for every kind of Wi-Fi card and so on available on the host, the host only ever needs to use the driver for the virtual network devices. If we connect each network client VM to the router VM using a bridge (i.e. a virtual Ethernet switch), we don't even need to worry about IP addresses or routing tables on the host. (There are some drawbacks to this approach, which are elaborated later in the series.) So that's what's implemented here. [1]: https://wiki.qemu.org/Features/VirtioVhostUser Implementation -------------- This series starts with some cherry-picks from upstream Nixpkgs. (We should sync with upstream soon, but I didn't want to delay this any longer by blocking it on a Nixpkgs sync.) Then there's some important refactoring to make it possible to fit multiple VM definitions into spectrumPackages. (Until now, there's only been one.) Finally, I implement two VMs -- one to act as the router and one to be the client -- as well as a service manager that can set everything up on the host to be able to run those VMs, and run them. Don't read too much into the structure of the Nix code for the VMs. I just needed _some_ structure, and this was what I came up with. As we explore the configuration aspect of Spectrum more, I expect it to change dramatically. There's quite a bit of duplication between VM definitions, but I don't think it's worth spending time getting rid of that for now when the entire structure could change before it becomes an issue. The important thing here is what's going on at runtime on the host and in the VMs, not the Nix code used to build that. Everything in the router VM is set up to be able to handle multiple clients coming and going at runtime (with the exception of a small issue with clients going away that I'll get into later in the series). The only reason the client VM is defined up front is because that's what fits our Nix code. There's no reason from the networking point of view that client VMs couldn't be instantiated entirely at runtime. One thing that might jump out to anybody skimming the series is that we're using cloud-hypervisor here. This is because there's a feature it supports that crosvm doesn't (I'll go into detail about what exactly later in the series), and it's more expedient to just use both VMMs as required by the needs of the VMs they're running than it would be to port features from cloud-hypervisor to crosvm or vice versa. Obviously using multiple different VMMs on the host is not good in the long term, but things are so in flux at the moment that it's very likely that any porting work will no longer be in use by the time we need to pare down to a single VMM anyway. Testing ------- If you want to try this out for yourself (and I encourage you to do so and reply to me with a Tested-by!), here's what you need to do: 1. Identify the PCI location of a physical Ethernet device on your system. `lspci -n' should help with this. 2. Modify pkgs/os-specific/linux/spectrum/testhost/default.nix to define PCI_LOCATION to be your one. In future, we can either have a configuration option for this sort of thing, or try to figure it out at runtime. 3. Start the service manager: sudo env XDG_RUNTIME_DIR=/run $(nix-build -A spectrumPackages.spectrum-testhost)/bin/spectrum-testhost A temporary directory will be created for service manager state. Its location will be printed to the terminal. 4. At another terminal, tell the service manager to start the "application VM". This is the client that will be connected to the router VM. s6-rc -u -l /run/spectrum.sy2huQuC3x/s6-rc/live change vm-app Remember to substitute in your temporary directory. You'll need the s6-rc command available (it's in Nixpkgs). This tells the service manager to start the service named vm-app. This service has a dependency on another VM called vm-net, which is the router VM that deals with the network hardware, so both will be started when you ask s6-rc to start vm-app. 5. The terminal running the service manager will be connected to the serial console of client VM. After a few seconds (because the router VM will probably have to sort out DHCP first), you should be able to ping hosts on the internet. Note that the client VM doesn't come with DNS set up. 6. Once you're done, you can tell the service manager to stop the VMs: s6-rc -da -l /run/spectrum.sy2huQuC3x/s6-rc/live change Again, remember to substitute your temporary directory. The service manager itself won't respond to ^C. I believe this is handled better in more recent versions of s6, which we'll get when we sync Nixpkgs. After this, it's a good idea to reboot, to restore all the network devices to their default state. Alyssa Ross (16): linux: enable Xen everywhere it can be cloud-hypervisor: 0.8.0 -> 0.14.1 mdevd: init at 0.1.3.0 spectrumPackages.linux_vm: fix cloud-hypervisor hotplug spectrumPackages.linux_vm: allow config overrides crosvm: support setting guest MAC from --tap-fd spectrumPackages: export makeRootfs spectrumPackages.rootfs: add s6-rc support spectrumPackages.rootfs: make /var/lib and /var/run spectrumPackages.rootfs: add dbus configuration spectrumPackages.rootfs: add connman dbus services spectrumPackages.sys-vms.comp: init spectrumPackages.makeRootfs: move to default.nix spectrumPackages.sys-vms.net: init spectrumPackages.sys-vms.app: init spectrumPackages.spectrum-testhost: init .../cargo-lock-vendor-fix.patch | 53 ---- .../cloud-hypervisor/default.nix | 15 +- ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++ .../linux/chromium-os/crosvm/default.nix | 1 + .../linux/kernel/common-config.nix | 13 +- pkgs/os-specific/linux/kernel/patches.nix | 9 + pkgs/os-specific/linux/mdevd/default.nix | 28 ++ pkgs/os-specific/linux/spectrum/default.nix | 6 +- pkgs/os-specific/linux/spectrum/linux/vm.nix | 7 +- .../linux/spectrum/rootfs/default.nix | 92 +++--- .../linux/spectrum/rootfs/etc/group | 1 + .../linux/spectrum/rootfs/etc/passwd | 1 + .../linux/spectrum/rootfs/generic.nix | 48 --- .../linux/spectrum/rootfs/rc-services.nix | 26 ++ .../linux/spectrum/rootfs/stage1.nix | 25 +- .../linux/spectrum/spectrum-vm/default.nix | 6 +- .../linux/spectrum/testhost/default.nix | 205 ++++++++++++ .../linux/spectrum/vm/app/default.nix | 63 ++++ .../linux/spectrum/vm/comp/default.nix | 86 +++++ .../os-specific/linux/spectrum/vm/default.nix | 9 + .../linux/spectrum/vm/net/default.nix | 165 ++++++++++ pkgs/top-level/aliases.nix | 6 + pkgs/top-level/all-packages.nix | 12 +- 23 files changed, 976 insertions(+), 195 deletions(-) delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch create mode 100644 pkgs/os-specific/linux/mdevd/default.nix delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/generic.nix create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix -- 2.30.0