From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id E07A25944E; Mon, 19 Sep 2022 07:38:28 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id 68C2159381; Mon, 19 Sep 2022 07:38:20 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by atuin.qyliss.net (Postfix) with ESMTPS id 87AFD59301 for ; Mon, 19 Sep 2022 07:38:10 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 1731D5C0288 for ; Mon, 19 Sep 2022 03:38:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 19 Sep 2022 03:38:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :content-transfer-encoding:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to; s=fm1; t=1663573089; x=1663659489; bh=rEikJWQiSR 9QBh3cyv3PcyGHiHa2KNvz19t1b1M4QjI=; b=m8EpOylzMFEhLx9Qw0QcATo1za AJNNf/ft5G6m1E3SC9vZ7s2V+osDSq1Ag9K3W/RRfek4h2Y7IbYNc36wN3v7WfaU rUrFoctLQ+Q4ogn4+uNO8AZBIVXfrlcTFn0OKTvyHAM3KD//If/IIQSrmMMbZV73 5VL80cCZN+JRQ2s+XkOQsBR+L4CCCINjmNfcguaK46i8xN3S5VVJv0A1CQHFqByO hwNBYRnJ1+421D7iECcATqNOfyLAxe+gplT90BlzUYm49TLyIlYxZxZcppUFGTSk fyFrjq0VR5qE8URZVsqlZYOfX4JJWHJCmgcFEqPmlxe5dEsr3mtTv+vYePbg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:date:feedback-id:feedback-id:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1663573089; x=1663659489; bh=rEikJWQiSR9QBh3cyv3PcyGHiHa2 KNvz19t1b1M4QjI=; b=u0tcs2X++U4Ce4wePR0me/H9FQmGJzAOHIlYJ6BxgO3L mhIet+Oexy1YsditDlHIwRNcn5Pj0/mZQWvXgm0FGfcO7bwgSFCEROdesTaUvLfG HjNqEiow3d6SdWr9zamdwbsku31sD5lGhBRnx+idF0fuVR8+c5JeX2VlRJ9B0rPG 7zmqPKidEid40iREKmx7pqBhPYMhJeg/99Nk5HUdZB4/5ekptieL+8ulnOTjOq/9 HPjMPDyUMuICbXXYu1Axh1xExl4atrSAEaPOF3ThPwvgxA8JDaOOBnM4JD6IIAwf +Ai2Y1lmBG8jx0u51w5/QwtwLlgtQjPKrPZWe0oC7A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfedviedguddvfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffvufffkffogggtgfesthekre dtredtjeenucfhrhhomheptehlhihsshgrucftohhsshcuoehhihesrghlhihsshgrrdhi sheqnecuggftrfgrthhtvghrnhephffhtdeijeelhefftdekkedvfedtfeeivdekudevhe eileevhfefveetveevjeeunecuffhomhgrihhnpehprhhovhhiuggvrhhsrdhnvghtpdhs phgvtghtrhhumhdqohhsrdhorhhgpdgtfhhgrdhinhenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehqhihlihhsshesgidvvddtrdhqhihlihhs shdrnhgvth X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 19 Sep 2022 03:38:08 -0400 (EDT) Received: by x220.qyliss.net (Postfix, from userid 1000) id 95D989814; Mon, 19 Sep 2022 07:38:06 +0000 (UTC) From: Alyssa Ross To: devel@spectrum-os.org Subject: [PATCH 0/6] Introduce a shared base for application VMs Date: Mon, 19 Sep 2022 07:36:54 +0000 Message-Id: <20220919073659.1703271-1-hi@alyssa.is> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Message-ID-Hash: RDBQ3F7OG5FPBHWA4FSCJFAGSESPCW63 X-Message-ID-Hash: RDBQ3F7OG5FPBHWA4FSCJFAGSESPCW63 X-MailFrom: qyliss@x220.qyliss.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.5 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Here are some changes I made months ago to enable work I never finished around building VMs inside the Spectrum system with Nix. I'm coming back to it now because it makes it much easier to add more applications for testing, and with upcoming features like Wayland and a persistent filesystem we're going to want more example applications. The idea here is to reduce duplication between application VMs, both in terms of source code size and output size. After this change, creating a new VM just requires writing a very small Nix file like this: { config ? import ../../../nix/eval-config.nix {} }: import ../../vm-lib/make-vm.nix { inherit config; } { name = "appvm-lynx"; providers.net = [ "netvm" ]; run = config.pkgs.pkgsStatic.callPackage ( { writeScript, lynx }: writeScript "run-lynx" '' #!/bin/execlineb -P ${lynx}/bin/lynx https://spectrum-os.org '' ) { }; } Rather than a whole big source tree as before, most of which was duplicated with every other application VM. When a VM generated this way is started, it gets two disk images. One is the shared base image, which is part of the Spectrum base system, and the other contains only the application-specific stuff: the run script, and any store path dependencies that are not already present in the base image. This means that the amount of storage required for each new application VM is substantially reduced. Of course, this isn't the only way to generate VMs. Monolithic VMs like we had before would still work, with some small adjustments for the new disk layout. I also see this fitting well into making it possible to configure extra VMs at build time. It doesn't directly help with that, but making it so that each VM doesn't need to provide everything itself will make creating external VMs easier when it does happen. In future we might want to apply a similar mechanism to service VMs, like netvm, but since we only have one of those so far, it's not clear which parts exactly would be duplicated, so I'm leaving it for now. Other future work is considering the impacts of the shared base image on guest isolation. Can guests observe whether reads of the shared base image hit the host page cache, or even an internal disk cache? At the moment I suspect that the base image doesn't have enough specialised code in it that there would be any interesting results, but it's worth thinking about if the shared image grows new functionality, whether it would be interesting to another guest to be able to observe whether those resources have previously been loaded or not. If this _does_ turn out to be a concern, it could be mitigated by simply copying the base image to temporary storage before booting a VM, and then booting the VM from the copy. Alyssa Ross (6): host/start-vm: support multiple block devices scripts/make-gpt.sh: add support for labels vm: build GPT images host/start-vm: boot using partition label release: rename from "img" img/app: extract from appvm-{lynx,catgirl} Documentation/creating-vms.adoc | 8 +- Documentation/getting-spectrum.adoc | 2 +- host/initramfs/extfs.nix | 11 +- host/rootfs/default.nix | 11 +- host/start-vm/start-vm.rs | 43 ++++-- {vm/app/lynx => img/app}/Makefile | 57 ++++---- {vm/app/catgirl => img/app}/bin | 0 {vm/app/catgirl => img/app}/default.nix | 22 ++-- img/app/etc/fstab | 8 ++ {vm/app/catgirl => img/app}/etc/init | 0 {vm/app/catgirl => img/app}/etc/mdev.conf | 0 {vm/app/lynx => img/app}/etc/mdev/iface | 2 +- {vm/app/catgirl => img/app}/etc/passwd | 0 .../catgirl => img/app}/etc/passwd.license | 0 {vm/app/catgirl => img/app}/etc/resolv.conf | 0 .../app}/etc/s6-linux-init/scripts/rc.init | 1 + .../s6-rc/lynx => img/app/etc/s6-rc/app}/run | 3 +- .../catgirl => img/app/etc/s6-rc/app}/type | 0 .../app/etc/s6-rc/app}/type.license | 0 .../etc/s6-rc/mdevd-coldplug/dependencies | 0 .../app}/etc/s6-rc/mdevd-coldplug/type | 0 .../etc/s6-rc/mdevd-coldplug/type.license | 0 .../app}/etc/s6-rc/mdevd-coldplug/up | 0 .../app}/etc/s6-rc/mdevd/notification-fd | 0 .../etc/s6-rc/mdevd/notification-fd.license | 0 .../catgirl => img/app}/etc/s6-rc/mdevd/run | 0 .../catgirl => img/app}/etc/s6-rc/mdevd/type | 0 .../app}/etc/s6-rc/mdevd/type.license | 0 .../app}/etc/s6-rc/ok-all/contents | 0 .../catgirl => img/app}/etc/s6-rc/ok-all/type | 0 .../app}/etc/s6-rc/ok-all/type.license | 0 .../app}/etc/ssl/certs/ca-certificates.crt | 0 {vm/app/lynx => img/app}/shell.nix | 11 +- release.nix | 2 +- {img => release}/combined/default.nix | 0 {img => release}/combined/eosimages.nix | 0 {img => release}/combined/grub.cfg.in | 0 {img => release}/combined/run-vm.nix | 0 ...ble-gpt-partition-attribute-55-check.patch | 0 ...pt-disable-partition-table-CRC-check.patch | 0 .../0003-install-remove-Endless-OS-ad.patch | 0 ...4-finished-don-t-run-eos-diagnostics.patch | 0 ...omote-spectrum-not-the-Endless-forum.patch | 0 {img => release}/installer/app/default.nix | 0 .../installer/app/vendor-customer-support.ini | 0 {img => release}/installer/configuration.nix | 0 {img => release}/installer/default.nix | 0 {img => release}/installer/run-vm.nix | 0 {img => release}/installer/seat.rules | 0 {img => release}/live/Makefile | 0 {img => release}/live/default.nix | 0 {img => release}/live/shell.nix | 0 scripts/make-gpt.sh | 4 +- scripts/sfdisk-field.awk | 2 +- vm-lib/make-vm.nix | 46 +++++++ vm/app/catgirl.nix | 18 +++ vm/app/catgirl/Makefile | 123 ------------------ vm/app/catgirl/etc/fstab | 6 - vm/app/catgirl/etc/mdev/iface | 36 ----- .../catgirl/etc/s6-linux-init/scripts/rc.init | 10 -- vm/app/catgirl/etc/s6-rc/catgirl/run | 31 ----- .../data/appvm-catgirl/providers/net/netvm | 0 vm/app/catgirl/shell.nix | 17 --- vm/app/lynx.nix | 16 +++ vm/app/lynx/bin | 1 - vm/app/lynx/default.nix | 92 ------------- vm/app/lynx/etc/fstab | 6 - vm/app/lynx/etc/init | 5 - vm/app/lynx/etc/mdev.conf | 5 - vm/app/lynx/etc/passwd | 1 - vm/app/lynx/etc/passwd.license | 2 - vm/app/lynx/etc/resolv.conf | 4 - vm/app/lynx/etc/s6-rc/lynx/type | 1 - vm/app/lynx/etc/s6-rc/lynx/type.license | 2 - .../etc/s6-rc/mdevd-coldplug/dependencies | 4 - vm/app/lynx/etc/s6-rc/mdevd-coldplug/type | 1 - .../etc/s6-rc/mdevd-coldplug/type.license | 2 - vm/app/lynx/etc/s6-rc/mdevd-coldplug/up | 4 - vm/app/lynx/etc/s6-rc/mdevd/notification-fd | 1 - .../etc/s6-rc/mdevd/notification-fd.license | 2 - vm/app/lynx/etc/s6-rc/mdevd/run | 5 - vm/app/lynx/etc/s6-rc/mdevd/type | 1 - vm/app/lynx/etc/s6-rc/mdevd/type.license | 2 - vm/app/lynx/etc/s6-rc/ok-all/contents | 4 - vm/app/lynx/etc/s6-rc/ok-all/type | 1 - vm/app/lynx/etc/s6-rc/ok-all/type.license | 2 - vm/app/lynx/etc/ssl/certs/ca-certificates.crt | 1 - .../host/data/appvm-lynx/providers/net/netvm | 0 vm/sys/net/Makefile | 23 ++-- vm/sys/net/default.nix | 10 +- 90 files changed, 221 insertions(+), 451 deletions(-) rename {vm/app/lynx => img/app}/Makefile (66%) rename {vm/app/catgirl => img/app}/bin (100%) rename {vm/app/catgirl => img/app}/default.nix (77%) create mode 100644 img/app/etc/fstab rename {vm/app/catgirl => img/app}/etc/init (100%) rename {vm/app/catgirl => img/app}/etc/mdev.conf (100%) rename {vm/app/lynx => img/app}/etc/mdev/iface (98%) rename {vm/app/catgirl => img/app}/etc/passwd (100%) rename {vm/app/catgirl => img/app}/etc/passwd.license (100%) rename {vm/app/catgirl => img/app}/etc/resolv.conf (100%) rename {vm/app/lynx => img/app}/etc/s6-linux-init/scripts/rc.init (90%) rename {vm/app/lynx/etc/s6-rc/lynx => img/app/etc/s6-rc/app}/run (80%) rename {vm/app/catgirl/etc/s6-rc/catgirl => img/app/etc/s6-rc/app}/type (100%) rename {vm/app/catgirl/etc/s6-rc/catgirl => img/app/etc/s6-rc/app}/type.license (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd-coldplug/dependencies (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd-coldplug/type (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd-coldplug/type.license (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd-coldplug/up (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd/notification-fd (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd/notification-fd.license (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd/run (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd/type (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/mdevd/type.license (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/ok-all/contents (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/ok-all/type (100%) rename {vm/app/catgirl => img/app}/etc/s6-rc/ok-all/type.license (100%) rename {vm/app/catgirl => img/app}/etc/ssl/certs/ca-certificates.crt (100%) rename {vm/app/lynx => img/app}/shell.nix (50%) rename {img => release}/combined/default.nix (100%) rename {img => release}/combined/eosimages.nix (100%) rename {img => release}/combined/grub.cfg.in (100%) rename {img => release}/combined/run-vm.nix (100%) rename {img => release}/installer/app/0001-gpt-disable-gpt-partition-attribute-55-check.patch (100%) rename {img => release}/installer/app/0002-gpt-disable-partition-table-CRC-check.patch (100%) rename {img => release}/installer/app/0003-install-remove-Endless-OS-ad.patch (100%) rename {img => release}/installer/app/0004-finished-don-t-run-eos-diagnostics.patch (100%) rename {img => release}/installer/app/0005-finished-promote-spectrum-not-the-Endless-forum.patch (100%) rename {img => release}/installer/app/default.nix (100%) rename {img => release}/installer/app/vendor-customer-support.ini (100%) rename {img => release}/installer/configuration.nix (100%) rename {img => release}/installer/default.nix (100%) rename {img => release}/installer/run-vm.nix (100%) rename {img => release}/installer/seat.rules (100%) rename {img => release}/live/Makefile (100%) rename {img => release}/live/default.nix (100%) rename {img => release}/live/shell.nix (100%) create mode 100644 vm-lib/make-vm.nix create mode 100644 vm/app/catgirl.nix delete mode 100644 vm/app/catgirl/Makefile delete mode 100644 vm/app/catgirl/etc/fstab delete mode 100755 vm/app/catgirl/etc/mdev/iface delete mode 100755 vm/app/catgirl/etc/s6-linux-init/scripts/rc.init delete mode 100755 vm/app/catgirl/etc/s6-rc/catgirl/run delete mode 100644 vm/app/catgirl/host/data/appvm-catgirl/providers/net/netvm delete mode 100644 vm/app/catgirl/shell.nix create mode 100644 vm/app/lynx.nix delete mode 120000 vm/app/lynx/bin delete mode 100644 vm/app/lynx/default.nix delete mode 100644 vm/app/lynx/etc/fstab delete mode 100755 vm/app/lynx/etc/init delete mode 100644 vm/app/lynx/etc/mdev.conf delete mode 100644 vm/app/lynx/etc/passwd delete mode 100644 vm/app/lynx/etc/passwd.license delete mode 100644 vm/app/lynx/etc/resolv.conf delete mode 100644 vm/app/lynx/etc/s6-rc/lynx/type delete mode 100644 vm/app/lynx/etc/s6-rc/lynx/type.license delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd-coldplug/dependencies delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd-coldplug/type delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd-coldplug/type.license delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd-coldplug/up delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd/notification-fd delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd/notification-fd.license delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd/run delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd/type delete mode 100644 vm/app/lynx/etc/s6-rc/mdevd/type.license delete mode 100644 vm/app/lynx/etc/s6-rc/ok-all/contents delete mode 100644 vm/app/lynx/etc/s6-rc/ok-all/type delete mode 100644 vm/app/lynx/etc/s6-rc/ok-all/type.license delete mode 120000 vm/app/lynx/etc/ssl/certs/ca-certificates.crt delete mode 100644 vm/app/lynx/host/data/appvm-lynx/providers/net/netvm base-commit: 2cc5bf637f944ec950f7a8162018f421e6671315 -- 2.37.1