From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 458BD90114; Mon, 10 Oct 2022 23:32:19 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id 98DA690103; Mon, 10 Oct 2022 23:32:16 +0000 (UTC) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by atuin.qyliss.net (Postfix) with ESMTPS id 71B52900C8 for ; Mon, 10 Oct 2022 23:32:12 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 099635C01A4 for ; Mon, 10 Oct 2022 19:32:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 10 Oct 2022 19:32:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h=cc :content-transfer-encoding:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to; s=fm2; t=1665444728; x=1665531128; bh=hxg1V4byF8 qwy1BVYXyPs18SbaDOV/JAEU98262zt/k=; b=WZE00/6ybvygDV+xv3VMW0m0iD KXqRKThGWi3ZoIxxgDClm6vlxtUx+8ZneE3m0j+b05gLpYwnfv64p28NAx8ezykN a3xmkBgrTntmThWq3PxTrLL7kOYzDZLBPLbMeo0H3h9aS1q6BL8k7Eg6gwlmL0ru xl/lUjgKjJT6W6aB4D+wCYtU8P/v0/5Q4Vxkb15TZpZBl659XBbRL/wlA3JhWeY9 K4DcOhEBowplvnX50ZuJRNpk9RbLbsLBMjHbNSp3L+qOVw92xI+/F+GHIb5l4OcY djp2Q9rN3fN8qlSmLWDsBe/OLBdumO9OQ9oBG3pERjnRAiUPuVf27JwYuscw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:date:feedback-id:feedback-id:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1665444728; x=1665531128; bh=hxg1V4byF8qwy1BVYXyPs18SbaDO V/JAEU98262zt/k=; b=SPRz1LAXWERWJXhZPvpWSHQDmoC/+AnLvPr8oipiotq0 I8JiPjd7ffUnmfj6JLVw3UthQV1Cx8kWsCsBqyBcp9o0/P8dvXJqNo9P2iHrkyHQ 3ELzuYFZzBDGEXVUUWjEFHo3nYt8CnV29DkFuUlDYgUt9bPOvnx9plUtOqCL36P6 I0O8yrM3JRLX+855ocR5Kb169Fqf888E/MKjbBk4aHV5O75KYdn/a1skB7Dcqvdl P++6Tew16++ujjweut8q70Ea8F5dHYVRmYlrmnkGH17JO/7GrjJ1cl3Tl72KrLdK 8Gxit6rI0Yncom8NEYaO0vBc1Wk/hD0jRv59uCmxQA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfeejhedgvddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofggtgfgsehtkeertd ertdejnecuhfhrohhmpeetlhihshhsrgcutfhoshhsuceohhhisegrlhihshhsrgdrihhs qeenucggtffrrghtthgvrhhnpedvheekfeeuvdeijeevjeeigfefteffffffkeekjeffhe ffgeefvdefveeuleegfeenucffohhmrghinhepshhpvggtthhruhhmqdhoshdrohhrghen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehqhihlih hsshesgidvvddtrdhqhihlihhsshdrnhgvth X-ME-Proxy: Feedback-ID: i12284293:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 10 Oct 2022 19:32:07 -0400 (EDT) Received: by x220.qyliss.net (Postfix, from userid 1000) id C035EC42; Mon, 10 Oct 2022 23:32:05 +0000 (UTC) From: Alyssa Ross To: devel@spectrum-os.org Subject: [PATCH 00/22] Implement managing VMs with Nix Date: Mon, 10 Oct 2022 23:28:40 +0000 Message-Id: <20221010232909.1953738-1-hi@alyssa.is> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Message-ID-Hash: GVEFXUICF2QAULH2GNE4EIZXLQQSVZGT X-Message-ID-Hash: GVEFXUICF2QAULH2GNE4EIZXLQQSVZGT X-MailFrom: qyliss@x220.qyliss.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-devel.spectrum-os.org-0; header-match-devel.spectrum-os.org-1; header-match-devel.spectrum-os.org-2; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.5 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: IMPORTANT NOTE: this series should be applied on top of v2 of my previous series "Introduce a shared base for application VMs" [1]. I'm much happier with v2 of that series, but I only posted it yesterday so I still want to leave a little more opportunity for comment before applying it. [1]: https://spectrum-os.org/lists/archives/spectrum-devel/20221009114036.463071-1-hi@alyssa.is/ This series contains the final big chunk of work I had left to do on Spectrum's original NLnet grant. It adds support for managing Spectrum VMs from the Spectrum system itself using Nix. Nix is optional, and can co-exist with VMs provided in some other way. More information is included in the new documentation. Most of this work was done earlier this year, but I got stuck on some implementation details that prevented me from getting over the last hurdle until I came up with a solution. That's explained in more detail in patch 15. Patches 1–10 add support for configuring VMs with read/write access to host directories using virtiofs. Then, in patches 11–14, come various changes that make the default user data partition more suitable as a mutable filesystem, which we haven't actually used it for before. And then the remaining patches actually implement support for a VM that can run Nix and easily build VMs that are available on the host. Alyssa Ross (22): host/start-vm: use MAP_SHARED memory for VMs host/start-vm: implement shared directories host/rootfs: generate virtiofsd services Documentation: explain VM shared directories vm-lib/make-vm.nix: support shared directories img/app: add support for testing virtiofs img/app: don't block app startup on network online img/app: auto-mount virtiofs0 filesystem vm/app/mg.nix: init vm/app/mg.nix: open virtio filesystem in dired host/rootfs: move ext mounting to s6-rc service host/rootfs: automatically grow user partition host/rootfs: use a bigger test ext partition host/initramfs/extfs.nix: tar2ext4 -> mkfs.ext4 -d host/start-vm: resolve VM symlinks with /ext root host/rootfs: resolve VM symlinks with /ext root Documentation: explain /ext symlink resolution host/start-vm: increase memory size to 512M vm/app/nix: add vm-lib/make-vms.nix: add host/initramfs/extfs.nix: add example Nix-built VM Documentation: add how-to guide for Nix-built VMs .gitignore | 5 +- Documentation/_sass/custom/custom.scss | 22 ++++++ Documentation/creating-vms.adoc | 12 ++- Documentation/nix-vms.adoc | 22 ++++++ host/initramfs/extfs.nix | 29 +++++-- host/rootfs/Makefile | 13 +++- host/rootfs/default.nix | 16 +++- host/rootfs/etc/mdev/block/add | 1 - host/rootfs/etc/s6-rc/ext-rc-init/up | 19 ++++- host/rootfs/etc/s6-rc/ext/up | 5 +- host/rootfs/etc/template/fs/notification-fd | 1 + .../etc/template/fs/notification-fd.license | 2 + host/rootfs/etc/template/fs/run | 11 +++ host/rootfs/etc/template/fs/type | 1 + host/rootfs/etc/template/fs/type.license | 2 + host/start-vm/fs.c | 17 +++++ host/start-vm/fs.rs | 68 +++++++++++++++++ host/start-vm/lib.rs | 54 ++++++++++--- host/start-vm/meson.build | 2 +- host/start-vm/start-vm.rs | 15 ++-- host/start-vm/tests/meson.build | 4 + host/start-vm/tests/vm_command-basic.rs | 6 +- .../tests/vm_command-config-symlink.rs | 30 ++++++++ host/start-vm/tests/vm_command-shared-dir.rs | 43 +++++++++++ img/app/Makefile | 24 +++++- img/app/etc/mdev.conf | 3 +- img/app/etc/mdev/iface | 4 +- img/app/etc/mdev/listen | 12 +++ img/app/etc/mdev/virtiofs | 10 +++ img/app/etc/mdev/wait | 15 ++++ img/app/etc/s6-rc/ok-all/contents | 1 + img/app/shell.nix | 6 +- tools/resolve_in_root/default.nix | 23 ++++++ tools/resolve_in_root/meson.build | 10 +++ tools/resolve_in_root/resolve_in_root.c | 76 +++++++++++++++++++ tools/resolve_in_root/test.sh | 11 +++ vm-lib/make-vm.nix | 20 ++++- vm-lib/make-vms.nix | 19 +++++ vm/app/catgirl.nix | 1 + vm/app/lynx.nix | 1 + vm/app/{lynx.nix => mg.nix} | 10 ++- vm/app/nix/bin/vm-rebuild | 25 ++++++ vm/app/nix/default.nix | 43 +++++++++++ vm/app/nix/example.nix | 13 ++++ 44 files changed, 673 insertions(+), 54 deletions(-) create mode 100644 Documentation/_sass/custom/custom.scss create mode 100644 Documentation/nix-vms.adoc create mode 100644 host/rootfs/etc/template/fs/notification-fd create mode 100644 host/rootfs/etc/template/fs/notification-fd.license create mode 100755 host/rootfs/etc/template/fs/run create mode 100644 host/rootfs/etc/template/fs/type create mode 100644 host/rootfs/etc/template/fs/type.license create mode 100644 host/start-vm/fs.c create mode 100644 host/start-vm/fs.rs create mode 100644 host/start-vm/tests/vm_command-config-symlink.rs create mode 100644 host/start-vm/tests/vm_command-shared-dir.rs create mode 100755 img/app/etc/mdev/listen create mode 100755 img/app/etc/mdev/virtiofs create mode 100755 img/app/etc/mdev/wait create mode 100644 tools/resolve_in_root/default.nix create mode 100644 tools/resolve_in_root/meson.build create mode 100644 tools/resolve_in_root/resolve_in_root.c create mode 100755 tools/resolve_in_root/test.sh create mode 100644 vm-lib/make-vms.nix copy vm/app/{lynx.nix => mg.nix} (52%) create mode 100755 vm/app/nix/bin/vm-rebuild create mode 100644 vm/app/nix/default.nix create mode 100644 vm/app/nix/example.nix -- 2.37.1