run: move /nix to / tmpfs

Since we started using the block device as swap, we've been building
on overlayfs, backed by tmpfs.  This has caused me purity options in
the past, and the backing tmpfs is also quite small and would need to
be resized.  So let's just get rid of that, and do everything on the /
tmpfs.  This means we need to copy the Nix store again, which makes
boot slower, and we also can't entirely get rid of the squashfs, since
the system is booted from it.  The latter has the potential to get
better if systemd makes it possible to restart userspace at some
point.

Signed-off-by: Alyssa Ross <hi@alyssa.is>

1 files changed, 40 insertions, 0 deletions

diff --git a/run b/run
index 9d890c4..2c24025 100644
--- a/run
+++ b/run
@@ -20,6 +20,46 @@ pagesize=$(getconf PAGESIZE)
 inodes=$((size / pagesize))
 mount -o remount,size="$size",nr_inodes="$inodes" /
 
+mkdir /nix2 /rshared
+
+# Make sure nothing is modifying the store while we're swapping it out.
+systemctl stop nix-daemon nix-daemon.socket
+
+# Create a copy of the store on the root tmpfs, so that builds are not
+# running on overlayfs, which is a bit weird and can introduce impurities.
+cp -a /nix/store /nix/var /nix2
+
+# Create another instance of the root filesystem, so the store
+# filesystems are still accessible for unmounting after we do a bind
+# mount over the top of them in /.
+mount --rbind / /rshared
+
+# Mounts should propagate from /rshared to /, so that unmounting the
+# filesystems in /rshared unmounts them in /, but they should not
+# propagate in the other direction, so that bind mounting /nix2 over
+# /nix does not obscure those filesytems in /rshared.
+mount --make-slave /
+mount --bind /nix2 /nix
+mount --make-shared /
+systemctl start nix-daemon.socket
+
+# For some reason, the overlay ends up being mounted several times, so
+# keep unmounting it until it's gone, then unmount its components.
+# The filesystems will probably still end up in use for as long as
+# we're up, because programs already running will have pages from
+# them.  We could fix this by restarting userspace after setting up
+# the filesystems, but we'd need to either do this earlier in boot
+# (which isn't feasible because we need systemd-run-generator to be
+# able to sequence commands), or systemd would need to support this,
+# e.g.  https://github.com/systemd/systemd/pull/27435
+while umount -l /rshared/nix/store; do :; done
+umount -l /rshared/nix/.rw-store /rshared/nix/.ro-store
+
+# Detach /rshared, without affecting /.
+mount --make-rprivate /rshared
+umount -l /rshared
+rmdir /rshared
+
 for o in $(</proc/cmdline); do
 	case $o in
 		configuration=*)