nixos-rebuild: run activation inside systemd-run

Right now, running `nixos-rebuild switch` in a remote system via SSH may
put the system in an unusable state by restarting services (e.g.:
network ones like systemd-networkd.service) during the update. This will
cause the SSH to lose access to the TTY, stopping the process entirely.

This commit wraps up the call to `switch-to-configuration` inside a
systemd-run call, making the switch resiliant against TTY loss, since it
will allocate its own TTY. For the user this should be entirely
transparent though, with the only visible change is that the
`switch-to-configuration` messages will be logged through journalctl
(e.g.: `journalctl -u nixos-rebuild-switch-to-configuration`).

2 files changed, 42 insertions, 4 deletions

diff --git a/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.8 b/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.8
index b0ff5b0a672..d947361b3bc 100644
--- a/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.8
+++ b/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.8
@@ -421,6 +421,14 @@ nixpkgs=./my-nixpkgs
 Additional options to be passed to
 .Ic ssh
 on the command line.
+.Ed
+.
+.It Ev NIXOS_SWITCH_USE_DIRTY_ENV
+Expose the the current environment variables to post activation scripts. Will
+skip usage of
+.Ic systemd-run
+during system activation. Possibly dangerous, specially in remote environments
+(e.g.: via SSH). Will be removed in the future.
 .El
 .
 .
diff --git a/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh b/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh
index 2f89642845e..68e00690652 100755
--- a/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh
+++ b/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh
@@ -653,18 +653,48 @@ fi
 # If we're not just building, then make the new configuration the boot
 # default and/or activate it now.
 if [[ "$action" = switch || "$action" = boot || "$action" = test || "$action" = dry-activate ]]; then
+    # Using systemd-run here to protect against PTY failures/network
+    # disconnections during rebuild.
+    # See: https://github.com/NixOS/nixpkgs/issues/39118
+    cmd=(
+        "systemd-run"
+        "-E" "LOCALE_ARCHIVE" # Will be set to new value early in switch-to-configuration script, but interpreter starts out with old value
+        "--collect"
+        "--no-ask-password"
+        "--pty"
+        "--quiet"
+        "--same-dir"
+        "--service-type=exec"
+        "--unit=nixos-rebuild-switch-to-configuration"
+        "--wait"
+    )
+    # Check if we have a working systemd-run. In chroot environments we may have
+    # a non-working systemd, so we fallback to not using systemd-run.
+    # You may also want to explicitly set NIXOS_SWITCH_USE_DIRTY_ENV environment
+    # variable, since systemd-run runs inside an isolated environment and
+    # this may break some post-switch scripts. However keep in mind that this
+    # may be dangerous in remote access (e.g. SSH).
+    if [[ -n "$NIXOS_SWITCH_USE_DIRTY_ENV" ]]; then
+        log "warning: skipping systemd-run since NIXOS_SWITCH_USE_DIRTY_ENV is set. This environment variable will be ignored in the future"
+        cmd=()
+    elif ! targetHostCmd "${cmd[@]}" true &>/dev/null; then
+        logVerbose "Skipping systemd-run to switch configuration since it is not working in target host."
+        cmd=("env" "-i" "LOCALE_ARCHIVE=$LOCALE_ARCHIVE")
+    else
+        logVerbose "Using systemd-run to switch configuration."
+    fi
     if [[ -z "$specialisation" ]]; then
-        cmd="$pathToConfig/bin/switch-to-configuration"
+        cmd+=("$pathToConfig/bin/switch-to-configuration")
     else
-        cmd="$pathToConfig/specialisation/$specialisation/bin/switch-to-configuration"
+        cmd+=("$pathToConfig/specialisation/$specialisation/bin/switch-to-configuration")
 
-        if [[ ! -f "$cmd" ]]; then
+        if [[ ! -f "${cmd[-1]}" ]]; then
             log "error: specialisation not found: $specialisation"
             exit 1
         fi
     fi
 
-    if ! targetHostCmd "$cmd" "$action"; then
+    if ! targetHostCmd "${cmd[@]}" "$action"; then
         log "warning: error(s) occurred while switching to the new configuration"
         exit 1
     fi