On Thu, Aug 18, 2022 at 12:15:28PM +0300, Ville Ilvonen wrote:
> On Wed, Aug 17, 2022 at 4:39 PM Alyssa Ross <hi@alyssa.is> wrote:
>
> > Yeah, I agree something like this would be good.  Especially when
> > testing on hardware as you say.  I would like to think more about
> > exactly how this should work.  Do you think that, if you it were
> > possible to develop Spectrum on Spectrum, it would be acceptable to have
> > to reboot into a new configuration if the host system was changed?
> > (Assume that the process of actually building the new system is fast —
> > the reboot would be the main overhead.)
>
> Option to develop on the target system is something people (not vocal
> here) already expect. I think booting into a new configuration is MCU
> style development process and not a fast enough iteration cycle on
> Linux user space in all scenarios. Even in kernel driver development,
> one may want to unload/load module during development but even disable
> module loading as a security hardening mechanism in
> deployment/production configuration. Another example is security
> policies development (SELinux) - you want to iterate and test them in
> user space during development but you want to deploy them immutable.
> Then again, kernel or device tree changes will require rebooting.

Okay.  Obviously we can support booting with the filesystem mounted
read-write and no dm-verity.  But the problem with that is that then
changes have to be manually copied back to the source tree.  We could
try to add some conveniences for that (e.g. a facility to diff the
modified root filesystem with the orginal version).  It still means
making changes to the compiled system, rather than working on the
source, but for the kind of changes you're talking about, maybe doing
it that way and then integrating the changes into the Spectrum source
at the end wouldn't be too bad?

Ideally it would be nice to have something like nixos-rebuild switch,
that can build a new system and then switch into it.  But the problem
is, you have to figure out how to *automatically* take the runtime state
of the system (the processes and services that are running), and somehow
carry that over into the new system, where anything could have changed.
So I'm not sure how we'd do that.  NixOS's implementation of this is
massive, tightly coupled to both NixOS and systemd, and still it's not
very difficult to get it into a weird state where you need to reboot
anyway to get an accurate idea of what the change does (or, if you
don't realise you need to do that, you just think the change hasn't
worked).  If a kernel module has changed, do you try to unload and
reload it?  That's pretty complicated to implement, because you have to
unbind the devices from it, unload the module, then rebind them, and
even then the module might have had important state that's been lost.
If the developer is working on that kernel module it's probably okay,
but what if they're working on something else and just pulled a bunch of
changes that happened to include a kernel module change?  Same with if
e.g. the Weston service changes, except in that case we have no way to
restore the previous state.  So either we choose not to automatically
restart Weston, in which case we're not really running the new system,
but a weird hybrid that may have bugs not present in either the old or
new system, or we do restart Weston, and risk an unsuspecting user
losing their whole windowing session.

If there's a way we could make this work well, I'm open to it, but it's
not at all obvious to me what that would look like.