general high-level discussion about spectrum
 help / color / mirror / Atom feed

* HW identification and configuration on Spectrum
@ 2022-08-16 15:50 Ville Ilvonen
  2022-08-17  7:52 ` Alyssa Ross
  0 siblings, 1 reply; 7+ messages in thread
From: Ville Ilvonen @ 2022-08-16 15:50 UTC (permalink / raw)
  To: discuss

Hi,

Now that we've been developing Spectrum ARM (aarch64) support
with iMX8 boards, I'd like to get back to Spectrum HW configuration design.

On x86 the generic image with kernel supporting most devices as modules can
make sense. On ARM, the vendor specific BSP HW quirks are more common.

As of now, the spectrum fork for aarch64 just adds another config
after rpi configs
and replaces the default config to use that to build. With small
changes this could
be handled like rpi configs. In addition, cloud-hypervisor accepts
kernel only in
EFI format for aarch64[1]. Anyway, this would allow us to build an
aarch64 Spectrum installer
- even make it with a more generic kernel. That takes us to ARM
vendor/device specific HW
quirks which would need to be handled anyway. I'll intentionally leave
device specific
kernel hardening and disabling kernel module loading for security
reasons for now.
As of now the vendor/device specifics are not supported unless one builds device
specific Spectrum image with all configs build-time and skips
installer altogether.

The other option that I see. We discussed earlier nix-hardware and
device specific modules.
That would bring nixos configuration.nix and installation supporting
scripts to Spectrum,
though. Those could be called from the Spectrum installer but it would
change the installer
logic from writing an image to dynamically configuring the device
during install based on user
selections.

Any thoughts which would be the preferred way? Maybe some other way?
In the end, HW specifics are needed also on x86 as we saw with NUCs
and different
Lenovo laptops in the spring. I'm not convinced one image to rule them
all is realistic or secure.

Finally, this is by no means blocking the hardened iMX8 based Spectrum
development
but will keep that work in Spectrum fork until there's an agreed path
to implement this.
Integrating this sooner and making it more generic would make Spectrum
more useful
for a wider audience.

Best regards,

-Ville

[1] https://github.com/tiiuae/spectrum/pull/3#issuecomment-1211834302


^ permalink raw reply	[flat|nested] 7+ messages in thread

• * Re: HW identification and configuration on Spectrum
    2022-08-16 15:50 HW identification and configuration on Spectrum Ville Ilvonen
  @ 2022-08-17  7:52 ` Alyssa Ross
    2022-08-17 13:25   ` Ville Ilvonen
    0 siblings, 1 reply; 7+ messages in thread
  From: Alyssa Ross @ 2022-08-17  7:52 UTC (permalink / raw)
    To: Ville Ilvonen; +Cc: discuss
  
  [-- Attachment #1: Type: text/plain, Size: 6042 bytes --]
  
  On Tue, Aug 16, 2022 at 06:50:48PM +0300, Ville Ilvonen wrote:
  > Hi,
  >
  > Now that we've been developing Spectrum ARM (aarch64) support
  > with iMX8 boards, I'd like to get back to Spectrum HW configuration design.
  >
  > On x86 the generic image with kernel supporting most devices as modules can
  > make sense. On ARM, the vendor specific BSP HW quirks are more common.
  
  What impact will Google's Generic Kernel Image[1] efforts have on this?
  As I understand it, going forward Android devices won't be allowed to
  make arbitrary kernel changes, and will be restricted to adding extra
  modules.  Which presumably means that SOCs aiming to be used for Android
  devices will have to work with the standard Android kernel, which is
  hopefully getting closer to mainline over time[2].
  
  Regardless, I understand that there will always be some cases where a
  non-upstream kernel is a necessity (even if the Android situation gets
  better, there will always be a new MacBook that doesn't have upstream
  drivers yet!) so I am keen to figure out how to support that well in
  Spectrum.
  
  [1]: https://source.android.com/docs/core/architecture/kernel/generic-kernel-image
  [2]: https://lwn.net/Articles/830979/
  
  > As of now, the spectrum fork for aarch64 just adds another config
  > after rpi configs
  > and replaces the default config to use that to build. With small
  > changes this could
  > be handled like rpi configs. In addition, cloud-hypervisor accepts
  > kernel only in
  > EFI format for aarch64[1]. Anyway, this would allow us to build an
  > aarch64 Spectrum installer
  > - even make it with a more generic kernel. That takes us to ARM
  > vendor/device specific HW
  > quirks which would need to be handled anyway. I'll intentionally leave
  > device specific
  > kernel hardening and disabling kernel module loading for security
  > reasons for now.
  > As of now the vendor/device specifics are not supported unless one builds device
  > specific Spectrum image with all configs build-time and skips
  > installer altogether.
  >
  > The other option that I see. We discussed earlier nix-hardware and
  > device specific modules.
  > That would bring nixos configuration.nix and installation supporting
  > scripts to Spectrum,
  > though. Those could be called from the Spectrum installer but it would
  > change the installer
  > logic from writing an image to dynamically configuring the device
  > during install based on user
  > selections.
  
  I don't think the full NixOS module system, with rebuilds, etc. belongs
  in Spectrum.  Being able to treat images as immutable makes it easier to
  provide various strong security guarantees.  But not wanting to
  integrate the full module system doesn't prevent us taking advantage of
  nixos-hardware.  It's possible to evaluate NixOS modules standalone in
  a Spectrum build, in fact we already do that to reuse NixOS's list of
  all redistributable firmware packages[3].  We could do a similar thing
  to extract the kernel that nixos-hardware configures for a particular
  device, something like this:
  
    inherit (nixos {
      configuration = [ <nixos-hardware/pine64/pinebook-pro/default.nix> ];
    }.config.boot.kernelPackages) kernel;
  
  And naturally which device that's pulling from should be configurable —
  we'll want to have a config file somewhere, just not a full NixOS one.
  
  In the medium term, I'd like to decouple nixos-hardware's custom kernel
  packages from NixOS configurations.  But that would require somebody
  finding the time to sit down and make the change, and also convince
  other nixos-hardware users that it's the way to go.  I don't think it
  would be a problem though, especially if it meant nixos-hardware getting
  more active maintenance, which it's lacking at the moment because it's
  not too well advertised and so not enough people are using it.
  
  I am intrigued by the idea of the installer being able to generate
  images, though.  Using Nix with a substituter configured on the
  installer image would mean that it could download a pre-built image if
  one exists for that platform, or fall back to generating one if not.
  (And if there was a pre-built image, it would even still be able to
  properly Secure Boot with a trusted key once we're in a position to do
  that.)  So I'm definitely keen on exploring that idea, but it might be
  something to do a bit down the road since the work to generate
  board-specific Spectrum images wouldn't be contingent on it.
  
  [3]: https://spectrum-os.org/git/spectrum/tree/host/rootfs/default.nix?id=b01594b2c089ce2434dacddccf9a285af7334d24#n64
  
  > Any thoughts which would be the preferred way? Maybe some other way?
  > In the end, HW specifics are needed also on x86 as we saw with NUCs
  > and different
  > Lenovo laptops in the spring. I'm not convinced one image to rule them
  > all is realistic or secure.
  
  The issues we saw with Lenovo laptops, etc. wouldn't have been solved by
  device specific images — those devices were broken because of bugs that
  hadn't affected any other systems I'd tried, but in the end the fixes
  were applicable everywhere.  That itself isn't an indication that we need
  device-specific images, just more hardware testing.
  
  But as I said above, I'm open to having an officially blessed
  configuration mechanism to make it possible to build custom images.
  
  > Finally, this is by no means blocking the hardened iMX8 based Spectrum
  > development
  > but will keep that work in Spectrum fork until there's an agreed path
  > to implement this.
  > Integrating this sooner and making it more generic would make Spectrum
  > more useful
  > for a wider audience.
  
  Makes sense — although of course, if any of the work that's been done so
  far is not i.MX8-specific, but is instead just generic stuff to make
  Spectrum more ARM-friendly or cross-compilable, I'd be happy to look at
  those patches already since they'll be relevant regardless of how we do
  device-specific stuff.
  
  > Best regards,
  >
  > -Ville
  >
  > [1] https://github.com/tiiuae/spectrum/pull/3#issuecomment-1211834302
  
  [-- Attachment #2: signature.asc --]
  [-- Type: application/pgp-signature, Size: 833 bytes --]
  
  ^ permalink raw reply	[flat|nested] 7+ messages in thread

• • * Re: HW identification and configuration on Spectrum
      2022-08-17  7:52 ` Alyssa Ross
    @ 2022-08-17 13:25   ` Ville Ilvonen
      2022-08-17 13:39     ` Alyssa Ross
      0 siblings, 1 reply; 7+ messages in thread
    From: Ville Ilvonen @ 2022-08-17 13:25 UTC (permalink / raw)
      To: Alyssa Ross; +Cc: discuss
    
    On Wed, Aug 17, 2022 at 10:52 AM Alyssa Ross <hi@alyssa.is> wrote:
    >
    > On Tue, Aug 16, 2022 at 06:50:48PM +0300, Ville Ilvonen wrote:
    > > Hi,
    > >
    > > Now that we've been developing Spectrum ARM (aarch64) support
    > > with iMX8 boards, I'd like to get back to Spectrum HW configuration design.
    > >
    > > On x86 the generic image with kernel supporting most devices as modules can
    > > make sense. On ARM, the vendor specific BSP HW quirks are more common.
    >
    > What impact will Google's Generic Kernel Image[1] efforts have on this?
    > As I understand it, going forward Android devices won't be allowed to
    > make arbitrary kernel changes, and will be restricted to adding extra
    > modules.  Which presumably means that SOCs aiming to be used for Android
    > devices will have to work with the standard Android kernel, which is
    > hopefully getting closer to mainline over time[2].
    
    ARM goes beyond Google Android ecosystem - including automotive and
    increasingly servers and other personal devices than smartphones. GKI is
    definitely the right direction but I'm afraid Google can't pull the
    BSPs on other
    sectors in. Which takes us to your point below. Nothing to add there.
    
    > Regardless, I understand that there will always be some cases where a
    > non-upstream kernel is a necessity (even if the Android situation gets
    > better, there will always be a new MacBook that doesn't have upstream
    > drivers yet!) so I am keen to figure out how to support that well in
    > Spectrum.
    >
    > [1]: https://source.android.com/docs/core/architecture/kernel/generic-kernel-image
    > [2]: https://lwn.net/Articles/830979/
    >
    > > As of now, the spectrum fork for aarch64 just adds another config
    > > after rpi configs
    > > and replaces the default config to use that to build. With small
    > > changes this could
    > > be handled like rpi configs. In addition, cloud-hypervisor accepts
    > > kernel only in
    > > EFI format for aarch64[1]. Anyway, this would allow us to build an
    > > aarch64 Spectrum installer
    > > - even make it with a more generic kernel. That takes us to ARM
    > > vendor/device specific HW
    > > quirks which would need to be handled anyway. I'll intentionally leave
    > > device specific
    > > kernel hardening and disabling kernel module loading for security
    > > reasons for now.
    > > As of now the vendor/device specifics are not supported unless one builds device
    > > specific Spectrum image with all configs build-time and skips
    > > installer altogether.
    > >
    > > The other option that I see. We discussed earlier nix-hardware and
    > > device specific modules.
    > > That would bring nixos configuration.nix and installation supporting
    > > scripts to Spectrum,
    > > though. Those could be called from the Spectrum installer but it would
    > > change the installer
    > > logic from writing an image to dynamically configuring the device
    > > during install based on user
    > > selections.
    >
    > I don't think the full NixOS module system, with rebuilds, etc. belongs
    > in Spectrum.  Being able to treat images as immutable makes it easier to
    > provide various strong security guarantees.  But not wanting to
    
    This was and still is one important design decision to build on Spectrum.
    Regardless, it makes development iterations on target HW more challenging
    than needed. Conceptually we've had discussions on separating concerns
    between "development system - writable, easily updatable" and
    "production system - immutable, updated as image". Latter could have more
    hardening, security policies etc. enabled which makes development more
    difficult by design. In practice, some developers have remounted the
    Spectrum file
    system as writable to make development iterations easier. In many cases,
    the development must be done on the target HW which brings us back to the need
    for the "development system" configuration. Update image iteration
    cycle is too slow.
    
    > integrate the full module system doesn't prevent us taking advantage of
    > nixos-hardware.  It's possible to evaluate NixOS modules standalone in
    > a Spectrum build, in fact we already do that to reuse NixOS's list of
    > all redistributable firmware packages[3].  We could do a similar thing
    > to extract the kernel that nixos-hardware configures for a particular
    > device, something like this:
    >
    >   inherit (nixos {
    >     configuration = [ <nixos-hardware/pine64/pinebook-pro/default.nix> ];
    >   }.config.boot.kernelPackages) kernel;
    >
    > And naturally which device that's pulling from should be configurable —
    > we'll want to have a config file somewhere, just not a full NixOS one.
    
    This made me propose nix-hardware usage more and think if we could have
    what I called "development configuration". In essence, nix-hardware is
    NixOS channel and we could have a custom channel to support development
    as well (e.g. dev git repo(s)).
    
    > In the medium term, I'd like to decouple nixos-hardware's custom kernel
    > packages from NixOS configurations.  But that would require somebody
    > finding the time to sit down and make the change, and also convince
    > other nixos-hardware users that it's the way to go.  I don't think it
    > would be a problem though, especially if it meant nixos-hardware getting
    > more active maintenance, which it's lacking at the moment because it's
    > not too well advertised and so not enough people are using it.
    
    Ideally yes and I hope we could contribute to that effort. However, we need
    to focus on getting Spectrum running on aarch64 with imx8 for now. For that
    I'm reading that nixos-hardware approach is preferred.
    
    > I am intrigued by the idea of the installer being able to generate
    > images, though.  Using Nix with a substituter configured on the
    > installer image would mean that it could download a pre-built image if
    > one exists for that platform, or fall back to generating one if not.
    > (And if there was a pre-built image, it would even still be able to
    > properly Secure Boot with a trusted key once we're in a position to do
    > that.)  So I'm definitely keen on exploring that idea, but it might be
    > something to do a bit down the road since the work to generate
    > board-specific Spectrum images wouldn't be contingent on it.
    
    Agree, but I'd also move this down the road.
    
    > [3]: https://spectrum-os.org/git/spectrum/tree/host/rootfs/default.nix?id=b01594b2c089ce2434dacddccf9a285af7334d24#n64
    >
    > > Any thoughts which would be the preferred way? Maybe some other way?
    > > In the end, HW specifics are needed also on x86 as we saw with NUCs
    > > and different
    > > Lenovo laptops in the spring. I'm not convinced one image to rule them
    > > all is realistic or secure.
    >
    > The issues we saw with Lenovo laptops, etc. wouldn't have been solved by
    > device specific images — those devices were broken because of bugs that
    > hadn't affected any other systems I'd tried, but in the end the fixes
    > were applicable everywhere.  That itself isn't an indication that we need
    > device-specific images, just more hardware testing.
    
    Fair enough, there were those as well. But there were also device
    specific quirks.
    There always will be. HW makers sell HW with their constraints. There
    will be hacks.
    
    > But as I said above, I'm open to having an officially blessed
    > configuration mechanism to make it possible to build custom images.
    
    Sounds good. Would you also please share your thoughts on this "development
    configuration"? Like some nix tooling enabled for "development" and not included
    in the immutable "production" image.
    
    > > Finally, this is by no means blocking the hardened iMX8 based Spectrum
    > > development
    > > but will keep that work in Spectrum fork until there's an agreed path
    > > to implement this.
    > > Integrating this sooner and making it more generic would make Spectrum
    > > more useful
    > > for a wider audience.
    >
    > Makes sense — although of course, if any of the work that's been done so
    > far is not i.MX8-specific, but is instead just generic stuff to make
    > Spectrum more ARM-friendly or cross-compilable, I'd be happy to look at
    > those patches already since they'll be relevant regardless of how we do
    > device-specific stuff.
    
    Thanks, I think it's time to start looking for those with generic
    stuff in mind so that
    the board bring-up won't diverge too much from the Spectrum mainline.
    
    -Ville
    
    
    ^ permalink raw reply	[flat|nested] 7+ messages in thread

  • • * Re: HW identification and configuration on Spectrum
        2022-08-17 13:25   ` Ville Ilvonen
      @ 2022-08-17 13:39     ` Alyssa Ross
        2022-08-18  9:15       ` Ville Ilvonen
        0 siblings, 1 reply; 7+ messages in thread
      From: Alyssa Ross @ 2022-08-17 13:39 UTC (permalink / raw)
        To: Ville Ilvonen; +Cc: discuss
      
      [-- Attachment #1: Type: text/plain, Size: 4967 bytes --]
      
      On Wed, Aug 17, 2022 at 04:25:20PM +0300, Ville Ilvonen wrote:
      > > > As of now, the spectrum fork for aarch64 just adds another config
      > > > after rpi configs
      > > > and replaces the default config to use that to build. With small
      > > > changes this could
      > > > be handled like rpi configs. In addition, cloud-hypervisor accepts
      > > > kernel only in
      > > > EFI format for aarch64[1]. Anyway, this would allow us to build an
      > > > aarch64 Spectrum installer
      > > > - even make it with a more generic kernel. That takes us to ARM
      > > > vendor/device specific HW
      > > > quirks which would need to be handled anyway. I'll intentionally leave
      > > > device specific
      > > > kernel hardening and disabling kernel module loading for security
      > > > reasons for now.
      > > > As of now the vendor/device specifics are not supported unless one builds device
      > > > specific Spectrum image with all configs build-time and skips
      > > > installer altogether.
      > > >
      > > > The other option that I see. We discussed earlier nix-hardware and
      > > > device specific modules.
      > > > That would bring nixos configuration.nix and installation supporting
      > > > scripts to Spectrum,
      > > > though. Those could be called from the Spectrum installer but it would
      > > > change the installer
      > > > logic from writing an image to dynamically configuring the device
      > > > during install based on user
      > > > selections.
      > >
      > > I don't think the full NixOS module system, with rebuilds, etc. belongs
      > > in Spectrum.  Being able to treat images as immutable makes it easier to
      > > provide various strong security guarantees.  But not wanting to
      >
      > This was and still is one important design decision to build on Spectrum.
      > Regardless, it makes development iterations on target HW more challenging
      > than needed. Conceptually we've had discussions on separating concerns
      > between "development system - writable, easily updatable" and
      > "production system - immutable, updated as image". Latter could have more
      > hardening, security policies etc. enabled which makes development more
      > difficult by design. In practice, some developers have remounted the
      > Spectrum file
      > system as writable to make development iterations easier. In many cases,
      > the development must be done on the target HW which brings us back to the need
      > for the "development system" configuration. Update image iteration
      > cycle is too slow.
      
      Yeah, I agree something like this would be good.  Especially when
      testing on hardware as you say.  I would like to think more about
      exactly how this should work.  Do you think that, if you it were
      possible to develop Spectrum on Spectrum, it would be acceptable to have
      to reboot into a new configuration if the host system was changed?
      (Assume that the process of actually building the new system is fast —
      the reboot would be the main overhead.)
      
      > > integrate the full module system doesn't prevent us taking advantage of
      > > nixos-hardware.  It's possible to evaluate NixOS modules standalone in
      > > a Spectrum build, in fact we already do that to reuse NixOS's list of
      > > all redistributable firmware packages[3].  We could do a similar thing
      > > to extract the kernel that nixos-hardware configures for a particular
      > > device, something like this:
      > >
      > >   inherit (nixos {
      > >     configuration = [ <nixos-hardware/pine64/pinebook-pro/default.nix> ];
      > >   }.config.boot.kernelPackages) kernel;
      > >
      > > And naturally which device that's pulling from should be configurable —
      > > we'll want to have a config file somewhere, just not a full NixOS one.
      >
      > This made me propose nix-hardware usage more and think if we could have
      > what I called "development configuration". In essence, nix-hardware is
      > NixOS channel and we could have a custom channel to support development
      > as well (e.g. dev git repo(s)).
      
      You mean set it to your own, custom version of nixos-hardware that
      included WIP support for the board you were working on?  Yeah, that
      wouldn't be a problem at all.
      
      > > In the medium term, I'd like to decouple nixos-hardware's custom kernel
      > > packages from NixOS configurations.  But that would require somebody
      > > finding the time to sit down and make the change, and also convince
      > > other nixos-hardware users that it's the way to go.  I don't think it
      > > would be a problem though, especially if it meant nixos-hardware getting
      > > more active maintenance, which it's lacking at the moment because it's
      > > not too well advertised and so not enough people are using it.
      >
      > Ideally yes and I hope we could contribute to that effort. However, we need
      > to focus on getting Spectrum running on aarch64 with imx8 for now. For that
      > I'm reading that nixos-hardware approach is preferred.
      
      Yeah, it's definitely the way to go.  And if we can make nixos-hardware
      better in future, that would just be further progress on top of
      integrating nixos-hardware as described above.
      
      [-- Attachment #2: signature.asc --]
      [-- Type: application/pgp-signature, Size: 833 bytes --]
      
      ^ permalink raw reply	[flat|nested] 7+ messages in thread

    • • * Re: HW identification and configuration on Spectrum
          2022-08-17 13:39     ` Alyssa Ross
        @ 2022-08-18  9:15       ` Ville Ilvonen
          2022-08-18 10:17         ` Development on the Spectrum host Alyssa Ross
          0 siblings, 1 reply; 7+ messages in thread
        From: Ville Ilvonen @ 2022-08-18  9:15 UTC (permalink / raw)
          To: Alyssa Ross; +Cc: discuss
        
        On Wed, Aug 17, 2022 at 4:39 PM Alyssa Ross <hi@alyssa.is> wrote:
        
        > Yeah, I agree something like this would be good.  Especially when
        > testing on hardware as you say.  I would like to think more about
        > exactly how this should work.  Do you think that, if you it were
        > possible to develop Spectrum on Spectrum, it would be acceptable to have
        > to reboot into a new configuration if the host system was changed?
        > (Assume that the process of actually building the new system is fast —
        > the reboot would be the main overhead.)
        
        Option to develop on the target system is something people (not vocal
        here) already expect. I think booting into a new configuration is MCU
        style development process and not a fast enough iteration cycle on
        Linux user space in all scenarios. Even in kernel driver development,
        one may want to unload/load module during development but even disable
        module loading as a security hardening mechanism in
        deployment/production configuration. Another example is security
        policies development (SELinux) - you want to iterate and test them in
        user space during development but you want to deploy them immutable.
        Then again, kernel or device tree changes will require rebooting.
        
        Point is - rebooting for everything is not development friendly enough
        mechanism alone.
        
        > You mean set it to your own, custom version of nixos-hardware that
        > included WIP support for the board you were working on?  Yeah, that
        
        Yeah, like an own git repo as source for nix-channel to pick the
        nix-hardware module. Ideally, some of that could be easily contributed
        to nix-hardware upstream - currently it seems to accept different
        quality support for different HW so that should not be an issue.
        
        > wouldn't be a problem at all.
        
        Sounds good.
        
        > > > In the medium term, I'd like to decouple nixos-hardware's custom kernel
        > > > packages from NixOS configurations.  But that would require somebody
        > > > finding the time to sit down and make the change, and also convince
        > > > other nixos-hardware users that it's the way to go.  I don't think it
        > > > would be a problem though, especially if it meant nixos-hardware getting
        > > > more active maintenance, which it's lacking at the moment because it's
        > > > not too well advertised and so not enough people are using it.
        > >
        > > Ideally yes and I hope we could contribute to that effort. However, we need
        > > to focus on getting Spectrum running on aarch64 with imx8 for now. For that
        > > I'm reading that nixos-hardware approach is preferred.
        >
        > Yeah, it's definitely the way to go.  And if we can make nixos-hardware
        > better in future, that would just be further progress on top of
        > integrating nixos-hardware as described above.
        
        Right. Even iMX8 dev board generic support might be interesting for
        other projects. I've seen iMX8M EVK being used at least with Zircon on
        Fuchsia.
        
        Best,
        
        -Ville
        
        
        ^ permalink raw reply	[flat|nested] 7+ messages in thread

      • • * Development on the Spectrum host
            2022-08-18  9:15       ` Ville Ilvonen
          @ 2022-08-18 10:17         ` Alyssa Ross
            2022-08-19  6:26           ` Ville Ilvonen
            0 siblings, 1 reply; 7+ messages in thread
          From: Alyssa Ross @ 2022-08-18 10:17 UTC (permalink / raw)
            To: Ville Ilvonen; +Cc: discuss
          
          [-- Attachment #1: Type: text/plain, Size: 3650 bytes --]
          
          On Thu, Aug 18, 2022 at 12:15:28PM +0300, Ville Ilvonen wrote:
          > On Wed, Aug 17, 2022 at 4:39 PM Alyssa Ross <hi@alyssa.is> wrote:
          >
          > > Yeah, I agree something like this would be good.  Especially when
          > > testing on hardware as you say.  I would like to think more about
          > > exactly how this should work.  Do you think that, if you it were
          > > possible to develop Spectrum on Spectrum, it would be acceptable to have
          > > to reboot into a new configuration if the host system was changed?
          > > (Assume that the process of actually building the new system is fast —
          > > the reboot would be the main overhead.)
          >
          > Option to develop on the target system is something people (not vocal
          > here) already expect. I think booting into a new configuration is MCU
          > style development process and not a fast enough iteration cycle on
          > Linux user space in all scenarios. Even in kernel driver development,
          > one may want to unload/load module during development but even disable
          > module loading as a security hardening mechanism in
          > deployment/production configuration. Another example is security
          > policies development (SELinux) - you want to iterate and test them in
          > user space during development but you want to deploy them immutable.
          > Then again, kernel or device tree changes will require rebooting.
          
          Okay.  Obviously we can support booting with the filesystem mounted
          read-write and no dm-verity.  But the problem with that is that then
          changes have to be manually copied back to the source tree.  We could
          try to add some conveniences for that (e.g. a facility to diff the
          modified root filesystem with the orginal version).  It still means
          making changes to the compiled system, rather than working on the
          source, but for the kind of changes you're talking about, maybe doing
          it that way and then integrating the changes into the Spectrum source
          at the end wouldn't be too bad?
          
          Ideally it would be nice to have something like nixos-rebuild switch,
          that can build a new system and then switch into it.  But the problem
          is, you have to figure out how to *automatically* take the runtime state
          of the system (the processes and services that are running), and somehow
          carry that over into the new system, where anything could have changed.
          So I'm not sure how we'd do that.  NixOS's implementation of this is
          massive, tightly coupled to both NixOS and systemd, and still it's not
          very difficult to get it into a weird state where you need to reboot
          anyway to get an accurate idea of what the change does (or, if you
          don't realise you need to do that, you just think the change hasn't
          worked).  If a kernel module has changed, do you try to unload and
          reload it?  That's pretty complicated to implement, because you have to
          unbind the devices from it, unload the module, then rebind them, and
          even then the module might have had important state that's been lost.
          If the developer is working on that kernel module it's probably okay,
          but what if they're working on something else and just pulled a bunch of
          changes that happened to include a kernel module change?  Same with if
          e.g. the Weston service changes, except in that case we have no way to
          restore the previous state.  So either we choose not to automatically
          restart Weston, in which case we're not really running the new system,
          but a weird hybrid that may have bugs not present in either the old or
          new system, or we do restart Weston, and risk an unsuspecting user
          losing their whole windowing session.
          
          If there's a way we could make this work well, I'm open to it, but it's
          not at all obvious to me what that would look like.
          
          [-- Attachment #2: signature.asc --]
          [-- Type: application/pgp-signature, Size: 833 bytes --]
          
          ^ permalink raw reply	[flat|nested] 7+ messages in thread

        • • * Re: Development on the Spectrum host
              2022-08-18 10:17         ` Development on the Spectrum host Alyssa Ross
            @ 2022-08-19  6:26           ` Ville Ilvonen
              0 siblings, 0 replies; 7+ messages in thread
            From: Ville Ilvonen @ 2022-08-19  6:26 UTC (permalink / raw)
              To: Alyssa Ross; +Cc: discuss
            
            On 18.8.2022 13.17, Alyssa Ross wrote:
            
            > Okay.  Obviously we can support booting with the filesystem mounted
            > read-write and no dm-verity.  But the problem with that is that then
            > changes have to be manually copied back to the source tree.  We could
            
            One  way to handle this is so that people git clone to the target and 
            build small incremental changes there.
            
            > try to add some conveniences for that (e.g. a facility to diff the
            > modified root filesystem with the orginal version).  It still means
            > making changes to the compiled system, rather than working on the
            > source, but for the kind of changes you're talking about, maybe doing
            > it that way and then integrating the changes into the Spectrum source
            > at the end wouldn't be too bad?
            
            When installation of development tools to the target is limited, 
            remotely mounting filesystem (e.g. sshfs) from the development host works.
            
            > Ideally it would be nice to have something like nixos-rebuild switch,
            > that can build a new system and then switch into it.  But the problem
            > is, you have to figure out how to *automatically* take the runtime state
            > of the system (the processes and services that are running), and somehow
            > carry that over into the new system, where anything could have changed.
            > So I'm not sure how we'd do that.  NixOS's implementation of this is
            > massive, tightly coupled to both NixOS and systemd, and still it's not
            > very difficult to get it into a weird state where you need to reboot
            > anyway to get an accurate idea of what the change does (or, if you
            > don't realise you need to do that, you just think the change hasn't
            > worked).  If a kernel module has changed, do you try to unload an > reload it?  That's pretty complicated to implement, because you have to
            > unbind the devices from it, unload the module, then rebind them, and
            > even then the module might have had important state that's been lost.
            
            Good examples but there's no one answer to these scenarios. Depends 
            really much on the case and developer preferences. Often services 
            restarting is enough, sometimes system reboot is required. Also, people 
            doing kernel development are usually fairly seasoned and consider which 
            makes sense to themselves. E.g. kernel modules run reset for the 
            hardware in init() and scripts/tests can be used to handle 
            unbind-unload-rebind in driver development. Managing state persistence 
            over resets or SW updates is an eternity problem. It's an interesting 
            problem - I studied dynamic software updates and state migration is one 
            key challenge there. DSU has not taken traction, though. It's tough to 
            update both code and the runtime state. We have legacy and culture of 
            restarts.
            
            In critical systems it's designed and handled with hardware, even system 
            redundancy - not at single component level.
            
            > If the developer is working on that kernel module it's probably okay,
            > but what if they're working on something else and just pulled a bunch of
            > changes that happened to include a kernel module change?  Same with if
            > e.g. the Weston service changes, except in that case we have no way to
            > restore the previous state.  So either we choose not to automatically
            > restart Weston, in which case we're not really running the new system,
            > but a weird hybrid that may have bugs not present in either the old or
            > new system, or we do restart Weston, and risk an unsuspecting user
            > losing their whole windowing session.
            
            True. In these scenarios I've learned not to try to cover every possible 
            scenario on behalf of other people. Developers are smart and they 
            understand and learn constraints of system changes.
            
            > If there's a way we could make this work well, I'm open to it, but it's
            > not at all obvious to me what that would look like.
            
            Given above, I try to keep this at high level design.
            Two configurations:
            a. development - enables development, testing and debugging with caveats
            b. user - immutable, hardened, strong security promises
            
            Now we have b. which makes a. by design impossible *on target*, making 
            development iterations slow. The question is if we want to generally 
            enable a. and then say - "there it is for anyone who needs it".
            
            Best, -Ville
            
            
            ^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2022-08-19  6:26 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-16 15:50 HW identification and configuration on Spectrum Ville Ilvonen
2022-08-17  7:52 ` Alyssa Ross
2022-08-17 13:25   ` Ville Ilvonen
2022-08-17 13:39     ` Alyssa Ross
2022-08-18  9:15       ` Ville Ilvonen
2022-08-18 10:17         ` Development on the Spectrum host Alyssa Ross
2022-08-19  6:26           ` Ville Ilvonen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).