[spectrum-devel] [PATCH www] design: state subdirectories, not block devices
Conversation on IRC has convinced me that this is the right thing to do after all: https://logs.spectrum-os.org/spectrum/2019-10-15#1571117332-1571144389; --- Jean-Phillipe, I'd be curious to hear your thoughts on the above discussion, since you recommended block devices to me when we talked. design.html | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/design.html b/design.html index 4b96a41..dc14cfe 100644 --- a/design.html +++ b/design.html @@ -43,23 +43,22 @@ one per application. Each virtual machine will be generated by a <a href="https://nixos.org/nix/">Nix</a> derivation, and will have a completely immutable root file system. Persistent storage will be -provided by virtual block devices, that arbitrary paths on the system -can be mapped to from the host. There may be other writable mount -points inside the virtual machine, but these will not persist between -reboots of the VM. Using Nix to generate virtual machines allows them -to be reproducibly built, rolled back, edited, and migrated as source -code, rather than large, opaque virtual machine images. +provided by mounting subdirectories of the global state directory into +virtual machines. There may be other writable mount points inside the +virtual machine, but these will not persist between reboots of the VM. +Using Nix to generate virtual machines allows them to be reproducibly +built, rolled back, edited, and migrated as source code, rather than +large, opaque virtual machine images. <p> -Virtual block devices will also be defined in Nix, and block devices -and applications will be <var>m</var>:<var>n</var>. Some virtual -machines may have no persistent storage, or even write access to a -disk, at all. In other cases, it might be desirable for multiple -applications to be able to access the same device, such as a local -mail store being shared by two mail clients. Other resources and -permissions, such as network cards and USB controllers, will similarly -be defined in Nix. There are three logical sections for the Nix -configuration -- applications, which are just packages, resources +State directories and applications will be <var>m</var>:<var>n</var>. +Some virtual machines may have no persistent storage, or even write +access to a disk, at all. In other cases, it might be desirable for +multiple applications to be able to access the same device, such as a +local mail store being shared by two mail clients. Other resources +and permissions, such as network cards and USB controllers, will +similarly be defined in Nix. There are three logical sections for the +Nix configuration -- applications, which are just packages, resources (virtual or physical devices), and <i>application instances</i>, which are mappings between applications and accessible resources. This structure allows users to have multiple instances of the same -- 2.23.0
Each virtual machine will be generated by a <a href="https://nixos.org/nix/">Nix</a> derivation, and will have a completely immutable root file system. Persistent storage will be -provided by virtual block devices, that arbitrary paths on the system -can be mapped to from the host. There may be other writable mount -points inside the virtual machine, but these will not persist between -reboots of the VM. Using Nix to generate virtual machines allows them -to be reproducibly built, rolled back, edited, and migrated as source -code, rather than large, opaque virtual machine images. +provided by mounting subdirectories of the global state directory into +virtual machines. There may be other writable mount points inside the
I guess in the eventual design we want VM-in-jail, i.e. the VM process itself (and the corresponding FS daemon, if separate from the VM) should not have any access to the resources not intended for the application.
+State directories and applications will be <var>m</var>:<var>n</var>.
Note (probably not affecting the design right now): this can also be a source of annoyance with running everything under unique single-use UIDs: to have multiple UIDs access the same resource, setting ACLs is a natural idea. It works well, but might require some cleanup of obsolete ACLs for a workflow with a large churn of instances ��� the maximum length of an ACL is limited.
+Some virtual machines may have no persistent storage, or even write +access to a disk, at all. In other cases, it might be desirable for +multiple applications to be able to access the same device, such as a +local mail store being shared by two mail clients. Other resources +and permissions, such as network cards and USB controllers, will +similarly be defined in Nix. There are three logical sections for the +Nix configuration -- applications, which are just packages, resources (virtual or physical devices), and <i>application instances</i>, which are mappings between applications and accessible resources. This structure allows users to have multiple instances of the same
Given that it is sometimes convenient to spawn multiple VMs with almost identical settings on the fly (e.g. Firefox instances but with different state directories for downloads/uploads), should we commit to define a CLI layer for starting VMs (probably similar to the underlying VMM software CLI, but adapted to the notions we want to use), with the Nix-based management of VM fleet built on top of this CLI? I suspect that building things the other way round is not much simpler, but building a single VM image usable with slightly different settings will have smaller overhead than evaluating a Nix expression for each extra VM. I guess it didn't make any difference in the previous design version where the maximum goal was a VM per application, not per application instance. It does matter now.
Michael Raskin <7c6f434c@mail.ru> writes:
I guess in the eventual design we want VM-in-jail, i.e. the VM process itself (and the corresponding FS daemon, if separate from the VM) should not have any access to the resources not intended for the application.
Yes. Layers are good.
+State directories and applications will be <var>m</var>:<var>n</var>.
Note (probably not affecting the design right now): this can also be a source of annoyance with running everything under unique single-use UIDs: to have multiple UIDs access the same resource, setting ACLs is a natural idea. It works well, but might require some cleanup of obsolete ACLs for a workflow with a large churn of instances ??? the maximum length of an ACL is limited.
Yeah, we'll need something like that. ACLs sound sensible.
+Some virtual machines may have no persistent storage, or even write +access to a disk, at all. In other cases, it might be desirable for +multiple applications to be able to access the same device, such as a +local mail store being shared by two mail clients. Other resources +and permissions, such as network cards and USB controllers, will +similarly be defined in Nix. There are three logical sections for the +Nix configuration -- applications, which are just packages, resources (virtual or physical devices), and <i>application instances</i>, which are mappings between applications and accessible resources. This structure allows users to have multiple instances of the same
Given that it is sometimes convenient to spawn multiple VMs with almost identical settings on the fly (e.g. Firefox instances but with different state directories for downloads/uploads), should we commit to define a CLI layer for starting VMs (probably similar to the underlying VMM software CLI, but adapted to the notions we want to use), with the Nix-based management of VM fleet built on top of this CLI?
Yeah, I've been thinking about this too and agree. We should have a nix run analogue for temporary VMs. It would be nice for scripts that use multiple VMs to be able to start them on the fly as well. (But we should definitely not have a nix-env -i analogue, because there should be no hidden state on a host.)>
I suspect that building things the other way round is not much simpler, but building a single VM image usable with slightly different settings will have smaller overhead than evaluating a Nix expression for each extra VM.
Maybe. I'll have to think more about that. tbh I think we can assume that the evaluator will get faster at some point (because there's big potential for optimisations and caching in safe mode). It might be nice to invent yet another way to do options, and just keep that in Nix. But I could be swayed either way.
I guess it didn't make any difference in the previous design version where the maximum goal was a VM per application, not per application instance. It does matter now.
Just to clarify, this was always the goal -- I just wrote the wrong thing when I was banging out the website initially.
+State directories and applications will be <var>m</var>:<var>n</var>.
Note (probably not affecting the design right now): this can also be a source of annoyance with running everything under unique single-use UIDs: to have multiple UIDs access the same resource, setting ACLs is a natural idea. It works well, but might require some cleanup of obsolete ACLs for a workflow with a large churn of instances ??? the maximum length of an ACL is limited.
Yeah, we'll need something like that. ACLs sound sensible.
Of course, this is a persistent state related to a large number of inodes that requires keeping track of used unique UIDs (to avoid reuse before cleanup) and cleanup (I have learned it hte hard way: one does run out of ACL capacity without removing the obsolete ones), including some kind of cleanup in case of VM crash or unclean host reboot.
Given that it is sometimes convenient to spawn multiple VMs with almost identical settings on the fly (e.g. Firefox instances but with different state directories for downloads/uploads), should we commit to define a CLI layer for starting VMs (probably similar to the underlying VMM software CLI, but adapted to the notions we want to use), with the Nix-based management of VM fleet built on top of this CLI?
Yeah, I've been thinking about this too and agree. We should have a nix run analogue for temporary VMs. It would be nice for scripts that use multiple VMs to be able to start them on the fly as well. (But we should definitely not have a nix-env -i analogue, because there should be no hidden state on a host.)>
I think it is better to avoid evaluation, actually. So I want a level where there is a Spectrum Profile Generation with named prebuilt VMs complexity levels (console-only, for stuff like ImageMagick on images downloaded from 4chan; GUI-but-no-HW, to remove GPU attack surface and guarantee lack of sound; full-GUI, which is still not guaranteed to get access to everything, of course), and prebuilt application sets (just buildEnv stuff, no fancy stuff). Then I say: spectrum-vm --base full-gui --net nat --app firefox-esr-js-profile \ --app gvim --dri card0 --rw-directory user:conf/nixcon-2020 \ -- firefox https://nixos.org/ This should not do anything in run time that can be safely cached. VM is prebuilt, application bunches are prebuilt and just mounted into store, even Firefox profile is prebuilt and changes are going to be discarded. No builds, no Nix evaluations. (yes, there is also the annoying question of RAM amount management, not sure how to do it well, ballooning would be the nicest thing if it is feasible)
I suspect that building things the other way round is not much simpler, but building a single VM image usable with slightly different settings will have smaller overhead than evaluating a Nix expression for each extra VM.
Maybe. I'll have to think more about that. tbh I think we can assume that the evaluator will get faster at some point (because there's big potential for optimisations and caching in safe mode). It might be nice
Oh well, in the safe mode there is even a potential for precompiling Nix functions. It's another question whether we want to experiment with workflows without having access to such speedups. And even a pure evaluation of a checkout needs to stat all the relevant files (even if we accept the risk of not checksumming them).
to invent yet another way to do options, and just keep that in Nix. But I could be swayed either way.
I assume there are good arguments for the config to be structured (and in my current system there are some interface boundaries with structured config and some wrapper scripts that manage to generate almost all the usecases using command-line arguments, but sometimes parts of the structure need to be passed to them). On the other hand, JSON can be parsed and written easily in all languages we might want to use, even in Bash using jq. And we need some code to setup the namespaces, but nsjail/firejail/bubblewrap are C code so in the long term there would be some rewrite, probably in Rust��� And having Nix preprocess arguments for Rust code sounds strange. And whatever you do, Nix evaluation is always just another layer before running the actual code for setting up the isolated environments that still needs to interpret its arguments. Also, I might want to use some binary cache, but when I am offline Nix builds wait for a long time for a reply from a binary cache. It's not so bad if I am intentionally building something and can pass an empty value for binary-caches via the command line, but doing it for each new command I execute sounds excessive. Also, it might be convenient to pass entire Nix store inside some VMs, but if all the _environments_ of all the currently running VMs are visible in the Nix store��� I really shouldn't. I guess if I mention multiuser systems you would say that we need to allow chmod a-x on /nix/store?
Michael Raskin <7c6f434c@mail.ru> writes:
Given that it is sometimes convenient to spawn multiple VMs with almost identical settings on the fly (e.g. Firefox instances but with different state directories for downloads/uploads), should we commit to define a CLI layer for starting VMs (probably similar to the underlying VMM software CLI, but adapted to the notions we want to use), with the Nix-based management of VM fleet built on top of this CLI?
Yeah, I've been thinking about this too and agree. We should have a nix run analogue for temporary VMs. It would be nice for scripts that use multiple VMs to be able to start them on the fly as well. (But we should definitely not have a nix-env -i analogue, because there should be no hidden state on a host.)>
I think it is better to avoid evaluation, actually.
So I want a level where there is a Spectrum Profile Generation with named prebuilt VMs complexity levels (console-only, for stuff like ImageMagick on images downloaded from 4chan; GUI-but-no-HW, to remove GPU attack surface and guarantee lack of sound; full-GUI, which is still not guaranteed to get access to everything, of course), and prebuilt application sets (just buildEnv stuff, no fancy stuff).
Then I say:
spectrum-vm --base full-gui --net nat --app firefox-esr-js-profile \ --app gvim --dri card0 --rw-directory user:conf/nixcon-2020 \ -- firefox https://nixos.org/
This should not do anything in run time that can be safely cached. VM is prebuilt, application bunches are prebuilt and just mounted into store, even Firefox profile is prebuilt and changes are going to be discarded. No builds, no Nix evaluations.
(yes, there is also the annoying question of RAM amount management, not sure how to do it well, ballooning would be the nicest thing if it is feasible)
I think I had imagined you building a "firefox" VM, but yeah, maybe there's really no point to that. In which case, I'd be happy for this sort of thing to be handled outside of Nix.
I suspect that building things the other way round is not much simpler, but building a single VM image usable with slightly different settings will have smaller overhead than evaluating a Nix expression for each extra VM.
Maybe. I'll have to think more about that. tbh I think we can assume that the evaluator will get faster at some point (because there's big potential for optimisations and caching in safe mode). It might be nice
Oh well, in the safe mode there is even a potential for precompiling Nix functions. It's another question whether we want to experiment with workflows without having access to such speedups.
And even a pure evaluation of a checkout needs to stat all the relevant files (even if we accept the risk of not checksumming them).
Another good point.
to invent yet another way to do options, and just keep that in Nix. But I could be swayed either way.
I assume there are good arguments for the config to be structured (and in my current system there are some interface boundaries with structured config and some wrapper scripts that manage to generate almost all the usecases using command-line arguments, but sometimes parts of the structure need to be passed to them).
On the other hand, JSON can be parsed and written easily in all languages we might want to use, even in Bash using jq. And we need some code to setup the namespaces, but nsjail/firejail/bubblewrap are C code so in the long term there would be some rewrite, probably in Rust??? And having Nix preprocess arguments for Rust code sounds strange.
Not sure about this -- I think it's something we'll have to experiment with. Maybe structured configuration isn't necessary -- it's a pain to do in a CLI...
And whatever you do, Nix evaluation is always just another layer before running the actual code for setting up the isolated environments that still needs to interpret its arguments.
Also, I might want to use some binary cache, but when I am offline Nix builds wait for a long time for a reply from a binary cache. It's not so bad if I am intentionally building something and can pass an empty value for binary-caches via the command line, but doing it for each new command I execute sounds excessive.
Yes, this is true, although I consider it a Nix bug. I shouldn't have to remember to say --option substitute false for trivial offline rebuilds.
Also, it might be convenient to pass entire Nix store inside some VMs, but if all the _environments_ of all the currently running VMs are visible in the Nix store??? I really shouldn't.
I guess if I mention multiuser systems you would say that we need to allow chmod a-x on /nix/store?
I don't plan on doing anything to accomodate multi-user systems, at least for now.
On the other hand, JSON can be parsed and written easily in all languages we might want to use, even in Bash using jq. And we need some code to setup the namespaces, but nsjail/firejail/bubblewrap are C code so in the long term there would be some rewrite, probably in Rust??? And having Nix preprocess arguments for Rust code sounds strange.
Not sure about this -- I think it's something we'll have to experiment with. Maybe structured configuration isn't necessary -- it's a pain to do in a CLI...
Sure, this is more of my ��last line of defense�� position. My preference is definitely ��nice CLI whenever possible, with a fallback to complex structure stuff when unavoidable, and this fallback can be used together with the ncie CLI for the rest of the options��
And whatever you do, Nix evaluation is always just another layer before running the actual code for setting up the isolated environments that still needs to interpret its arguments.
Also, I might want to use some binary cache, but when I am offline Nix builds wait for a long time for a reply from a binary cache. It's not so bad if I am intentionally building something and can pass an empty value for binary-caches via the command line, but doing it for each new command I execute sounds excessive.
Yes, this is true, although I consider it a Nix bug. I shouldn't have to remember to say --option substitute false for trivial offline rebuilds.
But there are cases where it would be hard for Nix to get right (the ��network won't become reachable�� part ��� the notion of triviality of the build is obviously hopeless even with manual declaration)
Also, I might want to use some binary cache, but when I am offline Nix builds wait for a long time for a reply from a binary cache. It's not so bad if I am intentionally building something and can pass an empty value for binary-caches via the command line, but doing it for each new command I execute sounds excessive.
Yes, this is true, although I consider it a Nix bug. I shouldn't have to remember to say --option substitute false for trivial offline rebuilds.
But there are cases where it would be hard for Nix to get right (the ??network won't become reachable?? part ??? the notion of triviality of the build is obviously hopeless even with manual declaration)
IMO it should start builds in parallel with trying to pull them from the network. It can abort the other after one finishes. But that's OT.
Also, I might want to use some binary cache, but when I am offline Nix builds wait for a long time for a reply from a binary cache. It's not so bad if I am intentionally building something and can pass an empty value for binary-caches via the command line, but doing it for each new command I execute sounds excessive.
Yes, this is true, although I consider it a Nix bug. I shouldn't have to remember to say --option substitute false for trivial offline rebuilds.
But there are cases where it would be hard for Nix to get right (the ??network won't become reachable?? part ??? the notion of triviality of the build is obviously hopeless even with manual declaration)
IMO it should start builds in parallel with trying to pull them from the network. It can abort the other after one finishes. But that's OT.
participants (2)
-
Alyssa Ross -
Michael Raskin