* [spectrum-devel] [PATCH www] design: state subdirectories, not block devices
@ 2019-10-16 20:28 Alyssa Ross
2019-10-16 21:33 ` Michael Raskin
0 siblings, 1 reply; 8+ messages in thread
From: Alyssa Ross @ 2019-10-16 20:28 UTC (permalink / raw)
To: devel; +Cc: Alyssa Ross, Jean-Philippe Ouellet, Michael Raskin
Conversation on IRC has convinced me that this is the right thing to
do after all:
<https://logs.spectrum-os.org/spectrum/2019-10-15#1571117332-1571144389;>
---
Jean-Phillipe, I'd be curious to hear your thoughts on the above
discussion, since you recommended block devices to me when we talked.
design.html | 29 ++++++++++++++---------------
1 file changed, 14 insertions(+), 15 deletions(-)
diff --git a/design.html b/design.html
index 4b96a41..dc14cfe 100644
--- a/design.html
+++ b/design.html
@@ -43,23 +43,22 @@ one per application.
Each virtual machine will be generated by
a <a href="https://nixos.org/nix/">Nix</a> derivation, and will have a
completely immutable root file system. Persistent storage will be
-provided by virtual block devices, that arbitrary paths on the system
-can be mapped to from the host. There may be other writable mount
-points inside the virtual machine, but these will not persist between
-reboots of the VM. Using Nix to generate virtual machines allows them
-to be reproducibly built, rolled back, edited, and migrated as source
-code, rather than large, opaque virtual machine images.
+provided by mounting subdirectories of the global state directory into
+virtual machines. There may be other writable mount points inside the
+virtual machine, but these will not persist between reboots of the VM.
+Using Nix to generate virtual machines allows them to be reproducibly
+built, rolled back, edited, and migrated as source code, rather than
+large, opaque virtual machine images.
<p>
-Virtual block devices will also be defined in Nix, and block devices
-and applications will be <var>m</var>:<var>n</var>. Some virtual
-machines may have no persistent storage, or even write access to a
-disk, at all. In other cases, it might be desirable for multiple
-applications to be able to access the same device, such as a local
-mail store being shared by two mail clients. Other resources and
-permissions, such as network cards and USB controllers, will similarly
-be defined in Nix. There are three logical sections for the Nix
-configuration -- applications, which are just packages, resources
+State directories and applications will be <var>m</var>:<var>n</var>.
+Some virtual machines may have no persistent storage, or even write
+access to a disk, at all. In other cases, it might be desirable for
+multiple applications to be able to access the same device, such as a
+local mail store being shared by two mail clients. Other resources
+and permissions, such as network cards and USB controllers, will
+similarly be defined in Nix. There are three logical sections for the
+Nix configuration -- applications, which are just packages, resources
(virtual or physical devices), and <i>application instances</i>, which
are mappings between applications and accessible resources. This
structure allows users to have multiple instances of the same
--
2.23.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] [PATCH www] design: state subdirectories, not block devices
2019-10-16 20:28 [spectrum-devel] [PATCH www] design: state subdirectories, not block devices Alyssa Ross
@ 2019-10-16 21:33 ` Michael Raskin
2019-10-16 23:15 ` [spectrum-devel] " Alyssa Ross
2019-10-17 7:47 ` Michael Raskin
0 siblings, 2 replies; 8+ messages in thread
From: Michael Raskin @ 2019-10-16 21:33 UTC (permalink / raw)
To: hi, devel; +Cc: hi, jpo
> Each virtual machine will be generated by
> a <a href="https://nixos.org/nix/">Nix</a> derivation, and will have a
> completely immutable root file system. Persistent storage will be
>-provided by virtual block devices, that arbitrary paths on the system
>-can be mapped to from the host. There may be other writable mount
>-points inside the virtual machine, but these will not persist between
>-reboots of the VM. Using Nix to generate virtual machines allows them
>-to be reproducibly built, rolled back, edited, and migrated as source
>-code, rather than large, opaque virtual machine images.
>+provided by mounting subdirectories of the global state directory into
>+virtual machines. There may be other writable mount points inside the
I guess in the eventual design we want VM-in-jail, i.e. the VM process
itself (and the corresponding FS daemon, if separate from the VM) should
not have any access to the resources not intended for the application.
>+State directories and applications will be <var>m</var>:<var>n</var>.
Note (probably not affecting the design right now): this can also be
a source of annoyance with running everything under unique single-use
UIDs: to have multiple UIDs access the same resource, setting ACLs is
a natural idea. It works well, but might require some cleanup of
obsolete ACLs for a workflow with a large churn of instances ??? the
maximum length of an ACL is limited.
>+Some virtual machines may have no persistent storage, or even write
>+access to a disk, at all. In other cases, it might be desirable for
>+multiple applications to be able to access the same device, such as a
>+local mail store being shared by two mail clients. Other resources
>+and permissions, such as network cards and USB controllers, will
>+similarly be defined in Nix. There are three logical sections for the
>+Nix configuration -- applications, which are just packages, resources
> (virtual or physical devices), and <i>application instances</i>, which
> are mappings between applications and accessible resources. This
> structure allows users to have multiple instances of the same
Given that it is sometimes convenient to spawn multiple VMs with almost
identical settings on the fly (e.g. Firefox instances but with different
state directories for downloads/uploads), should we commit to define
a CLI layer for starting VMs (probably similar to the underlying VMM
software CLI, but adapted to the notions we want to use), with the
Nix-based management of VM fleet built on top of this CLI?
I suspect that building things the other way round is not much simpler,
but building a single VM image usable with slightly different settings
will have smaller overhead than evaluating a Nix expression for each
extra VM.
I guess it didn't make any difference in the previous design version
where the maximum goal was a VM per application, not per application
instance. It does matter now.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-16 21:33 ` Michael Raskin
@ 2019-10-16 23:15 ` Alyssa Ross
2019-10-17 7:47 ` Michael Raskin
1 sibling, 0 replies; 8+ messages in thread
From: Alyssa Ross @ 2019-10-16 23:15 UTC (permalink / raw)
To: Michael Raskin, devel; +Cc: Jean-Philippe Ouellet
[-- Attachment #1: Type: text/plain, Size: 3175 bytes --]
Michael Raskin <7c6f434c@mail.ru> writes:
> I guess in the eventual design we want VM-in-jail, i.e. the VM process
> itself (and the corresponding FS daemon, if separate from the VM) should
> not have any access to the resources not intended for the application.
Yes. Layers are good.
>>+State directories and applications will be <var>m</var>:<var>n</var>.
>
> Note (probably not affecting the design right now): this can also be
> a source of annoyance with running everything under unique single-use
> UIDs: to have multiple UIDs access the same resource, setting ACLs is
> a natural idea. It works well, but might require some cleanup of
> obsolete ACLs for a workflow with a large churn of instances ??? the
> maximum length of an ACL is limited.
Yeah, we'll need something like that. ACLs sound sensible.
>>+Some virtual machines may have no persistent storage, or even write
>>+access to a disk, at all. In other cases, it might be desirable for
>>+multiple applications to be able to access the same device, such as a
>>+local mail store being shared by two mail clients. Other resources
>>+and permissions, such as network cards and USB controllers, will
>>+similarly be defined in Nix. There are three logical sections for the
>>+Nix configuration -- applications, which are just packages, resources
>> (virtual or physical devices), and <i>application instances</i>, which
>> are mappings between applications and accessible resources. This
>> structure allows users to have multiple instances of the same
>
> Given that it is sometimes convenient to spawn multiple VMs with almost
> identical settings on the fly (e.g. Firefox instances but with different
> state directories for downloads/uploads), should we commit to define
> a CLI layer for starting VMs (probably similar to the underlying VMM
> software CLI, but adapted to the notions we want to use), with the
> Nix-based management of VM fleet built on top of this CLI?
Yeah, I've been thinking about this too and agree. We should have a nix
run analogue for temporary VMs. It would be nice for scripts that use
multiple VMs to be able to start them on the fly as well. (But we
should definitely not have a nix-env -i analogue, because there should
be no hidden state on a host.)>
> I suspect that building things the other way round is not much simpler,
> but building a single VM image usable with slightly different settings
> will have smaller overhead than evaluating a Nix expression for each
> extra VM.
Maybe. I'll have to think more about that. tbh I think we can assume
that the evaluator will get faster at some point (because there's big
potential for optimisations and caching in safe mode). It might be nice
to invent yet another way to do options, and just keep that in Nix. But
I could be swayed either way.
> I guess it didn't make any difference in the previous design version
> where the maximum goal was a VM per application, not per application
> instance. It does matter now.
Just to clarify, this was always the goal -- I just wrote the wrong
thing when I was banging out the website initially.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-16 21:33 ` Michael Raskin
2019-10-16 23:15 ` [spectrum-devel] " Alyssa Ross
@ 2019-10-17 7:47 ` Michael Raskin
2019-10-17 11:36 ` Alyssa Ross
2019-10-17 12:09 ` Michael Raskin
1 sibling, 2 replies; 8+ messages in thread
From: Michael Raskin @ 2019-10-17 7:47 UTC (permalink / raw)
To: hi, devel; +Cc: jpo
>>>+State directories and applications will be <var>m</var>:<var>n</var>.
>>
>> Note (probably not affecting the design right now): this can also be
>> a source of annoyance with running everything under unique single-use
>> UIDs: to have multiple UIDs access the same resource, setting ACLs is
>> a natural idea. It works well, but might require some cleanup of
>> obsolete ACLs for a workflow with a large churn of instances ??? the
>> maximum length of an ACL is limited.
>
>Yeah, we'll need something like that. ACLs sound sensible.
Of course, this is a persistent state related to a large number of
inodes that requires keeping track of used unique UIDs (to avoid reuse
before cleanup) and cleanup (I have learned it hte hard way: one does
run out of ACL capacity without removing the obsolete ones), including
some kind of cleanup in case of VM crash or unclean host reboot.
>> Given that it is sometimes convenient to spawn multiple VMs with almost
>> identical settings on the fly (e.g. Firefox instances but with different
>> state directories for downloads/uploads), should we commit to define
>> a CLI layer for starting VMs (probably similar to the underlying VMM
>> software CLI, but adapted to the notions we want to use), with the
>> Nix-based management of VM fleet built on top of this CLI?
>
>Yeah, I've been thinking about this too and agree. We should have a nix
>run analogue for temporary VMs. It would be nice for scripts that use
>multiple VMs to be able to start them on the fly as well. (But we
>should definitely not have a nix-env -i analogue, because there should
>be no hidden state on a host.)>
I think it is better to avoid evaluation, actually.
So I want a level where there is a Spectrum Profile Generation with
named prebuilt VMs complexity levels (console-only, for stuff like
ImageMagick on images downloaded from 4chan; GUI-but-no-HW, to remove
GPU attack surface and guarantee lack of sound; full-GUI, which is still
not guaranteed to get access to everything, of course), and prebuilt
application sets (just buildEnv stuff, no fancy stuff).
Then I say:
spectrum-vm --base full-gui --net nat --app firefox-esr-js-profile \
--app gvim --dri card0 --rw-directory user:conf/nixcon-2020 \
-- firefox https://nixos.org/
This should not do anything in run time that can be safely cached. VM is
prebuilt, application bunches are prebuilt and just mounted into store,
even Firefox profile is prebuilt and changes are going to be discarded.
No builds, no Nix evaluations.
(yes, there is also the annoying question of RAM amount management, not
sure how to do it well, ballooning would be the nicest thing if it is
feasible)
>> I suspect that building things the other way round is not much simpler,
>> but building a single VM image usable with slightly different settings
>> will have smaller overhead than evaluating a Nix expression for each
>> extra VM.
>
>Maybe. I'll have to think more about that. tbh I think we can assume
>that the evaluator will get faster at some point (because there's big
>potential for optimisations and caching in safe mode). It might be nice
Oh well, in the safe mode there is even a potential for precompiling Nix
functions. It's another question whether we want to experiment with
workflows without having access to such speedups.
And even a pure evaluation of a checkout needs to stat all the relevant
files (even if we accept the risk of not checksumming them).
>to invent yet another way to do options, and just keep that in Nix. But
>I could be swayed either way.
I assume there are good arguments for the config to be structured (and
in my current system there are some interface boundaries with structured
config and some wrapper scripts that manage to generate almost all the
usecases using command-line arguments, but sometimes parts of the
structure need to be passed to them).
On the other hand, JSON can be parsed and written easily in all
languages we might want to use, even in Bash using jq. And we need some
code to setup the namespaces, but nsjail/firejail/bubblewrap are C code
so in the long term there would be some rewrite, probably in Rust??? And
having Nix preprocess arguments for Rust code sounds strange.
And whatever you do, Nix evaluation is always just another layer before
running the actual code for setting up the isolated environments that
still needs to interpret its arguments.
Also, I might want to use some binary cache, but when I am offline Nix
builds wait for a long time for a reply from a binary cache. It's not so
bad if I am intentionally building something and can pass an empty value
for binary-caches via the command line, but doing it for each new
command I execute sounds excessive.
Also, it might be convenient to pass entire Nix store inside some VMs,
but if all the _environments_ of all the currently running VMs are
visible in the Nix store??? I really shouldn't.
I guess if I mention multiuser systems you would say that we need to
allow chmod a-x on /nix/store?
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-17 7:47 ` Michael Raskin
@ 2019-10-17 11:36 ` Alyssa Ross
2019-10-17 12:09 ` Michael Raskin
1 sibling, 0 replies; 8+ messages in thread
From: Alyssa Ross @ 2019-10-17 11:36 UTC (permalink / raw)
To: 7c6f434c, devel; +Cc: jpo
[-- Attachment #1: Type: text/plain, Size: 5002 bytes --]
Michael Raskin <7c6f434c@mail.ru> writes:
>>> Given that it is sometimes convenient to spawn multiple VMs with almost
>>> identical settings on the fly (e.g. Firefox instances but with different
>>> state directories for downloads/uploads), should we commit to define
>>> a CLI layer for starting VMs (probably similar to the underlying VMM
>>> software CLI, but adapted to the notions we want to use), with the
>>> Nix-based management of VM fleet built on top of this CLI?
>>
>>Yeah, I've been thinking about this too and agree. We should have a nix
>>run analogue for temporary VMs. It would be nice for scripts that use
>>multiple VMs to be able to start them on the fly as well. (But we
>>should definitely not have a nix-env -i analogue, because there should
>>be no hidden state on a host.)>
>
> I think it is better to avoid evaluation, actually.
>
> So I want a level where there is a Spectrum Profile Generation with
> named prebuilt VMs complexity levels (console-only, for stuff like
> ImageMagick on images downloaded from 4chan; GUI-but-no-HW, to remove
> GPU attack surface and guarantee lack of sound; full-GUI, which is still
> not guaranteed to get access to everything, of course), and prebuilt
> application sets (just buildEnv stuff, no fancy stuff).
>
> Then I say:
>
> spectrum-vm --base full-gui --net nat --app firefox-esr-js-profile \
> --app gvim --dri card0 --rw-directory user:conf/nixcon-2020 \
> -- firefox https://nixos.org/
>
> This should not do anything in run time that can be safely cached. VM is
> prebuilt, application bunches are prebuilt and just mounted into store,
> even Firefox profile is prebuilt and changes are going to be discarded.
> No builds, no Nix evaluations.
>
> (yes, there is also the annoying question of RAM amount management, not
> sure how to do it well, ballooning would be the nicest thing if it is
> feasible)
I think I had imagined you building a "firefox" VM, but yeah, maybe
there's really no point to that. In which case, I'd be happy for this
sort of thing to be handled outside of Nix.
>>> I suspect that building things the other way round is not much simpler,
>>> but building a single VM image usable with slightly different settings
>>> will have smaller overhead than evaluating a Nix expression for each
>>> extra VM.
>>
>>Maybe. I'll have to think more about that. tbh I think we can assume
>>that the evaluator will get faster at some point (because there's big
>>potential for optimisations and caching in safe mode). It might be nice
>
> Oh well, in the safe mode there is even a potential for precompiling Nix
> functions. It's another question whether we want to experiment with
> workflows without having access to such speedups.
>
> And even a pure evaluation of a checkout needs to stat all the relevant
> files (even if we accept the risk of not checksumming them).
Another good point.
>>to invent yet another way to do options, and just keep that in Nix. But
>>I could be swayed either way.
>
> I assume there are good arguments for the config to be structured (and
> in my current system there are some interface boundaries with structured
> config and some wrapper scripts that manage to generate almost all the
> usecases using command-line arguments, but sometimes parts of the
> structure need to be passed to them).
>
> On the other hand, JSON can be parsed and written easily in all
> languages we might want to use, even in Bash using jq. And we need some
> code to setup the namespaces, but nsjail/firejail/bubblewrap are C code
> so in the long term there would be some rewrite, probably in Rust??? And
> having Nix preprocess arguments for Rust code sounds strange.
Not sure about this -- I think it's something we'll have to experiment
with. Maybe structured configuration isn't necessary -- it's a pain to
do in a CLI...
> And whatever you do, Nix evaluation is always just another layer before
> running the actual code for setting up the isolated environments that
> still needs to interpret its arguments.
>
> Also, I might want to use some binary cache, but when I am offline Nix
> builds wait for a long time for a reply from a binary cache. It's not so
> bad if I am intentionally building something and can pass an empty value
> for binary-caches via the command line, but doing it for each new
> command I execute sounds excessive.
Yes, this is true, although I consider it a Nix bug. I shouldn't have
to remember to say --option substitute false for trivial offline
rebuilds.
> Also, it might be convenient to pass entire Nix store inside some VMs,
> but if all the _environments_ of all the currently running VMs are
> visible in the Nix store??? I really shouldn't.
>
> I guess if I mention multiuser systems you would say that we need to
> allow chmod a-x on /nix/store?
I don't plan on doing anything to accomodate multi-user systems, at
least for now.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-17 7:47 ` Michael Raskin
2019-10-17 11:36 ` Alyssa Ross
@ 2019-10-17 12:09 ` Michael Raskin
2019-10-18 18:51 ` Alyssa Ross
2019-10-18 18:56 ` Alyssa Ross
1 sibling, 2 replies; 8+ messages in thread
From: Michael Raskin @ 2019-10-17 12:09 UTC (permalink / raw)
To: hi, devel; +Cc: jpo
>> On the other hand, JSON can be parsed and written easily in all
>> languages we might want to use, even in Bash using jq. And we need some
>> code to setup the namespaces, but nsjail/firejail/bubblewrap are C code
>> so in the long term there would be some rewrite, probably in Rust??? And
>> having Nix preprocess arguments for Rust code sounds strange.
>
>Not sure about this -- I think it's something we'll have to experiment
>with. Maybe structured configuration isn't necessary -- it's a pain to
>do in a CLI...
Sure, this is more of my ??last line of defense?? position. My preference
is definitely ??nice CLI whenever possible, with a fallback to complex
structure stuff when unavoidable, and this fallback can be used together
with the ncie CLI for the rest of the options??
>> And whatever you do, Nix evaluation is always just another layer before
>> running the actual code for setting up the isolated environments that
>> still needs to interpret its arguments.
>>
>> Also, I might want to use some binary cache, but when I am offline Nix
>> builds wait for a long time for a reply from a binary cache. It's not so
>> bad if I am intentionally building something and can pass an empty value
>> for binary-caches via the command line, but doing it for each new
>> command I execute sounds excessive.
>
>Yes, this is true, although I consider it a Nix bug. I shouldn't have
>to remember to say --option substitute false for trivial offline
>rebuilds.
But there are cases where it would be hard for Nix to get right (the
??network won't become reachable?? part ??? the notion of triviality of the
build is obviously hopeless even with manual declaration)
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-17 12:09 ` Michael Raskin
@ 2019-10-18 18:51 ` Alyssa Ross
2019-10-18 18:56 ` Alyssa Ross
1 sibling, 0 replies; 8+ messages in thread
From: Alyssa Ross @ 2019-10-18 18:51 UTC (permalink / raw)
To: Michael Raskin; +Cc: devel
[-- Attachment #1: Type: text/plain, Size: 865 bytes --]
>>> Also, I might want to use some binary cache, but when I am offline Nix
>>> builds wait for a long time for a reply from a binary cache. It's not so
>>> bad if I am intentionally building something and can pass an empty value
>>> for binary-caches via the command line, but doing it for each new
>>> command I execute sounds excessive.
>>
>>Yes, this is true, although I consider it a Nix bug. I shouldn't have
>>to remember to say --option substitute false for trivial offline
>>rebuilds.
>
> But there are cases where it would be hard for Nix to get right (the
> ??network won't become reachable?? part ??? the notion of triviality of the
> build is obviously hopeless even with manual declaration)
IMO it should start builds in parallel with trying to pull them from the
network. It can abort the other after one finishes. But that's OT.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* [spectrum-devel] Re: [PATCH www] design: state subdirectories, not block devices
2019-10-17 12:09 ` Michael Raskin
2019-10-18 18:51 ` Alyssa Ross
@ 2019-10-18 18:56 ` Alyssa Ross
1 sibling, 0 replies; 8+ messages in thread
From: Alyssa Ross @ 2019-10-18 18:56 UTC (permalink / raw)
To: Michael Raskin; +Cc: devel
[-- Attachment #1: Type: text/plain, Size: 865 bytes --]
>>> Also, I might want to use some binary cache, but when I am offline Nix
>>> builds wait for a long time for a reply from a binary cache. It's not so
>>> bad if I am intentionally building something and can pass an empty value
>>> for binary-caches via the command line, but doing it for each new
>>> command I execute sounds excessive.
>>
>>Yes, this is true, although I consider it a Nix bug. I shouldn't have
>>to remember to say --option substitute false for trivial offline
>>rebuilds.
>
> But there are cases where it would be hard for Nix to get right (the
> ??network won't become reachable?? part ??? the notion of triviality of the
> build is obviously hopeless even with manual declaration)
IMO it should start builds in parallel with trying to pull them from the
network. It can abort the other after one finishes. But that's OT.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-10-18 18:57 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-16 20:28 [spectrum-devel] [PATCH www] design: state subdirectories, not block devices Alyssa Ross
2019-10-16 21:33 ` Michael Raskin
2019-10-16 23:15 ` [spectrum-devel] " Alyssa Ross
2019-10-17 7:47 ` Michael Raskin
2019-10-17 11:36 ` Alyssa Ross
2019-10-17 12:09 ` Michael Raskin
2019-10-18 18:51 ` Alyssa Ross
2019-10-18 18:56 ` Alyssa Ross
Code repositories for project(s) associated with this public inbox
https://spectrum-os.org/git/crosvm
https://spectrum-os.org/git/doc
https://spectrum-os.org/git/mktuntap
https://spectrum-os.org/git/nixpkgs
https://spectrum-os.org/git/spectrum
https://spectrum-os.org/git/ucspi-vsock
https://spectrum-os.org/git/www
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).