Thanks for a useful reply Michał! Some responses below.

Michał "rysiek" Woźniak wrote:

Hi!

A QubesOS user here for about a year and a half, let's see if I can help out here.

On 6/12/20 11:06 AM, joweill(a)icloud.com wrote:

...

Going over each usability issue mentioned in the motivation doc:

• "Hardware compatibility is extremely limited": I don't believe this is

really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy support these. Is there anything I'm missing?

I got a shiny new Thinkpad T490 a few months ago. 3d acceleration (KDE user here, I demand my wobbly windows!) is simply not available, because the dom0 system is too old.

My other laptop, a T470, was specifically selected for QubesOS, and there were still issues (for instance, disabling Thuderbolt got me almost double the battery life).

Generally speaking, one can buy almost any laptop today and expect it to mostly work with plain GNU/Linux. However, most might not even be able to boot QubesOS.

...

• "People are reluctant to use Xen on their

computer for power management etc. reasons." Can you elaborate on these issues?

The T470 had easily a 10-12h battery life on plain Kubuntu. On Qubes, 4-5h is maximum I can squeeze out of it, and that's *after* the Tunderbolt fix.

Running virtual machines is extremely resource-intensive, there's no way around it.

But if the issues stem from running VMs (and not switching from Xen), they won't be resolved with Spectrum's current design, right?

...

• I know that Qubes considered using KVM and decided

against it for security reasons [2]. My understanding is that the downside of this decision is the limited hardware support, which is one of the things that Spectrum views as an opportunity for improvement. Can you elaborate on this decision?

Can't speak for the developers, but the way I see Spectrum is as a compromise between regular GNU/Linux distro (with all the related security problems) and QubesOS (with the limited hardware support and

...

• "VMs are heavy": How will Spectrum improve

on this without sacrificing security?

I'll leave this to the developers, but will say that I expect *some* security to be sacrificed. There are always trade-offs.

I feel one needs to be an expert to use QubesOS, but a regular user (with some basic training) can use a Mint or Ubuntu-based system. And I think it makes a lot of sense to offer a middle ground.

Agreed, but I think the current design of Spectrum may improve Qubes' hardware issues, but not the other issues the doc mentions. Possibly switching to containers (which something like gVisor) may solve some of the other issues of Qubes, though that would further degrade security.

...

• "GUI applications are buggy, command line tools

are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be better in the case of Spectrum which is a new project with one developer.

That's a fair point. Things to consider:

1. *probably* certain things can be easier (thus, less bug-prone) in SpectrumOS

than in QubesOS (kvm easier to work with than Xen, bigger potential community of users and developers due to improved hardware support, etc); 2. perhaps some QubesOS tools could be used in SpectrumOS, thus limiting the amount of work needed

...

More generally, I'm wondering whether this projects' goals couldn't be better achieved by trying to work with the Qubes developers to integrate Nix. It may very well be that they would reject it for some reason, but then the logical next step would be to fork Qubes.

My feel is that QubesOS and SpectrumOS might have a bit different threat models in mind, and thus things that make sense for SpectrumOS (like using kvm) are a no-go for Qubes. But that's just guesswork on my part.

...

Have you reached out to the Qubes developers?

Thanks in advance!

## References

[1] https://spectrum-os.org/motivation.html [2] https://www.qubes-os.org/faq/#why-does-qubes-use-xen-instead-of-kvm-or-some%...

Michał "rysiek" Woźniak rysiek@hackerspace.pl writes:

On 6/12/20 11:54 AM, infokiller wrote:

...

Michał "rysiek" Woźniak wrote:

...

Running virtual machines is extremely resource-intensive, there's no way around it.

But if the issues stem from running VMs (and not switching from Xen), they won't be resolved with Spectrum's current design, right?

Right you are about the performance issues *I think*. I would however expect hardware compatibility to improve with kvm.

But I'd love somebody more knowledgeable to shed some light here.

I am by no means a Xen expert (and so the following could be totally wrong, and I would appreciate being corrected), but my understanding is that Linux is likely to provide better power management than Xen due to being better optimised, and not being as focused as Xen on large data center deployments.

Additionally, there are interesting Linux virtualization features that may allow for reduced overall hardware requirements by securely sharing resources between VMs. For example, it may be possible to share identical read-only memory pages between VMs. There's not necessarily any reason that running identical copies of Firefox in two different VMs should require two copies of Firefox in host memory. The most promising application of this principle at the moment in virtio-fs[1][2], which supports this page sharing. DAX[3] may also be of interest. Of course, there will be security considerations when deciding whether to adopt these systems, but the point is that this sort of development just isn't happening with Xen. It's all focused on the KVM ecosystem.

[1]: https://virtio-fs.gitlab.io/ [2]: https://lwn.net/Articles/788333/ [3]: https://lwn.net/Articles/610174/

Hardware compatibility should certainly be better with KVM. Linux has extremely diverse hardware support, and a much larger user base than Xen, especially on consumer hardware. I'd be confident in saying most relatively recent consumer PC hardware will support Linux reasonably well (a notable exception of course being Nvidea GPUs), and I'd be surprised if Xen worked well on anywhere near as many devices.

...

...

I feel one needs to be an expert to use QubesOS, but a regular user (with some basic training) can use a Mint or Ubuntu-based system. And I think it makes a lot of sense to offer a middle ground.

Agreed, but I think the current design of Spectrum may improve Qubes' hardware issues, but not the other issues the doc mentions. Possibly switching to containers (which something like gVisor) may solve some of the other issues of Qubes, though that would further degrade security.

Fun fact, containers were in fact initially considered. :-)

Here's a good blogpost from Alyssa on why move to kvm, and the advantages of it over Xen: https://alyssa.is/back-from-cccamp-2019/

Indeed, Spectrum was originally conceived as a Qubes-style operating system that used containers instead of VMs. The reason the focus shifted from containers to KVM is that as I researched the topic, I became less confident in the ability of containers to offer meaningful security, and learned that the overhead of VMs doesn't have to be as big as I thought. Of particular inspiration was "My VM is Ligther (and Faster) than your Container"[4], which demonstrates Xen-based virtual machines with resource requirements roughly equivalent to containers.

I also think that Nix might put Spectrum in a unique position to provide lightweight virtual machines. Conventionally, VMs run full-featured Linux distributions, with all the memory and disk requirements that entails. It's not usually feasible to build tailored VM images running only the software that is critical to the task of the VM. With Nix, however, it suddenly is. A big part of the appeal of containers, I think, is that the tooling makes it easy to create small images focused on a single task. VM tooling has never really provided this. It's a sort of vicious cycle of assuming VMs are heavy and therefore it doesn't make sense to have a VM dedicated to a single task, which means that tooling for creating such a VM isn't developed, which means that VMs stay heavy.

[4]: http://cnp.neclab.eu/projects/lightvm/lightvm.pdf

Thanks for your detailed and well put responses!

Alyssa Ross wrote:

...

Hi there,

I just discovered this project and I'm really excited about it! I've long waited for an OS that combines Qubes-like compartmentalization with the reproducibility of Nix/GUIX.

Hi! Thank you for getting in touch. :)

...

I'm trying to understand what this project aims to improve over Qubes, other than the integration of Nix (which I do think is really important!). I read the motivation page [1], but I'm not yet convinced by most of the points mentioned that relate to Qubes. Going over each usability issue mentioned in the motivation doc:

• "Hardware compatibility is extremely limited": I don't believe this is

really the case for the minimum Qubes 4 requirements [4]: most modern computers people buy support these. Is there anything I'm missing?

• "People are reluctant to use Xen on their computer for power management etc.

reasons." Can you elaborate on these issues?

• I know that Qubes considered using KVM and decided against it for security reasons [2].

My understanding is that the downside of this decision is the limited hardware support, which is one of the things that Spectrum views as an opportunity for improvement. Can you elaborate on this decision?

• "VMs are heavy": How will Spectrum improve on this without sacrificing

security?

I already talked about Xen vs KVM elsewhere in the thread, but wanted to say that the impression I've got from talking to several Qubes developers is that they are at this point far less enthusiastic about Xen than their documentation might suggest.

Additionally, a couple of things that the Qubes Architecture Specification[1] mentions as being benefits of Xen over KVM are now, I believe, also available in KVM. Specifically, I believe it is possible to use PV drivers with KVM (although I wonder if maybe there is a terminology mismatch here?), and I believe it will be at least partially possible to have something resembling driver domains, although this is an area of active R&D within Spectrum. Other mentioned advantages of Xen, such as auditability, remain, of course.

...

• "GUI applications are buggy, command line tools

are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be better in the case of Spectrum which is a new project with one developer.

My understanding is that a lot of the instability I've encountered with Qubes's tools comes down to some severe technical debt with their inter-VM communication system. This is likely something that is very difficult to fix, but easy to learn from. Being a new project allows Spectrum to learn from Qubes' mistakes.

I agree, but I'm still not convinced this will be different in the case of Spectrum, having even more limited dev resources than Qubes'. I mean, if the Qubes folks could fix these issues without a huge effort, even if it meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation.

...

More generally, I'm wondering whether this projects' goals couldn't be better achieved by trying to work with the Qubes developers to integrate Nix. It may very well be that they would reject it for some reason, but then the logical next step would be to fork Qubes. Have you reached out to the Qubes developers?

I've had the pleasure of speaking with several Qubes developers on multiple occasions. I also know that there is work being done to support NixOS templates in Qubes.

Do you have any link you can share to work on Nix template being used in Qubes?

As I see it, though, the real benefit of Nix here is that it would allow defining the whole system in one single Nix configuration. Doing this in Qubes would require big fundamental changes to how Qubes works. I believe the idea has been brought up to the Qubes developers, and as I recall they were not keen. I believe there is room in this space for Qubes and Spectrum to coexist.

There's definitely a place for both efforts, but I'm just wondering if it would not be better to implement Spectrum's ideas while working more closely with Qubes. For example, creating a git branch or a fork that replaces Xen with KVM and integrates Nix. Now, that may very be not the best way to make progress on this project, because ramping up on the Qubes codebase will be an inefficient use of your time, or because you want to first experiment with some ideas in isolation, or for any other reason. However, the potential benefits of such a collaboration are, in my opinion, too big to pass on without strongly pursuing it. Just to name a few: - You will get feedback from top tier security experts - Qubes already has strong credibility which can be a huge driver of adoption, especially if your project will be endorsed by Qubes devs - You will be in a better position to integrate some of the tools of Qubes, which you likely would need to do, and it is less likely that changes in the Qubes tooling will break your project (if you will indeed integrate them) - You are less likely to be asked time and again "Why not Qubes"? :)

So, why not start a public discussion in Qubes' mailing list on issue tracker to figure out what is needed to accomplish Spectrum's goals? It will probably turn out that you made the right decision by starting a separate project, but at the very least: - You'll may get attention from people who can contribute to Spectrum - The issues involved with be publicly documented and searchable for future generations

Michael Raskin wrote:

...

  - "GUI applications are buggy, command line

tools are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be better in the case of Spectrum which is a new project with one developer. My understanding is that a lot of the instability I've encountered with Qubes's tools comes down to some severe technical debt with their inter-VM communication system. This is likely something that is very difficult to fix, but easy to learn from. Being a new project allows Spectrum to learn from Qubes' mistakes. I agree, but I'm still not convinced this will be different in the case of Spectrum, having even more limited dev resources than Qubes'. I mean, if the Qubes folks could fix these issues without a huge effort, even if it meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation.

Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes.

True, but I'm not sure this applies to making GUI tools less buggy, or having better documentation for CLI tools. Even if Spectrum development will make big compromises, I find it unlikely that these aspects will be improved over Qubes. Now, I'm not saying this is not possible at all. Maybe a 100 new contributors will join the project soon and will make this viable. I just think that this is not a strong selling point of Spectrum, which is somewhat implied in the motivation page.

A wl_roots based tooling is seriously considered for the first full release, after all���

Is using wl_roots a non-starter for Qubes?

A lot of problems for Qubes is that things are hard to do without making security slightly worse than what Qubes has now. Spectrum is definitely not going to achieve this level of security anytime soon for various reasons, and the goal is more to find the best trade-off between security and usability.

There is quite a bit of design space in the gap between ��quite a bit more secure than Firejail, with the ease of use around plain NixOS plus Firejail�� and ��less secure than Qubes, but easier to manage��.

What do you mean by "quite a bit more secure than firejail"? isn't this side of the spectrum actually "firejail-like security"?

The way I see it, the following are the major points on the usability-security spectrum for running desktop Linux (or another desktop OS for that matter), starting from the best usability and worst security, and ending in the worst usability and best security:

1. Run a regular Linux distro (some are better than others in providing quick security updates) 2. Harden the system: sandbox processes, harden the kernel and important userspace libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure boot, etc. 3. Use Qubes

Most users, of course, don't bother and just use (1), which is actually fine in most cases. (2) gives pretty good usability, with the main issues related to sandboxing, since most Linux desktop apps were not built with sandboxing in mind, and the overall experience does not support it well (for example, there's no standard permission dialogs like in Android, where sandboxing works much better in practice). Theoretically, people could use (2), run untrusted software in VMs to get strong sandboxing but still be able to use most software, and get pretty good security. In practice this doesn't quite work because running stuff in dedicated VMs is not streamlined enough, so people just don't do it. Then there's (3), which is the most secure, but has some usability issues that were already mentioned. So Spectrum can fill the gap between (2) and (3).

Michael Raskin wrote:

...

resources than Qubes'. I mean, if the Qubes folks

could fix these issues without a huge effort, even if it meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation. Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes. True, but I'm not sure this applies to making GUI tools less buggy, or having better documentation for CLI tools.

Yes, it does apply. SpectrumOS is willing to accept quite a bit more of external code with good in-the-wild track record inside the trusted code base, at least for _most_ tasks. So where Qubes needs to reimplement something with a completely different design (allowing minimisation of TCB), Spectrum can (at least in the beginning, sometimes grudgingly) take the closest thing in existence and add only critically missing parts.

I disagree. Spectrum still needs to write its own GUI tools. Since the Qubes CLI tools are not buggy, presumably the bugs Alyssa ran into are not related to the Qubes design, but rather to GUI programming. As for the CLI tools, I'm really not sure if she can actually reuse any docs, or write less of them. AFAIK, most of the tools in question are related to inter-VM comms, which are not just out there in a fully documented form. And even if it were the case that it would be possible to significantly reuse existing tools, it's still speculative at best to claim that this will overcome the clear disadvantage in development resources.

...

A wl_roots based tooling is seriously considered for the first full release, after all���������

Is using wl_roots a non-starter for Qubes?

Yes, and I hope it is a complete non-starter on the level of not being worth any discussion there.

I can only remind you, that Qubes has doubts about KVM, a widely used part of the Linux kernel, being sufficiently secure. Thety have the same stance about half of the features of Xen, the hypervisor they do use. This does make some things Spectrum plans to do for usability and performance much harder to implement, as the underlying features are undesirable in the TCB.

Compared to this, wl_roots is _way_ more dangerous. This is a library which has segfaults in relatively normal use, and apparently the users do hit these relatively frequently. Some of these seem to survive for months. I have not looked whether any of these are exploitable, but for the Qubes level of requirements one should just assume it is. Notice that for Spectrum one could also reuse some kind of VNC code if wl_roots is eventually considered too buggy. For Qubes even VNC libraries are far from good enough.

Well, Qubes does intend to experiment with Wayland [1], though possibly they could use smithay [2] or build their own, hopefully in a memory safe language. And there's also work in Qubes on moving the GUI part from dom0, which will lower the risk related to the GUI.

[1] https://www.qubes-os.org/gsoc/#wayland-support-in-gui-agent-andor-gui-daemon [2] https://github.com/Smithay/smithay

...

There is quite a bit of design space in the gap between ������quite a bit more secure than Firejail, with the ease of use around plain NixOS plus Firejail������ and ������less secure than Qubes, but easier to manage������. What do you mean by "quite a bit more secure than firejail"? isn't this side of the spectrum actually "firejail-like security"?

Firejail depends on namespaces, which still have some weird behaviours in some corner cases, there is a hope that VMs will be a simpler foundation.

I understand, but it sounded like this side of the spectrum should be "using NixOs, maybe with Firejail", in which case the security is Firejail-like.

...

The way I see it, the following are the major points on the usability-security spectrum for running desktop Linux (or another desktop OS for that matter), starting from the best usability and worst security, and ending in the worst usability and best security:

1. Run a regular Linux distro (some are better than others in providing quick security

updates) 2. Harden the system: sandbox processes, harden the kernel and important userspace libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure boot, etc.

If you do not understand what you are doing well enough, Wayland as you use it might end up being less secure than X11, by the way.

What do you mean?

...

Most users, of course, don't bother and just use (1), which is actually fine in most cases. (2) gives pretty good usability, with the main issues related to sandboxing, since most Linux desktop apps were not built with sandboxing in mind, and the overall experience does not support it well (for example, there's no standard permission dialogs like in Android, where sandboxing works much better in practice).

Actually, if you write a few relatively reasonable wrappers around some kind of namespace-based sandboxing, usability problems kind of become quite different from the normal ones (some UI flows are actually better when the choices are trimmed in advance��� and also suddenly a lot of things do not need to be chosen uniformly at the entire-system level anymore).

I'm curious about these wrappers, could you elaborate?

No idea how much effort is to make meaningful GUI permission dialogs. Android ones are clearly not meaningful enough, of course.

Android's permission system is far from perfect, but it also targets a very large userbase that doesn't care much about security (as is Windows). And it's still **much** better than desktop Linux.

Michał "rysiek" Woźniak wrote:

Hey,

On 6/15/20 1:44 PM, infokiller wrote:

...

Michael Raskin wrote:

resources than Qubes'. I mean, if the Qubes folks could fix these issues without a huge effort, even if it meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation. Note that SpectrumOS is going to make tradeoffs that are complete non-starters for Qubes. True, but I'm not sure this applies to making GUI tools less buggy, or having better documentation for CLI tools. Yes, it does apply. SpectrumOS is willing to accept quite a bit more of external code with good in-the-wild track record inside the trusted code base, at least for _most_ tasks. So where Qubes needs to reimplement something with a completely different design (allowing minimisation of TCB), Spectrum can (at least in the beginning, sometimes grudgingly) take the closest thing in existence and add only critically missing parts. I disagree.

And you have all the right to.

However, it seems to me that the reasons behind SpectrumOS being done the way it's done have been pretty extensively explained and documented in this thread. At this point the discussion seems to focus on back-of-a-napkin estimates regarding what is a more effective way of implementing certain things.

I was mainly responding to understand if I'm missing anything.

Clearly your estimates are different than those of others responding in this thread. You might be right and we might be wrong, but the only way to test that is to start implementing things one way or the other.

SpectrumOS made their estimations and arrived at a decision to do things one particular way. You may disagree this is the best way to do things, but I feel at this point a better way of making your point would be for you to start implementing things the way you believe is more effective.

I'm not trying to make any point, and I honestly don't have any stakes here. I'm trying to figure out why things are done the way they are. My ultimate goal is to have a secure, performant, and easy to manage OS. And so, I would be very happy if SpectrumOS will be able to satisfy these requirements.

SpectrumOS is basically a one-person team. If you're right, you should be able to overtake SpectrumOS development pretty quickly.

I'm not saying integrating with Qubes is quick or easy: I'm arguing that it may be the better approach to reach the project goals in a sustainable way. Either way, I won't have much time to work on something like this anytime soon. That said, though I'd like to figure out what it involves, and if it's reasonable to expect I'll be able to make incremental progress as a side project.

-- Best, r

Michael Raskin wrote:

...

I disagree. Spectrum still needs to write its own GUI tools.

There are some XDG tools that can be reused wholesale.

...

 Qubes, but easier to manage������������������. What

do you mean by "quite a bit more secure than firejail"? isn't this side of the spectrum actually "firejail-like security"? Firejail depends on namespaces, which still have some weird behaviours in some corner cases, there is a hope that VMs will be a simpler foundation. I understand, but it sounded like this side of the spectrum should be "using NixOs, maybe with Firejail", in which case the security is Firejail-like.

SpectrumOS will use small VMs, not containers.

I understand that: I was referring to a hypothetical system running NixOS with Firejail, which is lower security and higher usability than Spectrum/Qubes.

...

2. Harden the system: sandbox processes, harden the

kernel and important userspace libraries like libc, enforce MAC, use Wayland instead of X11, firewall, verified/secure boot, etc. If you do not understand what you are doing well enough, Wayland as you use it might end up being less secure than X11, by the way. What do you mean?

Well, most compositors allow clients quite a lot of things that theoretically should be permission-checked, so there is not as much good to overcome the drawbacks of a lot of code being still a bit raw.

...

 Most users, of course, don't bother and just use

(1), which is actually fine in most cases. (2) gives pretty good usability, with the main issues related to sandboxing, since most Linux desktop apps were not built with sandboxing in mind, and the overall experience does not support it well (for example, there's no standard permission dialogs like in Android, where sandboxing works much better in practice). Actually, if you write a few relatively reasonable wrappers around some kind of namespace-based sandboxing, usability problems kind of become quite different from the normal ones (some UI flows are actually better when the choices are trimmed in advance��������� and also suddenly a lot of things do not need to be chosen uniformly at the entire-system level anymore). I'm curious about these wrappers, could you elaborate?

Well, I run browser instances and PDF/image/video viewers inside nsjail, and the wrappers _mainly_ just prepare a reasonably restricted access to give the program running inside nsjail. Of course if I bother to do all that, there is also use of single-use UIDs on top of restrictions by mount namespaces.

It's not that much code, and in my case it is in Common Lisp.

And then it is just all the boring stuff that is annoying elsewhere like using different resolv.conf for different applications and giving applications access only through proxy and only picking from the files that the application should use and not navigating the entire storage.

Yeah, I find this is indeed the most annoying stuff when I'm using Firejail. It's especially annoying if I realize I want to open a file in the sandboxed app that I didn't whitelist, which means I rerun it (there may be a way to whitelist files after the app is already running, but I never bothered to check).

So I watch SpectrumOS mostly looking at the CrosVM work and when it will be sufficient for replacing containers and socket proxying with VMs and virtio-based socket proxying.

...

No idea how much effort is to make meaningful GUI permission dialogs. Android ones are clearly not meaningful enough, of course. Android's permission system is far from perfect, but it also targets a very large userbase that doesn't care much about security (as is Windows). And it's still **much** better than desktop Linux.

It is unclear that Android in practice (not how the intents should have been used) is better than desktop Linux with Flatpack everywhere, let alone the unfortunately rare case of desktop Linux where people _also_ use UID segregation. There is a chance that XDG-portal stuff ends up closer to how intents should have been used than what we currently see on Android. And recently it looks like Android is now going to give up the remaining appearances of being usable as a general-purpose computing platform, but this seems not to be forced by the permission system design in any way.

I think Flatpak is a step in the right direction, but is still far from being everywhere, and has its own share of issues. It also doesn't solve the issue of apps that were not built with a permission system. Android has other issues, of course, like very slow (it at all) security updates, but if you're using a recent Pixel you are probably much better off than using desktop Linux.

On 6/14/20 9:27 PM, Alyssa Ross wrote:

...

So, why not start a public discussion in Qubes' mailing list on issue tracker to figure out what is needed to accomplish Spectrum's goals?

It will probably turn out that you made the right decision by starting a separate project, but at the very least:

• You'll may get attention from people who can contribute to Spectrum
• The issues involved with be publicly documented and searchable for future generations

I think you maybe don't appreciate just how huge an undertaking this would be. There is so much that would have to change about how Qubes works that I think you'd end up having to reimplement most of it anyway, but you'd be doing it bit by bit, never having the opportunity to consider the system as a whole.

Plus, SpectrumOS does not have to deal with backwards compatibility. If QubesOS developers were to start implementing these changes, they would constantly have to deal with trade-offs between ease of implementing them and the cost of breaking backwards compatibility.

At the end of the day I just don't believe that trying to shoehorn these changes into Qubes is the best way to make progress. It might well be valuable to try that, but even so it would make much more sense for somebody who believes in that approach to dedicate the huge amount of effort required to attempt it, rather than me. This could be another effort that could be pursued in parallel to my work on Spectrum.

Very much this.

-- Best, r

Alyssa Ross wrote:

...

  - "GUI applications are buggy, command line

tools are mostly undocumented": I assume that the reason for this is the lack of resources the Qubes project has. However, I don't see how this will be be better in the case of Spectrum which is a new project with one developer. My understanding is that a lot of the instability I've encountered with Qubes's tools comes down to some severe technical debt with their inter-VM communication system. This is likely something that is very difficult to fix, but easy to learn from. Being a new project allows Spectrum to learn from Qubes' mistakes. I agree, but I'm still not convinced this will be different in the case of Spectrum, having even more limited dev resources than Qubes'. I mean, if the Qubes folks could fix these issues without a huge effort, even if it meant rewriting all the inter-VM communication tools, they probably would. If they didn't, I assume this is because this is just a huge undertaking (as is the whole project), and they're busy with other work which has higher priority. I would assume that you will end up in exactly the same situation.

Maybe eventually, but the flexibility that comes with being a new project still gives us a significant opportunity to advance the state of the art before that happens. At that point, maybe something else will come along to innovate free of all the technical debt Spectrum has acquired by then, and then the state of the art will advance even further. Such is the nature of software development.

...

  More generally, I'm wondering whether this

projects' goals couldn't be better achieved by trying to work with the Qubes developers to integrate Nix. It may very well be that they would reject it for some reason, but then the logical next step would be to fork Qubes. Have you reached out to the Qubes developers? I've had the pleasure of speaking with several Qubes developers on multiple occasions. I also know that there is work being done to support NixOS templates in Qubes. Do you have any link you can share to work on Nix template being used in Qubes?

This was the last I heard of it:

https://hackmd.shackspace.de/qubes-nixos

...

As I see it, though, the real benefit of Nix here is that it would allow defining the whole system in one single Nix configuration. Doing this in Qubes would require big fundamental changes to how Qubes works. I believe the idea has been brought up to the Qubes developers, and as I recall they were not keen. I believe there is room in this space for Qubes and Spectrum to coexist. There's definitely a place for both efforts, but I'm just wondering if it would not be better to implement Spectrum's ideas while working more closely with Qubes. For example, creating a git branch or a fork that replaces Xen with KVM and integrates Nix. Now, that may very be not the best way to make progress on this project, because ramping up on the Qubes codebase will be an inefficient use of your time, or because you want to first experiment with some ideas in isolation, or for any other reason. However, the potential benefits of such a collaboration are, in my opinion, too big to pass on without strongly pursuing it. Just to name a few:

• You will get feedback from top tier security experts
• Qubes already has strong credibility which can be a huge driver of adoption, especially if your project will be endorsed by Qubes devs
• You will be in a better position to integrate some of the tools of Qubes, which you likely would need to do, and it is less likely that changes in the Qubes tooling will break your project (if you will indeed integrate them)
• You are less likely to be asked time and again "Why not Qubes"? :)

So, why not start a public discussion in Qubes' mailing list on issue tracker to figure out what is needed to accomplish Spectrum's goals?

It will probably turn out that you made the right decision by starting a separate project, but at the very least:

• You'll may get attention from people who can contribute to Spectrum
• The issues involved with be publicly documented and searchable for future generations

I think you maybe don't appreciate just how huge an undertaking this would be. There is so much that would have to change about how Qubes works that I think you'd end up having to reimplement most of it anyway, but you'd be doing it bit by bit, never having the opportunity to consider the system as a whole.

At the end of the day I just don't believe that trying to shoehorn these changes into Qubes is the best way to make progress. It might well be valuable to try that, but even so it would make much more sense for somebody who believes in that approach to dedicate the huge amount of effort required to attempt it, rather than me. This could be another effort that could be pursued in parallel to my work on Spectrum.

The goal is not necessarily doing this work, but to better understand what it would require, which can guide the work on Spectrum, even if it won't share much with Qubes (since you are likely to run into some of the design decisions that Qubes had to make). Well, maybe you already figured this out. I know how Qubes works only superficially, so it may be obvious to more knowledgeable people. In any event, I personally would like to understand this better, as I'm interested in exploring the integration of Qubes and Nix. If you have any insights to share, it would be great, and I may also ask the Qubes devs.