general high-level discussion about spectrum
 help / color / mirror / Atom feed
* This Week in Spectrum
@ 2020-04-19 22:25 Alyssa Ross
  0 siblings, 0 replies; 3+ messages in thread
From: Alyssa Ross @ 2020-04-19 22:25 UTC (permalink / raw)
  To: devel, discuss

[-- Attachment #1: Type: text/plain, Size: 8312 bytes --]

Wow, a week isn't very long at all.  But still, there's definitely
enough to write about, even if I spent most of the week just reading
mailing list threads.  More on that in a bit.


Spectrum Live + nixpkgs
-----------------------

I did another stream.  This time, we looked at updating the Chromium OS
packages in Spectrum.  The script I used to update them had broken -- it
was stuck on an old version, so first we had to fix the script, and then
fix the package builds.  A lot of the Chromium OS packages aren't
actually necessary at the moment, but I made them a while ago when it
looked like they would be.  I couldn't get one particular such package,
libchrome, to build any more at all after spending quite a few hours on
it, so I ended up just dropping it, along with libbrillo, which depended
on it.  Some other package updates had some complexity, but we managed
to work through them all on the stream.

This stream was much more focused on what I was working on than the last
one.  The last one was more about people asking general questions and me
answering them and demonstrating things.  This time I stayed on the
package updates for the whole stream.  I figure people asked all their
questions last time.  But there were lots of questions about specific
issues in the package updates, etc.  It was nice to know that I had
explained things sufficiently well that people could follow along with
what I was doing.  Viewer numbers were up too, which was nice.  We were
on 15 for most of the stream.

The packaging work I mostly did on the stream, and then finished up
afterwards is here:

https://spectrum-os.org/git/nixpkgs/log/?h=2bd191e91f097ff2146099fae8b7c8bd90817808

And a couple of small upstream Nixpkgs PRs:

https://github.com/NixOS/nixpkgs/pull/85549
https://github.com/NixOS/nixpkgs/pull/85551

crosvm still needs to be updated in upstream Nixpkgs.  If you want to
help out, this might be a good thing to pick up!  You'd need to copy the
update.py change I made in Spectrum's Nixpkgs to the crosvm update.py in
upstream Nixpkgs, and then copy the changes in Spectrum nixpkgs' crosvm
to upstream Nixpkgs's crosvm.

If not, I'll probably get to it in a few days.


crosvm
------

Last week, I said that we might me able to use Virtio Wayland as a
transport for crosvm device <-> crosvm communication across a VM
boundary, but that I'd just had the idea and I might decide it couldn't
work.

Having researched it extensively this week, reading over the code for
the virtio_wl kernel driver with puck and consuming a huge number of
messages on assorted kernel-adjacent mailing lists, I think we can make
it work, and it's our best path forward, but I'm still not sure how
exactly.  The main problem is that Virtio Wayland can only send shared
memory from host to guest, not the other way around.  So when a Wayland
compositor in a guest hands the Wayland crosvm device, it's not going to
be able to pass it over the VM boundary to the crosvm instance running a
Wayland client.

Sharing a guest memory buffer with the host, or another guest, turns out
not to be a solved problem in general.  As an example, here is a very
long mailing list thread about how a generic way to do this could work,
from November/December, which reached no firm conclusions:

https://lore.kernel.org/linux-media/20191105105456.7xbhtistnbp272lj@sirius.home.kraxel.org/t/#u

I expect that for most devices, e.g. the network driver, this won't be a
problem.  But we're definitely going to need this for Wayland.  One
option would be to modify a Wayland compositor to make it ask the host
for some shared memory, rather than allocating its own shared memory
buffer.  That would work, but it's the last resort, because it would
have to be a modification made to any Wayland compositor we might want
to use.

At a high level, what we want to do is have a crosvm device running
inside a device VM tell crosvm that a certain address in guest memory
is some shared memory, and have it handle that as it would shared memory
it had allocated itself.

crosvm does support sending dmabufs from guest to host, because of
virtio-gpu.  There's a relatively new kernel API called udmabuf that
converts a memfd into a dmabuf, which sounds like it should solve
exactly this problem.  It has no documentation in the kernel, but I did
manage to find a small amount of documentation in a patch that was for
some reason never applied[1].  I then managed to write a small test
program that at least called all the ioctls properly and ended up with a
dmabuf.  Actually testing this any further will involve a VM, and
testing this sort of thing in a VM at the moment is a real pain, and I
want to make a small tooling improvement around that before I proceed
any further with testing udmabuf.  There are also good reasons to
suspect that the udmabuf idea won't work anyway.  puck thinks that the
only dmabufs Virtio Wayland will be able to transfer are those allocated
by virtio-gpu.  So, not dmabufs that we just made out of a memfd given
to us by a Wayland compositor.

If udmabuf ends up not working, I'll want to see whether we can use
virtio-gpu to allocate dmabufs.  The problem with this is that I don't
know how we'll bridge compositor shared memory to virtio-gpu dmabufs.
It'll presumably involve writing kernel code, which I've been pleased to
avoid so far.  I can at least probably use the udmabuf implementation as
a starting point.

The third least bad option would be to modify the virtio_wl kernel
driver so it supports shared memory originating from the guest.  This
would be very nice to use from userspace, but I just don't think there's
any way I'll be able to write the required kernel code.

And the fourth least bad option (the worst one) would be to modify the
compositor as mentioned above.


spectrum-vm
-----------

The current way of running a Spectrum test VM is to build and run a file
called start-vm.nix that lives in the root of a non-default branch of
Spectrum's nixpkgs called crosvm-demo.  Any time you want to try a
custom kernel, rootfs, or crosvm, you need to modify this file.  This is
extremely annoying.  Also, as time goes on, start-vm.nix is getting
longer and longer, and more and more difficult to understand.  So I
think it's finally time to do something better.  The something better is
that we introduce derivations for a Spectrum rootfs, guest kernel, and
crosvm, and then provide a program, spectrum-vm, that by default runs a
VM with those three components.  It would also have options for
overriding those components, so if you built a custom kernel you could
just drop that one in instead of the default one, for example.  And
finally, it would allow changing the flags passed to crosvm.

I started working on this this morning, and I'm quite excited about it.
It's not going to take very long at all to do, but it's going to be such
a big improvement.  It will also mean I no longer have a separate
crosvm-demo branch, to merge Spectrum nixpkgs master into every time I
update it.  So I'm going to spend the next couple of days on this, and
then get back into investigating udmabuf with a nice new tool.


So, that's what happened this week.  I feel like I just spent most of
the week reading mailing lists and not making much progress, but I've
managed to write quite a lot (again) so I suppose I've accomplished
something!

Now, I've been extremely immersed in this stuff all week, and I suspect
that some of this update, especially the crosvm stuff, will not be clear
to somebody who has not been similarly immersed.  And, it's getting
dangerously close to midnight UTC, my self-imposed This Week in Spectrum
deadline.  So I'm going to send this as is, even though it's probably
confusing.  But!  I do want you to understand what's going on.  So
please feel free to reply and point out anything I didn't explain very
well and ask me for a better explanation.  Just be prepared for me not
to know the answer, since I'm still trying to figure out a lot of this
stuff!  As a nice side effect, maybe we'll get a mailing list discussion
going. :)


[1]: https://lore.kernel.org/lkml/20180911134216.9760-14-kraxel@redhat.com/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: This week in Spectrum
  2020-04-12 16:30 This week " Alyssa Ross
@ 2020-04-12 22:19 ` Amanjeev Sethi
  0 siblings, 0 replies; 3+ messages in thread
From: Amanjeev Sethi @ 2020-04-12 22:19 UTC (permalink / raw)
  To: Alyssa Ross, devel; +Cc: discuss

I find these emails a nice read when I have time. I am perhaps not on the devel list (not a devel).



On Sun, Apr 12, 2020, at 12:30 PM, Alyssa Ross wrote:
> Before we begin, **if you are subscribed only to the discuss mailing
> list**, I'd like to hear from you whether you'd like future weekly
> development updates to come to you or not, because I'm not really sure
> whether this sort of update belongs on the devel mailing list, or the
> discuss one.  So for this initial edition, I'm sending to both, but I
> probably won't do that going forward.
> 
> So, as I wrote in the recent status update on my blog, I'm going to try
> weekly(?)  mailing updates instead of lengthy blog posts every few
> months.  I was going to start last week, but I ended up taking most of
> the week off because I'd been working every day for a long time and was
> starting to find it difficult to concentrate.  But I'm back now, and
> this week has been a good one.
> 
> 
> Spectrum Live
> -------------
> 
> (This was actually last week, but I felt this deserved a mention.)
> 
> A bit more than a week ago now, I did a live stream where I showed
> myself working on a fairly simple change to crosvm.  Viewer numbers
> stabilised at around 11, and we had a lively IRC conversation, which I
> was pleasantly surprised by.  I ended up spending most of the stream not
> working on that change, but instead explaining how various things in the
> system will work, and helping getting a couple of people started running
> the Spectrum code.  By the end of the stream two people had succeeded in
> this, and that means two people who are set up to work on Spectrum.  One
> was even starting work on trying to build a smaller kernel for Spectrum
> VMs!
> 
> Overall I think this was extremely successful, and I'll try to do it
> again soon.  I'll announce the stream a bit ahead of time in #spectrum
> on Freenode.
> 
> 
> crosvm
> ------
> 
> Having previously split crosvm into two separate programs -- one to run
> a virtual device, and one to run a device VM -- I further split it into
> three programs.  This might sound a bit silly, but here's the idea --
> one program is the crosvm that runs an application VM, the second will
> become part of the crosvm that runs a device VM, and the third will be
> the device implementation itself, that will run inside the device VM.
> As a reminder, the overall idea here is to put virtual devices into
> their own VMs, rather than being run directly on the host.
> 
> I also pulled in the latest changes from Google.  Of particular note is
> commit 146450b, which updates the msg_socket library to support
> dynamically sized messages.  If you're paying attention, you'll recall
> that I just wrote my own serde-based replacement for msg_socket because
> I needed to do exactly this.  That work might now have been superseded,
> but I'm not sure yet.  msg_socket2 uses SOCK_STREAM sockets (which can
> be used between VMs or VM<->guest), while msg_socket uses
> SOCK_SEQPACKET, which can't.  I haven't checked whether msg_socket's new
> implementation could work without SOCK_SEQPACKET-style preserved message
> boundaries -- if it can't, msg_socket2 will stick around.  msg_socket2
> also properly handles file descriptor lifetimes using the borrow
> checker.  This means that safe Rust code can't send closed or
> non-existent file descriptors over msg_socket2.  msg_socket does not do
> this, because it just represents file descriptors as integers with no
> lifetime information.  Whatever happens to msg_socket2, at some point
> I'll want to make msg_socket do file descriptor lifetimes properly.  In
> general, it's not clear from existing documentation how to properly
> handle file descriptor lifetimes in Rust, and so I plan a blog post on
> this topic soon.
> 
> The next problem is to figure out communication between the device VM
> and the device crosvm.  The protocols currently in use rely on being
> able to share memory using file descriptors, which is not possible
> between a VM and the host.  To avoid using file descriptors, I would
> have to change those protocols, possibly on a per-device basis, which
> would be a lot of work, and might have to be done again whenever crosvm
> adds a new device type.  But, I think I might have a better, if
> surprising solution.  Despite its name, the Virtio Wayland protocol has
> almost nothing to do with Wayland.  Instead, it's a fairly generic
> socket-like protocol that allows sending "virtual file descriptors" of
> certain specific types between a VM and its host.  So I think I should
> be able to send this sort of communication over "Virtio Wayland",
> without requiring any changes to the communications being channeled.  I
> just had this idea, so it's entirely possible it won't work out, but I
> think it's a neat unexpected reuse of a cool technology.
> 
> 
> Infrastructure
> --------------
> 
> The web interface for the Spectrum mailing lists has been upgraded from
> Django 1.11 (which is now EOL) to Django 2.2 (current LTS; Nixpkgs
> default).  I sent this upstream as Nixpkgs PR #84896.
> 
> Additionally, spectrum-os.org. now has an SPF header, which I thought
> was already there but apparently wasn't.  This should help with mailing
> list deliverability.
> 
> 
> That's it for this week.  Feel free to reply with any comments or
> insights, either on the content or the format.  I'll try to be back with
> another development update next week.
> 
> Attachments:
> * signature.asc

^ permalink raw reply	[flat|nested] 3+ messages in thread

* This week in Spectrum
@ 2020-04-12 16:30 Alyssa Ross
  2020-04-12 22:19 ` Amanjeev Sethi
  0 siblings, 1 reply; 3+ messages in thread
From: Alyssa Ross @ 2020-04-12 16:30 UTC (permalink / raw)
  To: devel; +Cc: discuss

[-- Attachment #1: Type: text/plain, Size: 5102 bytes --]

Before we begin, **if you are subscribed only to the discuss mailing
list**, I'd like to hear from you whether you'd like future weekly
development updates to come to you or not, because I'm not really sure
whether this sort of update belongs on the devel mailing list, or the
discuss one.  So for this initial edition, I'm sending to both, but I
probably won't do that going forward.

So, as I wrote in the recent status update on my blog, I'm going to try
weekly(?)  mailing updates instead of lengthy blog posts every few
months.  I was going to start last week, but I ended up taking most of
the week off because I'd been working every day for a long time and was
starting to find it difficult to concentrate.  But I'm back now, and
this week has been a good one.


Spectrum Live
-------------

(This was actually last week, but I felt this deserved a mention.)

A bit more than a week ago now, I did a live stream where I showed
myself working on a fairly simple change to crosvm.  Viewer numbers
stabilised at around 11, and we had a lively IRC conversation, which I
was pleasantly surprised by.  I ended up spending most of the stream not
working on that change, but instead explaining how various things in the
system will work, and helping getting a couple of people started running
the Spectrum code.  By the end of the stream two people had succeeded in
this, and that means two people who are set up to work on Spectrum.  One
was even starting work on trying to build a smaller kernel for Spectrum
VMs!

Overall I think this was extremely successful, and I'll try to do it
again soon.  I'll announce the stream a bit ahead of time in #spectrum
on Freenode.


crosvm
------

Having previously split crosvm into two separate programs -- one to run
a virtual device, and one to run a device VM -- I further split it into
three programs.  This might sound a bit silly, but here's the idea --
one program is the crosvm that runs an application VM, the second will
become part of the crosvm that runs a device VM, and the third will be
the device implementation itself, that will run inside the device VM.
As a reminder, the overall idea here is to put virtual devices into
their own VMs, rather than being run directly on the host.

I also pulled in the latest changes from Google.  Of particular note is
commit 146450b, which updates the msg_socket library to support
dynamically sized messages.  If you're paying attention, you'll recall
that I just wrote my own serde-based replacement for msg_socket because
I needed to do exactly this.  That work might now have been superseded,
but I'm not sure yet.  msg_socket2 uses SOCK_STREAM sockets (which can
be used between VMs or VM<->guest), while msg_socket uses
SOCK_SEQPACKET, which can't.  I haven't checked whether msg_socket's new
implementation could work without SOCK_SEQPACKET-style preserved message
boundaries -- if it can't, msg_socket2 will stick around.  msg_socket2
also properly handles file descriptor lifetimes using the borrow
checker.  This means that safe Rust code can't send closed or
non-existent file descriptors over msg_socket2.  msg_socket does not do
this, because it just represents file descriptors as integers with no
lifetime information.  Whatever happens to msg_socket2, at some point
I'll want to make msg_socket do file descriptor lifetimes properly.  In
general, it's not clear from existing documentation how to properly
handle file descriptor lifetimes in Rust, and so I plan a blog post on
this topic soon.

The next problem is to figure out communication between the device VM
and the device crosvm.  The protocols currently in use rely on being
able to share memory using file descriptors, which is not possible
between a VM and the host.  To avoid using file descriptors, I would
have to change those protocols, possibly on a per-device basis, which
would be a lot of work, and might have to be done again whenever crosvm
adds a new device type.  But, I think I might have a better, if
surprising solution.  Despite its name, the Virtio Wayland protocol has
almost nothing to do with Wayland.  Instead, it's a fairly generic
socket-like protocol that allows sending "virtual file descriptors" of
certain specific types between a VM and its host.  So I think I should
be able to send this sort of communication over "Virtio Wayland",
without requiring any changes to the communications being channeled.  I
just had this idea, so it's entirely possible it won't work out, but I
think it's a neat unexpected reuse of a cool technology.


Infrastructure
--------------

The web interface for the Spectrum mailing lists has been upgraded from
Django 1.11 (which is now EOL) to Django 2.2 (current LTS; Nixpkgs
default).  I sent this upstream as Nixpkgs PR #84896.

Additionally, spectrum-os.org. now has an SPF header, which I thought
was already there but apparently wasn't.  This should help with mailing
list deliverability.


That's it for this week.  Feel free to reply with any comments or
insights, either on the content or the format.  I'll try to be back with
another development update next week.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-04-19 22:25 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-19 22:25 This Week in Spectrum Alyssa Ross
  -- strict thread matches above, loose matches on Subject: below --
2020-04-12 16:30 This week " Alyssa Ross
2020-04-12 22:19 ` Amanjeev Sethi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).