On Sun, Apr 11, 2021 at 11:57:38AM +0000, Alyssa Ross wrote:
> This is a VM that acts as a router.  It handles talking to network
> hardware, and other VMs can be connected to it by attaching virtual
> ethernet devices.
>
> It expects to get a physical ethernet device passed through to it
> using VFIO.  Wi-Fi should work too, but would need to be configured so
> I've stuck with Ethernet for now.  We use ConnMan[1] to configure
> physical network interfaces, and it automatically takes care of DHCP
> and stuff for us.  I chose ConnMan over NetworkManager because it was
> easier to get set up.
>
> Virtual ethernet devices are identified by a specific OUI in their MAC
> address.  The NIC part of the MAC address is used to encode the last
> three octets of the IPv4 address this VM should assign to the
> interface.  This way, the host can tell this VM what the address of
> each virtual interface is without having to resort to a secondary
> communication channel.  The first octet will always be 100, as the
> intention is to use the IPv4 shared address space (aka the CGNAT
> space) for inter-VM networks to match the behaviour of Chromium OS[2].
>
> Every networking client will be connected to a router VM with a /31,
> where the low address is for the router, and the high address is for
> the client.  This way the host's job is as simple as possible -- it
> just has to connect two TAPs together, without worrying about any
> routing rules or anything -- that can all happen in the VM.  This does
> mean that the router has to have a virtio-net device for every client,
> though.  We may run into scaling limitations with this approach, in
> which case we might have to revisit how this works on the host.
>
> [1]: https://git.kernel.org/pub/scm/network/connman/connman.git/about/
> [2]: https://chromium.googlesource.com/chromiumos/platform2/+/39e48f668a937d266638f3f7d31d3427a4966464/patchpanel/address_manager.cc#13
> ---
>  .../os-specific/linux/spectrum/vm/default.nix |   2 +
>  .../linux/spectrum/vm/net/default.nix         | 165 ++++++++++++++++++
>  2 files changed, 167 insertions(+)
>  create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix

Cole, if you want to test this, you'll probably need the following
additional diff to make it work with your hardware.

It's quite clearly time to support loadable modules in this VM, but I'd
like to leave that for a follow up after this series is sorted.

diff --git i/pkgs/os-specific/linux/spectrum/vm/net/default.nix w/pkgs/os-specific/linux/spectrum/vm/net/default.nix
index 079311c80e6..1deb7031caf 100644
--- i/pkgs/os-specific/linux/spectrum/vm/net/default.nix
+++ w/pkgs/os-specific/linux/spectrum/vm/net/default.nix
@@ -6,6 +6,7 @@ runCommand "vm-net" rec {
   linux = linux_vm.override {
     structuredExtraConfig = with lib.kernel; {
       E1000E = yes;
+      IGB = yes;
       PACKET = yes;

       IP_NF_NAT = yes;