From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE,WEIRD_PORT autolearn=unavailable autolearn_force=no version=3.4.6 Received: by atuin.qyliss.net (Postfix, from userid 496) id B5A5B7CD06; Mon, 21 Mar 2022 12:11:36 +0000 (UTC) Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id CB0BD7CC53; Mon, 21 Mar 2022 12:11:08 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id 61E487CC85; Mon, 21 Mar 2022 12:11:05 +0000 (UTC) Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by atuin.qyliss.net (Postfix) with ESMTPS id 941DA7CC35 for ; Mon, 21 Mar 2022 12:11:00 +0000 (UTC) Received: by mail-lf1-x12b.google.com with SMTP id a26so6248651lfg.10 for ; Mon, 21 Mar 2022 05:11:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uaoPQFxmhrutTMahrT3iHKaUpeCQVsNTHmbv7dUSQcs=; b=cwO8wc7XcKlW5nsqMqLspBlzK9eCJx8LyrEWeCbZOgH9NdLGLLiOcOQYM9HmcnhJpw yonZkJtr3M2A9AyCQvqdTpUryaGGH5o2wN6lPNumx304SUSXQTjKky9/tnNbfxepiv1k vdw9QPH3sMfYix7oz8oE/JuYWPkytvVdV46aG3DQcjhwqVLCqQ8U2BZYFhyI0vAdEVVF 3/OeaN/cQo27TB+1V+t9+TlXHSCFqTDKCXunUVO9X06qaR+/gvGSQfIM76UPcrzo8y8z 2VV1ZBw4yaKAvGukRskyjis5vnUlKTiRuowLk2IXNxdQCK9930zI6zph6DFXi/v7QIxG hMdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uaoPQFxmhrutTMahrT3iHKaUpeCQVsNTHmbv7dUSQcs=; b=TnJa31F0N9tC4z4sDS2Em3EkBtmONALIaFP784IP6dfZb1a+IIuLdyqJK+k87566RO m8G3rKJEx629BfSBLU7/s9v1DvImUzYDUYCUm5WlRFuozdg2WoQBmp0ykRn0ugiDKlmL a9V0RVZYWIB9ijKdNpGQWBJErchGQOlh4BcPq5to44YKHh1scYqHzeM/ozF7+CYD1BS9 R7mJbFP3PweM7vtXddhR5OjKWAJB52iq1paCDAKKnOgxG6ExeWibYxUVKnnqBmUkmJx4 HD0pGRDW7IqxhiXMcOKfqy+7AZ5pn3eBK1DNI2vlaRXhET6Y9l8XDKtqEuFVAF5HMZbF H4IQ== X-Gm-Message-State: AOAM533rlviirqFPRMeDt1EpsTAouFjOTSQ/zKWuuePCSoRNE/4ojM0o bkg6EefcfaiXSnAG0+D35CR3yTmZSEk+BWtp+cBHV3LehI8= X-Google-Smtp-Source: ABdhPJxyCaZPITARK/ZizpCu+E1Dxbvqjrt86okWyzWgX/NoQCaNQ7XelMQWE1UhRXPaOjiXR7NMVrQ9dEUaFgpODWo= X-Received: by 2002:a05:6512:3d89:b0:44a:2ee3:cce7 with SMTP id k9-20020a0565123d8900b0044a2ee3cce7mr2491046lfv.6.1647864656754; Mon, 21 Mar 2022 05:10:56 -0700 (PDT) MIME-Version: 1.0 References: <20210309162556.ctiy3yfp7plkbdqs@x220.qyliss.net> <87o871tqqu.fsf@alyssa.is> <87bl2sp3p3.fsf@alyssa.is> <20220315140604.ta433j557jsoknkv@eve> <20220315202339.6bqrzsf4yjkggbup@eve> In-Reply-To: From: Thomas Leonard Date: Mon, 21 Mar 2022 12:10:43 +0000 Message-ID: Subject: Re: Using virtio-gpu instead of virtwl To: Alyssa Ross Content-Type: text/plain; charset="UTF-8" Message-ID-Hash: UM6HGQNKGXRMLZ6ZQ4C6JSMHI77J436J X-Message-ID-Hash: UM6HGQNKGXRMLZ6ZQ4C6JSMHI77J436J X-MailFrom: talex5@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-discuss.spectrum-os.org-0; header-match-discuss.spectrum-os.org-1; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: discuss@spectrum-os.org X-Mailman-Version: 3.3.5 Precedence: list List-Id: General high-level discussion about Spectrum Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 16 Mar 2022 at 16:18, Thomas Leonard wrote: > > On Tue, 15 Mar 2022 at 20:23, Alyssa Ross wrote: > > > > On Tue, Mar 15, 2022 at 02:06:04PM +0000, Alyssa Ross wrote: > > > On Sun, Mar 13, 2022 at 03:08:04PM +0000, Thomas Leonard wrote: > [ virtio-gpu ] > > > > I'm happy to test it now with Linux 5.16, but I think I need a newer > > > > crosvm too. Looks like support got added in > > > > https://github.com/google/crosvm/commit/231a54f36fbfb90ebac152d66f937af3644f6676, > > > > so it should be in 94.14150. > > > > > > > > Is there a new-enough package for crosvm available somewhere? NixOS > > > > only has 81.12871.0.0-rc1, and spectrum seems to be on > > > > 91.13904.0.0-rc2. > > > > > > I don't know of one. I haven't been keeping it up to date since we've > > > mostly switched over to cloud-hypervisor for Spectrum, but it doesn't > > > have any way of doing virtio-gpu (yet, porting crosvm's vhost-user-gpu > > > implementation is on my list). > > > > > > I'll have a go at updating it in Nixpkgs and report back. > > > > Okay, here's a package for crosvm 99.14468.0.0-rc1: > > https://github.com/NixOS/nixpkgs/pull/164306 > > Thanks - that's very useful! > > Testing it from NixOS stable, it crashed with [...] > [src/panic_hook.rs:90] thread 'virtio_blk' panicked at 'Failed to > create an executor: Uring(CreatingContext(Setup(12)))', > devices/src/virtio/block/asynchronous.rs:767:50 > > From a brief look at the code, it looks like crosvm sees that the host > is running Linux >= 5.10 and creates a test uring, which works. Then > it tries to create a real one, and that fails. I fixed it by > increasing the limit on locked memory in configuration.nix However, this led to another problem: after resuming the host from suspend, the crosvm drivers would crash with e.g. [devices/src/virtio/block/asynchronous.rs:792] An error with a uring source: URing::enter: Failed to enter io uring: 4 So a better fix is to run with "ulimit -l 0". That causes crosvm to decide that uring isn't available and makes it fall back to the old (working) system. > > I've tested some basic stuff, but haven't tested virtio-gpu. I don't > > see any reason it wouldn't work, though. > > I tried running with `crosvm --gpu`, but after `modprobe virtio-gpu`, > crosvm crashed with: > > [ 31.326763] Invalid ELF header magic: != ELF > [ 31.331450] [drm] pci: virtio-gpu-pci detected at 0000:00:07.0 > [ 31.333020] [drm] Host memory window: 0x200000000 +0x200000000 > [ 31.333983] [drm] features: +virgl -edid +resource_blob +host_visible > [ 31.333984] [drm] features: +context_init > [ 31.337289] [drm] number of scanouts: 1 > [ 31.337938] [drm] number of cap sets: 1 > [ERROR:src/linux.rs:3264] child pcivirtio-gpu (pid 63446) died: signo > 17, status 11, code 2 > [ERROR:devices/src/proxy.rs:212] failed write to child device process > pcivirtio-gpu: failed to send packet: Broken pipe (os error 32) > > Then I tried with `--gpu=backend=2d` and that didn't crash, but > instead opened a window showing some bootloader stuff. I now have > /dev/dri/{card0, renderD128} devices, so I guess the next step is > figuring out what they do! > > > Good luck! Let me know if I can help with anything else. Looking at the Linux virtio_gpu driver, it seems that using contexts requires virgl: static int virtio_gpu_context_init_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { ... if (!vgdev->has_context_init || !vgdev->has_virgl_3d) return -EINVAL; https://github.com/torvalds/linux/blob/f443e374ae131c168a065ea1748feac6b2e76613/drivers/gpu/drm/virtio/virtgpu_ioctl.c#L732 I think perhaps that crosvm is compiled without the "virgl_renderer" feature (it's not in the default set), and this is causing it to crash because that's also "self.default_component". I don't know how to compile crosvm with virgl enabled, though. -- talex5 (GitHub/Twitter) http://roscidus.com/blog/