From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on atuin.qyliss.net X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 Received: from atuin.qyliss.net (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id C5FAB28084; Sun, 15 May 2022 14:20:45 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id CC23227FE5; Sun, 15 May 2022 14:20:41 +0000 (UTC) Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) by atuin.qyliss.net (Postfix) with ESMTPS id 79EDC28075 for ; Sun, 15 May 2022 14:20:39 +0000 (UTC) Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-f189b07f57so3148062fac.1 for ; Sun, 15 May 2022 07:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=f8Kbdo2Wk3mH+0LexlJYNegGt4PIBFuCBkR//mwFroY=; b=bBxGEKFPq9nj6d5oEWtcYTJsFMJqc2hn/Jpzq40afDLs7ix3c2xaJRXOKr9IrwNs38 7GjeK0RPR1UkIzXjufakhmN510DccIaoZ0SYKLwBeLsMokA2BeLZuwJRGP5guiB8SQE1 gYF1usYeinV13kZOcPwbGFXDHNNme8+TYC6UIO8cy3cjjlPFbwnYirrLCE5eaOvTaKgp F4uG9G4pVNixXL5mHYerc8PQl3dBycgZNSPXH7b/rCxq/IMoj6tc74YLq2ALJbu+NXdZ nURUH59rUcSH82uZee9BHYrcxUBjAsSSEgRYTJP5E5BFvk1ZqOPTIvggYk6XjK08T6yd bXUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=f8Kbdo2Wk3mH+0LexlJYNegGt4PIBFuCBkR//mwFroY=; b=fMydVB6awg+YrGuvQKP4z0SXxRLwNwztYEQQFn4msxMNDIDQ7oAsaGMv9IP+bIbNY0 JsZH+iN13QoLefG/w5ES8gatdN997V+azfijT1USG2H9VAoNLxG1uYpnYdHwsFtoLo4d lCk/e3W0MBDw4dsXYqQB9UmRhatSb9ShPY3z2d4ocxxBhJ7RuojFRc11ZJi/VrA5D2rl 4fQLoJMahv7EDphoXCUnAUUoYKGTVWQJAkeb9s1fF7KGAnDRn56G9m4aQ/Rj0Cnk1HOJ gP2n0RQjC73bh7Pm91w3zTqQLFvotmjEetvbBJKmBNVSK5TM/Kj1Hob5QvjNsrzHanbl nM9g== X-Gm-Message-State: AOAM530QCtxbd0e/9AjPJCZ3eakzuyOooQ3m6mwR/RO9968Fl74TAs1A hn5DIeGPZFna/S3yQ00fvjdM9upVnrZ7BKsx3GA= X-Google-Smtp-Source: ABdhPJwaMmLzJjkCGYLt7oiZiNPyAJCgIlQ8cttYT02kd2OmTEYheo15bPwDwJ/iSaxDzVvb97UHwd2ptsNCgzcHc/4= X-Received: by 2002:a05:6870:538d:b0:de:aa91:898e with SMTP id h13-20020a056870538d00b000deaa91898emr6826014oan.54.1652624436315; Sun, 15 May 2022 07:20:36 -0700 (PDT) MIME-Version: 1.0 References: <20220315140604.ta433j557jsoknkv@eve> <20220315202339.6bqrzsf4yjkggbup@eve> <20220321160534.w4oifq3cdygrobrm@eve> <20220322111639.pvr5gxz6zmeu5oua@eve> <20220414135737.lvfbrch34kjojn3s@eve> In-Reply-To: <20220414135737.lvfbrch34kjojn3s@eve> From: Thomas Leonard Date: Sun, 15 May 2022 15:20:24 +0000 Message-ID: Subject: Re: Using virtio-gpu instead of virtwl To: Alyssa Ross Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: BK4WYBFHJ264NRFOJZBPYW6XV6RWSCGE X-Message-ID-Hash: BK4WYBFHJ264NRFOJZBPYW6XV6RWSCGE X-MailFrom: talex5@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-discuss.spectrum-os.org-0; header-match-discuss.spectrum-os.org-1; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: discuss@spectrum-os.org X-Mailman-Version: 3.3.5 Precedence: list List-Id: General high-level discussion about Spectrum Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 14 Apr 2022 at 13:57, Alyssa Ross wrote: > > On Wed, Apr 13, 2022 at 05:12:13PM +0000, Thomas Leonard wrote: > > On Wed, 6 Apr 2022 at 12:19, Thomas Leonard wrote: > > [ converting from virtwl to virtio-gpu ] > > > I tried, but failed, to figure out the protocol. I did manage to get = a > > > test application showing a little animation, but it crashes after a > > > few seconds. > > > > OK, I found a solution to this: you can just open the device file > > twice and use one instance for Wayland messages and the other for > > allocating images. This avoids the first race. With that, I got the > > proxy converted: > > > > https://github.com/talex5/wayland-proxy-virtwl/pull/28 > > > > Though I'm not sure it's an improvement: +1,819 =E2=88=92577 lines! > > > > Instructions for configuring crosvm to use it: > > > > https://github.com/talex5/wayland-proxy-virtwl#virtio-gpu-support > > > > And I wrote up my guesses about the protocol here: > > > > https://github.com/talex5/wayland-proxy-virtwl/blob/master/virtio-spe= c.md > > That's extremely helpful, thanks for writing it up! A small update on this: First, I got virtio-gpu working with the jail. It just needs a couple of extra paths for NixOS: diff --git a/src/linux.rs b/src/linux.rs index ad031749..52d3142f 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -790,6 +790,8 @@ fn gpu_jail(cfg: &Config, policy: &str) -> Result> { jail_mount_bind_if_exists( &mut jail, &[ + "/run/opengl-driver", + "/nix/store", "/usr/lib", "/usr/lib64", "/lib", Secondly, I realised that the "video" memory returned by virtio-gpu was just regular host memory (from stracing crosvm). This is because crosvm is compiled without minigbm support and falls back to this. After enabling minigbm and its amdgpu support (which also required adding a dependency on mesa) it started allocating vram. However, this broke the proxy because vram can't be used with the Wl_shm protocol. I updated the proxy's test.ml code (which opens a window with a scrolling pattern) to use Linux_dmabuf_unstable_v1, and that almost worked. But some of the pixels are wrong and it often crashes the whole VM. Possibly some of the output is getting stuck in a cache somewhere and is not actually flushed to the graphics card in time? Unfortunately, I don't know anything about GPUs. Anyway, I merged the changes I had, in case anyone else wants to try to get it working: https://github.com/talex5/wayland-proxy-virtwl/pull/33 If it did work, it should make things a bit more efficient by avoiding one copy. At the moment, I think the process is: 1. Application renders to guest memory. 2. Proxy copies to host memory. 3. Compositor copies to graphics card. If the mingbm stuff worked, the proxy could copy directly to the graphics card. And possibly the application could render there directly in the first place, but that would need more work. --=20 talex5 (GitHub/Twitter) http://roscidus.com/blog/