From: Alyssa Ross <hi@alyssa.is>
To: Ryan Lahfa <ryan@lahfa.xyz>
Cc: devel@spectrum-os.org
Subject: Re: [PATCH] release/checks/try.nix: init
Date: Sat, 27 May 2023 15:07:20 +0000 [thread overview]
Message-ID: <87y1l995jr.fsf@alyssa.is> (raw)
In-Reply-To: <wjfkrqljbbkrm6kcgpiyronk5fnqiyvhxt4ji5sgaoienujp65@ospb75g7l7o3>
[-- Attachment #1: Type: text/plain, Size: 4561 bytes --]
Ryan Lahfa <ryan@lahfa.xyz> writes:
> On Sat, May 27, 2023 at 02:38:22PM +0000, Alyssa Ross wrote:
>> Ryan Lahfa <ryan@lahfa.xyz> writes:
>>
>> > On Fri, May 26, 2023 at 09:07:58PM +0000, Alyssa Ross wrote:
>> >> This is a regression test for c7f87f3 ("host/rootfs: allow growing ext
>> >> partition to fail"). It's the first time we're actually doing
>> >> automated tests of a Spectrum boot. For now, I'm using the NixOS test
>> >> framework, but because we're not using NixOS and not setting any NixOS
>> >> options, it feels to me like it doesn't actually buy us very much, so
>> >> if it doesn't start adding more value as we add more (or more complex)
>> >> tests, it might be simpler to just use a shell/execline script for
>> >> tests.
>> >
>> > Are you only interested into tests that works without any
>> > instrumentation?
>> >
>> > e.g. what if Spectrum added the backdoor.service in their service
>> > management? That'd repair the ability of the test framework to have
>> > better interactions with the guest without QEMU Agent.
>>
>> At least until I really need tests that depend on guest cooperation, yes.
>> Because to have either the backdoor service or the QEMU agent, I'd
>> either have to build custom images just for testing (which would mean
>> the real images were not actually tested), or I'd have to build those
>> things into all Spectrum images.
>>
>> At some point, it might not be possible to get away from this, but the
>> basic tests I've written so far have all gone fine without the need for
>> any guest cooperation beyond a serial shell.
>
> Makes sense, supporting "minimal" friction for this usecase is
> desirable.
>
> I guess we could have Python *and* Bash/execline as "test scripts" if we want.
>
> But I would need to see more about what would Bash/execline-oriented
> test scripts would look like in practice.
>
> Python is suboptimal for various things but the "batteries included" are
> very useful for getting a OK-grade thing running.
I don't think there's much of a reason to support shell/execline scripts
for NixOS tests. My point here is more that I'm not really getting
anything out of the NixOS test setup at all. I already have to
construct my own QEMU command line, so the only thing I'm really getting
out of it is the wait_for_console_text() function, which wouldn't be
_that_ hard to do without in a shell/execline script either.
It would probably be different if I _was_ using a guest agent / backdoor
service, so I do think there's value to the NixOS test framework trying
to support other distros. It would be useful when the tests would
benefit from its built-in systemd integration, or OCR functionality, etc.
>> >> +config.pkgs.nixosTest ({ pkgs, ... }: {
>> >> + name = "try-spectrum-test";
>> >> + nodes = {};
>> >> +
>> >> + testScript = ''
>> >> + import shlex
>> >> + import subprocess
>> >> +
>> >> + conf = subprocess.run([
>> >> + "${pkgs.mtools}/bin/mcopy",
>> >> + "-i",
>> >> + "${live}@@1M",
>> >> + "::loader/entries/spectrum.conf",
>> >> + "-",
>> >> + ], stdout=subprocess.PIPE)
>> >> + conf.check_returncode()
>> >> +
>> >> + cmdline = None
>> >> + for line in conf.stdout.decode('utf-8').splitlines():
>> >> + key, value = line.split(' ', 1)
>> >> + if key == 'options':
>> >> + cmdline = value
>> >
>> > Is there any reason to not have `conf` / `cmdline`
>> > being derived in a derivation?
>>
>> We can't know it at eval time, because it includes the verity hash. So
>> our options are to recompute it here, or reuse the results of the
>> derivation that already computed it, which is the approach I've taken
>> here. It's a bit unweildy, but I blame Python for that. If we do end
>> up switching to shell it'd just be:
>>
>> mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p'
>
> Makes sense, but do you need it at evaltime?
>
> Wouldn't this be a possibility:
>
> ```nix
> let optionsFile = pkgs.runCommand "extract-options" {} ''mcopy -i ${live}@@1M ::loader/entries/spectrum.conf - | sed -n 's/^options //p' > $out'';
> in
> ''
> with open(${optionsFile}, "r") as f:
> options = f.read()
> ''
> ```
>
> Ideally, with the https://github.com/NixOS/nixpkgs/issues/156563
> proposal, this could even become easier IMHO.
Well yeah, I could, but I think that's just extra complexity over doing
it in the driver script. It's not _that_ bad in Python.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
next prev parent reply other threads:[~2023-05-27 15:07 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-26 21:07 Alyssa Ross
2023-05-27 13:48 ` Ryan Lahfa
2023-05-27 14:38 ` Alyssa Ross
2023-05-27 14:53 ` Ryan Lahfa
2023-05-27 15:07 ` Alyssa Ross [this message]
2023-05-27 15:09 ` Ryan Lahfa
2023-05-27 15:10 ` Ryan Lahfa
2023-05-27 21:39 ` Alyssa Ross
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87y1l995jr.fsf@alyssa.is \
--to=hi@alyssa.is \
--cc=devel@spectrum-os.org \
--cc=ryan@lahfa.xyz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://spectrum-os.org/git/crosvm
https://spectrum-os.org/git/doc
https://spectrum-os.org/git/mktuntap
https://spectrum-os.org/git/nixpkgs
https://spectrum-os.org/git/spectrum
https://spectrum-os.org/git/ucspi-vsock
https://spectrum-os.org/git/www
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).