summary refs log tree commit diff
path: root/nixos/lib
Commit message (Collapse)AuthorAge
* nixos/testing: fix invalid node names detectionDavid Arnold2021-07-28
| | | | | | | | | | | | | the use of python further restricts possible RFC1035 host labels since dash is not allowed for use in python identifiers. The previous implementation of this check was flawed, since it did not check the `hostName` value that is actually used to construe the identifier, but the node name, which can be anything, e.g. just `machine`. The previous implementation, by further restricting RFC1035 labels, only for the sake of testing seems to be an unacceptable restriction and should be addressed separately.
* tigervnc, tightvnc: add basic testsIngo Blechschmidt2021-07-20
| | | | Co-Authored-By: Ingo Blechschmidt <iblech@web.de>
* nixos/test-driver: allow overriding qemu binary in create_startcommandmisuzu2021-07-14
|
* Merge pull request #127744 from cransom/disk-image-fasterRobert Hensing2021-06-26
|\ | | | | make-disk-image: Reduce forking
| * make-disk-image: Reduce forkingCasey Ransom2021-06-21
| | | | | | | | | | | | | | | | | | | | | | | | The disk image calculator was using find + exec forking du for every file in the disk image, making it very slow. Change du to accept files, nul delimeted on stdin to speed it back up. Before change: nix-build nixos/tests/image-contents.nix 9.71s user 1.06s system 8% cpu 2:13.11 total After change: nix-build nixos/tests/image-contents.nix 9.93s user 1.23s system 21% cpu 51.601 total
* | nixos/test-driver: replace termlib with socatJörg Thalheim2021-06-22
|/ | | | | telnetlib does not handle unicode, which is annoying when using systemctl. Also this gives us a nice readline with history.
* nixosTest: Force system.nixos.revision constantDavid Arnold2021-06-21
| | | | | | | | | | | | | | nixos tests are blended with other system configurations, hence their settings must be either enforced or defaulted. This particular setting is set via lib.nixosSystem as `system.nixos.revision = final.mkIf (self ? rev) self.rev;` which would mean that without this change no flake generated nixos could be blended with nixos testing. This setting was made previously constant in 169c6b4b1478a3a0c823c99ea39d4082f76a2708 in order to avoid pointless rebuilds of the testing VMs, but was set without enforcing it.
* Merge pull request #127309 from jakobrs/nixos-normalise-nullKim Lindberger2021-06-18
|\ | | | | Resolve evaluation failure when filesystem device option is null
| * nixos/lib: Handle null `device` correctlyjakobrs2021-06-18
| |
* | nixos/test-driver: Fix passing passthru attributeaszlig2021-06-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Apparently this looks like it was forgotten when doing commit 3884ff70badca0c93c717e6190946a9a2846e948, which refactored the test runner and driver a bit. The passthru argument actually was correctly reintroduced in setupDriverForTest, but the actual makeTest function didn't use it. This fixes the nixpkgs tarball job, which previously failed with: attribute 'elkPackages' missing, at /build/source/pkgs/tools/misc/logstash/6.x.nix:58:30 Signed-off-by: aszlig <aszlig@nix.build> Acked-by: David Arnold <dar@xoe.solutions> Fixes: https://github.com/NixOS/nixpkgs/issues/127274 Merges: https://github.com/NixOS/nixpkgs/pull/127346
* | Merge pull request #125973 from ↵Jacek Galowicz2021-06-16
|\ \ | | | | | | | | | | | | blaggacao/nixos-test-ref/02-normalise-test-driver-nix-expressions nixos test ref/02 normalise test driver nix expressions
| * | nixos/tests/test-driver: cleanup nix expressionDavid Arnold2021-06-13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Less nesting, where that improves readability. More nesteing, where that improves readability, but most importantly: Expose individual functions separately so that they can be more easily built directly, eg.: `nix build --impure --expr '(import ./testing-python.nix {system = builtins.currentSystem;}).mkTestDriver'`
* | | Merge pull request #86967 from jakobrs/more-general-fsbeforeKim Lindberger2021-06-14
|\ \ \ | |/ / |/| / | |/ nixos/lib/utils: Add `fileSystems.<name>.depends` option and generalise fsBefore (fixes #86955)
| * nixos/lib, nixos/filesystems: Make fsBefore more stable, and add `depends` ↵jakobrs2021-06-08
| | | | | | | | option
* | nixos/tests/test-driver: cleanup "dead" code (USE_SERIAL)David Arnold2021-06-06
| | | | | | | | | | | | | | | | | | | | At nixpkgs root: `rg redirectSerial ./` does not result in any other match nor does `rg USE_SERIAL ./` except for an unrelated match in: pkgs/tools/graphics/argyllcms/default.nix
* | nixos/test-driver: Run commands with error handlingtalyz2021-06-05
| | | | | | | | | | | | | | | | | | | | | | | | Bash's standard behavior of not propagating non-zero exit codes through a pipeline is unexpected and almost universally unwanted. Default to setting `pipefail` for the command being run; it can still be turned off by prefixing the pipeline with `set +o pipefail` if needed. Also, set `errexit` and `nonunset` options to make the first command of consecutive commands separated by `;` fail, and disallow dereferencing unset variables respectively.
* | make-test-python: disallow aliasesFrederik Rietdijk2021-06-03
| | | | | | | | | | | | | | When importing Nixpkgs within Nixpkgs, we should not consider aliases to ensure we don't rely on them internally. There are probably more places that need to be converted.
* | Merge pull request #125372 from Synthetica9/shell_interactDomen Kožar2021-06-03
|\ \ | | | | | | nixos/tests/test-driver: add shell_interact
| * | nixos/tests/test-driver: make it clear when shell is readyPatrick Hilhorst2021-06-03
| | | | | | | | | Co-authored-by: Domen Kožar <domen@enlambda.com>
| * | nixos/tests/test-driver: document shell_interactPatrick Hilhorst2021-06-02
| | |
| * | nixos/tests/test-driver: add shell_interactPatrick Hilhorst2021-06-02
| |/
* | nixos/lib/make-ext4-fs: Fix: `resize2fs -M' can leave insufficient slackDamien Diederen2021-05-31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The root filesystem resizing step, `resize2fs -M', does not provide any control over the amount of slack left in the result. It can produce an arbitrarily tight fit, depending on how well the payload aligns with ext4 data structures. This is problematic, as NixOS must create a few files and directories during its first boot, before the root is enlarged to match the size of the containing SD card. An overly tight fit can cause failures in the first stage: mkdir: can't create directory '/mnt-root/proc': No space left on device or in the second stage: install: cannot create directory '/var': No space left on device A previous version of `make-ext4-fs' (before PR #79368) was explicitly "reserving" 16 MiB of free space in the final filesystem. Manually calculating the size of an ext4 filesystem is a perilous endeavor, however, and the method it employed was apparently unreliable. Reverting is consequently not a good option. A solution would be to create some sort of "balloon" occupying inodes and blocks in the image prior to invoking `resize2fs -M', and to remove these temporary files/directories before the compression step. This changeset takes the simpler approach of simply dropping the resizing step. Note that this does *not* result in a larger image in general, as the current procedure does not truncate the `.img' file anyway. In fact, it has been observed to yield *smaller* compressed images---probably because of some "noise" left after resizing. E.g., before-vs-after: -r--r--r-- 2 root root 607M 1. Jan 1970 nixos-sd-image-21.11pre-git-x86_64-linux.img.zst -r--r--r-- 2 root root 606M 1. Jan 1970 nixos-sd-image-21.11pre-git-x86_64-linux.img.zst
* | nixos/test-driver: mention the elapsed time when it times outAndreas Rammhold2021-05-30
|/ | | | | | | | | | | | | | | | | | | | | | | For now you had to know that the actions are retried for 900s when seeing an error like > Traceback (most recent call last): > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 927, in run_tests > exec(tests, globals()) > File "<string>", line 1, in <module> > File "<string>", line 31, in <module> > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 565, in wait_for_file > retry(check_file) > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 142, in retry > raise Exception("action timed out") > Exception: action timed out in your (hydra) build failure. Due to the absence of timestamps you were left guessing if the machine was just slow, someone passed a low timeout value (which they couldn't until now) or whatever might have happened. By making this error a bit more descriptive (by including the elapsed time) these hopefully become more useful.
* Merge pull request #123823 from misuzu/test-driver-usb-boot-speedupJacek Galowicz2021-05-21
|\ | | | | nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
| * nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhcimisuzu2021-05-20
| | | | | | | | | | On my system this change offers ~5X speed up of nixosTests.boot.biosUsb and nixosTests.boot.uefiUsb tests.
* | Merge pull request #123052 from xoe-labs/da-test-vm-innteractive-log-switchRobert Hensing2021-05-20
|\ \ | | | | | | nixos/testing: add interactive serial stdout logs switch and dim them
| * | nixos/testing: add interactive serial stdout logs switch and dim themDavid Arnold2021-05-14
| |/
* | Merge pull request #117275 from hercules-ci/nixosTest-remove-nixpkgs-commit-hashRobert Hensing2021-05-20
|\ \ | |/ |/| nixosTest: Make system.nixos.revision constant
| * nixosTest: Make system.nixos.revision constantRobert Hensing2021-03-22
| | | | | | | | | | | | | | | | Make the revision metadata constant, in order to avoid needless retesting. The human version (e.g. 21.05-pre) is left as is, because it is useful for external modules that test with e.g. nixosTest and rely on that version number.
* | nixos/testing-python.nix: Move makeWrapper to nativeBuildInputsRobert Hensing2021-05-09
| |
* | nixos/testing: Set up scope for testScript linterRobert Hensing2021-05-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Our test driver exposes a bunch of variables and functions, which pyflakes doesn't recognise by default because it assumes that the test script is executed standalone. In reality however the test driver script is using exec() on the testScript. Fortunately pyflakes has $PYFLAKES_BUILTINS, which are the attributes that are globally available on all modules to be checked. Since we only have one module, using this environment variable is fine as opposed to my first approach to this, which tried to use the unstable internal API of pyflakes. The attributes are gathered by the main derivation of the test driver, because we don't want to end up defining a new attribute in the test driver module just to being confused why using it in a test will result in an error. Another way we could have gathered these attributes would be in mkDriver, which is where the linting takes place. However, we do have a different set of Python dependencies in scope and duplicating these will again just cause confusion over having it at one location only. Signed-off-by: aszlig <aszlig@nix.build> Co-Authored-By: aszlig <aszlig@nix.build>
* | nixos/testing-python.nix: Expose driverRobert Hensing2021-05-09
| | | | | | | | (cherry picked from commit a2c9220568648b4528154ebd8e657add243ed0b4)
* | nixos/testing: Switch from black to pyflakesaszlig2021-05-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | So far, we have used "black" for formatting the test code, which is rather strict and opinionated and when used inline in Nix expressions it creates all sorts of trouble. One of the main annoyances is that when using strings coming from Nix expressions (eg. store paths or option definitions from NixOS modules), completely unrelated changes could cause tests to fail, since eg. black wants lines to be broken. Another downside of enforcing a certain kind of formatting is that it makes the Nix expression code inconsistent because we're mixing two spaces of indentation (common in nixpkgs) with four spaces of indentation as defined in PEP-8. While this is perfectly fine for standalone Python files, it really looks ugly and inconsistent IMO when used within Nix strings. What we actually want though is a linter that catches problems early on before actually running the test, because this is *actually* helping in development because running the actual VM test takes much longer. This is the reason why I switched from black to pyflakes, because the latter actually has useful checks, eg. usage of undefined variables, invalid format arguments, duplicate arguments, shadowed loop vars and more. Signed-off-by: aszlig <aszlig@nix.build> Closes: https://github.com/NixOS/nixpkgs/issues/72964
* | nixos/test-driver: Allow interactive testing on Wayland-only setupsMichael Weiss2021-05-04
| | | | | | | | | | | | On my system I have XWayland disabled and therefore only WAYLAND_DISPLAY is set. This ensures that the graphical output will still be enabled on such setups (both Wayland and X11 are supported by the viewer).
* | testing-python.nix: document runTests pos argumentRobert Hensing2021-05-03
| |
* | Merge pull request #112504 from hercules-ci/fix-nixosTest-meta-positionRobert Hensing2021-05-03
|\ \ | | | | | | nixosTest: fix meta.position
| * | nixosTest: fix meta.positionRobert Hensing2021-03-23
| |/
* | treewide: use lib.warnIf where appropriateAlyssa Ross2021-04-28
| |
* | Merge pull request #120489 from samueldr/fix/make-disk-image-auto-sizeLassulus2021-04-26
|\ \ | | | | | | Fix make disk image automatic size
| * | make-disk-image: Round image size to the next mebibyteSamuel Dionne-Riel2021-04-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This ensures the following gptfdisk warning won't happen: ``` Warning: File size is not a multiple of 512 bytes! Misbehavior is likely! ``` Additionally, helps towards aligning the partition to be more optimal for the underlying storage. It is actually impossible to align for the actual underlying storage optimally because we don't know what the block device will be! But aligning on 1MiB should help.
| * | make-disk-image: Account for reserved disk spaceSamuel Dionne-Riel2021-04-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a bit of a thorny issue. See, the actual `diskSize` variable is for the *total* disk size, not for the filesystem! The automatic numbers are meant to compute the *filesystem* required space. So we have to add any other reserved space! We have different requirements for reserved space. E.g. there could be none (when it's actually a filesystem image). There could also be 1MiB for alignment for an MBR image, legacy+gpt needs 2MiB, then GPT with an ESP ("bootSize") needs to take the boot partition and GPT size into account too! Though luckily(?) for this latter situation we can cheat! As noted in the change, `bootSize` is NOT the boot partition size. It is actually the offset where the target filesystem starts.
| * | make-disk-image: Account for the ext4 reserved spaceSamuel Dionne-Riel2021-04-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reserved space includes: - inodes space in use (2 blocks per) - about 5.2% of the space The 5.2% reserved space was computed empirically when working on a previous EXT4 image builder. It seems to stabilize around 5% even for much larger filesystems.
| * | make-disk-image: Get proper size for automatic sizeSamuel Dionne-Riel2021-04-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On some filesystems, `du` without `--apparent-size` will not give the actual size for a file. Using `--apparent-size` will give us the actual file size. Though, this is not actually correct still. 1000 × 1 bytes is not 1000 bytes. It is 1000 × ceil(filesize/blockSize)*blockSize. So instead of adding up the actual file sizes. We are adding up the block sizes. Note that this also changes the builder to work with *bytes*, rather than with any other units. Doing maths on bytes is less likely to go awry than doing it on other units.
* | | Merge pull request #120251 from mschwaig/fix-make-disk-image-for-efi-2Lassulus2021-04-26
|\ \ \ | |/ / |/| | make-disk-image: fix broken EFI image builds
| * | make-disk-image: fix broken EFI image buildsMartin Schwaighofer2021-04-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Work around missing /dev files inside runInLinuxVM by creating a symlink before calling nixos-enter. This fixes https://github.com/NixOS/nixpkgs/issues/93381. I ran into this issue when trying to create a VMware image that boots from EFI. Thanks @colemickens for reporting this and @danielfullmer for fixing the same thing in in qemu-vm.nix (37676e77cb5adab936606838d6465faafda54f61) and explaining what the issue was.
* | | nixos/test-driver: use a variety of different Tesseract settings for OCRLuke Granger-Brown2021-04-23
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When performing OCR, some of the Tesseract settings perform better than others on a variety of different workloads, but they mostly take ~negligible incremental time to run compared to the overhead of running the ImageMagick filters. After this commit, we try using all three of the current Tesseract models (classic, LSTM, and classic+LSTM) to generate output text. This fixes chromium-90's tests at release-20.09, and should make cases where you're looking for *specific* text better, with the tradeoff of running Tesseract multiple times. To make it sensible to cherrypick this into release-20.09, this doesn't change the existing API surface for the test driver. In particular, get_screen_text continues to have the existing behaviour.
* / nixosOptionsDoc: add markdown formattingPacman992021-04-13
|/
* nixos/qemu-flags: add ppc64Ryan Burns2021-03-05
| | | | | * PPC uses ttyAMA0 as its serial device, similar to ARM. * PowerNV is a typical platform for testing and distro development
* squashfs: use -no-hardlinks for reproducible squashfs images (#114454)Arnout Engelen2021-02-28
| | | | | | | | | | | | | | | | | the nix store may contain hardlinks: derivations may output them directly, or users may be using store optimization which automatically hardlinks identical files in the nix store. The presence of these links are intended to be a 'transparent' optimization. However, when creating a squashfs image, the image will be different depending on whether hard links were present on the filesystem, leading to reproducibility problems. By passing '-no-hardlinks' to mksquashfs the files are stored as duplicates in the squashfs image. Since squashfs has support for duplicate files this does not lead to a larger image. For more details see https://github.com/NixOS/nixpkgs/issues/114331
* Merge pull request #67493 from clefru/make-disk-imageJörg Thalheim2021-01-29
|\
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-05-11 08:12:20 +0000