diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README.md | 21 | ||||
| -rw-r--r-- | doc/build-helpers.md | 28 | ||||
| -rw-r--r-- | doc/build-helpers/fetchers.chapter.md (renamed from doc/builders/fetchers.chapter.md) | 33 | ||||
| -rw-r--r-- | doc/build-helpers/images.md (renamed from doc/builders/images.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/appimagetools.section.md (renamed from doc/builders/images/appimagetools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/binarycache.section.md (renamed from doc/builders/images/binarycache.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/dockertools.section.md (renamed from doc/builders/images/dockertools.section.md) | 2 | ||||
| -rw-r--r-- | doc/build-helpers/images/makediskimage.section.md (renamed from doc/builders/images/makediskimage.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/ocitools.section.md (renamed from doc/builders/images/ocitools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/portableservice.section.md (renamed from doc/builders/images/portableservice.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/snaptools.section.md (renamed from doc/builders/images/snaptools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special.md (renamed from doc/builders/special.md) | 5 | ||||
| -rw-r--r-- | doc/build-helpers/special/fhs-environments.section.md (renamed from doc/builders/special/fhs-environments.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special/makesetuphook.section.md (renamed from doc/builders/special/makesetuphook.section.md) | 2 | ||||
| -rw-r--r-- | doc/build-helpers/special/mkshell.section.md (renamed from doc/builders/special/mkshell.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special/vm-tools.section.md (renamed from doc/builders/special/vm-tools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/testers.chapter.md (renamed from doc/builders/testers.chapter.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/trivial-build-helpers.chapter.md (renamed from doc/builders/trivial-builders.chapter.md) | 2 | ||||
| -rw-r--r-- | doc/builders.md | 12 | ||||
| -rw-r--r-- | doc/default.nix | 1 | ||||
| -rw-r--r-- | doc/functions/fileset.section.md | 3 | ||||
| -rw-r--r-- | doc/hooks/autopatchelf.section.md | 2 | ||||
| -rw-r--r-- | doc/hooks/index.md | 1 | ||||
| -rw-r--r-- | doc/hooks/qt-4.section.md | 3 | ||||
| -rw-r--r-- | doc/languages-frameworks/agda.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/beam.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/dart.section.md | 9 | ||||
| -rw-r--r-- | doc/languages-frameworks/emscripten.section.md | 297 | ||||
| -rw-r--r-- | doc/languages-frameworks/go.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/haskell.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/javascript.section.md | 3 | ||||
| -rw-r--r-- | doc/languages-frameworks/lisp.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/lua.section.md | 6 | ||||
| -rw-r--r-- | doc/languages-frameworks/maven.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/php.section.md | 6 | ||||
| -rw-r--r-- | doc/languages-frameworks/python.section.md | 63 | ||||
| -rw-r--r-- | doc/languages-frameworks/ruby.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/rust.section.md | 65 | ||||
| -rw-r--r-- | doc/languages-frameworks/swift.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/texlive.section.md | 58 | ||||
| -rw-r--r-- | doc/manual.md.in | 2 | ||||
| -rw-r--r-- | doc/packages/cataclysm-dda.section.md (renamed from doc/builders/packages/cataclysm-dda.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/citrix.section.md (renamed from doc/builders/packages/citrix.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/darwin-builder.section.md (renamed from doc/builders/special/darwin-builder.section.md) | 30 | ||||
| -rw-r--r-- | doc/packages/dlib.section.md (renamed from doc/builders/packages/dlib.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/eclipse.section.md (renamed from doc/builders/packages/eclipse.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/elm.section.md (renamed from doc/builders/packages/elm.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/emacs.section.md (renamed from doc/builders/packages/emacs.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/etc-files.section.md (renamed from doc/builders/packages/etc-files.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/firefox.section.md (renamed from doc/builders/packages/firefox.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/fish.section.md (renamed from doc/builders/packages/fish.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/fuse.section.md (renamed from doc/builders/packages/fuse.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/ibus.section.md (renamed from doc/builders/packages/ibus.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/index.md (renamed from doc/builders/packages/index.md) | 1 | ||||
| -rw-r--r-- | doc/packages/kakoune.section.md (renamed from doc/builders/packages/kakoune.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/linux.section.md (renamed from doc/builders/packages/linux.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/locales.section.md (renamed from doc/builders/packages/locales.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/nginx.section.md (renamed from doc/builders/packages/nginx.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/opengl.section.md (renamed from doc/builders/packages/opengl.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/shell-helpers.section.md (renamed from doc/builders/packages/shell-helpers.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/steam.section.md (renamed from doc/builders/packages/steam.section.md) | 2 | ||||
| -rw-r--r-- | doc/packages/urxvt.section.md (renamed from doc/builders/packages/urxvt.section.md) | 2 | ||||
| -rw-r--r-- | doc/packages/weechat.section.md (renamed from doc/builders/packages/weechat.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/xorg.section.md (renamed from doc/builders/packages/xorg.section.md) | 0 | ||||
| -rw-r--r-- | doc/stdenv/stdenv.chapter.md | 59 | ||||
| -rw-r--r-- | doc/using/overlays.chapter.md | 4 |
66 files changed, 481 insertions, 269 deletions
diff --git a/doc/README.md b/doc/README.md index 3f9aff1a38a..9dee2d30d73 100644 --- a/doc/README.md +++ b/doc/README.md @@ -114,3 +114,24 @@ pear watermelon : green fruit with red flesh ``` + +## Commit conventions + +- Make sure you read about the [commit conventions](../CONTRIBUTING.md#commit-conventions) common to Nixpkgs as a whole. + +- If creating a commit purely for documentation changes, format the commit message in the following way: + + ``` + doc: (documentation summary) + + (Motivation for change, relevant links, additional information.) + ``` + + Examples: + + * doc: update the kernel config documentation to use `nix-shell` + * doc: add information about `nix-update-script` + + Closes #216321. + +- If the commit contains more than just documentation changes, follow the commit message format relevant for the rest of the changes. diff --git a/doc/build-helpers.md b/doc/build-helpers.md new file mode 100644 index 00000000000..06737e16676 --- /dev/null +++ b/doc/build-helpers.md @@ -0,0 +1,28 @@ +# Build helpers {#part-builders} + +A build helper is a function that produces derivations. + +:::{.warning} +This is not to be confused with the [`builder` argument of the Nix `derivation` primitive](https://nixos.org/manual/nix/unstable/language/derivations.html), which refers to the executable that produces the build result, or [remote builder](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html), which refers to a remote machine that could run such an executable. +::: + +Such a function is usually designed to abstract over a typical workflow for a given programming language or framework. +This allows declaring a build recipe by setting a limited number of options relevant to the particular use case instead of using the `derivation` function directly. + +[`stdenv.mkDerivation`](#part-stdenv) is the most widely used build helper, and serves as a basis for many others. +In addition, it offers various options to customize parts of the builds. + +There is no uniform interface for build helpers. +[Trivial build helpers](#chap-trivial-builders) and [fetchers](#chap-pkgs-fetchers) have various input types for convenience. +[Language- or framework-specific build helpers](#chap-language-support) usually follow the style of `stdenv.mkDerivation`, which accepts an attribute set or a fixed-point function taking an attribute set. + +```{=include=} chapters +build-helpers/fetchers.chapter.md +build-helpers/trivial-build-helpers.chapter.md +build-helpers/testers.chapter.md +build-helpers/special.md +build-helpers/images.md +hooks/index.md +languages-frameworks/index.md +packages/index.md +``` diff --git a/doc/builders/fetchers.chapter.md b/doc/build-helpers/fetchers.chapter.md index ba7b1b19012..7bd1bbd6de0 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/build-helpers/fetchers.chapter.md @@ -1,12 +1,27 @@ # Fetchers {#chap-pkgs-fetchers} Building software with Nix often requires downloading source code and other files from the internet. -`nixpkgs` provides *fetchers* for different protocols and services. Fetchers are functions that simplify downloading files. +To this end, Nixpkgs provides *fetchers*: functions to obtain remote sources via various protocols and services. -## Caveats {#chap-pkgs-fetchers-caveats} +Nixpkgs fetchers differ from built-in fetchers such as [`builtins.fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball): +- A built-in fetcher will download and cache files at evaluation time and produce a [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path). + A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/language/derivations), and files are downloaded at build time. +- Built-in fetchers will invalidate their cache after [`tarball-ttl`](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-tarball-ttl) expires, and will require network activity to check if the cache entry is up to date. + Nixpkgs fetchers only re-download if the specified hash changes or the store object is not otherwise available. +- Built-in fetchers do not use [substituters](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-substituters). + Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently. + +This significantly reduces the time needed to evaluate the entirety of Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org). +For these reasons, built-in fetchers are not allowed in Nixpkgs source code. + +The following table shows an overview of the differences: -Fetchers create [fixed output derivations](https://nixos.org/manual/nix/stable/#fixed-output-drvs) from downloaded files. -Nix can reuse the downloaded files via the hash of the resulting derivation. +| Fetchers | Download | Output | Cache | Re-download when | +|-|-|-|-|-| +| `builtins.fetch*` | evaluation time | store path | `/nix/store`, `~/.cache/nix` | `tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store | +| `pkgs.fetch*` | build time | derivation | `/nix/store`, substituters | output store object not available | + +## Caveats {#chap-pkgs-fetchers-caveats} The fact that the hash belongs to the Nix derivation output and not the file itself can lead to confusion. For example, consider the following fetcher: @@ -243,21 +258,21 @@ or *** ``` -## `fetchFromBittorrent` {#fetchfrombittorrent} +## `fetchtorrent` {#fetchtorrent} -`fetchFromBittorrent` expects two arguments. `url` which can either be a Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. It can also take a `config` argument which will craft a `settings.json` configuration file and give it to `transmission`, the underlying program that is performing the fetch. The available config options for `transmission` can be found [here](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options) +`fetchtorrent` expects two arguments. `url` which can either be a Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. It can also take a `config` argument which will craft a `settings.json` configuration file and give it to `transmission`, the underlying program that is performing the fetch. The available config options for `transmission` can be found [here](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options) ``` -{ fetchFromBittorrent }: +{ fetchtorrent }: -fetchFromBittorrent { +fetchtorrent { config = { peer-limit-global = 100; }; url = "magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c"; sha256 = ""; } ``` -### Parameters {#fetchfrombittorrent-parameters} +### Parameters {#fetchtorrent-parameters} - `url`: Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. diff --git a/doc/builders/images.md b/doc/build-helpers/images.md index 5596784bfa4..5596784bfa4 100644 --- a/doc/builders/images.md +++ b/doc/build-helpers/images.md diff --git a/doc/builders/images/appimagetools.section.md b/doc/build-helpers/images/appimagetools.section.md index 0c72315a26e..0c72315a26e 100644 --- a/doc/builders/images/appimagetools.section.md +++ b/doc/build-helpers/images/appimagetools.section.md diff --git a/doc/builders/images/binarycache.section.md b/doc/build-helpers/images/binarycache.section.md index 62e47dad7c6..62e47dad7c6 100644 --- a/doc/builders/images/binarycache.section.md +++ b/doc/build-helpers/images/binarycache.section.md diff --git a/doc/builders/images/dockertools.section.md b/doc/build-helpers/images/dockertools.section.md index 3ac4f224b5d..42d6e297f52 100644 --- a/doc/builders/images/dockertools.section.md +++ b/doc/build-helpers/images/dockertools.section.md @@ -275,7 +275,7 @@ pullImage { `nix-prefetch-docker` command can be used to get required image parameters: ```ShellSession -$ nix run nixpkgs.nix-prefetch-docker -c nix-prefetch-docker --image-name mysql --image-tag 5 +$ nix run nixpkgs#nix-prefetch-docker -- --image-name mysql --image-tag 5 ``` Since a given `imageName` may transparently refer to a manifest list of images which support multiple architectures and/or operating systems, you can supply the `--os` and `--arch` arguments to specify exactly which image you want. By default it will match the OS and architecture of the host the command is run on. diff --git a/doc/builders/images/makediskimage.section.md b/doc/build-helpers/images/makediskimage.section.md index e50479c4e83..e50479c4e83 100644 --- a/doc/builders/images/makediskimage.section.md +++ b/doc/build-helpers/images/makediskimage.section.md diff --git a/doc/builders/images/ocitools.section.md b/doc/build-helpers/images/ocitools.section.md index c35f65bce00..c35f65bce00 100644 --- a/doc/builders/images/ocitools.section.md +++ b/doc/build-helpers/images/ocitools.section.md diff --git a/doc/builders/images/portableservice.section.md b/doc/build-helpers/images/portableservice.section.md index 5400928b158..5400928b158 100644 --- a/doc/builders/images/portableservice.section.md +++ b/doc/build-helpers/images/portableservice.section.md diff --git a/doc/builders/images/snaptools.section.md b/doc/build-helpers/images/snaptools.section.md index 259fa1b0618..259fa1b0618 100644 --- a/doc/builders/images/snaptools.section.md +++ b/doc/build-helpers/images/snaptools.section.md diff --git a/doc/builders/special.md b/doc/build-helpers/special.md index 6d07fa87f3f..f88648207fd 100644 --- a/doc/builders/special.md +++ b/doc/build-helpers/special.md @@ -1,11 +1,10 @@ -# Special builders {#chap-special} +# Special build helpers {#chap-special} -This chapter describes several special builders. +This chapter describes several special build helpers. ```{=include=} sections special/fhs-environments.section.md special/makesetuphook.section.md special/mkshell.section.md -special/darwin-builder.section.md special/vm-tools.section.md ``` diff --git a/doc/builders/special/fhs-environments.section.md b/doc/build-helpers/special/fhs-environments.section.md index 8145fbd730f..8145fbd730f 100644 --- a/doc/builders/special/fhs-environments.section.md +++ b/doc/build-helpers/special/fhs-environments.section.md diff --git a/doc/builders/special/makesetuphook.section.md b/doc/build-helpers/special/makesetuphook.section.md index eb042412137..e83164b7eb7 100644 --- a/doc/builders/special/makesetuphook.section.md +++ b/doc/build-helpers/special/makesetuphook.section.md @@ -1,6 +1,6 @@ # pkgs.makeSetupHook {#sec-pkgs.makeSetupHook} -`pkgs.makeSetupHook` is a builder that produces hooks that go in to `nativeBuildInputs` +`pkgs.makeSetupHook` is a build helper that produces hooks that go in to `nativeBuildInputs` ## Usage {#sec-pkgs.makeSetupHook-usage} diff --git a/doc/builders/special/mkshell.section.md b/doc/build-helpers/special/mkshell.section.md index 96d43535955..96d43535955 100644 --- a/doc/builders/special/mkshell.section.md +++ b/doc/build-helpers/special/mkshell.section.md diff --git a/doc/builders/special/vm-tools.section.md b/doc/build-helpers/special/vm-tools.section.md index 8feab04902d..8feab04902d 100644 --- a/doc/builders/special/vm-tools.section.md +++ b/doc/build-helpers/special/vm-tools.section.md diff --git a/doc/builders/testers.chapter.md b/doc/build-helpers/testers.chapter.md index b2a581c3dd8..b2a581c3dd8 100644 --- a/doc/builders/testers.chapter.md +++ b/doc/build-helpers/testers.chapter.md diff --git a/doc/builders/trivial-builders.chapter.md b/doc/build-helpers/trivial-build-helpers.chapter.md index 2cb1f2debcb..a0cda86a660 100644 --- a/doc/builders/trivial-builders.chapter.md +++ b/doc/build-helpers/trivial-build-helpers.chapter.md @@ -1,4 +1,4 @@ -# Trivial builders {#chap-trivial-builders} +# Trivial build helpers {#chap-trivial-builders} Nixpkgs provides a couple of functions that help with building derivations. The most important one, `stdenv.mkDerivation`, has already been documented above. The following functions wrap `stdenv.mkDerivation`, making it easier to use in certain cases. diff --git a/doc/builders.md b/doc/builders.md deleted file mode 100644 index 2e959422405..00000000000 --- a/doc/builders.md +++ /dev/null @@ -1,12 +0,0 @@ -# Builders {#part-builders} - -```{=include=} chapters -builders/fetchers.chapter.md -builders/trivial-builders.chapter.md -builders/testers.chapter.md -builders/special.md -builders/images.md -hooks/index.md -languages-frameworks/index.md -builders/packages/index.md -``` diff --git a/doc/default.nix b/doc/default.nix index 18e12c1a8ac..61bbd2ba8dc 100644 --- a/doc/default.nix +++ b/doc/default.nix @@ -23,6 +23,7 @@ let { name = "sources"; description = "source filtering functions"; } { name = "cli"; description = "command-line serialization functions"; } { name = "gvariant"; description = "GVariant formatted string serialization functions"; } + { name = "customisation"; description = "Functions to customise (derivation-related) functions, derivatons, or attribute sets"; } ]; }; diff --git a/doc/functions/fileset.section.md b/doc/functions/fileset.section.md index 08b9ba9eaed..c42337feaba 100644 --- a/doc/functions/fileset.section.md +++ b/doc/functions/fileset.section.md @@ -6,11 +6,8 @@ The [`lib.fileset`](#sec-functions-library-fileset) library allows you to work w A file set is a mathematical set of local files that can be added to the Nix store for use in Nix derivations. File sets are easy and safe to use, providing obvious and composable semantics with good error messages to prevent mistakes. -These sections apply to the entire library. See the [function reference](#sec-functions-library-fileset) for function-specific documentation. -The file set library is currently somewhat limited but is being expanded to include more functions over time. - ## Implicit coercion from paths to file sets {#sec-fileset-path-coercion} All functions accepting file sets as arguments can also accept [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) as arguments. diff --git a/doc/hooks/autopatchelf.section.md b/doc/hooks/autopatchelf.section.md index 008a90d4614..995204b9021 100644 --- a/doc/hooks/autopatchelf.section.md +++ b/doc/hooks/autopatchelf.section.md @@ -6,6 +6,6 @@ You can also specify a `runtimeDependencies` variable which lists dependencies t In certain situations you may want to run the main command (`autoPatchelf`) of the setup hook on a file or a set of directories instead of unconditionally patching all outputs. This can be done by setting the `dontAutoPatchelf` environment variable to a non-empty value. -By default `autoPatchelf` will fail as soon as any ELF file requires a dependency which cannot be resolved via the given build inputs. In some situations you might prefer to just leave missing dependencies unpatched and continue to patch the rest. This can be achieved by setting the `autoPatchelfIgnoreMissingDeps` environment variable to a non-empty value. `autoPatchelfIgnoreMissingDeps` can be set to a list like `autoPatchelfIgnoreMissingDeps = [ "libcuda.so.1" "libcudart.so.1" ];` or to simply `[ "*" ]` to ignore all missing dependencies. +By default `autoPatchelf` will fail as soon as any ELF file requires a dependency which cannot be resolved via the given build inputs. In some situations you might prefer to just leave missing dependencies unpatched and continue to patch the rest. This can be achieved by setting the `autoPatchelfIgnoreMissingDeps` environment variable to a non-empty value. `autoPatchelfIgnoreMissingDeps` can be set to a list like `autoPatchelfIgnoreMissingDeps = [ "libcuda.so.1" "libcudart.so.1" ];` or to `[ "*" ]` to ignore all missing dependencies. The `autoPatchelf` command also recognizes a `--no-recurse` command line flag, which prevents it from recursing into subdirectories. diff --git a/doc/hooks/index.md b/doc/hooks/index.md index 363d627e525..1534ef85ccb 100644 --- a/doc/hooks/index.md +++ b/doc/hooks/index.md @@ -25,7 +25,6 @@ perl.section.md pkg-config.section.md postgresql-test-hook.section.md python.section.md -qt-4.section.md scons.section.md tetex-tex-live.section.md unzip.section.md diff --git a/doc/hooks/qt-4.section.md b/doc/hooks/qt-4.section.md deleted file mode 100644 index 4b704df4959..00000000000 --- a/doc/hooks/qt-4.section.md +++ /dev/null @@ -1,3 +0,0 @@ -# Qt 4 {#qt-4} - -Sets the `QTDIR` environment variable to Qt’s path. diff --git a/doc/languages-frameworks/agda.section.md b/doc/languages-frameworks/agda.section.md index ff3d70ef0c6..cb1f12eec23 100644 --- a/doc/languages-frameworks/agda.section.md +++ b/doc/languages-frameworks/agda.section.md @@ -146,7 +146,7 @@ agdaPackages.mkDerivation { ### Building Agda packages {#building-agda-packages} -The default build phase for `agdaPackages.mkDerivation` simply runs `agda` on the `Everything.agda` file. +The default build phase for `agdaPackages.mkDerivation` runs `agda` on the `Everything.agda` file. If something else is needed to build the package (e.g. `make`) then the `buildPhase` should be overridden. Additionally, a `preBuild` or `configurePhase` can be used if there are steps that need to be done prior to checking the `Everything.agda` file. `agda` and the Agda libraries contained in `buildInputs` are made available during the build phase. @@ -250,7 +250,7 @@ Usually, the maintainers will answer within a week or two with a new release. Bumping the version of that reverse dependency should be a further commit on your PR. In the rare case that a new release is not to be expected within an acceptable time, -simply mark the broken package as broken by setting `meta.broken = true;`. +mark the broken package as broken by setting `meta.broken = true;`. This will exclude it from the build test. It can be added later when it is fixed, and does not hinder the advancement of the whole package set in the meantime. diff --git a/doc/languages-frameworks/beam.section.md b/doc/languages-frameworks/beam.section.md index 2cb4863fc53..1e83d4b93c7 100644 --- a/doc/languages-frameworks/beam.section.md +++ b/doc/languages-frameworks/beam.section.md @@ -44,7 +44,7 @@ There is also a `buildMix` helper, whose behavior is closer to that of `buildErl ## How to Install BEAM Packages {#how-to-install-beam-packages} -BEAM builders are not registered at the top level, simply because they are not relevant to the vast majority of Nix users. +BEAM builders are not registered at the top level, because they are not relevant to the vast majority of Nix users. To use any of those builders into your environment, refer to them by their attribute path under `beamPackages`, e.g. `beamPackages.rebar3`: ::: {.example #ex-beam-ephemeral-shell} diff --git a/doc/languages-frameworks/dart.section.md b/doc/languages-frameworks/dart.section.md index b00327b78eb..9da43714a16 100644 --- a/doc/languages-frameworks/dart.section.md +++ b/doc/languages-frameworks/dart.section.md @@ -8,10 +8,12 @@ It fetches its Dart dependencies automatically through `fetchDartDeps`, and (thr If you are packaging a Flutter desktop application, use [`buildFlutterApplication`](#ssec-dart-flutter) instead. -`vendorHash`: is the hash of the output of the dependency fetcher derivation. To obtain it, simply set it to `lib.fakeHash` (or omit it) and run the build ([more details here](#sec-source-hashes)). +`vendorHash`: is the hash of the output of the dependency fetcher derivation. To obtain it, set it to `lib.fakeHash` (or omit it) and run the build ([more details here](#sec-source-hashes)). If the upstream source is missing a `pubspec.lock` file, you'll have to vendor one and specify it using `pubspecLockFile`. If it is needed, one will be generated for you and printed when attempting to build the derivation. +The `depsListFile` must always be provided when packaging in Nixpkgs. It will be generated and printed if the derivation is attempted to be built without one. Alternatively, `autoDepsList` may be set to `true` only when outside of Nixpkgs, as it relies on import-from-derivation. + The `dart` commands run can be overridden through `pubGetScript` and `dartCompileCommand`, you can also add flags using `dartCompileFlags` or `dartJitFlags`. Dart supports multiple [outputs types](https://dart.dev/tools/dart-compile#types-of-output), you can choose between them using `dartOutputType` (defaults to `exe`). If you want to override the binaries path or the source path they come from, you can use `dartEntryPoints`. Outputs that require a runtime will automatically be wrapped with the relevant runtime (`dartaotruntime` for `aot-snapshot`, `dart run` for `jit-snapshot` and `kernel`, `node` for `js`), this can be overridden through `dartRuntimeCommand`. @@ -31,6 +33,7 @@ buildDartApplication rec { }; pubspecLockFile = ./pubspec.lock; + depsListFile = ./deps.json; vendorHash = "sha256-Atm7zfnDambN/BmmUf4BG0yUz/y6xWzf0reDw3Ad41s="; } ``` @@ -39,9 +42,7 @@ buildDartApplication rec { The function `buildFlutterApplication` builds Flutter applications. -The deps.json file must always be provided when packaging in Nixpkgs. It will be generated and printed if the derivation is attempted to be built without one. Alternatively, `autoDepsList` may be set to `true` when outside of Nixpkgs, as it relies on import-from-derivation. - -A `pubspec.lock` file must be available. See the [Dart documentation](#ssec-dart-applications) for more details. +See the [Dart documentation](#ssec-dart-applications) for more details on required files and arguments. ```nix { flutter, fetchFromGitHub }: diff --git a/doc/languages-frameworks/emscripten.section.md b/doc/languages-frameworks/emscripten.section.md index 5f93dd5ff31..20d358f2e9e 100644 --- a/doc/languages-frameworks/emscripten.section.md +++ b/doc/languages-frameworks/emscripten.section.md @@ -2,168 +2,159 @@ [Emscripten](https://github.com/kripken/emscripten): An LLVM-to-JavaScript Compiler -This section of the manual covers how to use `emscripten` in nixpkgs. +If you want to work with `emcc`, `emconfigure` and `emmake` as you are used to from Ubuntu and similar distributions, -Minimal requirements: - -* nix -* nixpkgs - -Modes of use of `emscripten`: - -* **Imperative usage** (on the command line): - - If you want to work with `emcc`, `emconfigure` and `emmake` as you are used to from Ubuntu and similar distributions you can use these commands: - - * `nix-env -f "<nixpkgs>" -iA emscripten` - * `nix-shell -p emscripten` - -* **Declarative usage**: - - This mode is far more power full since this makes use of `nix` for dependency management of emscripten libraries and targets by using the `mkDerivation` which is implemented by `pkgs.emscriptenStdenv` and `pkgs.buildEmscriptenPackage`. The source for the packages is in `pkgs/top-level/emscripten-packages.nix` and the abstraction behind it in `pkgs/development/em-modules/generic/default.nix`. From the root of the nixpkgs repository: - * build and install all packages: - * `nix-env -iA emscriptenPackages` - - * dev-shell for zlib implementation hacking: - * `nix-shell -A emscriptenPackages.zlib` - -## Imperative usage {#imperative-usage} +```console +nix-shell -p emscripten +``` A few things to note: * `export EMCC_DEBUG=2` is nice for debugging -* `~/.emscripten`, the build artifact cache sometimes creates issues and needs to be removed from time to time +* The build artifact cache in `~/.emscripten` sometimes creates issues and needs to be removed from time to time -## Declarative usage {#declarative-usage} +## Examples {#declarative-usage} Let's see two different examples from `pkgs/top-level/emscripten-packages.nix`: * `pkgs.zlib.override` * `pkgs.buildEmscriptenPackage` -Both are interesting concepts. - -A special requirement of the `pkgs.buildEmscriptenPackage` is the `doCheck = true` is a default meaning that each emscriptenPackage requires a `checkPhase` implemented. - -* Use `export EMCC_DEBUG=2` from within a emscriptenPackage's `phase` to get more detailed debug output what is going wrong. -* ~/.emscripten cache is requiring us to set `HOME=$TMPDIR` in individual phases. This makes compilation slower but also makes it more deterministic. - -### Usage 1: pkgs.zlib.override {#usage-1-pkgs.zlib.override} - -This example uses `zlib` from nixpkgs but instead of compiling **C** to **ELF** it compiles **C** to **JS** since we were using `pkgs.zlib.override` and changed stdenv to `pkgs.emscriptenStdenv`. A few adaptions and hacks were set in place to make it working. One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. However, this can also be the downside... - -See the `zlib` example: - - zlib = (pkgs.zlib.override { - stdenv = pkgs.emscriptenStdenv; - }).overrideAttrs - (old: rec { - buildInputs = old.buildInputs ++ [ pkg-config ]; - # we need to reset this setting! - env = (old.env or { }) // { NIX_CFLAGS_COMPILE = ""; }; - configurePhase = '' - # FIXME: Some tests require writing at $HOME - HOME=$TMPDIR - runHook preConfigure - - #export EMCC_DEBUG=2 - emconfigure ./configure --prefix=$out --shared - - runHook postConfigure - ''; - dontStrip = true; - outputs = [ "out" ]; - buildPhase = '' - emmake make - ''; - installPhase = '' - emmake make install - ''; - checkPhase = '' - echo "================= testing zlib using node =================" - - echo "Compiling a custom test" - set -x - emcc -O2 -s EMULATE_FUNCTION_POINTER_CASTS=1 test/example.c -DZ_SOLO \ - libz.so.${old.version} -I . -o example.js - - echo "Using node to execute the test" - ${pkgs.nodejs}/bin/node ./example.js - - set +x - if [ $? -ne 0 ]; then - echo "test failed for some reason" - exit 1; - else - echo "it seems to work! very good." - fi - echo "================= /testing zlib using node =================" - ''; - - postPatch = pkgs.lib.optionalString pkgs.stdenv.isDarwin '' - substituteInPlace configure \ - --replace '/usr/bin/libtool' 'ar' \ - --replace 'AR="libtool"' 'AR="ar"' \ - --replace 'ARFLAGS="-o"' 'ARFLAGS="-r"' - ''; - }); - -### Usage 2: pkgs.buildEmscriptenPackage {#usage-2-pkgs.buildemscriptenpackage} - -This `xmlmirror` example features a emscriptenPackage which is defined completely from this context and no `pkgs.zlib.override` is used. - - xmlmirror = pkgs.buildEmscriptenPackage rec { - name = "xmlmirror"; - - buildInputs = [ pkg-config autoconf automake libtool gnumake libxml2 nodejs openjdk json_c ]; - nativeBuildInputs = [ pkg-config zlib ]; - - src = pkgs.fetchgit { - url = "https://gitlab.com/odfplugfest/xmlmirror.git"; - rev = "4fd7e86f7c9526b8f4c1733e5c8b45175860a8fd"; - hash = "sha256-i+QgY+5PYVg5pwhzcDnkfXAznBg3e8sWH2jZtixuWsk="; - }; - - configurePhase = '' - rm -f fastXmlLint.js* - # a fix for ERROR:root:For asm.js, TOTAL_MEMORY must be a multiple of 16MB, was 234217728 - # https://gitlab.com/odfplugfest/xmlmirror/issues/8 - sed -e "s/TOTAL_MEMORY=234217728/TOTAL_MEMORY=268435456/g" -i Makefile.emEnv - # https://github.com/kripken/emscripten/issues/6344 - # https://gitlab.com/odfplugfest/xmlmirror/issues/9 - sed -e "s/\$(JSONC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(LIBXML20_LDFLAGS)/\$(JSONC_LDFLAGS) \$(LIBXML20_LDFLAGS) \$(ZLIB_LDFLAGS) /g" -i Makefile.emEnv - # https://gitlab.com/odfplugfest/xmlmirror/issues/11 - sed -e "s/-o fastXmlLint.js/-s EXTRA_EXPORTED_RUNTIME_METHODS='[\"ccall\", \"cwrap\"]' -o fastXmlLint.js/g" -i Makefile.emEnv - ''; - - buildPhase = '' - HOME=$TMPDIR - make -f Makefile.emEnv - ''; - - outputs = [ "out" "doc" ]; - - installPhase = '' - mkdir -p $out/share - mkdir -p $doc/share/${name} - - cp Demo* $out/share - cp -R codemirror-5.12 $out/share - cp fastXmlLint.js* $out/share - cp *.xsd $out/share - cp *.js $out/share - cp *.xhtml $out/share - cp *.html $out/share - cp *.json $out/share - cp *.rng $out/share - cp README.md $doc/share/${name} - ''; - checkPhase = '' - - ''; - }; - -### Declarative debugging {#declarative-debugging} +A special requirement of the `pkgs.buildEmscriptenPackage` is the `doCheck = true`. +This means each Emscripten package requires that a [`checkPhase`](#ssec-check-phase) is implemented. + +* Use `export EMCC_DEBUG=2` from within a phase to get more detailed debug output what is going wrong. +* The cache at `~/.emscripten` requires to set `HOME=$TMPDIR` in individual phases. + This makes compilation slower but also more deterministic. + +::: {.example #usage-1-pkgs.zlib.override} + +# Using `pkgs.zlib.override {}` + +This example uses `zlib` from Nixpkgs, but instead of compiling **C** to **ELF** it compiles **C** to **JavaScript** since we were using `pkgs.zlib.override` and changed `stdenv` to `pkgs.emscriptenStdenv`. + +A few adaptions and hacks were put in place to make it work. +One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. + + +```nix +(pkgs.zlib.override { + stdenv = pkgs.emscriptenStdenv; +}).overrideAttrs +(old: rec { + buildInputs = old.buildInputs ++ [ pkg-config ]; + # we need to reset this setting! + env = (old.env or { }) // { NIX_CFLAGS_COMPILE = ""; }; + configurePhase = '' + # FIXME: Some tests require writing at $HOME + HOME=$TMPDIR + runHook preConfigure + + #export EMCC_DEBUG=2 + emconfigure ./configure --prefix=$out --shared + + runHook postConfigure + ''; + dontStrip = true; + outputs = [ "out" ]; + buildPhase = '' + emmake make + ''; + installPhase = '' + emmake make install + ''; + checkPhase = '' + echo "================= testing zlib using node =================" + + echo "Compiling a custom test" + set -x + emcc -O2 -s EMULATE_FUNCTION_POINTER_CASTS=1 test/example.c -DZ_SOLO \ + libz.so.${old.version} -I . -o example.js + + echo "Using node to execute the test" + ${pkgs.nodejs}/bin/node ./example.js + + set +x + if [ $? -ne 0 ]; then + echo "test failed for some reason" + exit 1; + else + echo "it seems to work! very good." + fi + echo "================= /testing zlib using node =================" + ''; + + postPatch = pkgs.lib.optionalString pkgs.stdenv.isDarwin '' + substituteInPlace configure \ + --replace '/usr/bin/libtool' 'ar' \ + --replace 'AR="libtool"' 'AR="ar"' \ + --replace 'ARFLAGS="-o"' 'ARFLAGS="-r"' + ''; +}) +``` + +:::{.example #usage-2-pkgs.buildemscriptenpackage} + +# Using `pkgs.buildEmscriptenPackage {}` + +This `xmlmirror` example features an Emscripten package that is defined completely from this context and no `pkgs.zlib.override` is used. + +```nix +pkgs.buildEmscriptenPackage rec { + name = "xmlmirror"; + + buildInputs = [ pkg-config autoconf automake libtool gnumake libxml2 nodejs openjdk json_c ]; + nativeBuildInputs = [ pkg-config zlib ]; + + src = pkgs.fetchgit { + url = "https://gitlab.com/odfplugfest/xmlmirror.git"; + rev = "4fd7e86f7c9526b8f4c1733e5c8b45175860a8fd"; + hash = "sha256-i+QgY+5PYVg5pwhzcDnkfXAznBg3e8sWH2jZtixuWsk="; + }; + + configurePhase = '' + rm -f fastXmlLint.js* + # a fix for ERROR:root:For asm.js, TOTAL_MEMORY must be a multiple of 16MB, was 234217728 + # https://gitlab.com/odfplugfest/xmlmirror/issues/8 + sed -e "s/TOTAL_MEMORY=234217728/TOTAL_MEMORY=268435456/g" -i Makefile.emEnv + # https://github.com/kripken/emscripten/issues/6344 + # https://gitlab.com/odfplugfest/xmlmirror/issues/9 + sed -e "s/\$(JSONC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(LIBXML20_LDFLAGS)/\$(JSONC_LDFLAGS) \$(LIBXML20_LDFLAGS) \$(ZLIB_LDFLAGS) /g" -i Makefile.emEnv + # https://gitlab.com/odfplugfest/xmlmirror/issues/11 + sed -e "s/-o fastXmlLint.js/-s EXTRA_EXPORTED_RUNTIME_METHODS='[\"ccall\", \"cwrap\"]' -o fastXmlLint.js/g" -i Makefile.emEnv + ''; + + buildPhase = '' + HOME=$TMPDIR + make -f Makefile.emEnv + ''; + + outputs = [ "out" "doc" ]; + + installPhase = '' + mkdir -p $out/share + mkdir -p $doc/share/${name} + + cp Demo* $out/share + cp -R codemirror-5.12 $out/share + cp fastXmlLint.js* $out/share + cp *.xsd $out/share + cp *.js $out/share + cp *.xhtml $out/share + cp *.html $out/share + cp *.json $out/share + cp *.rng $out/share + cp README.md $doc/share/${name} + ''; + checkPhase = '' + + ''; +} +``` + +::: + +## Debugging {#declarative-debugging} Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from there you can go trough the individual steps. This makes it easy to build a good `unit test` or list the files of the project. @@ -174,9 +165,3 @@ Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from 5. `configurePhase` 6. `buildPhase` 7. ... happy hacking... - -## Summary {#summary} - -Using this toolchain makes it easy to leverage `nix` from NixOS, MacOSX or even Windows (WSL+ubuntu+nix). This toolchain is reproducible, behaves like the rest of the packages from nixpkgs and contains a set of well working examples to learn and adapt from. - -If in trouble, ask the maintainers. diff --git a/doc/languages-frameworks/go.section.md b/doc/languages-frameworks/go.section.md index 7fd38a7d21c..884ebcebf7f 100644 --- a/doc/languages-frameworks/go.section.md +++ b/doc/languages-frameworks/go.section.md @@ -18,9 +18,9 @@ In the following is an example expression using `buildGoModule`, the following a To avoid updating this field when dependencies change, run `go mod vendor` in your source repo and set `vendorHash = null;` - To obtain the actual hash, set `vendorHash = lib.fakeSha256;` and run the build ([more details here](#sec-source-hashes)). + To obtain the actual hash, set `vendorHash = lib.fakeHash;` and run the build ([more details here](#sec-source-hashes)). - `proxyVendor`: Fetches (go mod download) and proxies the vendor directory. This is useful if your code depends on c code and go mod tidy does not include the needed sources to build or if any dependency has case-insensitive conflicts which will produce platform-dependent `vendorHash` checksums. -- `modPostBuild`: Shell commands to run after the build of the goModules executes `go mod vendor`, and before calculating fixed output derivation's `vendorHash` (or `vendorSha256`). Note that if you change this attribute, you need to update `vendorHash` (or `vendorSha256`) attribute. +- `modPostBuild`: Shell commands to run after the build of the goModules executes `go mod vendor`, and before calculating fixed output derivation's `vendorHash`. Note that if you change this attribute, you need to update `vendorHash` attribute. ```nix pet = buildGoModule rec { diff --git a/doc/languages-frameworks/haskell.section.md b/doc/languages-frameworks/haskell.section.md index 6b9ce32d173..b0b5f5c3bb2 100644 --- a/doc/languages-frameworks/haskell.section.md +++ b/doc/languages-frameworks/haskell.section.md @@ -177,7 +177,7 @@ exactly one version. Those versions need to satisfy all the version constraints given in the `.cabal` file of your package and all its dependencies. The [Haskell builder in nixpkgs](#haskell-mkderivation) does no such thing. -It will simply take as input packages with names off the desired dependencies +It will take as input packages with names off the desired dependencies and just check whether they fulfill the version bounds and fail if they don’t (by default, see `jailbreak` to circumvent this). @@ -780,7 +780,7 @@ there instead. The top level `pkgs.haskell-language-server` attribute is just a convenience wrapper to make it possible to install HLS for multiple GHC versions at the same time. If you know, that you only use one GHC version, e.g., in a project -specific `nix-shell` you can simply use +specific `nix-shell` you can use `pkgs.haskellPackages.haskell-language-server` or `pkgs.haskell.packages.*.haskell-language-server` from the package set you use. diff --git a/doc/languages-frameworks/javascript.section.md b/doc/languages-frameworks/javascript.section.md index f35fd83cc59..0bb9959a510 100644 --- a/doc/languages-frameworks/javascript.section.md +++ b/doc/languages-frameworks/javascript.section.md @@ -13,7 +13,7 @@ If you find you are lacking inspiration for packing javascript applications, the ### Github {#javascript-finding-examples-github} - Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code> -- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+filename%3Aflake.nix&type=code> +- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+path%3A**%2Fflake.nix&type=code> ### Gitlab {#javascript-finding-examples-gitlab} @@ -209,6 +209,7 @@ In the default `installPhase` set by `buildNpmPackage`, it uses `npm pack --json * `npmPackFlags`: Flags to pass to `npm pack`. * `npmPruneFlags`: Flags to pass to `npm prune`. Defaults to the value of `npmInstallFlags`. * `makeWrapperArgs`: Flags to pass to `makeWrapper`, added to executable calling the generated `.js` with `node` as an interpreter. These scripts are defined in `package.json`. +* `nodejs`: The `nodejs` package to build against, using the corresponding `npm` shipped with that version of `node`. Defaults to `pkgs.nodejs`. #### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps} diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index 8712c341206..09193093b08 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -66,7 +66,7 @@ buildPhase = '' To save some work of writing Nix expressions, there is a script that imports all the packages distributed by Quicklisp into `imported.nix`. This works by parsing its `releases.txt` and `systems.txt` files, which are published every couple of -months on [quicklisp.org](http://beta.quicklisp.org/dist/quicklisp.txt). +months on [quicklisp.org](https://beta.quicklisp.org/dist/quicklisp.txt). The import process is implemented in the `import` directory as Common Lisp code in the `org.lispbuilds.nix` ASDF system. To run the script, one can @@ -268,7 +268,7 @@ getting an environment variable for `ext:getenv`. This will load the ### Loading systems {#lisp-loading-systems} -There, you can simply use `asdf:load-system`. This works by setting the right +There, you can use `asdf:load-system`. This works by setting the right values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment variables, so that systems are found in the Nix store and pre-compiled FASLs are loaded. diff --git a/doc/languages-frameworks/lua.section.md b/doc/languages-frameworks/lua.section.md index c5049326a77..310ea88a86d 100644 --- a/doc/languages-frameworks/lua.section.md +++ b/doc/languages-frameworks/lua.section.md @@ -134,11 +134,11 @@ The site proposes two types of packages, the `rockspec` and the `src.rock` Luarocks-based packages are generated in [pkgs/development/lua-modules/generated-packages.nix](https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/lua-modules/generated-packages.nix) from the whitelist maintainers/scripts/luarocks-packages.csv and updated by running -the script -[maintainers/scripts/update-luarocks-packages](https://github.com/NixOS/nixpkgs/tree/master/maintainers/scripts/update-luarocks-packages): +the package `luarocks-packages-updater`: ```sh -./maintainers/scripts/update-luarocks-packages update + +nix-shell -p luarocks-packages-updater --run luarocks-packages-updater ``` [luarocks2nix](https://github.com/nix-community/luarocks) is a tool capable of generating nix derivations from both rockspec and src.rock (and favors the src.rock). diff --git a/doc/languages-frameworks/maven.section.md b/doc/languages-frameworks/maven.section.md index 7e287a097c7..b86733a7589 100644 --- a/doc/languages-frameworks/maven.section.md +++ b/doc/languages-frameworks/maven.section.md @@ -53,7 +53,7 @@ After setting `maven.buildMavenPackage`, we then do standard Java `.jar` install Maven defines default versions for its core plugins, e.g. `maven-compiler-plugin`. If your project does not override these versions, an upgrade of Maven will change the version of the used plugins, and therefore the derivation and hash. -When `maven` is upgraded, `mvnHash` for the derivation must be updated as well: otherwise, the project will simply be built on the derivation of old plugins, and fail because the requested plugins are missing. +When `maven` is upgraded, `mvnHash` for the derivation must be updated as well: otherwise, the project will be built on the derivation of old plugins, and fail because the requested plugins are missing. This clearly prevents automatic upgrades of Maven: a manual effort must be made throughout nixpkgs by any maintainer wishing to push the upgrades. diff --git a/doc/languages-frameworks/php.section.md b/doc/languages-frameworks/php.section.md index 377e3947b2a..154d8174f9a 100644 --- a/doc/languages-frameworks/php.section.md +++ b/doc/languages-frameworks/php.section.md @@ -58,7 +58,7 @@ php.withExtensions ({ enabled, all }: ++ [ all.imagick ]) ``` -To build your list of extensions from the ground up, you can simply +To build your list of extensions from the ground up, you can ignore `enabled`: ```nix @@ -140,7 +140,7 @@ Example of building `composer` with additional extensions: ### Overriding PHP packages {#ssec-php-user-guide-overriding-packages} `php-packages.nix` form a scope, allowing us to override the packages defined -within. For example, to apply a patch to a `mysqlnd` extension, you can simply +within. For example, to apply a patch to a `mysqlnd` extension, you can pass an overlay-style function to `php`’s `packageOverrides` argument: ```nix @@ -191,7 +191,7 @@ using the `bin` attribute in `composer.json`, these binaries will be automatically linked and made accessible in the derivation. In this context, "binaries" refer to PHP scripts that are intended to be executable. -To use the helper effectively, simply add the `vendorHash` attribute, which +To use the helper effectively, add the `vendorHash` attribute, which enables the wrapper to handle the heavy lifting. Internally, the helper operates in three stages: diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index 40236d141d3..19d4496eef5 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -9,9 +9,10 @@ | python27 | python2, python | CPython 2.7 | | python38 | | CPython 3.8 | | python39 | | CPython 3.9 | -| python310 | python3 | CPython 3.10 | -| python311 | | CPython 3.11 | +| python310 | | CPython 3.10 | +| python311 | python3 | CPython 3.11 | | python312 | | CPython 3.12 | +| python313 | | CPython 3.13 | | pypy27 | pypy2, pypy | PyPy2.7 | | pypy39 | pypy3 | PyPy 3.9 | @@ -63,12 +64,14 @@ sets are * `pkgs.python39Packages` * `pkgs.python310Packages` * `pkgs.python311Packages` +* `pkgs.python312Packages` +* `pkgs.python313Packages` * `pkgs.pypyPackages` and the aliases * `pkgs.python2Packages` pointing to `pkgs.python27Packages` -* `pkgs.python3Packages` pointing to `pkgs.python310Packages` +* `pkgs.python3Packages` pointing to `pkgs.python311Packages` * `pkgs.pythonPackages` pointing to `pkgs.python2Packages` #### `buildPythonPackage` function {#buildpythonpackage-function} @@ -141,7 +144,7 @@ buildPythonPackage rec { The `buildPythonPackage` mainly does four things: -* In the [`buildPhase`](#build-phase), it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to +* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} setup.py bdist_wheel` to build a wheel binary zipfile. * In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `pip install *.whl`. * In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to @@ -261,7 +264,7 @@ python3MyBlas = pkgs.python3.override { ``` This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations. -Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in +Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in compilation issues, because scipy dependencies need to use the same blas implementation as well. #### `buildPythonApplication` function {#buildpythonapplication-function} @@ -277,16 +280,16 @@ the packages with the version of the interpreter. Because this is irrelevant for applications, the prefix is omitted. When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be -called with `callPackage` and passed `python` or `pythonPackages` (possibly +called with `callPackage` and passed `python3` or `python3Packages` (possibly specifying an interpreter version), like this: ```nix { lib -, python3 +, python3Packages , fetchPypi }: -python3.pkgs.buildPythonApplication rec { +python3Packages.buildPythonApplication rec { pname = "luigi"; version = "2.7.9"; pyproject = true; @@ -297,13 +300,13 @@ python3.pkgs.buildPythonApplication rec { }; nativeBuildInputs = [ - python3.pkgs.setuptools - python3.pkgs.wheel + python3Packages.setuptools + python3Packages.wheel ]; - propagatedBuildInputs = with python3.pkgs; [ - tornado - python-daemon + propagatedBuildInputs = [ + python3Packages.tornado + python3Packages.python-daemon ]; meta = with lib; { @@ -319,7 +322,7 @@ luigi = callPackage ../applications/networking/cluster/luigi { }; ``` Since the package is an application, a consumer doesn't need to care about -Python versions or modules, which is why they don't go in `pythonPackages`. +Python versions or modules, which is why they don't go in `python3Packages`. #### `toPythonApplication` function {#topythonapplication-function} @@ -335,7 +338,7 @@ the attribute in `python-packages.nix`, and the `toPythonApplication` shall be applied to the reference: ```nix -youtube-dl = with pythonPackages; toPythonApplication youtube-dl; +youtube-dl = with python3Packages; toPythonApplication youtube-dl; ``` #### `toPythonModule` function {#topythonmodule-function} @@ -364,8 +367,8 @@ Saving the following as `default.nix` ```nix with import <nixpkgs> {}; -python.buildEnv.override { - extraLibs = [ pythonPackages.pyramid ]; +python3.buildEnv.override { + extraLibs = [ python3Packages.pyramid ]; ignoreCollisions = true; } ``` @@ -430,7 +433,7 @@ python3.withPackages (ps: [ ps.pyramid ]) Now, `ps` is set to `python3Packages`, matching the version of the interpreter. -As [`python.withPackages`](#python.withpackages-function) simply uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also +As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also supports the `env` attribute. The `shell.nix` file from the previous section can thus be also written like this: @@ -495,9 +498,9 @@ Given a `default.nix`: ```nix with import <nixpkgs> {}; -pythonPackages.buildPythonPackage { +python3Packages.buildPythonPackage { name = "myproject"; - buildInputs = with pythonPackages; [ pyramid ]; + buildInputs = with python3Packages; [ pyramid ]; src = ./.; } @@ -509,7 +512,7 @@ the package would be built with `nix-build`. Shortcut to setup environments with C headers/libraries and Python packages: ```shell -nix-shell -p pythonPackages.pyramid zlib libjpeg git +nix-shell -p python3Packages.pyramid zlib libjpeg git ``` ::: {.note} @@ -524,7 +527,7 @@ There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. Several versions of the Python interpreter are available on Nix, as well as a high amount of packages. The attribute `python3` refers to the default -interpreter, which is currently CPython 3.10. The attribute `python` refers to +interpreter, which is currently CPython 3.11. The attribute `python` refers to CPython 2.7 for backwards-compatibility. It is also possible to refer to specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to the default PyPy interpreter. @@ -542,7 +545,7 @@ however, are in separate sets, with one set per interpreter version. The interpreters have several common attributes. One of these attributes is `pkgs`, which is a package set of Python libraries for this specific interpreter. E.g., the `toolz` package corresponding to the default interpreter -is `python.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`. +is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`. The main package set contains aliases to these package sets, e.g. `pythonPackages` refers to `python.pkgs` and `python311Packages` to `python311.pkgs`. @@ -679,7 +682,7 @@ b = np.array([3,4]) print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") ``` -Then we simply execute it, without requiring any environment setup at all! +Then we execute it, without requiring any environment setup at all! ```sh $ ./foo.py @@ -1681,7 +1684,7 @@ of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. As workaround install it as an extra `preInstall` step: ```shell -${python.pythonForBuild.interpreter} setup.py install_data --install-dir=$out --root=$out +${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out sed -i '/ = data\_files/d' setup.py ``` @@ -1710,7 +1713,7 @@ This is an example of a `default.nix` for a `nix-shell`, which allows to consume a virtual environment created by `venv`, and install Python modules through `pip` the traditional way. -Create this `default.nix` file, together with a `requirements.txt` and simply +Create this `default.nix` file, together with a `requirements.txt` and execute `nix-shell`. ```nix @@ -1834,7 +1837,7 @@ If you need to change a package's attribute(s) from `configuration.nix` you coul }; ``` -`pythonPackages.twisted` is now globally overridden. +`python3Packages.twisted` is now globally overridden. All packages and also all NixOS services that reference `twisted` (such as `services.buildbot-worker`) now use the new definition. Note that `python-super` refers to the old package set and `python-self` @@ -1844,7 +1847,7 @@ To modify only a Python package set instead of a whole Python derivation, use this snippet: ```nix - myPythonPackages = pythonPackages.override { + myPythonPackages = python3Packages.override { overrides = self: super: { twisted = ...; }; @@ -2024,7 +2027,9 @@ The following rules are desired to be respected: disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why. * Commit names of Python libraries should reflect that they are Python - libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`. + libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`. + It is highly recommended to specify the current default version to enable + automatic build by ofborg. * Attribute names in `python-packages.nix` as well as `pname`s should match the library's name on PyPI, but be normalized according to [PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means diff --git a/doc/languages-frameworks/ruby.section.md b/doc/languages-frameworks/ruby.section.md index d3b896686c0..920c84eee68 100644 --- a/doc/languages-frameworks/ruby.section.md +++ b/doc/languages-frameworks/ruby.section.md @@ -94,7 +94,7 @@ $ bundle lock $ bundix ``` -If you already have a `Gemfile.lock`, you can simply run `bundix` and it will work the same. +If you already have a `Gemfile.lock`, you can run `bundix` and it will work the same. To update the gems in your `Gemfile.lock`, you may use the `bundix -l` flag, which will create a new `Gemfile.lock` in case the `Gemfile` has a more recent time of modification. @@ -251,7 +251,7 @@ source 'https://rubygems.org' do end ``` -If you want to package a specific version, you can use the standard Gemfile syntax for that, e.g. `gem 'mdl', '0.5.0'`, but if you want the latest stable version anyway, it's easier to update by simply running the `bundle lock` and `bundix` steps again. +If you want to package a specific version, you can use the standard Gemfile syntax for that, e.g. `gem 'mdl', '0.5.0'`, but if you want the latest stable version anyway, it's easier to update by running the `bundle lock` and `bundix` steps again. Now you can also make a `default.nix` that looks like this: diff --git a/doc/languages-frameworks/rust.section.md b/doc/languages-frameworks/rust.section.md index 3bd8e1c7651..d18b048b911 100644 --- a/doc/languages-frameworks/rust.section.md +++ b/doc/languages-frameworks/rust.section.md @@ -939,3 +939,68 @@ Fenix also has examples with `buildRustPackage`, [crane](https://github.com/ipetkov/crane), [naersk](https://github.com/nix-community/naersk), and cross compilation in its [Examples](https://github.com/nix-community/fenix#examples) section. + +## Using `git bisect` on the Rust compiler {#using-git-bisect-on-the-rust-compiler} + +Sometimes an upgrade of the Rust compiler (`rustc`) will break a +downstream package. In these situations, being able to `git bisect` +the `rustc` version history to find the offending commit is quite +useful. Nixpkgs makes it easy to do this. + +First, roll back your nixpkgs to a commit in which its `rustc` used +*the most recent one which doesn't have the problem.* You'll need +to do this because of `rustc`'s extremely aggressive +version-pinning. + +Next, add the following overlay, updating the Rust version to the +one in your rolled-back nixpkgs, and replacing `/git/scratch/rust` +with the path into which you have `git clone`d the `rustc` git +repository: + +```nix + (final: prev: /*lib.optionalAttrs prev.stdenv.targetPlatform.isAarch64*/ { + rust_1_72 = + lib.updateManyAttrsByPath [{ + path = [ "packages" "stable" ]; + update = old: old.overrideScope(final: prev: { + rustc = prev.rustc.overrideAttrs (_: { + src = lib.cleanSource /git/scratch/rust; + # do *not* put passthru.isReleaseTarball=true here + }); + }); + }] + prev.rust_1_72; + }) +``` + +If the problem you're troubleshooting only manifests when +cross-compiling you can uncomment the `lib.optionalAttrs` in the +example above, and replace `isAarch64` with the target that is +having problems. This will speed up your bisect quite a bit, since +the host compiler won't need to be rebuilt. + +Now, you can start a `git bisect` in the directory where you checked +out the `rustc` source code. It is recommended to select the +endpoint commits by searching backwards from `origin/master` for the +*commits which added the release notes for the versions in +question.* If you set the endpoints to commits on the release +branches (i.e. the release tags), git-bisect will often get confused +by the complex merge-commit structures it will need to traverse. + +The command loop you'll want to use for bisecting looks like this: + +```bash +git bisect {good,bad} # depending on result of last build +git submodule update --init +CARGO_NET_OFFLINE=false cargo vendor \ + --sync ./src/tools/cargo/Cargo.toml \ + --sync ./src/tools/rust-analyzer/Cargo.toml \ + --sync ./compiler/rustc_codegen_cranelift/Cargo.toml \ + --sync ./src/bootstrap/Cargo.toml +nix-build $NIXPKGS -A package-broken-by-rust-changes +``` + +The `git submodule update --init` and `cargo vendor` commands above +require network access, so they can't be performed from within the +`rustc` derivation, unfortunately. + diff --git a/doc/languages-frameworks/swift.section.md b/doc/languages-frameworks/swift.section.md index 1cc452cc9b9..213d444f499 100644 --- a/doc/languages-frameworks/swift.section.md +++ b/doc/languages-frameworks/swift.section.md @@ -32,7 +32,7 @@ look for the following directories: (If not targeting macOS, replace `macosx` with the Xcode platform name.) - On other platforms: `lib/swift/linux/x86_64` (Where `linux` and `x86_64` are from lowercase `uname -sm`.) -- For convenience, Nixpkgs also adds simply `lib/swift` to the search path. +- For convenience, Nixpkgs also adds `lib/swift` to the search path. This can save a bit of work packaging Swift modules, because many Nix builds will produce output for just one target any way. @@ -123,7 +123,7 @@ swiftpmFlags = [ "--disable-dead-strip" ]; The default `buildPhase` already passes `-j` for parallel building. -If these two customization options are insufficient, simply provide your own +If these two customization options are insufficient, provide your own `buildPhase` that invokes `swift build`. ### Running tests {#ssec-swiftpm-running-tests} diff --git a/doc/languages-frameworks/texlive.section.md b/doc/languages-frameworks/texlive.section.md index a4c81daa54b..2ba846dc492 100644 --- a/doc/languages-frameworks/texlive.section.md +++ b/doc/languages-frameworks/texlive.section.md @@ -2,6 +2,46 @@ Since release 15.09 there is a new TeX Live packaging that lives entirely under attribute `texlive`. +## User's guide (experimental new interface) {#sec-language-texlive-user-guide-experimental} + +Release 23.11 ships with a new interface that will eventually replace `texlive.combine`. + +- For basic usage, use some of the prebuilt environments available at the top level, such as `texliveBasic`, `texliveSmall`. For the full list of prebuilt environments, inspect `texlive.schemes`. + +- Packages cannot be used directly but must be assembled in an environment. To create or add packages to an environment, use + ```nix + texliveSmall.withPackages (ps: with ps; [ collection-langkorean algorithms cm-super ]) + ``` + The function `withPackages` can be called multiple times to add more packages. + + - **Note.** Within Nixpkgs, packages should only use prebuilt environments as inputs, such as `texliveSmall` or `texliveInfraOnly`, and should not depend directly on `texlive`. Further dependencies should be added by calling `withPackages`. This is to ensure that there is a consistent and simple way to override the inputs. + +- `texlive.withPackages` uses the same logic as `buildEnv`. Only parts of a package are installed in an environment: its 'runtime' files (`tex` output), binaries (`out` output), and support files (`tlpkg` output). Moreover, man and info pages are assembled into separate `man` and `info` outputs. To add only the TeX files of a package, or its documentation (`texdoc` output), just specify the outputs: + ```nix + texlive.withPackages (ps: with ps; [ + texdoc # recommended package to navigate the documentation + perlPackages.LaTeXML.tex # tex files of LaTeXML, omit binaries + cm-super + cm-super.texdoc # documentation of cm-super + ]) + ``` + +- All packages distributed by TeX Live, which contains most of CTAN, are available and can be found under `texlive.pkgs`: + ```ShellSession + $ nix repl + nix-repl> :l <nixpkgs> + nix-repl> texlive.pkgs.[TAB] + ``` + Note that the packages in `texlive.pkgs` are only provided for search purposes and must not be used directly. + +- **Experimental and subject to change without notice:** to add the documentation for all packages in the environment, use + ```nix + texliveSmall.__overrideTeXConfig { withDocs = true; } + ``` + This can be applied before or after calling `withPackages`. + + The function currently support the parameters `withDocs`, `withSources`, and `requireTeXPackages`. + ## User's guide {#sec-language-texlive-user-guide} - For basic usage just pull `texlive.combined.scheme-basic` for an environment with basic LaTeX support. @@ -38,6 +78,24 @@ Since release 15.09 there is a new TeX Live packaging that lives entirely under - Note that the wrapper assumes that the result has a chance to be useful. For example, the core executables should be present, as well as some core data files. The supported way of ensuring this is by including some scheme, for example `scheme-basic`, into the combination. +- TeX Live packages are also available under `texlive.pkgs` as derivations with outputs `out`, `tex`, `texdoc`, `texsource`, `tlpkg`, `man`, `info`. They cannot be installed outside of `texlive.combine` but are available for other uses. To repackage a font, for instance, use + + ```nix + stdenvNoCC.mkDerivation rec { + src = texlive.pkgs.iwona; + + inherit (src) pname version; + + installPhase = '' + runHook preInstall + install -Dm644 fonts/opentype/nowacki/iwona/*.otf -t $out/share/fonts/opentype + runHook postInstall + ''; + } + ``` + + See `biber`, `iwona` for complete examples. + ## Custom packages {#sec-language-texlive-custom-packages} You may find that you need to use an external TeX package. A derivation for such package has to provide the contents of the "texmf" directory in its output and provide the appropriate `tlType` attribute (one of `"run"`, `"bin"`, `"doc"`, `"source"`). Dependencies on other TeX packages can be listed in the attribute `tlDeps`. diff --git a/doc/manual.md.in b/doc/manual.md.in index 6b8d351380f..52971ff526c 100644 --- a/doc/manual.md.in +++ b/doc/manual.md.in @@ -9,7 +9,7 @@ preface.chapter.md using-nixpkgs.md lib.md stdenv.md -builders.md +build-helpers.md development.md contributing.md ``` diff --git a/doc/builders/packages/cataclysm-dda.section.md b/doc/packages/cataclysm-dda.section.md index f401e9b9efa..f401e9b9efa 100644 --- a/doc/builders/packages/cataclysm-dda.section.md +++ b/doc/packages/cataclysm-dda.section.md diff --git a/doc/builders/packages/citrix.section.md b/doc/packages/citrix.section.md index bcf0924249b..bcf0924249b 100644 --- a/doc/builders/packages/citrix.section.md +++ b/doc/packages/citrix.section.md diff --git a/doc/builders/special/darwin-builder.section.md b/doc/packages/darwin-builder.section.md index 13d01a0e3af..89c2445667d 100644 --- a/doc/builders/special/darwin-builder.section.md +++ b/doc/packages/darwin-builder.section.md @@ -1,10 +1,10 @@ # darwin.linux-builder {#sec-darwin-builder} -`darwin.linux-builder` provides a way to bootstrap a Linux builder on a macOS machine. +`darwin.linux-builder` provides a way to bootstrap a Linux remote builder on a macOS machine. This requires macOS version 12.4 or later. -The builder runs on host port 31022 by default. +The remote builder runs on host port 31022 by default. You can change it by overriding `virtualisation.darwin-builder.hostPort`. See the [example](#sec-darwin-builder-example-flake). @@ -15,7 +15,7 @@ words, your `/etc/nix/nix.conf` should have something like: extra-trusted-users = <your username goes here> ``` -To launch the builder, run the following flake: +To launch the remote builder, run the following flake: ```ShellSession $ nix run nixpkgs#darwin.linux-builder @@ -57,7 +57,7 @@ builders = ssh-ng://builder@linux-builder ${ARCH}-linux /etc/nix/builder_ed25519 builders-use-substitutes = true ``` -To allow Nix to connect to a builder not running on port 22, you will also need to create a new file at `/etc/ssh/ssh_config.d/100-linux-builder.conf`: +To allow Nix to connect to a remote builder not running on port 22, you will also need to create a new file at `/etc/ssh/ssh_config.d/100-linux-builder.conf`: ``` Host linux-builder @@ -130,11 +130,11 @@ $ sudo launchctl kickstart -k system/org.nixos.nix-daemon } ``` -## Reconfiguring the builder {#sec-darwin-builder-reconfiguring} +## Reconfiguring the remote builder {#sec-darwin-builder-reconfiguring} -Initially you should not change the builder configuration else you will not be -able to use the binary cache. However, after you have the builder running locally -you may use it to build a modified builder with additional storage or memory. +Initially you should not change the remote builder configuration else you will not be +able to use the binary cache. However, after you have the remote builder running locally +you may use it to build a modified remote builder with additional storage or memory. To do this, you just need to set the `virtualisation.darwin-builder.*` parameters as in the example below and rebuild. @@ -157,3 +157,17 @@ in the example below and rebuild. You may make any other changes to your VM in this attribute set. For example, you could enable Docker or X11 forwarding to your Darwin host. +## Troubleshooting the generated configuration {#sec-darwin-builder-troubleshoot} + +The `linux-builder` package exposes the attributes `nixosConfig` and `nixosOptions` that allow you to inspect the generated NixOS configuration in the `nix repl`. For example: + +``` +$ nix repl --file ~/src/nixpkgs --argstr system aarch64-darwin + +nix-repl> darwin.linux-builder.nixosConfig.nix.package +«derivation /nix/store/...-nix-2.17.0.drv» + +nix-repl> :p darwin.linux-builder.nixosOptions.virtualisation.memorySize.definitionsWithLocations +[ { file = "/home/user/src/nixpkgs/nixos/modules/profiles/macos-builder.nix"; value = 3072; } ] + +``` diff --git a/doc/builders/packages/dlib.section.md b/doc/packages/dlib.section.md index bd5b1a20a4d..bd5b1a20a4d 100644 --- a/doc/builders/packages/dlib.section.md +++ b/doc/packages/dlib.section.md diff --git a/doc/builders/packages/eclipse.section.md b/doc/packages/eclipse.section.md index e19510e131a..e19510e131a 100644 --- a/doc/builders/packages/eclipse.section.md +++ b/doc/packages/eclipse.section.md diff --git a/doc/builders/packages/elm.section.md b/doc/packages/elm.section.md index 063dd73d9de..063dd73d9de 100644 --- a/doc/builders/packages/elm.section.md +++ b/doc/packages/elm.section.md diff --git a/doc/builders/packages/emacs.section.md b/doc/packages/emacs.section.md index c50c7815537..c50c7815537 100644 --- a/doc/builders/packages/emacs.section.md +++ b/doc/packages/emacs.section.md diff --git a/doc/builders/packages/etc-files.section.md b/doc/packages/etc-files.section.md index 94a769ed335..94a769ed335 100644 --- a/doc/builders/packages/etc-files.section.md +++ b/doc/packages/etc-files.section.md diff --git a/doc/builders/packages/firefox.section.md b/doc/packages/firefox.section.md index 46bc0457a3d..46bc0457a3d 100644 --- a/doc/builders/packages/firefox.section.md +++ b/doc/packages/firefox.section.md diff --git a/doc/builders/packages/fish.section.md b/doc/packages/fish.section.md index 85b57acd109..85b57acd109 100644 --- a/doc/builders/packages/fish.section.md +++ b/doc/packages/fish.section.md diff --git a/doc/builders/packages/fuse.section.md b/doc/packages/fuse.section.md index 6deea6b5626..6deea6b5626 100644 --- a/doc/builders/packages/fuse.section.md +++ b/doc/packages/fuse.section.md diff --git a/doc/builders/packages/ibus.section.md b/doc/packages/ibus.section.md index 817e55d56f1..817e55d56f1 100644 --- a/doc/builders/packages/ibus.section.md +++ b/doc/packages/ibus.section.md diff --git a/doc/builders/packages/index.md b/doc/packages/index.md index 1f443570240..1f45018ffc4 100644 --- a/doc/builders/packages/index.md +++ b/doc/packages/index.md @@ -4,6 +4,7 @@ This chapter contains information about how to use and maintain the Nix expressi ```{=include=} sections citrix.section.md +darwin-builder.section.md dlib.section.md eclipse.section.md elm.section.md diff --git a/doc/builders/packages/kakoune.section.md b/doc/packages/kakoune.section.md index 8e054777a75..8e054777a75 100644 --- a/doc/builders/packages/kakoune.section.md +++ b/doc/packages/kakoune.section.md diff --git a/doc/builders/packages/linux.section.md b/doc/packages/linux.section.md index b64da85791a..b64da85791a 100644 --- a/doc/builders/packages/linux.section.md +++ b/doc/packages/linux.section.md diff --git a/doc/builders/packages/locales.section.md b/doc/packages/locales.section.md index 3a983f13a39..3a983f13a39 100644 --- a/doc/builders/packages/locales.section.md +++ b/doc/packages/locales.section.md diff --git a/doc/builders/packages/nginx.section.md b/doc/packages/nginx.section.md index 0704b534e5f..0704b534e5f 100644 --- a/doc/builders/packages/nginx.section.md +++ b/doc/packages/nginx.section.md diff --git a/doc/builders/packages/opengl.section.md b/doc/packages/opengl.section.md index f4d282267a0..f4d282267a0 100644 --- a/doc/builders/packages/opengl.section.md +++ b/doc/packages/opengl.section.md diff --git a/doc/builders/packages/shell-helpers.section.md b/doc/packages/shell-helpers.section.md index e7c2b0abebf..e7c2b0abebf 100644 --- a/doc/builders/packages/shell-helpers.section.md +++ b/doc/packages/shell-helpers.section.md diff --git a/doc/builders/packages/steam.section.md b/doc/packages/steam.section.md index 25728aa52ae..a1e88b0d971 100644 --- a/doc/builders/packages/steam.section.md +++ b/doc/packages/steam.section.md @@ -11,7 +11,7 @@ Nix problems and constraints: - The `steam.sh` script in `$HOME` cannot be patched, as it is checked and rewritten by steam. - The steam binary cannot be patched, it's also checked. -The current approach to deploy Steam in NixOS is composing a FHS-compatible chroot environment, as documented [here](http://sandervanderburg.blogspot.nl/2013/09/composing-fhs-compatible-chroot.html). This allows us to have binaries in the expected paths without disrupting the system, and to avoid patching them to work in a non FHS environment. +The current approach to deploy Steam in NixOS is composing a FHS-compatible chroot environment, as documented [here](https://sandervanderburg.blogspot.com/2013/09/composing-fhs-compatible-chroot.html). This allows us to have binaries in the expected paths without disrupting the system, and to avoid patching them to work in a non FHS environment. ## How to play {#sec-steam-play} diff --git a/doc/builders/packages/urxvt.section.md b/doc/packages/urxvt.section.md index 507feaa6fd8..7aff0997dd2 100644 --- a/doc/builders/packages/urxvt.section.md +++ b/doc/packages/urxvt.section.md @@ -34,7 +34,7 @@ $ nix repl map (p: p.name) pkgs.rxvt-unicode.plugins ``` -Alternatively, if your shell is bash or zsh and have completion enabled, simply type `nixpkgs.rxvt-unicode.plugins.<tab>`. +Alternatively, if your shell is bash or zsh and have completion enabled, type `nixpkgs.rxvt-unicode.plugins.<tab>`. In addition to `plugins` the options `extraDeps` and `perlDeps` can be used to install extra packages. `extraDeps` can be used, for example, to provide `xsel` (a clipboard manager) to the clipboard plugin, without installing it globally: diff --git a/doc/builders/packages/weechat.section.md b/doc/packages/weechat.section.md index 755b6e6ad1e..755b6e6ad1e 100644 --- a/doc/builders/packages/weechat.section.md +++ b/doc/packages/weechat.section.md diff --git a/doc/builders/packages/xorg.section.md b/doc/packages/xorg.section.md index ae885f92346..ae885f92346 100644 --- a/doc/builders/packages/xorg.section.md +++ b/doc/packages/xorg.section.md diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 366c519751c..26c43bd9e94 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -101,25 +101,62 @@ genericBuild ### Building a `stdenv` package in `nix-shell` {#sec-building-stdenv-package-in-nix-shell} -To build a `stdenv` package in a [`nix-shell`](https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html), use +To build a `stdenv` package in a [`nix-shell`](https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html), enter a shell, find the [phases](#sec-stdenv-phases) you wish to build, then invoke `genericBuild` manually: + +Go to an empty directory, invoke `nix-shell` with the desired package, and from inside the shell, set the output variables to a writable directory: ```bash +cd "$(mktemp -d)" nix-shell '<nixpkgs>' -A some_package -eval "${unpackPhase:-unpackPhase}" -cd $sourceRoot -eval "${patchPhase:-patchPhase}" -eval "${configurePhase:-configurePhase}" -eval "${buildPhase:-buildPhase}" +export out=$(pwd)/out +``` + +Next, invoke the desired parts of the build. +First, run the phases that generate a working copy of the sources, which will change directory to the sources for you: + +```bash +phases="${prePhases[*]:-} unpackPhase patchPhase" genericBuild +``` + +Then, run more phases up until the failure is reached. +For example, if the failure is in the build phase, the following phases would be required: + +```bash +phases="${preConfigurePhases[*]:-} configurePhase ${preBuildPhases[*]:-} buildPhase" genericBuild +``` + +Re-run a single phase as many times as necessary to examine the failure like so: + +```bash +phases="buildPhase" genericBuild ``` To modify a [phase](#sec-stdenv-phases), first print it with ```bash +echo "$buildPhase" +``` + +Or, if that is empty, for instance, if it is using a function: + +```bash type buildPhase ``` then change it in a text editor, and paste it back to the terminal. +::: {.note} +This method may have some inconsistencies in environment variables and behaviour compared to a normal build within the [Nix build sandbox](https://nixos.org/manual/nix/unstable/language/derivations#builder-execution). +The following is a non-exhaustive list of such differences: + +- `TMP`, `TMPDIR`, and similar variables likely point to non-empty directories that the build might conflict with files in. +- Output store paths are not writable, so the variables for outputs need to be overridden to writable paths. +- Other environment variables may be inconsistent with a `nix-build` either due to `nix-shell`'s initialization script or due to the use of `nix-shell` without the `--pure` option. + +If the build fails differently inside the shell than in the sandbox, consider using [`breakpointHook`](#breakpointhook) and invoking `nix-build` instead. +The [`--keep-failed`](https://nixos.org/manual/nix/unstable/command-ref/conf-file#opt--keep-failed) option for `nix-build` may also be useful to examine the build directory of a failed build. +::: + ## Tools provided by `stdenv` {#sec-tools-of-stdenv} The standard environment provides the following packages: @@ -282,7 +319,7 @@ let f(h, h + 1, i) = i + (if i <= 0 then h else h) let f(h, h + 1, i) = i + h ``` -This is where “sum-like” comes in from above: We can just sum all of the host offsets to get the host offset of the transitive dependency. The target offset is the transitive dependency is simply the host offset + 1, just as it was with the dependencies composed to make this transitive one; it can be ignored as it doesn’t add any new information. +This is where “sum-like” comes in from above: We can just sum all of the host offsets to get the host offset of the transitive dependency. The target offset is the transitive dependency is the host offset + 1, just as it was with the dependencies composed to make this transitive one; it can be ignored as it doesn’t add any new information. Because of the bounds checks, the uncommon cases are `h = t` and `h + 2 = t`. In the former case, the motivation for `mapOffset` is that since its host and target platforms are the same, no transitive dependency of it should be able to “discover” an offset greater than its reduced target offsets. `mapOffset` effectively “squashes” all its transitive dependencies’ offsets so that none will ever be greater than the target offset of the original `h = t` package. In the other case, `h + 1` is skipped over between the host and target offsets. Instead of squashing the offsets, we need to “rip” them apart so no transitive dependencies’ offset is that one. @@ -491,7 +528,7 @@ If the returned array contains exactly one object (e.g. `[{}]`), all values are ``` ::: -### Recursive attributes in `mkDerivation` {#mkderivation-recursive-attributes} +### Fixed-point arguments of `mkDerivation` {#mkderivation-recursive-attributes} If you pass a function to `mkDerivation`, it will receive as its argument the final arguments, including the overrides when reinvoked via `overrideAttrs`. For example: @@ -612,7 +649,7 @@ Zip files are unpacked using `unzip`. However, `unzip` is not in the standard en #### Directories in the Nix store {#directories-in-the-nix-store} -These are simply copied to the current directory. The hash part of the file name is stripped, e.g. `/nix/store/1wydxgby13cz...-my-sources` would be copied to `my-sources`. +These are copied to the current directory. The hash part of the file name is stripped, e.g. `/nix/store/1wydxgby13cz...-my-sources` would be copied to `my-sources`. Additional file types can be supported by setting the `unpackCmd` variable (see below). @@ -751,7 +788,7 @@ Hook executed at the end of the configure phase. ### The build phase {#build-phase} -The build phase is responsible for actually building the package (e.g. compiling it). The default `buildPhase` simply calls `make` if a file named `Makefile`, `makefile` or `GNUmakefile` exists in the current directory (or the `makefile` is explicitly set); otherwise it does nothing. +The build phase is responsible for actually building the package (e.g. compiling it). The default `buildPhase` calls `make` if a file named `Makefile`, `makefile` or `GNUmakefile` exists in the current directory (or the `makefile` is explicitly set); otherwise it does nothing. #### Variables controlling the build phase {#variables-controlling-the-build-phase} @@ -1280,7 +1317,7 @@ Nix itself considers a build-time dependency as merely something that should pre In order to alleviate this burden, the setup hook mechanism was written, where any package can include a shell script that \[by convention rather than enforcement by Nix\], any downstream reverse-dependency will source as part of its build process. That allows the downstream dependency to merely specify its dependencies, and lets those dependencies effectively initialize themselves. No boilerplate mirroring the list of dependencies is needed. -The setup hook mechanism is a bit of a sledgehammer though: a powerful feature with a broad and indiscriminate area of effect. The combination of its power and implicit use may be expedient, but isn’t without costs. Nix itself is unchanged, but the spirit of added dependencies being effect-free is violated even if the latter isn’t. For example, if a derivation path is mentioned more than once, Nix itself doesn’t care and simply makes sure the dependency derivation is already built just the same—depending is just needing something to exist, and needing is idempotent. However, a dependency specified twice will have its setup hook run twice, and that could easily change the build environment (though a well-written setup hook will therefore strive to be idempotent so this is in fact not observable). More broadly, setup hooks are anti-modular in that multiple dependencies, whether the same or different, should not interfere and yet their setup hooks may well do so. +The setup hook mechanism is a bit of a sledgehammer though: a powerful feature with a broad and indiscriminate area of effect. The combination of its power and implicit use may be expedient, but isn’t without costs. Nix itself is unchanged, but the spirit of added dependencies being effect-free is violated even if the latter isn’t. For example, if a derivation path is mentioned more than once, Nix itself doesn’t care and makes sure the dependency derivation is already built just the same—depending is just needing something to exist, and needing is idempotent. However, a dependency specified twice will have its setup hook run twice, and that could easily change the build environment (though a well-written setup hook will therefore strive to be idempotent so this is in fact not observable). More broadly, setup hooks are anti-modular in that multiple dependencies, whether the same or different, should not interfere and yet their setup hooks may well do so. The most typical use of the setup hook is actually to add other hooks which are then run (i.e. after all the setup hooks) on each dependency. For example, the C compiler wrapper’s setup hook feeds itself flags for each dependency that contains relevant libraries and headers. This is done by defining a bash function, and appending its name to one of `envBuildBuildHooks`, `envBuildHostHooks`, `envBuildTargetHooks`, `envHostHostHooks`, `envHostTargetHooks`, or `envTargetTargetHooks`. These 6 bash variables correspond to the 6 sorts of dependencies by platform (there’s 12 total but we ignore the propagated/non-propagated axis). diff --git a/doc/using/overlays.chapter.md b/doc/using/overlays.chapter.md index 6ee52215a4e..1bec6586f28 100644 --- a/doc/using/overlays.chapter.md +++ b/doc/using/overlays.chapter.md @@ -77,7 +77,7 @@ In Nixpkgs, we have multiple implementations of the BLAS/LAPACK numerical linear The Nixpkgs attribute is `openblas` for ILP64 (integer width = 64 bits) and `openblasCompat` for LP64 (integer width = 32 bits). `openblasCompat` is the default. -- [LAPACK reference](http://www.netlib.org/lapack/) (also provides BLAS and CBLAS) +- [LAPACK reference](https://www.netlib.org/lapack/) (also provides BLAS and CBLAS) The Nixpkgs attribute is `lapack-reference`. @@ -156,7 +156,7 @@ All programs that are built with [MPI](https://en.wikipedia.org/wiki/Message_Pas - [MVAPICH](https://mvapich.cse.ohio-state.edu/), attribute name `mvapich` -To provide MPI enabled applications that use `MPICH`, instead of the default `Open MPI`, simply use the following overlay: +To provide MPI enabled applications that use `MPICH`, instead of the default `Open MPI`, use the following overlay: ```nix self: super: |