diff options
Diffstat (limited to 'doc')
28 files changed, 832 insertions, 1364 deletions
diff --git a/doc/builders/images/dockertools.xml b/doc/builders/images/dockertools.xml index 126698d0a9e..d881e712a04 100644 --- a/doc/builders/images/dockertools.xml +++ b/doc/builders/images/dockertools.xml @@ -132,11 +132,11 @@ buildImage { <para> By default <function>buildImage</function> will use a static date of one second past the UNIX Epoch. This allows <function>buildImage</function> to produce binary reproducible images. When listing images with <command>docker images</command>, the newly created images will be listed like this: </para> -<screen><![CDATA[ -$ docker images +<screen> +<prompt>$ </prompt>docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello latest 08c791c7846e 48 years ago 25.2MB -]]></screen> +</screen> <para> You can break binary reproducibility but have a sorted, meaningful <literal>CREATED</literal> column by setting <literal>created</literal> to <literal>now</literal>. </para> @@ -152,11 +152,11 @@ pkgs.dockerTools.buildImage { ]]></programlisting> <para> and now the Docker CLI will display a reasonable date and sort the images as expected: -<screen><![CDATA[ -$ docker images +<screen> +<prompt>$ </prompt>docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello latest de2bf4786de6 About a minute ago 25.2MB -]]></screen> +</screen> however, the produced images will not be binary reproducible. </para> </example> diff --git a/doc/builders/images/ocitools.xml b/doc/builders/images/ocitools.xml index e8cd3472f54..f26ed864427 100644 --- a/doc/builders/images/ocitools.xml +++ b/doc/builders/images/ocitools.xml @@ -38,8 +38,7 @@ buildContainer { readonly = false; <co xml:id='ex-ociTools-buildContainer-3' /> } - - </programlisting> +</programlisting> <calloutlist> <callout arearefs='ex-ociTools-buildContainer-1'> <para> diff --git a/doc/builders/packages/citrix.xml b/doc/builders/packages/citrix.xml index 16f1bc6f8f2..803eb2e4fc4 100644 --- a/doc/builders/packages/citrix.xml +++ b/doc/builders/packages/citrix.xml @@ -22,10 +22,10 @@ </para> <para> In order to set this up, you first have to <link xlink:href="https://its.uiowa.edu/support/article/102186">download the <literal>.cr</literal> file from the Netscaler Gateway</link>. After that you can configure the <command>selfservice</command> like this: - <screen> - <prompt>$ </prompt>storebrowse -C ~/Downloads/receiverconfig.cr - <prompt>$ </prompt>selfservice - </screen> +<screen> +<prompt>$ </prompt>storebrowse -C ~/Downloads/receiverconfig.cr +<prompt>$ </prompt>selfservice +</screen> </para> </section> diff --git a/doc/builders/packages/kakoune.xml b/doc/builders/packages/kakoune.xml index 728d40dacc9..045dbd0a653 100644 --- a/doc/builders/packages/kakoune.xml +++ b/doc/builders/packages/kakoune.xml @@ -6,9 +6,7 @@ <para> Kakoune can be built to autoload plugins: <programlisting>(kakoune.override { - configure = { - plugins = with pkgs.kakounePlugins; [ parinfer-rust ]; - }; + plugins = with pkgs.kakounePlugins; [ parinfer-rust ]; })</programlisting> </para> </section> diff --git a/doc/builders/packages/opengl.xml b/doc/builders/packages/opengl.xml index 5f4433a2884..dfd64b18858 100644 --- a/doc/builders/packages/opengl.xml +++ b/doc/builders/packages/opengl.xml @@ -4,6 +4,6 @@ <title>OpenGL</title> <para> - Packages that use OpenGL have NixOS desktop as their primary target. The current solution for loading the GPU-specific drivers is based on <literal>libglvnd</literal> and looks for the driver implementation in <literal>LD_LIBRARY_PATH</literal>. If you are using a non-NixOS GNU/Linux/X11 desktop with free software video drivers, consider launching OpenGL-dependent programs from Nixpkgs with Nixpkgs versions of <literal>libglvnd</literal> and <literal>mesa_drivers</literal> in <literal>LD_LIBRARY_PATH</literal>. For proprietary video drivers you might have luck with also adding the corresponding video driver package. + Packages that use OpenGL have NixOS desktop as their primary target. The current solution for loading the GPU-specific drivers is based on <literal>libglvnd</literal> and looks for the driver implementation in <literal>LD_LIBRARY_PATH</literal>. If you are using a non-NixOS GNU/Linux/X11 desktop with free software video drivers, consider launching OpenGL-dependent programs from Nixpkgs with Nixpkgs versions of <literal>libglvnd</literal> and <literal>mesa.drivers</literal> in <literal>LD_LIBRARY_PATH</literal>. For proprietary video drivers you might have luck with also adding the corresponding video driver package. </para> </section> diff --git a/doc/builders/packages/urxvt.xml b/doc/builders/packages/urxvt.xml index 135cc82a0b5..330e056b656 100644 --- a/doc/builders/packages/urxvt.xml +++ b/doc/builders/packages/urxvt.xml @@ -18,10 +18,13 @@ includes all available plugins. To make use of this functionality, use an overlay or directly install an expression that overrides its configuration, such as - <programlisting>rxvt-unicode.override { configure = { availablePlugins, ... }: { +<programlisting> +rxvt-unicode.override { + configure = { availablePlugins, ... }: { plugins = with availablePlugins; [ perls resize-font vtwheel ]; - } -}</programlisting> + }; +} +</programlisting> If the <literal>configure</literal> function returns an attrset without the <literal>plugins</literal> attribute, <literal>availablePlugins</literal> will be used automatically. @@ -30,18 +33,22 @@ <para> In order to add plugins but also keep all default plugins installed, it is possible to use the following method: - <programlisting>rxvt-unicode.override { configure = { availablePlugins, ... }: { - plugins = (builtins.attrValues availablePlugins) ++ [ custom-plugin ]; - }; -}</programlisting> +<programlisting> +rxvt-unicode.override { + configure = { availablePlugins, ... }: { + plugins = (builtins.attrValues availablePlugins) ++ [ custom-plugin ]; + }; +} +</programlisting> </para> <para> To get a list of all the plugins available, open the Nix REPL and run - <programlisting>$ nix repl +<screen> +<prompt>$ </prompt>nix repl :l <nixpkgs> map (p: p.name) pkgs.rxvt-unicode.plugins - </programlisting> +</screen> Alternatively, if your shell is bash or zsh and have completion enabled, simply type <literal>nixpkgs.rxvt-unicode.plugins.<tab></literal>. </para> @@ -53,18 +60,24 @@ map (p: p.name) pkgs.rxvt-unicode.plugins <literal>extraDeps</literal> can be used, for example, to provide <literal>xsel</literal> (a clipboard manager) to the clipboard plugin, without installing it globally: - <programlisting>rxvt-unicode.override { configure = { availablePlugins, ... }: { - pluginsDeps = [ xsel ]; - } -}</programlisting> +<programlisting> +rxvt-unicode.override { + configure = { availablePlugins, ... }: { + pluginsDeps = [ xsel ]; + }; +} +</programlisting> <literal>perlDeps</literal> is a handy way to provide Perl packages to your custom plugins (in <literal>$HOME/.urxvt/ext</literal>). For example, if you need <literal>AnyEvent</literal> you can do: - <programlisting>rxvt-unicode.override { configure = { availablePlugins, ... }: { - perlDeps = with perlPackages; [ AnyEvent ]; - } -}</programlisting> +<programlisting> +rxvt-unicode.override { + configure = { availablePlugins, ... }: { + perlDeps = with perlPackages; [ AnyEvent ]; + }; +} +</programlisting> </para> </section> @@ -90,7 +103,8 @@ map (p: p.name) pkgs.rxvt-unicode.plugins <para> If the plugin is itself a perl package that needs to be imported from other plugins or scripts, add the following passthrough: - <programlisting>passthru.perlPackages = [ "self" ]; +<programlisting> +passthru.perlPackages = [ "self" ]; </programlisting> This will make the urxvt wrapper pick up the dependency and set up the perl path accordingly. diff --git a/doc/contributing/coding-conventions.xml b/doc/contributing/coding-conventions.xml index fcb6501f615..e587275a638 100644 --- a/doc/contributing/coding-conventions.xml +++ b/doc/contributing/coding-conventions.xml @@ -524,6 +524,16 @@ args.stdenv.mkDerivation (args // { </varlistentry> <varlistentry> <term> + If it’s a <emphasis>terminal emulator</emphasis>: + </term> + <listitem> + <para> + <filename>applications/terminal-emulators</filename> (e.g. <filename>alacritty</filename> or <filename>rxvt</filename> or <filename>termite</filename>) + </para> + </listitem> + </varlistentry> + <varlistentry> + <term> If it’s for <emphasis>video playback / editing</emphasis>: </term> <listitem> diff --git a/doc/contributing/submitting-changes.xml b/doc/contributing/submitting-changes.xml index a88965f5cc6..22389c24ea2 100644 --- a/doc/contributing/submitting-changes.xml +++ b/doc/contributing/submitting-changes.xml @@ -209,12 +209,12 @@ Additional information. </para> <programlisting> - (fetchpatch { - name = "CVE-2019-11068.patch"; - url = "https://gitlab.gnome.org/GNOME/libxslt/commit/e03553605b45c88f0b4b2980adfbbb8f6fca2fd6.patch"; - sha256 = "0pkpb4837km15zgg6h57bncp66d5lwrlvkr73h0lanywq7zrwhj8"; - }) - </programlisting> +(fetchpatch { + name = "CVE-2019-11068.patch"; + url = "https://gitlab.gnome.org/GNOME/libxslt/commit/e03553605b45c88f0b4b2980adfbbb8f6fca2fd6.patch"; + sha256 = "0pkpb4837km15zgg6h57bncp66d5lwrlvkr73h0lanywq7zrwhj8"; +}) +</programlisting> <para> If a security fix applies to both master and a stable release then, similar to regular changes, they are preferably delivered via master first and cherry-picked to the release branch. diff --git a/doc/languages-frameworks/agda.section.md b/doc/languages-frameworks/agda.section.md index 9ce046d05b6..5ba4e285f42 100644 --- a/doc/languages-frameworks/agda.section.md +++ b/doc/languages-frameworks/agda.section.md @@ -12,12 +12,13 @@ Agda can be installed from `agda`: $ nix-env -iA agda ``` -To use agda with libraries, the `agda.withPackages` function can be used. This function either takes: -+ A list of packages, -+ or a function which returns a list of packages when given the `agdaPackages` attribute set, -+ or an attribute set containing a list of packages and a GHC derivation for compilation (see below). +To use Agda with libraries, the `agda.withPackages` function can be used. This function either takes: -For example, suppose we wanted a version of agda which has access to the standard library. This can be obtained with the expressions: +* A list of packages, +* or a function which returns a list of packages when given the `agdaPackages` attribute set, +* or an attribute set containing a list of packages and a GHC derivation for compilation (see below). + +For example, suppose we wanted a version of Agda which has access to the standard library. This can be obtained with the expressions: ``` agda.withPackages [ agdaPackages.standard-library ] @@ -33,18 +34,19 @@ or can be called as in the [Compiling Agda](#compiling-agda) section. If you want to use a library in your home directory (for instance if it is a development version) then typecheck it manually (using `agda.withPackages` if necessary) and then override the `src` attribute of the package to point to your local repository. -Agda will not by default use these libraries. To tell agda to use the library we have some options: -- Call `agda` with the library flag: +Agda will not by default use these libraries. To tell Agda to use the library we have some options: + +* Call `agda` with the library flag: ``` $ agda -l standard-library -i . MyFile.agda ``` -- Write a `my-library.agda-lib` file for the project you are working on which may look like: +* Write a `my-library.agda-lib` file for the project you are working on which may look like: ``` name: my-library include: . depend: standard-library ``` -- Create the file `~/.agda/defaults` and add any libraries you want to use by default. +* Create the file `~/.agda/defaults` and add any libraries you want to use by default. More information can be found in the [official Agda documentation on library management](https://agda.readthedocs.io/en/v2.6.1/tools/package-system.html). @@ -60,12 +62,13 @@ agda.withPackages { ``` ## Writing Agda packages -To write a nix derivation for an agda library, first check that the library has a `*.agda-lib` file. +To write a nix derivation for an Agda library, first check that the library has a `*.agda-lib` file. A derivation can then be written using `agdaPackages.mkDerivation`. This has similar arguments to `stdenv.mkDerivation` with the following additions: -+ `everythingFile` can be used to specify the location of the `Everything.agda` file, defaulting to `./Everything.agda`. If this file does not exist then either it should be patched in or the `buildPhase` should be overridden (see below). -+ `libraryName` should be the name that appears in the `*.agda-lib` file, defaulting to `pname`. -+ `libraryFile` should be the file name of the `*.agda-lib` file, defaulting to `${libraryName}.agda-lib`. + +* `everythingFile` can be used to specify the location of the `Everything.agda` file, defaulting to `./Everything.agda`. If this file does not exist then either it should be patched in or the `buildPhase` should be overridden (see below). +* `libraryName` should be the name that appears in the `*.agda-lib` file, defaulting to `pname`. +* `libraryFile` should be the file name of the `*.agda-lib` file, defaulting to `${libraryName}.agda-lib`. ### Building Agda packages The default build phase for `agdaPackages.mkDerivation` simply runs `agda` on the `Everything.agda` file. @@ -74,12 +77,14 @@ Additionally, a `preBuild` or `configurePhase` can be used if there are steps th `agda` and the Agda libraries contained in `buildInputs` are made available during the build phase. ### Installing Agda packages -The default install phase copies agda source files, agda interface files (`*.agdai`) and `*.agda-lib` files to the output directory. +The default install phase copies Agda source files, Agda interface files (`*.agdai`) and `*.agda-lib` files to the output directory. This can be overridden. -By default, agda sources are files ending on `.agda`, or literate agda files ending on `.lagda`, `.lagda.tex`, `.lagda.org`, `.lagda.md`, `.lagda.rst`. The list of recognised agda source extensions can be extended by setting the `extraExtensions` config variable. +By default, Agda sources are files ending on `.agda`, or literate Agda files ending on `.lagda`, `.lagda.tex`, `.lagda.org`, `.lagda.md`, `.lagda.rst`. The list of recognised Agda source extensions can be extended by setting the `extraExtensions` config variable. + +## Adding Agda packages to Nixpkgs -To add an agda package to `nixpkgs`, the derivation should be written to `pkgs/development/libraries/agda/${library-name}/` and an entry should be added to `pkgs/top-level/agda-packages.nix`. Here it is called in a scope with access to all other agda libraries, so the top line of the `default.nix` can look like: +To add an Agda package to `nixpkgs`, the derivation should be written to `pkgs/development/libraries/agda/${library-name}/` and an entry should be added to `pkgs/top-level/agda-packages.nix`. Here it is called in a scope with access to all other Agda libraries, so the top line of the `default.nix` can look like: ``` { mkDerivation, standard-library, fetchFromGitHub }: ``` @@ -103,4 +108,4 @@ mkDerivation { ``` This library has a file called `.agda-lib`, and so we give an empty string to `libraryFile` as nothing precedes `.agda-lib` in the filename. This file contains `name: IAL-1.3`, and so we let `libraryName = "IAL-1.3"`. This library does not use an `Everything.agda` file and instead has a Makefile, so there is no need to set `everythingFile` and we set a custom `buildPhase`. -When writing an agda package it is essential to make sure that no `.agda-lib` file gets added to the store as a single file (for example by using `writeText`). This causes agda to think that the nix store is a agda library and it will attempt to write to it whenever it typechecks something. See [https://github.com/agda/agda/issues/4613](https://github.com/agda/agda/issues/4613). +When writing an Agda package it is essential to make sure that no `.agda-lib` file gets added to the store as a single file (for example by using `writeText`). This causes Agda to think that the nix store is a Agda library and it will attempt to write to it whenever it typechecks something. See [https://github.com/agda/agda/issues/4613](https://github.com/agda/agda/issues/4613). diff --git a/doc/languages-frameworks/beam.xml b/doc/languages-frameworks/beam.xml index 1d307e1d6dc..addab24f7f6 100644 --- a/doc/languages-frameworks/beam.xml +++ b/doc/languages-frameworks/beam.xml @@ -72,9 +72,9 @@ To install any of those builders into your profile, refer to them by their attribute path <literal>beamPackages.rebar3</literal>: </para> - <screen> - <prompt>$ </prompt>nix-env -f "<nixpkgs>" -iA beamPackages.rebar3 - </screen> +<screen> +<prompt>$ </prompt>nix-env -f "<nixpkgs>" -iA beamPackages.rebar3 +</screen> </section> <section xml:id="packaging-beam-applications"> diff --git a/doc/languages-frameworks/go.xml b/doc/languages-frameworks/go.xml index 26414f23c7d..ebdcf616054 100644 --- a/doc/languages-frameworks/go.xml +++ b/doc/languages-frameworks/go.xml @@ -38,11 +38,7 @@ pet = buildGoModule rec { vendorSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; <co xml:id='ex-buildGoModule-1' /> - subPackages = [ "." ]; <co xml:id='ex-buildGoModule-2' /> - - deleteVendor = true; <co xml:id='ex-buildGoModule-3' /> - - runVend = true; <co xml:id='ex-buildGoModule-4' /> + runVend = true; <co xml:id='ex-buildGoModule-2' /> meta = with lib; { description = "Simple command-line snippet manager, written in Go"; @@ -65,16 +61,6 @@ pet = buildGoModule rec { </callout> <callout arearefs='ex-buildGoModule-2'> <para> - <varname>subPackages</varname> limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built. - </para> - </callout> - <callout arearefs='ex-buildGoModule-3'> - <para> - <varname>deleteVendor</varname> removes the pre-existing vendor directory and fetches the dependencies. This should only be used if the dependencies included in the vendor folder are broken or incomplete. - </para> - </callout> - <callout arearefs='ex-buildGoModule-4'> - <para> <varname>runVend</varname> runs the vend command to generate the vendor directory. This is useful if your code depends on c code and go mod tidy does not include the needed sources to build. </para> </callout> @@ -82,12 +68,7 @@ pet = buildGoModule rec { </para> <para> - <varname>vendorSha256</varname> can also take <varname>null</varname> as an input. - - When `null` is used as a value, rather than fetching the dependencies - and vendoring them, we use the vendoring included within the source repo. - If you'd like to not have to update this field on dependency changes, - run `go mod vendor` in your source repo and set 'vendorSha256 = null;' + <varname>vendorSha256</varname> can also take <varname>null</varname> as an input. When `null` is used as a value, rather than fetching the dependencies and vendoring them, we use the vendoring included within the source repo. If you'd like to not have to update this field on dependency changes, run `go mod vendor` in your source repo and set 'vendorSha256 = null;' </para> </section> @@ -106,7 +87,6 @@ deis = buildGoPackage rec { version = "1.13.0"; goPackagePath = "github.com/deis/deis"; <co xml:id='ex-buildGoPackage-1' /> - subPackages = [ "client" ]; <co xml:id='ex-buildGoPackage-2' /> src = fetchFromGitHub { owner = "deis"; @@ -115,9 +95,7 @@ deis = buildGoPackage rec { sha256 = "1qv9lxqx7m18029lj8cw3k7jngvxs4iciwrypdy0gd2nnghc68sw"; }; - goDeps = ./deps.nix; <co xml:id='ex-buildGoPackage-3' /> - - buildFlags = [ "--tags" "release" ]; <co xml:id='ex-buildGoPackage-4' /> + goDeps = ./deps.nix; <co xml:id='ex-buildGoPackage-2' /> } </programlisting> </example> @@ -132,22 +110,9 @@ deis = buildGoPackage rec { </callout> <callout arearefs='ex-buildGoPackage-2'> <para> - <varname>subPackages</varname> limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built. - </para> - <para> - In this example only <literal>github.com/deis/deis/client</literal> will be built. - </para> - </callout> - <callout arearefs='ex-buildGoPackage-3'> - <para> <varname>goDeps</varname> is where the Go dependencies of a Go program are listed as a list of package source identified by Go import path. It could be imported as a separate <varname>deps.nix</varname> file for readability. The dependency data structure is described below. </para> </callout> - <callout arearefs='ex-buildGoPackage-4'> - <para> - <varname>buildFlags</varname> is a list of flags passed to the go build command. - </para> - </callout> </calloutlist> </para> @@ -214,4 +179,70 @@ done </screen> </para> </section> + + <section xml:id="ssec-go-common-attributes"> + <title>Attributes used by the builders</title> + + <para> + Both <link xlink:href="#ssec-go-modules"><varname>buildGoModule</varname></link> and <link xlink:href="#ssec-go-modules"><varname>buildGoPackage</varname></link> can be tweaked to behave slightly differently, if the following attributes are used: + </para> + + <variablelist> + <varlistentry xml:id="var-go-buildFlagsArray"> + <term> + <varname>buildFlagsArray</varname> and <varname>buildFlags</varname> + </term> + <listitem> + <para> + These attributes set build flags supported by <varname>go build</varname>. We recommend using <varname>buildFlagsArray</varname>. The most common use case of these attributes is to make the resulting executable aware of its own version. For example: + </para> + <example xml:id='ex-goBuildFlags-nospaces'> + <title>buildFlagsArray</title> +<programlisting> + buildFlagsArray = [ + "-ldflags=-X main.Version=${version} -X main.Commit=${version}" <co xml:id='ex-goBuildFlags-1' /> + ]; +</programlisting> + </example> + <calloutlist> + <callout arearefs='ex-goBuildFlags-1'> + <para> + Note: single quotes are not needed. + </para> + </callout> + </calloutlist> + <example xml:id='ex-goBuildFlags-noarray'> + <title>buildFlagsArray</title> +<programlisting> + buildFlagsArray = '' + -ldflags= + -X main.Version=${version} + -X main.Commit=${version} + ''; +</programlisting> + </example> + </listitem> + </varlistentry> + <varlistentry xml:id="var-go-deleteVendor"> + <term> + <varname>deleteVendor</varname> + </term> + <listitem> + <para> + Removes the pre-existing vendor directory. This should only be used if the dependencies included in the vendor folder are broken or incomplete. + </para> + </listitem> + </varlistentry> + <varlistentry xml:id="var-go-subPackages"> + <term> + <varname>subPackages</varname> + </term> + <listitem> + <para> + Limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built. + </para> + </listitem> + </varlistentry> + </variablelist> + </section> </section> diff --git a/doc/languages-frameworks/haskell.section.md b/doc/languages-frameworks/haskell.section.md index cba4d0561b0..67318a5b746 100644 --- a/doc/languages-frameworks/haskell.section.md +++ b/doc/languages-frameworks/haskell.section.md @@ -3,1092 +3,11 @@ title: User's Guide for Haskell in Nixpkgs author: Peter Simons date: 2015-06-01 --- -# Haskell - - -## How to install Haskell packages - -Nixpkgs distributes build instructions for all Haskell packages registered on -[Hackage](http://hackage.haskell.org/), but strangely enough normal Nix package -lookups don't seem to discover any of them, except for the default version of ghc, cabal-install, and stack: -``` -$ nix-env -i alex -error: selector ‘alex’ matches no derivations -$ nix-env -qa ghc -ghc-7.10.2 -``` - -The Haskell package set is not registered in the top-level namespace because it -is *huge*. If all Haskell packages were visible to these commands, then -name-based search/install operations would be much slower than they are now. We -avoided that by keeping all Haskell-related packages in a separate attribute -set called `haskellPackages`, which the following command will list: -``` -$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages -haskellPackages.a50 a50-0.5 -haskellPackages.AAI AAI-0.2.0.1 -haskellPackages.abacate abacate-0.0.0.0 -haskellPackages.abc-puzzle abc-puzzle-0.2.1 -haskellPackages.abcBridge abcBridge-0.15 -haskellPackages.abcnotation abcnotation-1.9.0 -haskellPackages.abeson abeson-0.1.0.1 -[... some 14000 entries omitted ...] -``` - -To install any of those packages into your profile, refer to them by their -attribute path (first column): -```shell -nix-env -f "<nixpkgs>" -iA haskellPackages.Allure ... -``` - -The attribute path of any Haskell packages corresponds to the name of that -particular package on Hackage: the package `cabal-install` has the attribute -`haskellPackages.cabal-install`, and so on. (Actually, this convention causes -trouble with packages like `3dmodels` and `4Blocks`, because these names are -invalid identifiers in the Nix language. The issue of how to deal with these -rare corner cases is currently unresolved.) - -Haskell packages whose Nix name (second column) begins with a `haskell-` prefix -are packages that provide a library whereas packages without that prefix -provide just executables. Libraries may provide executables too, though: the -package `haskell-pandoc`, for example, installs both a library and an -application. You can install and use Haskell executables just like any other -program in Nixpkgs, but using Haskell libraries for development is a bit -trickier and we'll address that subject in great detail in section [How to -create a development environment](#how-to-create-a-development-environment). - -Attribute paths are deterministic inside of Nixpkgs, but the path necessary to -reach Nixpkgs varies from system to system. We dodged that problem by giving -`nix-env` an explicit `-f "<nixpkgs>"` parameter, but if you call `nix-env` -without that flag, then chances are the invocation fails: -``` -$ nix-env -iA haskellPackages.cabal-install -error: attribute ‘haskellPackages’ in selection path - ‘haskellPackages.cabal-install’ not found -``` - -On NixOS, for example, Nixpkgs does *not* exist in the top-level namespace by -default. To figure out the proper attribute path, it's easiest to query for the -path of a well-known Nixpkgs package, i.e.: -``` -$ nix-env -qaP coreutils -nixos.coreutils coreutils-8.23 -``` - -If your system responds like that (most NixOS installations will), then the -attribute path to `haskellPackages` is `nixos.haskellPackages`. Thus, if you -want to use `nix-env` without giving an explicit `-f` flag, then that's the way -to do it: -```shell -nix-env -qaP -A nixos.haskellPackages -nix-env -iA nixos.haskellPackages.cabal-install -``` - -Our current default compiler is GHC 8.8.x and the `haskellPackages` set -contains packages built with that particular version. Nixpkgs contains the last -three major releases of GHC and there is a whole family of package sets -available that defines Hackage packages built with each of those compilers, -too: -```shell -nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc865 -nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc8101 -``` - -The name `haskellPackages` is really just a synonym for -`haskell.packages.ghc882`, because we prefer that package set internally and -recommend it to our users as their default choice, but ultimately you are free -to compile your Haskell packages with any GHC version you please. The following -command displays the complete list of available compilers: -``` -$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler -haskell.compiler.ghc8101 ghc-8.10.1 -haskell.compiler.integer-simple.ghc8101 ghc-8.10.1 -haskell.compiler.ghcHEAD ghc-8.11.20200505 -haskell.compiler.integer-simple.ghcHEAD ghc-8.11.20200505 -haskell.compiler.ghc822Binary ghc-8.2.2-binary -haskell.compiler.ghc844 ghc-8.4.4 -haskell.compiler.ghc863Binary ghc-8.6.3-binary -haskell.compiler.ghc865 ghc-8.6.5 -haskell.compiler.integer-simple.ghc865 ghc-8.6.5 -haskell.compiler.ghc882 ghc-8.8.2 -haskell.compiler.integer-simple.ghc882 ghc-8.8.2 -haskell.compiler.ghc883 ghc-8.8.3 -haskell.compiler.integer-simple.ghc883 ghc-8.8.3 -haskell.compiler.ghcjs ghcjs-8.6.0.1 -``` - -We have no package sets for `jhc` or `uhc` yet, unfortunately, but for every -version of GHC listed above, there exists a package set based on that compiler. -Also, the attributes `haskell.compiler.ghcXYC` and -`haskell.packages.ghcXYC.ghc` are synonymous for the sake of convenience. - -## How to create a development environment - -### How to install a compiler - -A simple development environment consists of a Haskell compiler and one or both -of the tools `cabal-install` and `stack`. We saw in section -[How to install Haskell packages](#how-to-install-haskell-packages) how you can install those programs into your -user profile: -```shell -nix-env -f "<nixpkgs>" -iA haskellPackages.ghc haskellPackages.cabal-install -``` - -Instead of the default package set `haskellPackages`, you can also use the more -precise name `haskell.compiler.ghc7102`, which has the advantage that it refers -to the same GHC version regardless of what Nixpkgs considers "default" at any -given time. - -Once you've made those tools available in `$PATH`, it's possible to build -Hackage packages the same way people without access to Nix do it all the time: -```shell -cabal get lens-4.11 && cd lens-4.11 -cabal install -j --dependencies-only -cabal configure -cabal build -``` - -If you enjoy working with Cabal sandboxes, then that's entirely possible too: -just execute the command -```shell -cabal sandbox init -``` -before installing the required dependencies. - -The `nix-shell` utility makes it easy to switch to a different compiler -version; just enter the Nix shell environment with the command -```shell -nix-shell -p haskell.compiler.ghc784 -``` -to bring GHC 7.8.4 into `$PATH`. Alternatively, you can use Stack instead of -`nix-shell` directly to select compiler versions and other build tools -per-project. It uses `nix-shell` under the hood when Nix support is turned on. -See [How to build a Haskell project using Stack](#how-to-build-a-haskell-project-using-stack). - -If you're using `cabal-install`, re-running `cabal configure` inside the spawned -shell switches your build to use that compiler instead. If you're working on -a project that doesn't depend on any additional system libraries outside of GHC, -then it's even sufficient to just run the `cabal configure` command inside of -the shell: -```shell -nix-shell -p haskell.compiler.ghc784 --command "cabal configure" -``` - -Afterwards, all other commands like `cabal build` work just fine in any shell -environment, because the configure phase recorded the absolute paths to all -required tools like GHC in its build configuration inside of the `dist/` -directory. Please note, however, that `nix-collect-garbage` can break such an -environment because the Nix store paths created by `nix-shell` aren't "alive" -anymore once `nix-shell` has terminated. If you find that your Haskell builds -no longer work after garbage collection, then you'll have to re-run `cabal -configure` inside of a new `nix-shell` environment. - -### How to install a compiler with libraries - -GHC expects to find all installed libraries inside of its own `lib` directory. -This approach works fine on traditional Unix systems, but it doesn't work for -Nix, because GHC's store path is immutable once it's built. We cannot install -additional libraries into that location. As a consequence, our copies of GHC -don't know any packages except their own core libraries, like `base`, -`containers`, `Cabal`, etc. - -We can register additional libraries to GHC, however, using a special build -function called `ghcWithPackages`. That function expects one argument: a -function that maps from an attribute set of Haskell packages to a list of -packages, which determines the libraries known to that particular version of -GHC. For example, the Nix expression `ghcWithPackages (pkgs: [pkgs.mtl])` -generates a copy of GHC that has the `mtl` library registered in addition to -its normal core packages: -``` -$ nix-shell -p "haskellPackages.ghcWithPackages (pkgs: [pkgs.mtl])" - -[nix-shell:~]$ ghc-pkg list mtl -/nix/store/zy79...-ghc-7.10.2/lib/ghc-7.10.2/package.conf.d: - mtl-2.2.1 -``` - -This function allows users to define their own development environment by means -of an override. After adding the following snippet to `~/.config/nixpkgs/config.nix`, -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - myHaskellEnv = self.haskell.packages.ghc7102.ghcWithPackages - (haskellPackages: with haskellPackages; [ - # libraries - arrows async cgi criterion - # tools - cabal-install haskintex - ]); - }; -} -``` -it's possible to install that compiler with `nix-env -f "<nixpkgs>" -iA -myHaskellEnv`. If you'd like to switch that development environment to a -different version of GHC, just replace the `ghc7102` bit in the previous -definition with the appropriate name. Of course, it's also possible to define -any number of these development environments! (You can't install two of them -into the same profile at the same time, though, because that would result in -file conflicts.) - -The generated `ghc` program is a wrapper script that re-directs the real -GHC executable to use a new `lib` directory --- one that we specifically -constructed to contain all those packages the user requested: -``` -$ cat $(type -p ghc) -#! /nix/store/xlxj...-bash-4.3-p33/bin/bash -e -export NIX_GHC=/nix/store/19sm...-ghc-7.10.2/bin/ghc -export NIX_GHCPKG=/nix/store/19sm...-ghc-7.10.2/bin/ghc-pkg -export NIX_GHC_DOCDIR=/nix/store/19sm...-ghc-7.10.2/share/doc/ghc/html -export NIX_GHC_LIBDIR=/nix/store/19sm...-ghc-7.10.2/lib/ghc-7.10.2 -exec /nix/store/j50p...-ghc-7.10.2/bin/ghc "-B$NIX_GHC_LIBDIR" "$@" -``` - -The variables `$NIX_GHC`, `$NIX_GHCPKG`, etc. point to the *new* store path -`ghcWithPackages` constructed specifically for this environment. The last line -of the wrapper script then executes the real `ghc`, but passes the path to the -new `lib` directory using GHC's `-B` flag. - -The purpose of those environment variables is to work around an impurity in the -popular [ghc-paths](http://hackage.haskell.org/package/ghc-paths) library. That -library promises to give its users access to GHC's installation paths. Only, -the library can't possible know that path when it's compiled, because the path -GHC considers its own is determined only much later, when the user configures -it through `ghcWithPackages`. So we [patched -ghc-paths](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/patches/ghc-paths-nix.patch) -to return the paths found in those environment variables at run-time rather -than trying to guess them at compile-time. - -To make sure that mechanism works properly all the time, we recommend that you -set those variables to meaningful values in your shell environment, too, i.e. -by adding the following code to your `~/.bashrc`: -```bash -if type >/dev/null 2>&1 -p ghc; then - eval "$(egrep ^export "$(type -p ghc)")" -fi -``` - -If you are certain that you'll use only one GHC environment which is located in -your user profile, then you can use the following code, too, which has the -advantage that it doesn't contain any paths from the Nix store, i.e. those -settings always remain valid even if a `nix-env -u` operation updates the GHC -environment in your profile: -```bash -if [ -e ~/.nix-profile/bin/ghc ]; then - export NIX_GHC="$HOME/.nix-profile/bin/ghc" - export NIX_GHCPKG="$HOME/.nix-profile/bin/ghc-pkg" - export NIX_GHC_DOCDIR="$HOME/.nix-profile/share/doc/ghc/html" - export NIX_GHC_LIBDIR="$HOME/.nix-profile/lib/ghc-$($NIX_GHC --numeric-version)" -fi -``` - -### How to install a compiler with libraries, hoogle and documentation indexes - -If you plan to use your environment for interactive programming, not just -compiling random Haskell code, you might want to replace `ghcWithPackages` in -all the listings above with `ghcWithHoogle`. - -This environment generator not only produces an environment with GHC and all -the specified libraries, but also generates a `hoogle` and `haddock` indexes -for all the packages, and provides a wrapper script around `hoogle` binary that -uses all those things. A precise name for this thing would be -"`ghcWithPackagesAndHoogleAndDocumentationIndexes`", which is, regrettably, too -long and scary. - -For example, installing the following environment -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - myHaskellEnv = self.haskellPackages.ghcWithHoogle - (haskellPackages: with haskellPackages; [ - # libraries - arrows async cgi criterion - # tools - cabal-install haskintex - ]); - }; -} -``` -allows one to browse module documentation index [not too dissimilar to -this](https://downloads.haskell.org/~ghc/latest/docs/html/libraries/index.html) -for all the specified packages and their dependencies by directing a browser of -choice to `~/.nix-profile/share/doc/hoogle/index.html` (or -`/run/current-system/sw/share/doc/hoogle/index.html` in case you put it in -`environment.systemPackages` in NixOS). - -After you've marveled enough at that try adding the following to your -`~/.ghc/ghci.conf` -``` -:def hoogle \s -> return $ ":! hoogle search -cl --count=15 \"" ++ s ++ "\"" -:def doc \s -> return $ ":! hoogle search -cl --info \"" ++ s ++ "\"" -``` -and test it by typing into `ghci`: -``` -:hoogle a -> a -:doc a -> a -``` - -Be sure to note the links to `haddock` files in the output. With any modern and -properly configured terminal emulator you can just click those links to -navigate there. - -Finally, you can run -```shell -hoogle server --local -p 8080 -``` -and navigate to http://localhost:8080/ for your own local -[Hoogle](https://www.haskell.org/hoogle/). The `--local` flag makes the hoogle -server serve files from your nix store over http, without the flag it will use -`file://` URIs. Note, however, that Firefox and possibly other browsers -disallow navigation from `http://` to `file://` URIs for security reasons, -which might be quite an inconvenience. Versions before v5 did not have this -flag. See -[this page](http://kb.mozillazine.org/Links_to_local_pages_do_not_work) for -workarounds. - -For NixOS users there's a service which runs this exact command for you. -Specify the `packages` you want documentation for and the `haskellPackages` set -you want them to come from. Add the following to `configuration.nix`. - -```nix -services.hoogle = { - enable = true; - packages = (hpkgs: with hpkgs; [text cryptonite]); - haskellPackages = pkgs.haskellPackages; -}; -``` - -### How to build a Haskell project using Stack - -[Stack](http://haskellstack.org) is a popular build tool for Haskell projects. -It has first-class support for Nix. Stack can optionally use Nix to -automatically select the right version of GHC and other build tools to build, -test and execute apps in an existing project downloaded from somewhere on the -Internet. Pass the `--nix` flag to any `stack` command to do so, e.g. -```shell -git clone --recurse-submodules https://github.com/yesodweb/wai.git -cd wai -stack --nix build -``` - -If you want `stack` to use Nix by default, you can add a `nix` section to the -`stack.yaml` file, as explained in the [Stack documentation][stack-nix-doc]. For -example: -```yaml -nix: - enable: true - packages: [pkgconfig zeromq zlib] -``` - -The example configuration snippet above tells Stack to create an ad hoc -environment for `nix-shell` as in the below section, in which the `pkgconfig`, -`zeromq` and `zlib` packages from Nixpkgs are available. All `stack` commands -will implicitly be executed inside this ad hoc environment. - -Some projects have more sophisticated needs. For examples, some ad hoc -environments might need to expose Nixpkgs packages compiled in a certain way, or -with extra environment variables. In these cases, you'll need a `shell` field -instead of `packages`: -```yaml -nix: - enable: true - shell-file: shell.nix -``` - -For more on how to write a `shell.nix` file see the below section. You'll need -to express a derivation. Note that Nixpkgs ships with a convenience wrapper -function around `mkDerivation` called `haskell.lib.buildStackProject` to help you -create this derivation in exactly the way Stack expects. However for this to work -you need to disable the sandbox, which you can do by using `--option sandbox relaxed` -or `--option sandbox false` to the Nix command. All of the same inputs -as `mkDerivation` can be provided. For example, to build a Stack project that -including packages that link against a version of the R library compiled with -special options turned on: -```nix -with (import <nixpkgs> { }); - -let R = pkgs.R.override { enableStrictBarrier = true; }; -in -haskell.lib.buildStackProject { - name = "HaskellR"; - buildInputs = [ R zeromq zlib ]; -} -``` - -You can select a particular GHC version to compile with by setting the -`ghc` attribute as an argument to `buildStackProject`. Better yet, let -Stack choose what GHC version it wants based on the snapshot specified -in `stack.yaml` (only works with Stack >= 1.1.3): -```nix -{nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}: - -with nixpkgs; - -let R = pkgs.R.override { enableStrictBarrier = true; }; -in -haskell.lib.buildStackProject { - name = "HaskellR"; - buildInputs = [ R zeromq zlib ]; - inherit ghc; -} -``` - -[stack-nix-doc]: http://docs.haskellstack.org/en/stable/nix_integration.html - -### How to create ad hoc environments for `nix-shell` - -The easiest way to create an ad hoc development environment is to run -`nix-shell` with the appropriate GHC environment given on the command-line: -```shell -nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])" -``` - -For more sophisticated use-cases, however, it's more convenient to save the -desired configuration in a file called `shell.nix` that looks like this: -```nix -{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: -let - inherit (nixpkgs) pkgs; - ghc = pkgs.haskell.packages.${compiler}.ghcWithPackages (ps: with ps; [ - monad-par mtl - ]); -in -pkgs.stdenv.mkDerivation { - name = "my-haskell-env-0"; - buildInputs = [ ghc ]; - shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)"; -} -``` - -Now run `nix-shell` --- or even `nix-shell --pure` --- to enter a shell -environment that has the appropriate compiler in `$PATH`. If you use `--pure`, -then add all other packages that your development environment needs into the -`buildInputs` attribute. If you'd like to switch to a different compiler -version, then pass an appropriate `compiler` argument to the expression, i.e. -`nix-shell --argstr compiler ghc784`. - -If you need such an environment because you'd like to compile a Hackage package -outside of Nix --- i.e. because you're hacking on the latest version from Git ----, then the package set provides suitable nix-shell environments for you -already! Every Haskell package has an `env` attribute that provides a shell -environment suitable for compiling that particular package. If you'd like to -hack the `lens` library, for example, then you just have to check out the -source code and enter the appropriate environment: -``` -$ cabal get lens-4.11 && cd lens-4.11 -Downloading lens-4.11... -Unpacking to lens-4.11/ - -$ nix-shell "<nixpkgs>" -A haskellPackages.lens.env -[nix-shell:/tmp/lens-4.11]$ -``` - -At point, you can run `cabal configure`, `cabal build`, and all the other -development commands. Note that you need `cabal-install` installed in your -`$PATH` already to use it here --- the `nix-shell` environment does not provide -it. - -## How to create Nix builds for your own private Haskell packages - -If your own Haskell packages have build instructions for Cabal, then you can -convert those automatically into build instructions for Nix using the -`cabal2nix` utility, which you can install into your profile by running -`nix-env -i cabal2nix`. - -### How to build a stand-alone project - -For example, let's assume that you're working on a private project called -`foo`. To generate a Nix build expression for it, change into the project's -top-level directory and run the command: -```shell -cabal2nix . > foo.nix -``` -Then write the following snippet into a file called `default.nix`: -```nix -{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: -nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./foo.nix { } -``` - -Finally, store the following code in a file called `shell.nix`: -```nix -{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: -(import ./default.nix { inherit nixpkgs compiler; }).env -``` - -At this point, you can run `nix-build` to have Nix compile your project and -install it into a Nix store path. The local directory will contain a symlink -called `result` after `nix-build` returns that points into that location. Of -course, passing the flag `--argstr compiler ghc763` allows switching the build -to any version of GHC currently supported. - -Furthermore, you can call `nix-shell` to enter an interactive development -environment in which you can use `cabal configure` and `cabal build` to develop -your code. That environment will automatically contain a proper GHC derivation -with all the required libraries registered as well as all the system-level -libraries your package might need. - -If your package does not depend on any system-level libraries, then it's -sufficient to run -```shell -nix-shell --command "cabal configure" -``` -once to set up your build. `cabal-install` determines the absolute paths to all -resources required for the build and writes them into a config file in the -`dist/` directory. Once that's done, you can run `cabal build` and any other -command for that project even outside of the `nix-shell` environment. This -feature is particularly nice for those of us who like to edit their code with -an IDE, like Emacs' `haskell-mode`, because it's not necessary to start Emacs -inside of nix-shell just to make it find out the necessary settings for -building the project; `cabal-install` has already done that for us. - -If you want to do some quick-and-dirty hacking and don't want to bother setting -up a `default.nix` and `shell.nix` file manually, then you can use the -`--shell` flag offered by `cabal2nix` to have it generate a stand-alone -`nix-shell` environment for you. With that feature, running -```shell -cabal2nix --shell . > shell.nix -nix-shell --command "cabal configure" -``` -is usually enough to set up a build environment for any given Haskell package. -You can even use that generated file to run `nix-build`, too: -```shell -nix-build shell.nix -``` - -### How to build projects that depend on each other - -If you have multiple private Haskell packages that depend on each other, then -you'll have to register those packages in the Nixpkgs set to make them visible -for the dependency resolution performed by `callPackage`. First of all, change -into each of your projects top-level directories and generate a `default.nix` -file with `cabal2nix`: -```shell -cd ~/src/foo && cabal2nix . > default.nix -cd ~/src/bar && cabal2nix . > default.nix -``` -Then edit your `~/.config/nixpkgs/config.nix` file to register those builds in the -default Haskell package set: -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - haskellPackages = super.haskellPackages.override { - overrides = self: super: { - foo = self.callPackage ../src/foo {}; - bar = self.callPackage ../src/bar {}; - }; - }; - }; -} -``` -Once that's accomplished, `nix-env -f "<nixpkgs>" -qA haskellPackages` will -show your packages like any other package from Hackage, and you can build them -```shell -nix-build "<nixpkgs>" -A haskellPackages.foo -``` -or enter an interactive shell environment suitable for building them: -```shell -nix-shell "<nixpkgs>" -A haskellPackages.bar.env -``` - -## Miscellaneous Topics - -### How to build with profiling enabled - -Every Haskell package set takes a function called `overrides` that you can use -to manipulate the package as much as you please. One useful application of this -feature is to replace the default `mkDerivation` function with one that enables -library profiling for all packages. To accomplish that add the following -snippet to your `~/.config/nixpkgs/config.nix` file: -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - profiledHaskellPackages = self.haskellPackages.override { - overrides = self: super: { - mkDerivation = args: super.mkDerivation (args // { - enableLibraryProfiling = true; - }); - }; - }; - }; -} -``` -Then, replace instances of `haskellPackages` in the `cabal2nix`-generated -`default.nix` or `shell.nix` files with `profiledHaskellPackages`. - -### How to override package versions in a compiler-specific package set -Nixpkgs provides the latest version of -[`ghc-events`](http://hackage.haskell.org/package/ghc-events), which is 0.4.4.0 -at the time of this writing. This is fine for users of GHC 7.10.x, but GHC -7.8.4 cannot compile that binary. Now, one way to solve that problem is to -register an older version of `ghc-events` in the 7.8.x-specific package set. -The first step is to generate Nix build instructions with `cabal2nix`: -```shell -cabal2nix cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix -``` -Then add the override in `~/.config/nixpkgs/config.nix`: -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - haskell = super.haskell // { - packages = super.haskell.packages // { - ghc784 = super.haskell.packages.ghc784.override { - overrides = self: super: { - ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {}; - }; - }; - }; - }; - }; -} -``` - -This code is a little crazy, no doubt, but it's necessary because the intuitive -version -```nix -{ # ... - - haskell.packages.ghc784 = super.haskell.packages.ghc784.override { - overrides = self: super: { - ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {}; - }; - }; -} -``` -doesn't do what we want it to: that code replaces the `haskell` package set in -Nixpkgs with one that contains only one entry,`packages`, which contains only -one entry `ghc784`. This override loses the `haskell.compiler` set, and it -loses the `haskell.packages.ghcXYZ` sets for all compilers but GHC 7.8.4. To -avoid that problem, we have to perform the convoluted little dance from above, -iterating over each step in hierarchy. - -Once it's accomplished, however, we can install a variant of `ghc-events` -that's compiled with GHC 7.8.4: -```shell -nix-env -f "<nixpkgs>" -iA haskell.packages.ghc784.ghc-events -``` -Unfortunately, it turns out that this build fails again while executing the -test suite! Apparently, the release archive on Hackage is missing some data -files that the test suite requires, so we cannot run it. We accomplish that by -re-generating the Nix expression with the `--no-check` flag: -```shell -cabal2nix --no-check cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix -``` -Now the builds succeeds. - -Of course, in the concrete example of `ghc-events` this whole exercise is not -an ideal solution, because `ghc-events` can analyze the output emitted by any -version of GHC later than 6.12 regardless of the compiler version that was used -to build the `ghc-events` executable, so strictly speaking there's no reason to -prefer one built with GHC 7.8.x in the first place. However, for users who -cannot use GHC 7.10.x at all for some reason, the approach of downgrading to an -older version might be useful. - -### How to override packages in all compiler-specific package sets - -In the previous section we learned how to override a package in a single -compiler-specific package set. You may have some overrides defined that you want -to use across multiple package sets. To accomplish this you could use the -technique that we learned in the previous section by repeating the overrides for -all the compiler-specific package sets. For example: - -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - haskell = super.haskell // { - packages = super.haskell.packages // { - ghc784 = super.haskell.packages.ghc784.override { - overrides = self: super: { - my-package = ...; - my-other-package = ...; - }; - }; - ghc822 = super.haskell.packages.ghc784.override { - overrides = self: super: { - my-package = ...; - my-other-package = ...; - }; - }; - ... - }; - }; - }; -} -``` - -However there's a more convenient way to override all compiler-specific package -sets at once: - -```nix -{ - packageOverrides = super: let self = super.pkgs; in - { - haskell = super.haskell // { - packageOverrides = self: super: { - my-package = ...; - my-other-package = ...; - }; - }; - }; -} -``` - -### How to specify source overrides for your Haskell package - -When starting a Haskell project you can use `developPackage` -to define a derivation for your package at the `root` path -as well as source override versions for Hackage packages, like so: - -```nix -# default.nix -{ compilerVersion ? "ghc842" }: -let - # pinning nixpkgs using new Nix 2.0 builtin `fetchGit` - pkgs = import (fetchGit (import ./version.nix)) { }; - compiler = pkgs.haskell.packages."${compilerVersion}"; - pkg = compiler.developPackage { - root = ./.; - source-overrides = { - # Let's say the GHC 8.4.2 haskellPackages uses 1.6.0.0 and your test suite is incompatible with >= 1.6.0.0 - HUnit = "1.5.0.0"; - }; - }; -in pkg -``` - -This could be used in place of a simplified `stack.yaml` defining a Nix -derivation for your Haskell package. - -As you can see this allows you to specify only the source version found on -Hackage and nixpkgs will take care of the rest. - -You can also specify `buildInputs` for your Haskell derivation for packages -that directly depend on external libraries like so: - -```nix -# default.nix -{ compilerVersion ? "ghc842" }: -let - # pinning nixpkgs using new Nix 2.0 builtin `fetchGit` - pkgs = import (fetchGit (import ./version.nix)) { }; - compiler = pkgs.haskell.packages."${compilerVersion}"; - pkg = compiler.developPackage { - root = ./.; - source-overrides = { - HUnit = "1.5.0.0"; # Let's say the GHC 8.4.2 haskellPackages uses 1.6.0.0 and your test suite is incompatible with >= 1.6.0.0 - }; - }; - # in case your package source depends on any libraries directly, not just transitively. - buildInputs = [ zlib ]; -in pkg.overrideAttrs(attrs: { - buildInputs = attrs.buildInputs ++ buildInputs; -}) -``` - -Notice that you will need to override (via `overrideAttrs` or similar) the -derivation returned by the `developPackage` Nix lambda as there is no `buildInputs` -named argument you can pass directly into the `developPackage` lambda. - -### How to recover from GHC's infamous non-deterministic library ID bug - -GHC and distributed build farms don't get along well: - - - https://ghc.haskell.org/trac/ghc/ticket/4012 - -When you see an error like this one -``` -package foo-0.7.1.0 is broken due to missing package -text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91 -``` -then you have to download and re-install `foo` and all its dependents from -scratch: -```shell -nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \ - | xargs -L 1 nix-store --repair-path -``` - -If you're using additional Hydra servers other than `hydra.nixos.org`, then it -might be necessary to purge the local caches that store data from those -machines to disable these binary channels for the duration of the previous -command, i.e. by running: -```shell -rm ~/.cache/nix/binary-cache*.sqlite -``` - -### Builds on Darwin fail with `math.h` not found - -Users of GHC on Darwin have occasionally reported that builds fail, because the -compiler complains about a missing include file: -``` -fatal error: 'math.h' file not found -``` -The issue has been discussed at length in [ticket -6390](https://github.com/NixOS/nixpkgs/issues/6390), and so far no good -solution has been proposed. As a work-around, users who run into this problem -can configure the environment variables -```shell -export NIX_CFLAGS_COMPILE="-idirafter /usr/include" -export NIX_CFLAGS_LINK="-L/usr/lib" -``` -in their `~/.bashrc` file to avoid the compiler error. - -### Builds using Stack complain about missing system libraries - -``` --- While building package zlib-0.5.4.2 using: - runhaskell -package=Cabal-1.22.4.0 -clear-package-db [... lots of flags ...] -Process exited with code: ExitFailure 1 -Logs have been written to: /home/foo/src/stack-ide/.stack-work/logs/zlib-0.5.4.2.log - -Configuring zlib-0.5.4.2... -Setup.hs: Missing dependency on a foreign library: -* Missing (or bad) header file: zlib.h -This problem can usually be solved by installing the system package that -provides this library (you may need the "-dev" version). If the library is -already installed but in a non-standard location then you can use the flags ---extra-include-dirs= and --extra-lib-dirs= to specify where it is. -If the header file does exist, it may contain errors that are caught by the C -compiler at the preprocessing stage. In this case you can re-run configure -with the verbosity flag -v3 to see the error messages. -``` - -When you run the build inside of the nix-shell environment, the system -is configured to find `libz.so` without any special flags -- the compiler -and linker "just know" how to find it. Consequently, Cabal won't record -any search paths for `libz.so` in the package description, which means -that the package works fine inside of nix-shell, but once you leave the -shell the shared object can no longer be found. That issue is by no -means specific to Stack: you'll have that problem with any other -Haskell package that's built inside of nix-shell but run outside of that -environment. - -You can remedy this issue in several ways. The easiest is to add a `nix` section -to the `stack.yaml` like the following: -```yaml -nix: - enable: true - packages: [ zlib ] -``` - -Stack's Nix support knows to add `${zlib.out}/lib` and `${zlib.dev}/include` -as an `--extra-lib-dirs` and `extra-include-dirs`, respectively. -Alternatively, you can achieve the same effect by hand. First of all, run -``` -$ nix-build --no-out-link "<nixpkgs>" -A zlib -/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8 -``` -to find out the store path of the system's zlib library. Now, you can - - 1. add that path (plus a "/lib" suffix) to your `$LD_LIBRARY_PATH` - environment variable to make sure your system linker finds `libz.so` - automatically. It's no pretty solution, but it will work. - - 2. As a variant of (1), you can also install any number of system - libraries into your user's profile (or some other profile) and point - `$LD_LIBRARY_PATH` to that profile instead, so that you don't have to - list dozens of those store paths all over the place. - - 3. The solution I prefer is to call stack with an appropriate - --extra-lib-dirs flag like so: - ```shell - stack --extra-lib-dirs=/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8/lib build - ``` - -Typically, you'll need `--extra-include-dirs` as well. It's possible -to add those flag to the project's `stack.yaml` or your user's -global `~/.stack/global/stack.yaml` file so that you don't have to -specify them manually every time. But again, you're likely better off -using Stack's Nix support instead. - -The same thing applies to `cabal configure`, of course, if you're -building with `cabal-install` instead of Stack. - -### Creating statically linked binaries - -There are two levels of static linking. The first option is to configure the -build with the Cabal flag `--disable-executable-dynamic`. In Nix expressions, -this can be achieved by setting the attribute: -``` -enableSharedExecutables = false; -``` -That gives you a binary with statically linked Haskell libraries and -dynamically linked system libraries. - -To link both Haskell libraries and system libraries statically, the additional -flags `--ghc-option=-optl=-static --ghc-option=-optl=-pthread` need to be used. -In Nix, this is accomplished with: -``` -configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ]; -``` - -It's important to realize, however, that most system libraries in Nix are -built as shared libraries only, i.e. there is just no static library -available that Cabal could link! - -### Building GHC with integer-simple - -By default GHC implements the Integer type using the -[GNU Multiple Precision Arithmetic (GMP) library](https://gmplib.org/). -The implementation can be found in the -[integer-gmp](http://hackage.haskell.org/package/integer-gmp) package. - -A potential problem with this is that GMP is licensed under the -[GNU Lesser General Public License (LGPL)](https://www.gnu.org/copyleft/lesser.html), -a kind of "copyleft" license. According to the terms of the LGPL, paragraph 5, -you may distribute a program that is designed to be compiled and dynamically -linked with the library under the terms of your choice (i.e., commercially) but -if your program incorporates portions of the library, if it is linked -statically, then your program is a "derivative"--a "work based on the -library"--and according to paragraph 2, section c, you "must cause the whole of -the work to be licensed" under the terms of the LGPL (including for free). - -The LGPL licensing for GMP is a problem for the overall licensing of binary -programs compiled with GHC because most distributions (and builds) of GHC use -static libraries. (Dynamic libraries are currently distributed only for macOS.) -The LGPL licensing situation may be worse: even though -[The Glasgow Haskell Compiler License](https://www.haskell.org/ghc/license) -is essentially a "free software" license (BSD3), according to -paragraph 2 of the LGPL, GHC must be distributed under the terms of the LGPL! - -To work around these problems GHC can be build with a slower but LGPL-free -alternative implementation for Integer called -[integer-simple](http://hackage.haskell.org/package/integer-simple). - -To get a GHC compiler build with `integer-simple` instead of `integer-gmp` use -the attribute: `haskell.compiler.integer-simple."${ghcVersion}"`. -For example: -``` -$ nix-build -E '(import <nixpkgs> {}).haskell.compiler.integer-simple.ghc802' -... -$ result/bin/ghc-pkg list | grep integer - integer-simple-0.1.1.1 -``` -The following command displays the complete list of GHC compilers build with `integer-simple`: -``` -$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler.integer-simple -haskell.compiler.integer-simple.ghc7102 ghc-7.10.2 -haskell.compiler.integer-simple.ghc7103 ghc-7.10.3 -haskell.compiler.integer-simple.ghc722 ghc-7.2.2 -haskell.compiler.integer-simple.ghc742 ghc-7.4.2 -haskell.compiler.integer-simple.ghc783 ghc-7.8.3 -haskell.compiler.integer-simple.ghc784 ghc-7.8.4 -haskell.compiler.integer-simple.ghc801 ghc-8.0.1 -haskell.compiler.integer-simple.ghc802 ghc-8.0.2 -haskell.compiler.integer-simple.ghcHEAD ghc-8.1.20170106 -``` - -To get a package set supporting `integer-simple` use the attribute: -`haskell.packages.integer-simple."${ghcVersion}"`. For example -use the following to get the `scientific` package build with `integer-simple`: -```shell -nix-build -A haskell.packages.integer-simple.ghc802.scientific -``` - -### Quality assurance - -The `haskell.lib` library includes a number of functions for checking for -various imperfections in Haskell packages. It's useful to apply these functions -to your own Haskell packages and integrate that in a Continuous Integration -server like [hydra](https://nixos.org/hydra/) to assure your packages maintain a -minimum level of quality. This section discusses some of these functions. - -#### failOnAllWarnings - -Applying `haskell.lib.failOnAllWarnings` to a Haskell package enables the -`-Wall` and `-Werror` GHC options to turn all warnings into build failures. - -#### buildStrictly - -Applying `haskell.lib.buildStrictly` to a Haskell package calls -`failOnAllWarnings` on the given package to turn all warnings into build -failures. Additionally the source of your package is gotten from first invoking -`cabal sdist` to ensure all needed files are listed in the Cabal file. - -#### checkUnusedPackages - -Applying `haskell.lib.checkUnusedPackages` to a Haskell package invokes -the [packunused](http://hackage.haskell.org/package/packunused) tool on the -package. `packunused` complains when it finds packages listed as build-depends -in the Cabal file which are redundant. For example: - -``` -$ nix-build -E 'let pkgs = import <nixpkgs> {}; in pkgs.haskell.lib.checkUnusedPackages {} pkgs.haskellPackages.scientific' -these derivations will be built: - /nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv -... -detected package components -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - - library - - testsuite(s): test-scientific - - benchmark(s): bench-scientific* - -(component names suffixed with '*' are not configured to be built) - -library -~~~~~~~ - -The following package dependencies seem redundant: - - - ghc-prim-0.5.0.0 - -testsuite(test-scientific) -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -no redundant packages dependencies found - -builder for ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed with exit code 1 -error: build of ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed -``` - -As you can see, `packunused` finds out that although the testsuite component has -no redundant dependencies the library component of `scientific-0.3.5.1` depends -on `ghc-prim` which is unused in the library. - -### Using hackage2nix with nixpkgs - -Hackage package derivations are found in the -[`hackage-packages.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix) -file within `nixpkgs` and are used as the initial package set for -`haskellPackages`. The `hackage-packages.nix` file is not meant to be edited -by hand, but rather autogenerated by [`hackage2nix`](https://github.com/NixOS/cabal2nix/tree/master/hackage2nix), -which by default uses the [`configuration-hackage2nix.yaml`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml) -file to generate all the derivations. - -To modify the contents `configuration-hackage2nix.yaml`, follow the -instructions on [`hackage2nix`](https://github.com/NixOS/cabal2nix/tree/master/hackage2nix). - -## Other resources - - - The Youtube video [Nix Loves Haskell](https://www.youtube.com/watch?v=BsBhi_r-OeE) - provides an introduction into Haskell NG aimed at beginners. The slides are - available at http://cryp.to/nixos-meetup-3-slides.pdf and also -- in a form - ready for cut & paste -- at - https://github.com/NixOS/cabal2nix/blob/master/doc/nixos-meetup-3-slides.md. - - - Another Youtube video is [Escaping Cabal Hell with Nix](https://www.youtube.com/watch?v=mQd3s57n_2Y), - which discusses the subject of Haskell development with Nix but also provides - a basic introduction to Nix as well, i.e. it's suitable for viewers with - almost no prior Nix experience. - - - Oliver Charles wrote a very nice [Tutorial how to develop Haskell packages with Nix](http://wiki.ocharles.org.uk/Nix). - - - The *Journey into the Haskell NG infrastructure* series of postings - describe the new Haskell infrastructure in great detail: - - - [Part 1](https://nixos.org/nix-dev/2015-January/015591.html) - explains the differences between the old and the new code and gives - instructions how to migrate to the new setup. - - - [Part 2](https://nixos.org/nix-dev/2015-January/015608.html) - looks in-depth at how to tweak and configure your setup by means of - overrides. +# Haskell - - [Part 3](https://nixos.org/nix-dev/2015-April/016912.html) - describes the infrastructure that keeps the Haskell package set in Nixpkgs - up-to-date. +The documentation for the Haskell infrastructure is published at +<https://haskell4nix.readthedocs.io/>. The source code for that +site lives in the `doc/` sub-directory of the +[`cabal2nix` Git repository](https://github.com/NixOS/cabal2nix) +and changes can be submitted there. diff --git a/doc/languages-frameworks/index.xml b/doc/languages-frameworks/index.xml index 728a38c264a..3366108cd93 100644 --- a/doc/languages-frameworks/index.xml +++ b/doc/languages-frameworks/index.xml @@ -19,6 +19,7 @@ <xi:include href="ios.section.xml" /> <xi:include href="java.xml" /> <xi:include href="lua.section.xml" /> + <xi:include href="maven.section.xml" /> <xi:include href="node.section.xml" /> <xi:include href="ocaml.xml" /> <xi:include href="perl.xml" /> diff --git a/doc/languages-frameworks/java.xml b/doc/languages-frameworks/java.xml index bf0fc488392..881d492b5bf 100644 --- a/doc/languages-frameworks/java.xml +++ b/doc/languages-frameworks/java.xml @@ -32,7 +32,7 @@ nativeBuildInputs = [ jdk ]; </para> <para> - If your Java package provides a program, you need to generate a wrapper script to run it using the OpenJRE. You can use <literal>makeWrapper</literal> for this: + If your Java package provides a program, you need to generate a wrapper script to run it using a JRE. You can use <literal>makeWrapper</literal> for this: <programlisting> nativeBuildInputs = [ makeWrapper ]; @@ -43,7 +43,21 @@ installPhase = --add-flags "-cp $out/share/java/foo.jar org.foo.Main" ''; </programlisting> - Note the use of <literal>jre</literal>, which is the part of the OpenJDK package that contains the Java Runtime Environment. By using <literal>${jre}/bin/java</literal> instead of <literal>${jdk}/bin/java</literal>, you prevent your package from depending on the JDK at runtime. +Since the introduction of the Java Platform Module System in Java 9, Java distributions typically no longer ship with a general-purpose JRE: instead, they allow generating a JRE with only the modules required for your application(s). Because we can't predict what modules will be needed on a general-purpose system, the default <package>jre</package> package is the full JDK. When building a minimal system/image, you can override the <literal>modules</literal> parameter on <literal>jre_minimal</literal> to build a JRE with only the modules relevant for you: +<programlisting> +let + my_jre = pkgs.jre_minimal.override { + modules = [ + # The modules used by 'something' and 'other' combined: + "java.base" + "java.logging" + ]; + }; + something = (pkgs.something.override { jre = my_jre; }); + other = (pkgs.other.override { jre = my_jre; }); +in + ... +</programlisting> </para> <para> diff --git a/doc/languages-frameworks/maven.section.md b/doc/languages-frameworks/maven.section.md new file mode 100644 index 00000000000..fe183e7ba3c --- /dev/null +++ b/doc/languages-frameworks/maven.section.md @@ -0,0 +1,354 @@ +--- +title: Maven +author: Farid Zakaria +date: 2020-10-15 +--- + +# Maven + +Maven is a well-known build tool for the Java ecosystem however it has some challenges when integrating into the Nix build system. + +The following provides a list of common patterns with how to package a Maven project (or any JVM language that can export to Maven) as a Nix package. + +For the purposes of this example let's consider a very basic Maven project with the following `pom.xml` with a single dependency on [emoji-java](https://github.com/vdurmont/emoji-java). + +```xml +<?xml version="1.0" encoding="UTF-8"?> +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <modelVersion>4.0.0</modelVersion> + <groupId>io.github.fzakaria</groupId> + <artifactId>maven-demo</artifactId> + <version>1.0</version> + <packaging>jar</packaging> + <name>NixOS Maven Demo</name> + + <dependencies> + <dependency> + <groupId>com.vdurmont</groupId> + <artifactId>emoji-java</artifactId> + <version>5.1.1</version> + </dependency> + </dependencies> +</project> +``` + +Our main class file will be very simple: + +```java +import com.vdurmont.emoji.EmojiParser; + +public class Main { + public static void main(String[] args) { + String str = "NixOS :grinning: is super cool :smiley:!"; + String result = EmojiParser.parseToUnicode(str); + System.out.println(result); + } +} +``` + +You find this demo project at https://github.com/fzakaria/nixos-maven-example + +## Solving for dependencies + +### buildMaven with NixOS/mvn2nix-maven-plugin + +> ⚠️ Although `buildMaven` is the "blessed" way within nixpkgs, as of 2020, it hasn't seen much activity in quite a while. + +`buildMaven` is an alternative method that tries to follow similar patterns of other programming languages by generating a lock file. It relies on the maven plugin [mvn2nix-maven-plugin](https://github.com/NixOS/mvn2nix-maven-plugin). + +First you generate a `project-info.json` file using the maven plugin. + +> This should be executed in the project's source repository or be told which `pom.xml` to execute with. + +```bash +# run this step within the project's source repository +❯ mvn org.nixos.mvn2nix:mvn2nix-maven-plugin:mvn2nix + +❯ cat project-info.json | jq | head +{ + "project": { + "artifactId": "maven-demo", + "groupId": "org.nixos", + "version": "1.0", + "classifier": "", + "extension": "jar", + "dependencies": [ + { + "artifactId": "maven-resources-plugin", +``` + +This file is then given to the `buildMaven` function, and it returns 2 attributes. + +**`repo`**: + A Maven repository that is a symlink farm of all the dependencies found in the `project-info.json` + + +**`build`**: + A simple derivation that runs through `mvn compile` & `mvn package` to build the JAR. You may use this as inspiration for more complicated derivations. + +Here is an [example](https://github.com/fzakaria/nixos-maven-example/blob/main/build-maven-repository.nix) of building the Maven repository +```nix +{ pkgs ? import <nixpkgs> { } }: +with pkgs; +(buildMaven ./project-info.json).repo +``` + +The benefit over the _double invocation_ as we will see below, is that the _/nix/store_ entry is a _linkFarm_ of every package, so that changes to your dependency set doesn't involve downloading everything from scratch. + +```bash +❯ tree $(nix-build --no-out-link build-maven-repository.nix) | head +/nix/store/g87va52nkc8jzbmi1aqdcf2f109r4dvn-maven-repository +├── antlr +│ └── antlr +│ └── 2.7.2 +│ ├── antlr-2.7.2.jar -> /nix/store/d027c8f2cnmj5yrynpbq2s6wmc9cb559-antlr-2.7.2.jar +│ └── antlr-2.7.2.pom -> /nix/store/mv42fc5gizl8h5g5vpywz1nfiynmzgp2-antlr-2.7.2.pom +├── avalon-framework +│ └── avalon-framework +│ └── 4.1.3 +│ ├── avalon-framework-4.1.3.jar -> /nix/store/iv5fp3955w3nq28ff9xfz86wvxbiw6n9-avalon-framework-4.1.3.jar +``` +### Double Invocation + +> ⚠️ This pattern is the simplest but may cause unnecessary rebuilds due to the output hash changing. + +The double invocation is a _simple_ way to get around the problem that `nix-build` may be sandboxed and have no Internet connectivity. + +It treats the entire Maven repository as a single source to be downloaded, relying on Maven's dependency resolution to satisfy the output hash. This is similar to fetchers like `fetchgit`, except it has to run a Maven build to determine what to download. + +The first step will be to build the Maven project as a fixed-output derivation in order to collect the Maven repository -- below is an [example](https://github.com/fzakaria/nixos-maven-example/blob/main/double-invocation-repository.nix). + +> Traditionally the Maven repository is at `~/.m2/repository`. We will override this to be the `$out` directory. + +```nix +{ stdenv, maven }: +stdenv.mkDerivation { + name = "maven-repository"; + buildInputs = [ maven ]; + src = ./.; # or fetchFromGitHub, cleanSourceWith, etc + buildPhase = '' + mvn package -Dmaven.repo.local=$out + ''; + + # keep only *.{pom,jar,sha1,nbm} and delete all ephemeral files with lastModified timestamps inside + installPhase = '' + find $out -type f \ + -name \*.lastUpdated -or \ + -name resolver-status.properties -or \ + -name _remote.repositories \ + -delete + ''; + + # don't do any fixup + dontFixup = true; + outputHashAlgo = "sha256"; + outputHashMode = "recursive"; + # replace this with the correct SHA256 + outputHash = stdenv.lib.fakeSha256; +} +``` + +The build will fail, and tell you the expected `outputHash` to place. When you've set the hash, the build will return with a `/nix/store` entry whose contents are the full Maven repository. + +> Some additional files are deleted that would cause the output hash to change potentially on subsequent runs. + +```bash +❯ tree $(nix-build --no-out-link double-invocation-repository.nix) | head +/nix/store/8kicxzp98j68xyi9gl6jda67hp3c54fq-maven-repository +├── backport-util-concurrent +│ └── backport-util-concurrent +│ └── 3.1 +│ ├── backport-util-concurrent-3.1.pom +│ └── backport-util-concurrent-3.1.pom.sha1 +├── classworlds +│ └── classworlds +│ ├── 1.1 +│ │ ├── classworlds-1.1.jar +``` + +If your package uses _SNAPSHOT_ dependencies or _version ranges_; there is a strong likelihood that over-time your output hash will change since the resolved dependencies may change. Hence this method is less recommended then using `buildMaven`. + +## Building a JAR + +Regardless of which strategy is chosen above, the step to build the derivation is the same. + +```nix +{ stdenv, lib, maven, callPackage }: +# pick a repository derivation, here we will use buildMaven +let repository = callPackage ./build-maven-repository.nix { }; +in stdenv.mkDerivation rec { + pname = "maven-demo"; + version = "1.0"; + + src = builtins.fetchTarball "https://github.com/fzakaria/nixos-maven-example/archive/main.tar.gz"; + buildInputs = [ maven ]; + + buildPhase = '' + echo "Using repository ${repository}" + mvn --offline -Dmaven.repo.local=${repository} package; + ''; + + installPhase = '' + install -Dm644 target/${pname}-${version}.jar $out/share/java + ''; +} +``` + +> We place the library in `$out/share/java` since JDK package has a _stdenv setup hook_ that adds any JARs in the `share/java` directories of the build inputs to the CLASSPATH environment. + +```bash +❯ tree $(nix-build --no-out-link build-jar.nix) +/nix/store/7jw3xdfagkc2vw8wrsdv68qpsnrxgvky-maven-demo-1.0 +└── share + └── java + └── maven-demo-1.0.jar + +2 directories, 1 file +``` + +## Runnable JAR + +The previous example builds a `jar` file but that's not a file one can run. + +You need to use it with `java -jar $out/share/java/output.jar` and make sure to provide the required dependencies on the classpath. + +The following explains how to use `makeWrapper` in order to make the derivation produce an executable that will run the JAR file you created. + +We will use the same repository we built above (either _double invocation_ or _buildMaven_) to setup a CLASSPATH for our JAR. + +The following two methods are more suited to Nix then building an [UberJar](https://imagej.net/Uber-JAR) which may be the more traditional approach. + +### CLASSPATH + +> This is ideal if you are providing a derivation for _nixpkgs_ and don't want to patch the project's `pom.xml`. + +We will read the Maven repository and flatten it to a single list. This list will then be concatenated with the _CLASSPATH_ separator to create the full classpath. + +We make sure to provide this classpath to the `makeWrapper`. + +```nix +{ stdenv, lib, maven, callPackage, makeWrapper, jre }: +let + repository = callPackage ./build-maven-repository.nix { }; +in stdenv.mkDerivation rec { + pname = "maven-demo"; + version = "1.0"; + + src = builtins.fetchTarball + "https://github.com/fzakaria/nixos-maven-example/archive/main.tar.gz"; + buildInputs = [ maven makeWrapper ]; + + buildPhase = '' + echo "Using repository ${repository}" + mvn --offline -Dmaven.repo.local=${repository} package; + ''; + + installPhase = '' + mkdir -p $out/bin + + classpath=$(find ${repository} -name "*.jar" -printf ':%h/%f'); + install -Dm644 target/${pname}-${version}.jar $out/share/java + # create a wrapper that will automatically set the classpath + # this should be the paths from the dependency derivation + makeWrapper ${jre}/bin/java $out/bin/${pname} \ + --add-flags "-classpath $out/share/java/${pname}-${version}.jar:''${classpath#:}" \ + --add-flags "Main" + ''; +} +``` + +### MANIFEST file via Maven Plugin + +> This is ideal if you are the project owner and want to change your `pom.xml` to set the CLASSPATH within it. + +Augment the `pom.xml` to create a JAR with the following manifest: +```xml +<build> + <plugins> + <plugin> + <artifactId>maven-jar-plugin</artifactId> + <configuration> + <archive> + <manifest> + <addClasspath>true</addClasspath> + <classpathPrefix>../../repository/</classpathPrefix> + <classpathLayoutType>repository</classpathLayoutType> + <mainClass>Main</mainClass> + </manifest> + <manifestEntries> + <Class-Path>.</Class-Path> + </manifestEntries> + </archive> + </configuration> + </plugin> + </plugins> +</build> +``` + +The above plugin instructs the JAR to look for the necessary dependencies in the `lib/` relative folder. The layout of the folder is also in the _maven repository_ style. + +```bash +❯ unzip -q -c $(nix-build --no-out-link runnable-jar.nix)/share/java/maven-demo-1.0.jar META-INF/MANIFEST.MF + +Manifest-Version: 1.0 +Archiver-Version: Plexus Archiver +Built-By: nixbld +Class-Path: . ../../repository/com/vdurmont/emoji-java/5.1.1/emoji-jav + a-5.1.1.jar ../../repository/org/json/json/20170516/json-20170516.jar +Created-By: Apache Maven 3.6.3 +Build-Jdk: 1.8.0_265 +Main-Class: Main +``` + +We will modify the derivation above to add a symlink to our repository so that it's accessible to our JAR during the `installPhase`. + +```nix +{ stdenv, lib, maven, callPackage, makeWrapper, jre }: +# pick a repository derivation, here we will use buildMaven +let repository = callPackage ./build-maven-repository.nix { }; +in stdenv.mkDerivation rec { + pname = "maven-demo"; + version = "1.0"; + + src = builtins.fetchTarball + "https://github.com/fzakaria/nixos-maven-example/archive/main.tar.gz"; + buildInputs = [ maven makeWrapper ]; + + buildPhase = '' + echo "Using repository ${repository}" + mvn --offline -Dmaven.repo.local=${repository} package; + ''; + + installPhase = '' + mkdir -p $out/bin + + # create a symbolic link for the repository directory + ln -s ${repository} $out/repository + + install -Dm644 target/${pname}-${version}.jar $out/share/java + # create a wrapper that will automatically set the classpath + # this should be the paths from the dependency derivation + makeWrapper ${jre}/bin/java $out/bin/${pname} \ + --add-flags "-jar $out/share/java/${pname}-${version}.jar" + ''; +} +``` + +> Our script produces a dependency on `jre` rather than `jdk` to restrict the runtime closure necessary to run the application. + +This will give you an executable shell-script that launches your JAR with all the dependencies available. + +```bash +❯ tree $(nix-build --no-out-link runnable-jar.nix) +/nix/store/8d4c3ibw8ynsn01ibhyqmc1zhzz75s26-maven-demo-1.0 +├── bin +│ └── maven-demo +├── repository -> /nix/store/g87va52nkc8jzbmi1aqdcf2f109r4dvn-maven-repository +└── share + └── java + └── maven-demo-1.0.jar + +❯ $(nix-build --no-out-link --option tarball-ttl 1 runnable-jar.nix)/bin/maven-demo +NixOS 😀 is super cool 😃! +``` diff --git a/doc/languages-frameworks/node.section.md b/doc/languages-frameworks/node.section.md index c1f4294711a..2120adfc0b4 100644 --- a/doc/languages-frameworks/node.section.md +++ b/doc/languages-frameworks/node.section.md @@ -25,12 +25,13 @@ build system it uses. Here are some examples: After you have identified the correct system, you need to override your package expression while adding in build system as a build input. For example, `dat` -requires `node-gyp-build`, so we override its expression in `default.nix`: +requires `node-gyp-build`, so [we override](https://github.com/NixOS/nixpkgs/blob/32f5e5da4a1b3f0595527f5195ac3a91451e9b56/pkgs/development/node-packages/default.nix#L37-L40) its expression in [`default.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix): ```nix -dat = nodePackages.dat.override (oldAttrs: { - buildInputs = oldAttrs.buildInputs ++ [ nodePackages.node-gyp-build ]; -}); + dat = super.dat.override { + buildInputs = [ self.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ]; + meta.broken = since "12"; + }; ``` To add a package from NPM to nixpkgs: diff --git a/doc/languages-frameworks/perl.xml b/doc/languages-frameworks/perl.xml index ff0f350e99c..b017c028f64 100644 --- a/doc/languages-frameworks/perl.xml +++ b/doc/languages-frameworks/perl.xml @@ -8,28 +8,28 @@ <para> When executing a Perl script, it is possible you get an error such as <literal>./myscript.pl: bad interpreter: /usr/bin/perl: no such file or directory</literal>. This happens when the script expects Perl to be installed at <filename>/usr/bin/perl</filename>, which is not the case when using Perl from nixpkgs. You can fix the script by changing the first line to: - <programlisting> - #!/usr/bin/env perl - </programlisting> +<programlisting> +#!/usr/bin/env perl +</programlisting> to take the Perl installation from the <literal>PATH</literal> environment variable, or invoke Perl directly with: - <screen> - <prompt>$ </prompt>perl ./myscript.pl - </screen> +<screen> +<prompt>$ </prompt>perl ./myscript.pl +</screen> </para> <para> When the script is using a Perl library that is not installed globally, you might get an error such as <literal>Can't locate DB_File.pm in @INC (you may need to install the DB_File module)</literal>. In that case, you can use <command>nix-shell</command> to start an ad-hoc shell with that library installed, for instance: - <screen> - <prompt>$ </prompt>nix-shell -p perl perlPackages.DBFile --run ./myscript.pl - </screen> +<screen> +<prompt>$ </prompt>nix-shell -p perl perlPackages.DBFile --run ./myscript.pl +</screen> </para> <para> If you are always using the script in places where <command>nix-shell</command> is available, you can embed the <command>nix-shell</command> invocation in the shebang like this: - <programlisting> - #!/usr/bin/env nix-shell - #! nix-shell -i perl -p perl perlPackages.DBFile - </programlisting> +<programlisting> +#!/usr/bin/env nix-shell +#! nix-shell -i perl -p perl perlPackages.DBFile +</programlisting> </para> </section> @@ -44,30 +44,30 @@ <para> Perl packages from CPAN are defined in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/perl-packages.nix"><filename>pkgs/top-level/perl-packages.nix</filename></link>, rather than <filename>pkgs/all-packages.nix</filename>. Most Perl packages are so straight-forward to build that they are defined here directly, rather than having a separate function for each package called from <filename>perl-packages.nix</filename>. However, more complicated packages should be put in a separate file, typically in <filename>pkgs/development/perl-modules</filename>. Here is an example of the former: - <programlisting> - ClassC3 = buildPerlPackage rec { - name = "Class-C3-0.21"; - src = fetchurl { - url = "mirror://cpan/authors/id/F/FL/FLORA/${name}.tar.gz"; - sha256 = "1bl8z095y4js66pwxnm7s853pi9czala4sqc743fdlnk27kq94gz"; - }; - }; - </programlisting> +<programlisting> +ClassC3 = buildPerlPackage rec { + name = "Class-C3-0.21"; + src = fetchurl { + url = "mirror://cpan/authors/id/F/FL/FLORA/${name}.tar.gz"; + sha256 = "1bl8z095y4js66pwxnm7s853pi9czala4sqc743fdlnk27kq94gz"; + }; +}; +</programlisting> Note the use of <literal>mirror://cpan/</literal>, and the <literal>${name}</literal> in the URL definition to ensure that the name attribute is consistent with the source that we’re actually downloading. Perl packages are made available in <filename>all-packages.nix</filename> through the variable <varname>perlPackages</varname>. For instance, if you have a package that needs <varname>ClassC3</varname>, you would typically write - <programlisting> - foo = import ../path/to/foo.nix { - inherit stdenv fetchurl ...; - inherit (perlPackages) ClassC3; - }; - </programlisting> +<programlisting> +foo = import ../path/to/foo.nix { + inherit stdenv fetchurl ...; + inherit (perlPackages) ClassC3; +}; +</programlisting> in <filename>all-packages.nix</filename>. You can test building a Perl package as follows: - <screen> - <prompt>$ </prompt>nix-build -A perlPackages.ClassC3 - </screen> +<screen> +<prompt>$ </prompt>nix-build -A perlPackages.ClassC3 +</screen> <varname>buildPerlPackage</varname> adds <literal>perl-</literal> to the start of the name attribute, so the package above is actually called <literal>perl-Class-C3-0.21</literal>. So to install it, you can say: - <screen> - <prompt>$ </prompt>nix-env -i perl-Class-C3 - </screen> +<screen> +<prompt>$ </prompt>nix-env -i perl-Class-C3 +</screen> (Of course you can also install using the attribute name: <literal>nix-env -i -A perlPackages.ClassC3</literal>.) </para> @@ -94,61 +94,61 @@ <para> <varname>buildPerlPackage</varname> is built on top of <varname>stdenv</varname>, so everything can be customised in the usual way. For instance, the <literal>BerkeleyDB</literal> module has a <varname>preConfigure</varname> hook to generate a configuration file used by <filename>Makefile.PL</filename>: - <programlisting> - { buildPerlPackage, fetchurl, db }: - - buildPerlPackage rec { - name = "BerkeleyDB-0.36"; - - src = fetchurl { - url = "mirror://cpan/authors/id/P/PM/PMQS/${name}.tar.gz"; - sha256 = "07xf50riarb60l1h6m2dqmql8q5dij619712fsgw7ach04d8g3z1"; - }; - - preConfigure = '' - echo "LIB = ${db.out}/lib" > config.in - echo "INCLUDE = ${db.dev}/include" >> config.in - ''; - } - </programlisting> +<programlisting> +{ buildPerlPackage, fetchurl, db }: + +buildPerlPackage rec { + name = "BerkeleyDB-0.36"; + + src = fetchurl { + url = "mirror://cpan/authors/id/P/PM/PMQS/${name}.tar.gz"; + sha256 = "07xf50riarb60l1h6m2dqmql8q5dij619712fsgw7ach04d8g3z1"; + }; + + preConfigure = '' + echo "LIB = ${db.out}/lib" > config.in + echo "INCLUDE = ${db.dev}/include" >> config.in + ''; +} +</programlisting> </para> <para> Dependencies on other Perl packages can be specified in the <varname>buildInputs</varname> and <varname>propagatedBuildInputs</varname> attributes. If something is exclusively a build-time dependency, use <varname>buildInputs</varname>; if it’s (also) a runtime dependency, use <varname>propagatedBuildInputs</varname>. For instance, this builds a Perl module that has runtime dependencies on a bunch of other modules: - <programlisting> - ClassC3Componentised = buildPerlPackage rec { - name = "Class-C3-Componentised-1.0004"; - src = fetchurl { - url = "mirror://cpan/authors/id/A/AS/ASH/${name}.tar.gz"; - sha256 = "0xql73jkcdbq4q9m0b0rnca6nrlvf5hyzy8is0crdk65bynvs8q1"; - }; - propagatedBuildInputs = [ - ClassC3 ClassInspector TestException MROCompat - ]; - }; - </programlisting> +<programlisting> +ClassC3Componentised = buildPerlPackage rec { + name = "Class-C3-Componentised-1.0004"; + src = fetchurl { + url = "mirror://cpan/authors/id/A/AS/ASH/${name}.tar.gz"; + sha256 = "0xql73jkcdbq4q9m0b0rnca6nrlvf5hyzy8is0crdk65bynvs8q1"; + }; + propagatedBuildInputs = [ + ClassC3 ClassInspector TestException MROCompat + ]; +}; +</programlisting> </para> <para> On Darwin, if a script has too many <literal>-I<replaceable>dir</replaceable></literal> flags in its first line (its “shebang line”), it will not run. This can be worked around by calling the <literal>shortenPerlShebang</literal> function from the <literal>postInstall</literal> phase: - <programlisting> - { stdenv, buildPerlPackage, fetchurl, shortenPerlShebang }: - - ImageExifTool = buildPerlPackage { - pname = "Image-ExifTool"; - version = "11.50"; - - src = fetchurl { - url = "https://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-11.50.tar.gz"; - sha256 = "0d8v48y94z8maxkmw1rv7v9m0jg2dc8xbp581njb6yhr7abwqdv3"; - }; - - buildInputs = stdenv.lib.optional stdenv.isDarwin shortenPerlShebang; - postInstall = stdenv.lib.optional stdenv.isDarwin '' - shortenPerlShebang $out/bin/exiftool - ''; - }; - </programlisting> +<programlisting> +{ stdenv, buildPerlPackage, fetchurl, shortenPerlShebang }: + +ImageExifTool = buildPerlPackage { + pname = "Image-ExifTool"; + version = "11.50"; + + src = fetchurl { + url = "https://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-11.50.tar.gz"; + sha256 = "0d8v48y94z8maxkmw1rv7v9m0jg2dc8xbp581njb6yhr7abwqdv3"; + }; + + buildInputs = stdenv.lib.optional stdenv.isDarwin shortenPerlShebang; + postInstall = stdenv.lib.optional stdenv.isDarwin '' + shortenPerlShebang $out/bin/exiftool + ''; +}; +</programlisting> This will remove the <literal>-I</literal> flags from the shebang line, rewrite them in the <literal>use lib</literal> form, and put them on the next line instead. This function can be given any number of Perl scripts as arguments; it will modify them in-place. </para> @@ -159,27 +159,27 @@ Nix expressions for Perl packages can be generated (almost) automatically from CPAN. This is done by the program <command>nix-generate-from-cpan</command>, which can be installed as follows: </para> - <screen> - <prompt>$ </prompt>nix-env -i nix-generate-from-cpan - </screen> +<screen> +<prompt>$ </prompt>nix-env -i nix-generate-from-cpan +</screen> <para> This program takes a Perl module name, looks it up on CPAN, fetches and unpacks the corresponding package, and prints a Nix expression on standard output. For example: - <screen> - <prompt>$ </prompt>nix-generate-from-cpan XML::Simple - XMLSimple = buildPerlPackage rec { - name = "XML-Simple-2.22"; - src = fetchurl { - url = "mirror://cpan/authors/id/G/GR/GRANTM/${name}.tar.gz"; - sha256 = "b9450ef22ea9644ae5d6ada086dc4300fa105be050a2030ebd4efd28c198eb49"; - }; - propagatedBuildInputs = [ XMLNamespaceSupport XMLSAX XMLSAXExpat ]; - meta = { - description = "An API for simple XML files"; - license = with stdenv.lib.licenses; [ artistic1 gpl1Plus ]; - }; - }; - </screen> +<screen> +<prompt>$ </prompt>nix-generate-from-cpan XML::Simple + XMLSimple = buildPerlPackage rec { + name = "XML-Simple-2.22"; + src = fetchurl { + url = "mirror://cpan/authors/id/G/GR/GRANTM/${name}.tar.gz"; + sha256 = "b9450ef22ea9644ae5d6ada086dc4300fa105be050a2030ebd4efd28c198eb49"; + }; + propagatedBuildInputs = [ XMLNamespaceSupport XMLSAX XMLSAXExpat ]; + meta = { + description = "An API for simple XML files"; + license = with stdenv.lib.licenses; [ artistic1 gpl1Plus ]; + }; + }; +</screen> The output can be pasted into <filename>pkgs/top-level/perl-packages.nix</filename> or wherever else you need it. </para> </section> diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index f189ce31448..59f7389b9ad 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -567,7 +567,7 @@ test run would be: checkPhase = "pytest"; ``` -However, many repositories' test suites do not translate well to nix's build +However, many repositories' test suites do not translate well to nix's build sandbox, and will generally need many tests to be disabled. To filter tests using pytest, one can do the following: @@ -755,8 +755,8 @@ and in this case the `python38` interpreter is automatically used. ### Interpreters -Versions 2.7, 3.6, 3.7 and 3.8 of the CPython interpreter are available as -respectively `python27`, `python36`, `python37` and `python38`. The +Versions 2.7, 3.6, 3.7, 3.8 and 3.9 of the CPython interpreter are available as +respectively `python27`, `python36`, `python37`, `python38` and `python39`. The aliases `python2` and `python3` correspond to respectively `python27` and `python38`. The default interpreter, `python`, maps to `python2`. The PyPy interpreters compatible with Python 2.7 and 3 are available as `pypy27` and diff --git a/doc/languages-frameworks/qt.xml b/doc/languages-frameworks/qt.xml index 8d97de504ad..ec95621d8ff 100644 --- a/doc/languages-frameworks/qt.xml +++ b/doc/languages-frameworks/qt.xml @@ -18,7 +18,7 @@ mkDerivation { <co xml:id='qt-default-nix-co-2' /> buildInputs = [ qtbase ]; <co xml:id='qt-default-nix-co-3' /> } - </programlisting> +</programlisting> </example> <calloutlist> diff --git a/doc/languages-frameworks/ruby.xml b/doc/languages-frameworks/ruby.xml index 9b36801fb96..9b579d6804f 100644 --- a/doc/languages-frameworks/ruby.xml +++ b/doc/languages-frameworks/ruby.xml @@ -12,14 +12,14 @@ </para> <screen> -<![CDATA[$ cd pkgs/servers/monitoring -$ mkdir sensu -$ cd sensu -$ cat > Gemfile +<prompt>$ </prompt>cd pkgs/servers/monitoring +<prompt>$ </prompt>mkdir sensu +<prompt>$ </prompt>cd sensu +<prompt>$ </prompt>cat > Gemfile source 'https://rubygems.org' gem 'sensu' -$ $(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic -$ cat > default.nix +<prompt>$ </prompt>$(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic +<prompt>$ </prompt>cat > default.nix { lib, bundlerEnv, ruby }: bundlerEnv rec { @@ -37,7 +37,7 @@ bundlerEnv rec { maintainers = with maintainers; [ theuni ]; platforms = platforms.unix; }; -}]]> +} </screen> <para> @@ -49,17 +49,16 @@ bundlerEnv rec { </para> <screen> -<![CDATA[$ cd pkgs/servers/monitoring/sensu -$ nix-shell -p bundler --run 'bundle lock --update' -$ nix-shell -p bundix --run 'bundix' -]]> +<prompt>$ </prompt>cd pkgs/servers/monitoring/sensu +<prompt>$ </prompt>nix-shell -p bundler --run 'bundle lock --update' +<prompt>$ </prompt>nix-shell -p bundix --run 'bundix' </screen> <para> For tools written in Ruby - i.e. where the desire is to install a package and then execute e.g. <command>rake</command> at the command line, there is an alternative builder called <literal>bundlerApp</literal>. Set up the <filename>gemset.nix</filename> the same way, and then, for example: </para> -<screen> +<programlisting> <![CDATA[{ lib, bundlerApp }: bundlerApp { @@ -75,7 +74,7 @@ bundlerApp { platforms = platforms.unix; }; }]]> -</screen> +</programlisting> <para> The chief advantage of <literal>bundlerApp</literal> over <literal>bundlerEnv</literal> is the executables introduced in the environment are precisely those selected in the <literal>exes</literal> list, as opposed to <literal>bundlerEnv</literal> which adds all the executables made available by gems in the gemset, which can mean e.g. <command>rspec</command> or <command>rake</command> in unpredictable versions available from various packages. diff --git a/doc/languages-frameworks/rust.section.md b/doc/languages-frameworks/rust.section.md index 6c51da87cab..400f39c345c 100644 --- a/doc/languages-frameworks/rust.section.md +++ b/doc/languages-frameworks/rust.section.md @@ -16,9 +16,9 @@ cargo into the `environment.systemPackages` or bring them into scope with `nix-shell -p rustc cargo`. -For daily builds (beta and nightly) use either rustup from -nixpkgs or use the [Rust nightlies -overlay](#using-the-rust-nightlies-overlay). +For other versions such as daily builds (beta and nightly), +use either `rustup` from nixpkgs (which will manage the rust installation in your home directory), +or use Mozilla's [Rust nightlies overlay](#using-the-rust-nightlies-overlay). ## Compiling Rust applications with Cargo @@ -50,7 +50,7 @@ rustPlatform.buildRustPackage rec { `buildRustPackage` requires a `cargoSha256` attribute which is computed over all crate sources of this package. Currently it is obtained by inserting a fake checksum into the expression and building the package once. The correct -checksum can be then take from the failed build. +checksum can then be taken from the failed build. Per the instructions in the [Cargo Book](https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html) best practices guide, Rust applications should always commit the `Cargo.lock` @@ -119,6 +119,18 @@ The above are just guidelines, and exceptions may be granted on a case-by-case b However, please check if it's possible to disable a problematic subset of the test suite and leave a comment explaining your reasoning. +#### Setting `test-threads` + +`buildRustPackage` will use parallel test threads by default, +sometimes it may be necessary to disable this so the tests run consecutively. + +```nix +rustPlatform.buildRustPackage { + /* ... */ + cargoParallelTestThreads = false; +} +``` + ### Building a package in `debug` mode By default, `buildRustPackage` will use `release` mode for builds. If a package @@ -466,8 +478,15 @@ Mozilla provides an overlay for nixpkgs to bring a nightly version of Rust into This overlay can _also_ be used to install recent unstable or stable versions of Rust, if desired. -To use this overlay, clone -[nixpkgs-mozilla](https://github.com/mozilla/nixpkgs-mozilla), +### Rust overlay installation + +You can use this overlay by either changing your local nixpkgs configuration, +or by adding the overlay declaratively in a nix expression, e.g. in `configuration.nix`. +For more information see [#sec-overlays-install](the manual on installing overlays). + +#### Imperative rust overlay installation + +Clone [nixpkgs-mozilla](https://github.com/mozilla/nixpkgs-mozilla), and create a symbolic link to the file [rust-overlay.nix](https://github.com/mozilla/nixpkgs-mozilla/blob/master/rust-overlay.nix) in the `~/.config/nixpkgs/overlays` directory. @@ -476,7 +495,34 @@ in the `~/.config/nixpkgs/overlays` directory. $ mkdir -p ~/.config/nixpkgs/overlays $ ln -s $(pwd)/nixpkgs-mozilla/rust-overlay.nix ~/.config/nixpkgs/overlays/rust-overlay.nix -The latest version can be installed with the following command: +### Declarative rust overlay installation + +Add the following to your `configuration.nix`, `home-configuration.nix`, `shell.nix`, or similar: + +``` + nixpkgs = { + overlays = [ + (import (builtins.fetchTarball https://github.com/mozilla/nixpkgs-mozilla/archive/master.tar.gz)) + # Further overlays go here + ]; + }; +``` + +Note that this will fetch the latest overlay version when rebuilding your system. + +### Rust overlay usage + +The overlay contains attribute sets corresponding to different versions of the rust toolchain, such as: + +* `latest.rustChannels.stable` +* `latest.rustChannels.nightly` +* a function `rustChannelOf`, called as `(rustChannelOf { date = "2018-04-11"; channel = "nightly"; })`, or... +* `(nixpkgs.rustChannelOf { rustToolchain = ./rust-toolchain; })` if you have a local `rust-toolchain` file (see https://github.com/mozilla/nixpkgs-mozilla#using-in-nix-expressions for an example) + +Each of these contain packages such as `rust`, which contains your usual rust development tools with the respective toolchain chosen. +For example, you might want to add `latest.rustChannels.stable.rust` to the list of packages in your configuration. + +Imperatively, the latest stable version can be installed with the following command: $ nix-env -Ai nixos.latest.rustChannels.stable.rust diff --git a/doc/languages-frameworks/texlive.xml b/doc/languages-frameworks/texlive.xml index a581ec5911c..141c46e5a62 100644 --- a/doc/languages-frameworks/texlive.xml +++ b/doc/languages-frameworks/texlive.xml @@ -44,11 +44,11 @@ texlive.combine { <listitem> <para> You can list packages e.g. by <command>nix repl</command>. -<programlisting><![CDATA[ -$ nix repl -nix-repl> :l <nixpkgs> -nix-repl> texlive.collection-<TAB> -]]></programlisting> +<programlisting> +<prompt>$ </prompt>nix repl +<prompt>nix-repl> </prompt>:l <nixpkgs> +<prompt>nix-repl> </prompt>texlive.collection-<keycap function="tab" /> +</programlisting> </para> </listitem> <listitem> diff --git a/doc/languages-frameworks/vim.section.md b/doc/languages-frameworks/vim.section.md index 4911509212e..84ad567e8c2 100644 --- a/doc/languages-frameworks/vim.section.md +++ b/doc/languages-frameworks/vim.section.md @@ -263,6 +263,22 @@ Sometimes plugins require an override that must be changed when the plugin is up To add a new plugin, run `./update.py --add "[owner]/[name]"`. **NOTE**: This script automatically commits to your git repository. Be sure to check out a fresh branch before running. +Finally, there are some plugins that are also packaged in nodePackages because they have Javascript-related build steps, such as running webpack. Those plugins are not listed in `vim-plugin-names` or managed by `update.py` at all, and are included separately in `overrides.nix`. Currently, all these plugins are related to the `coc.nvim` ecosystem of Language Server Protocol integration with vim/neovim. + +## Updating plugins in nixpkgs + +Run the update script with a GitHub API token that has at least `public_repo` access. Running the script without the token is likely to result in rate-limiting (429 errors). For steps on creating an API token, please refer to [GitHub's token documentation](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token). + +```sh +GITHUB_API_TOKEN=my_token ./pkgs/misc/vim-plugins/update.py +``` + +Alternatively, set the number of processes to a lower count to avoid rate-limiting. + +```sh +./pkgs/misc/vim-plugins/update.py --proc 1 +``` + ## Important repositories - [vim-pi](https://bitbucket.org/vimcommunity/vim-pi) is a plugin repository diff --git a/doc/manual.xml b/doc/manual.xml index 1f69872d2a7..4ca497e234e 100644 --- a/doc/manual.xml +++ b/doc/manual.xml @@ -1,7 +1,7 @@ <book xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"> <info> - <title>Nixpkgs Users and Contributors Guide</title> + <title>Nixpkgs Manual</title> <subtitle>Version <xi:include href=".version" parse="text" /> </subtitle> </info> diff --git a/doc/stdenv/multiple-output.xml b/doc/stdenv/multiple-output.xml index 51e1cc2e024..20658918db7 100644 --- a/doc/stdenv/multiple-output.xml +++ b/doc/stdenv/multiple-output.xml @@ -22,39 +22,69 @@ The reduction effects could be instead achieved by building the parts in completely separate derivations. That would often additionally reduce build-time closures, but it tends to be much harder to write such derivations, as build systems typically assume all parts are being built at once. This compromise approach of single source package producing multiple binary packages is also utilized often by rpm and deb. </para> </note> + + <para> + A number of attributes can be used to work with a derivation with multiple outputs. The attribute <varname>outputs</varname> is a list of strings, which are the names of the outputs. For each of these names, an identically named attribute is created, corresponding to that output. The attribute <varname>meta.outputsToInstall</varname> is used to determine the default set of outputs to install when using the derivation name unqualified. + </para> + </section> <section xml:id="sec-multiple-outputs-installing"> <title>Installing a split package</title> <para> - When installing a package via <varname>systemPackages</varname> or <command>nix-env</command> you have several options: + When installing a package with multiple outputs, the package's <varname>meta.outputsToInstall</varname> attribute determines which outputs are actually installed. <varname>meta.outputsToInstall</varname> is a list whose <link xlink:href="https://github.com/NixOS/nixpkgs/blob/f1680774340d5443a1409c3421ced84ac1163ba9/pkgs/stdenv/generic/make-derivation.nix#L310-L320">default installs binaries and the associated man pages</link>. The following sections describe ways to install different outputs. </para> - <itemizedlist> - <listitem> - <para> - You can install particular outputs explicitly, as each is available in the Nix language as an attribute of the package. The <varname>outputs</varname> attribute contains a list of output names. - </para> - </listitem> - <listitem> - <para> - You can let it use the default outputs. These are handled by <varname>meta.outputsToInstall</varname> attribute that contains a list of output names. - </para> + <section xml:id="sec-multiple-outputs-installing-nixos"> + <title>Selecting outputs to install via NixOS</title> + + <para> + NixOS provides two ways to select the outputs to install for packages listed in <varname>environment.systemPackages</varname>: + </para> + + <itemizedlist> + <listitem> + <para> + The configuration option <varname>environment.extraOutputsToInstall</varname> is appended to each package's <varname>meta.outputsToInstall</varname> attribute to determine the outputs to install. It can for example be used to install <literal>info</literal> documentation or debug symbols for all packages. + </para> + </listitem> + <listitem> + <para> + The outputs can be listed as packages in <varname>environment.systemPackages</varname>. For example, the <literal>"out"</literal> and <literal>"info"</literal> outputs for the <varname>coreutils</varname> package can be installed by including <varname>coreutils</varname> and <varname>coreutils.info</varname> in <varname>environment.systemPackages</varname>. + </para> + </listitem> + </itemizedlist> + </section> + + <section xml:id="sec-multiple-outputs-installing-nix-env"> + <title>Selecting outputs to install via <command>nix-env</command></title> + + <para> + <command>nix-env</command> lacks an easy way to select the outputs to install. When installing a package, <command>nix-env</command> always installs the outputs listed in <varname>meta.outputsToInstall</varname>, even when the user explicitly selects an output. + </para> + + <warning> <para> - TODO: more about tweaking the attribute, etc. + <command>nix-env</command> silenty disregards the outputs selected by the user, and instead installs the outputs from <varname>meta.outputsToInstall</varname>. For example, </para> - </listitem> - <listitem> +<screen><prompt>$ </prompt>nix-env -iA nixpkgs.coreutils.info</screen> <para> - NixOS provides configuration option <varname>environment.extraOutputsToInstall</varname> that allows adding extra outputs of <varname>environment.systemPackages</varname> atop the default ones. It's mainly meant for documentation and debug symbols, and it's also modified by specific options. + installs the <literal>"out"</literal> output (<varname>coreutils.meta.outputsToInstall</varname> is <literal>[ "out" ]</literal>) instead of the requested <literal>"info"</literal>. </para> - <note> - <para> - At this moment there is no similar configurability for packages installed by <command>nix-env</command>. You can still use approach from <xref linkend="sec-modify-via-packageOverrides" /> to override <varname>meta.outputsToInstall</varname> attributes, but that's a rather inconvenient way. - </para> - </note> - </listitem> - </itemizedlist> + </warning> + + <para> + The only recourse to select an output with <command>nix-env</command> is to override the package's <varname>meta.outputsToInstall</varname>, using the functions described in <xref linkend="chap-overrides" />. For example, the following overlay adds the <literal>"info"</literal> output for the <varname>coreutils</varname> package: + </para> + +<programlisting>self: super: +{ + coreutils = super.coreutils.overrideAttrs (oldAttrs: { + meta = oldAttrs.meta // { outputsToInstall = oldAttrs.meta.outputsToInstall or [ "out" ] ++ [ "info" ]; }; + }); +} +</programlisting> + </section> </section> <section xml:id="sec-multiple-outputs-using-split-packages"> <title>Using a split package</title> diff --git a/doc/stdenv/stdenv.xml b/doc/stdenv/stdenv.xml index f97c2a145af..46ee97927ea 100644 --- a/doc/stdenv/stdenv.xml +++ b/doc/stdenv/stdenv.xml @@ -475,9 +475,12 @@ passthru.updateScript = writeScript "update-zoom-us" '' <programlisting> passthru.updateScript = [ ../../update.sh pname "--requested-release=unstable" ]; </programlisting> - </para> - <para> - The script will be usually run from the root of the Nixpkgs repository but you should not rely on that. Also note that the update scripts will be run in parallel by default; you should avoid running <command>git commit</command> or any other commands that cannot handle that. + The script will be run with <varname>UPDATE_NIX_ATTR_PATH</varname> environment variable set to the attribute path it is supposed to update. + <note> + <para> + The script will be usually run from the root of the Nixpkgs repository but you should not rely on that. Also note that the update scripts will be run in parallel by default; you should avoid running <command>git commit</command> or any other commands that cannot handle that. + </para> + </note> </para> <para> For information about how to run the updates, execute <command>nix-shell maintainers/scripts/update.nix</command>. @@ -1636,10 +1639,6 @@ substitute ./foo.in ./foo.out \ --subst-var someVar </programlisting> </para> - <para> - <function>substitute</function> is implemented using the <command - xlink:href="http://replace.richardlloyd.org.uk/">replace</command> command. Unlike with the <command>sed</command> command, you don’t have to worry about escaping special characters. It supports performing substitutions on binary files (such as executables), though there you’ll probably want to make sure that the replacement string is as long as the replaced string. - </para> </listitem> </varlistentry> <varlistentry xml:id='fun-substituteInPlace'> @@ -1836,6 +1835,19 @@ addEnvHooks "$hostOffset" myBashFunction </varlistentry> <varlistentry> <term> + <literal>move-systemd-user-units.sh</literal> + </term> + <listitem> + <para> + This setup hook moves any systemd user units installed in the lib + subdirectory into share. In addition, a link is provided from share to + lib for compatibility. This is needed for systemd to find user services + when installed into the user profile. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term> <literal>set-source-date-epoch-to-latest.sh</literal> </term> <listitem> @@ -2058,7 +2070,7 @@ nativeBuildInputs = [ breakpointHook ]; The <literal>installManPage</literal> function takes one or more paths to manpages to install. The manpages must have a section suffix, and may optionally be compressed (with <literal>.gz</literal> suffix). This function will place them into the correct directory. </para> <para> - The <literal>installShellCompletion</literal> function takes one or more paths to shell completion files. By default it will autodetect the shell type from the completion file extension, but you may also specify it by passing one of <literal>--bash</literal>, <literal>--fish</literal>, or <literal>--zsh</literal>. These flags apply to all paths listed after them (up until another shell flag is given). Each path may also have a custom installation name provided by providing a flag <literal>--name NAME</literal> before the path. If this flag is not provided, zsh completions will be renamed automatically such that <literal>foobar.zsh</literal> becomes <literal>_foobar</literal>. + The <literal>installShellCompletion</literal> function takes one or more paths to shell completion files. By default it will autodetect the shell type from the completion file extension, but you may also specify it by passing one of <literal>--bash</literal>, <literal>--fish</literal>, or <literal>--zsh</literal>. These flags apply to all paths listed after them (up until another shell flag is given). Each path may also have a custom installation name provided by providing a flag <literal>--name NAME</literal> before the path. If this flag is not provided, zsh completions will be renamed automatically such that <literal>foobar.zsh</literal> becomes <literal>_foobar</literal>. A root name may be provided for all paths using the flag <literal>--cmd NAME</literal>; this synthesizes the appropriate name depending on the shell (e.g. <literal>--cmd foo</literal> will synthesize the name <literal>foo.bash</literal> for bash and <literal>_foo</literal> for zsh). The path may also be a fifo or named fd (such as produced by <literal><(cmd)</literal>), in which case the shell and name must be provided. <programlisting> nativeBuildInputs = [ installShellFiles ]; postInstall = '' @@ -2069,6 +2081,11 @@ postInstall = '' installShellCompletion --zsh --name _foobar share/completions.zsh # implicit behavior installShellCompletion share/completions/foobar.{bash,fish,zsh} + # using named fd + installShellCompletion --cmd foobar \ + --bash <($out/bin/foobar --bash-completion) \ + --fish <($out/bin/foobar --fish-completion) \ + --zsh <($out/bin/foobar --zsh-completion) ''; </programlisting> </para> diff --git a/doc/using/configuration.xml b/doc/using/configuration.xml index b670f78f28b..3e21b0e2284 100644 --- a/doc/using/configuration.xml +++ b/doc/using/configuration.xml @@ -66,7 +66,7 @@ <listitem> <para> For allowing the build of a broken package once, you can use an environment variable for a single invocation of the nix tools: -<programlisting>$ export NIXPKGS_ALLOW_BROKEN=1</programlisting> +<screen><prompt>$ </prompt>export NIXPKGS_ALLOW_BROKEN=1</screen> </para> </listitem> <listitem> @@ -92,7 +92,7 @@ <listitem> <para> For allowing the build of an unsupported package once, you can use an environment variable for a single invocation of the nix tools: -<programlisting>$ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1</programlisting> +<screen><prompt>$ </prompt>export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1</screen> </para> </listitem> <listitem> @@ -122,7 +122,7 @@ <listitem> <para> To temporarily allow all unfree packages, you can use an environment variable for a single invocation of the nix tools: -<programlisting>$ export NIXPKGS_ALLOW_UNFREE=1</programlisting> +<screen><prompt>$ </prompt>export NIXPKGS_ALLOW_UNFREE=1</screen> </para> </listitem> <listitem> @@ -187,7 +187,7 @@ <listitem> <para> To temporarily allow all insecure packages, you can use an environment variable for a single invocation of the nix tools: -<programlisting>$ export NIXPKGS_ALLOW_INSECURE=1</programlisting> +<screen><prompt>$ </prompt>export NIXPKGS_ALLOW_INSECURE=1</screen> </para> </listitem> <listitem> diff --git a/doc/using/overlays.xml b/doc/using/overlays.xml index 7f6ee040c7c..4937e950885 100644 --- a/doc/using/overlays.xml +++ b/doc/using/overlays.xml @@ -187,14 +187,28 @@ self: super: <listitem> <para> <link + xlink:href="https://github.com/flame/blis">BLIS</link> + </para> + <para> + BLIS, available through the attribute + <literal>blis</literal>, is a framework for linear algebra kernels. In + addition, it implements the BLAS interface. + </para> + </listitem> + <listitem> + <para> + <link xlink:href="https://developer.amd.com/amd-aocl/blas-library/">AMD - BLIS/LIBFLAME</link> (optimized for modern AMD x86_64 CPUs) + BLIS/LIBFLAME</link> (optimized for modern AMD x86_64 CPUs) </para> <para> - The AMD BLIS library, with attribute <literal>amd-blis</literal>, - provides a BLAS implementation. The complementary AMD LIBFLAME - library, with attribute <literal>amd-libflame</literal>, provides - a LAPACK implementation. + The AMD fork of the BLIS library, with attribute + <literal>amd-blis</literal>, extends BLIS with optimizations for + modern AMD CPUs. The changes are usually submitted to + the upstream BLIS project after some time. However, AMD BLIS + typically provides some performance improvements on AMD Zen CPUs. + The complementary AMD LIBFLAME library, with attribute + <literal>amd-libflame</literal>, provides a LAPACK implementation. </para> </listitem> </itemizedlist> @@ -226,7 +240,7 @@ self: super: lapackProvider = self.mkl; } } - </programlisting> +</programlisting> <para> This overlay uses Intel’s MKL library for both BLAS and LAPACK interfaces. Note that the same can be accomplished at runtime @@ -234,9 +248,9 @@ self: super: <literal>libblas.so.3</literal> and <literal>liblapack.so.3</literal>. For instance: </para> - <programlisting> -$ LD_LIBRARY_PATH=$(nix-build -A mkl)/lib:$LD_LIBRARY_PATH nix-shell -p octave --run octave - </programlisting> +<screen> +<prompt>$ </prompt>LD_LIBRARY_PATH=$(nix-build -A mkl)/lib:$LD_LIBRARY_PATH nix-shell -p octave --run octave +</screen> <para> Intel MKL requires an <literal>openmp</literal> implementation when running with multiple processors. By default, @@ -274,7 +288,7 @@ assert (!blas.isILP64) && (!lapack.isILP64); stdenv.mkDerivation { ... } - </programlisting> +</programlisting> </section> </section> </chapter> |